uWSGI HARAKIRI ON WORKER: requests killed for exceeding the timeout

HARAKIRI ON WORKER N (pid: XXXX, try: 1) !!!

You see it in the uWSGI error log, followed by the master reporting the worker died by signal 9. Your monitoring shows a spike in 502 or 504 responses from the reverse proxy. Something in the request path is hanging long enough to exceed the configured harakiri limit, and the master process is killing workers to prevent total pool exhaustion.

Harakiri is doing its job. It is not the problem. The problem is whatever is causing requests to run past the timeout.

What this means

When harakiri fires, the uWSGI master sends SIGKILL to the worker that has been processing a request longer than the configured harakiri value (in seconds). The worker dies immediately. There is no graceful shutdown, no request completion, and no cleanup of resources: database connections, file locks, network sockets. The upstream proxy sees a broken connection. Nginx typically logs recv() failed (104: Connection reset by peer) and returns 502 Bad Gateway. If the proxy’s own upstream timeout fires before harakiri, the user sees 504 Gateway Timeout instead.

The master respawns the worker immediately after the kill. If the root cause is systemic, the replacement picks up the next queued request, hangs on the same dependency, and gets killed again. This cycle is the harakiri death spiral: workers churn through kill-respawn cycles while serving zero useful traffic.

The harakiri directive operates in two modes:

• SIGALRM-based: The worker sets an alarm timer for itself. Unreliable because application code or C extensions can override or mask SIGALRM, disabling the watchdog silently.
• Master/shared-memory-based: Each worker writes a per-request timestamp to shared memory. The master periodically checks whether the timestamp plus the harakiri interval has elapsed. The worker cannot interfere. This is the reliable mode, used when the master is present.

Each harakiri event increments two per-worker counters in the stats server JSON:

• harakiri_count: per-worker, monotonic. It does not reset on respawn, so track the delta between polls, not the absolute value. A worker showing harakiri_count: 47 might have accumulated those over months with zero events in the last week.
• respawn_count: per-worker, monotonic. Every harakiri increments this by one, but so do other worker exits (crashes, max-requests recycling). Subtract harakiri_count delta from respawn_count delta to isolate non-harakiri churn.

Common causes

Quick checks

All commands assume the stats server is enabled with --stats and accessible. Adjust the address to match your deployment (TCP or UNIX socket).

# Total harakiri count (sum across all workers; delta between polls gives the rate)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].harakiri_count] | add'

# Per-worker breakdown
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.workers[] | {id: .id, harakiri: .harakiri_count, respawn: .respawn_count, status: .status}'

# Worker busy ratio (how saturated is the pool?)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '([.workers[] | select(.status == "busy")] | length) as $busy | ([.workers[] | select(.pid > 0 and .status != "cheap")] | length) as $alive | if $alive > 0 then ($busy / $alive * 100) else 0 end'

# Which URIs are busy workers processing?
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.workers[] | select(.status == "busy") | {id: .id, uri: .uri, avg_rt: .avg_rt}'

<!-- TODO: verify whether avg_rt in stats JSON is milliseconds or microseconds -->

# Stuck request ages (how long has each in-flight request been running?)
# cores[] is suppressed if uWSGI was started with --stats-no-cores
uwsgi --connect-and-read 127.0.0.1:9191 | jq --argjson now "$(date +%s)" '[.workers[] | select(.pid > 0) | .id as $wid | .cores[] | select(.in_request == 1) | {worker: $wid, age_seconds: ($now - .req_info.request_start)}]'

# Check configured harakiri timeout
grep -i harakiri /etc/uwsgi/*.ini

# Search logs for harakiri-verbose output (syscall and wchan)
grep -A10 "HARAKIRI" /var/log/uwsgi/*.log | tail -80

How to diagnose it

flowchart td
    A["HARAKIRI log lines appearing"] --> B{"All workers affected?"}
    B -->|Yes| C{"Throughput near zero?"}
    B -->|No| D["Single worker or specific URI"]
    C -->|Yes| E["Systemic: downstream dependency outage"]
    C -->|No| F["Intermittent: DNS, pool, or contention"]
    D --> G["Check harakiri-verbose for syscall"]
    G --> H["Check workers.uri for slow endpoint"]
    E --> I["Check DB and API connectivity"]
    F --> J["Check DNS, pool limits, locks"]

1. Confirm the harakiri timeout. Run grep -i harakiri /etc/uwsgi/*.ini or check your config path. If harakiri is not set, it is disabled by default and these kills are not coming from uWSGI. Something else is killing workers.

2. Determine scope: death spiral or isolated kills. Poll harakiri_count twice, 60 seconds apart. If the delta is 3 or more across the pool and worker busy ratio is at or near 100%, you are in a death spiral. The service is effectively down. If only one or two workers show increments while others serve normally, the problem is endpoint-specific or worker-specific.

3. Enable harakiri-verbose if not already on. Add harakiri-verbose = true to your config and reload. This causes uWSGI to log the blocked syscall and kernel wait channel at harakiri time (Linux only; it reads /proc/<pid>/syscall and /proc/<pid>/wchan). You cannot retroactively get this data. If it was not enabled before the event, the information is gone. After enabling, wait for the next harakiri and check the log.

4. Identify the blocked syscall. With harakiri-verbose enabled, the log shows lines like HARAKIRI: -- syscall> 232 0x27... and HARAKIRI: -- wchan> sys_epoll_wait. Common patterns:

   • sys_epoll_wait, poll_schedule_timeout, or sys_read: blocked on network I/O, likely waiting on a downstream dependency that is not responding.
   • futex_wait: lock contention or a threading primitive deadlock.
   • io_schedule: blocked on disk I/O, possibly an NFS stall or synchronous file operation.

5. Check which endpoints are timing out. Poll workers[].uri on busy workers. If all busy workers show the same URI, that endpoint is the bottleneck. If URIs are diverse, the problem is systemic, not endpoint-specific.

6. Check downstream dependencies. The most common root cause is a downstream service (database, cache, external API) that is slow or unresponsive. Verify database connectivity, check connection pool utilization against configured maximums, and test external API endpoints independently.

7. Rule out known bugs. If you run threaded mode (threads > 1) with post-buffering > 0, a race condition in uWSGI’s per-worker harakiri field can cause premature kills. One thread completes a quick request and zeroes the shared harakiri field; another thread then adds the harakiri interval to that zeroed field, producing an already-expired timestamp. The master sees this as expired and kills the worker. The workaround is threads = 1 or post-buffering = 0.

Metrics and signals to monitor

Fixes

Downstream dependency is down or slow

The most common cause. The fix is at the application and dependency level, not in uWSGI configuration.

• Add explicit timeouts to all outbound calls. Database queries, HTTP client requests, and DNS lookups should all have timeouts shorter than the harakiri limit. A database query with no timeout will hang until harakiri kills the worker, leaking the connection when the worker dies.
• Fail fast. If a dependency health check fails, return 503 immediately instead of attempting the request and waiting for it to hang. This preserves worker capacity for requests that can succeed.
• Check connection pool sizing. If workers are waiting for connections from an exhausted pool, increasing the pool size or reducing the worker count can help. Total demand is workers multiplied by connections per worker, across all application servers sharing the downstream resource.

Slow endpoint is hitting the timeout

If harakiri clusters on a specific URI (reports, exports, bulk operations), the endpoint legitimately needs more time than the global harakiri allows.

• Use per-route harakiri. uWSGI supports route-level harakiri overrides via setharakiri in the internal routing subsystem. Set a longer timeout for known slow endpoints and keep the global timeout tight for everything else.
• Move long-running work to the spooler or mules. Operations that take minutes should not occupy request workers. Write the job to the spooler and return immediately with a job identifier.

Harakiri timeout is misconfigured

• Too low: Kills legitimate requests. If your p99 latency is 8 seconds and harakiri is 10 seconds, normal traffic spikes will trigger kills. A reasonable starting point is 2-3x your expected maximum legitimate request duration.
• Too high: Workers hang for a long time before recovery. A 300-second timeout means a stuck worker occupies a slot for 5 minutes before harakiri frees it. During that time, effective capacity is reduced by one worker.
• Consider graceful harakiri. The harakiri-graceful-timeout directive sends SIGTERM first, giving the worker a chance to clean up (release connections, log the request) before SIGKILL. This reduces resource leaks from violent kills.

Container CPU throttling is causing premature kills

In containerized environments with CPU limits, the kernel throttles CPU time in cgroup scheduling periods. uWSGI’s harakiri timer measures wall-clock time, not CPU time. If a worker is throttled and receives less CPU than expected, it may exceed the harakiri timeout despite doing legitimate work. Check cgroup CPU throttling counters and consider increasing CPU limits if throttling is frequent.

Prevention

• Always configure harakiri. Without it, a stuck worker hangs forever, silently consuming a worker slot until all workers are exhausted and the service is unresponsive. A harakiri_count of 0 when harakiri is not configured is not a sign of health. It is a monitoring blind spot.
• Enable harakiri-verbose from day one. You lose the syscall and wchan data if it was not enabled before the event. Retroactive diagnosis without it is guesswork.
• Set application-level timeouts on every outbound call. Database query timeout, HTTP client timeout, DNS resolver timeout. All should be shorter than harakiri. This ensures the application recovers before the master kills the worker.
• Monitor harakiri rate as a trend, not individual events. Harakiri is a safety mechanism. Occasional kills on legitimately slow requests are expected. A rising rate indicates systemic degradation.
• Correlate harakiri with downstream dependency metrics. Harakiri rate alongside database connection count, query latency, and external API response time immediately reveals whether the root cause is in the application or in a dependency.

How Netdata helps

Netdata collects harakiri_count per worker from the uWSGI stats server at per-second resolution, computing the delta automatically so you see the kill rate without manual polling.

Correlating harakiri rate with worker busy ratio, avg_rt, and exception rate in a single view makes the death spiral pattern immediately visible: all three rise together when a downstream dependency fails.

Per-worker breakdowns let you distinguish systemic harakiri (all workers affected) from endpoint-specific kills (one or two workers), narrowing the investigation before you touch a shell.

Netdata also surfaces respawn_count, so you can verify that respawns track harakiri 1:1 (confirming kills as the cause) versus tracking max-requests recycling (normal worker lifecycle).

Related guides

• How uWSGI actually works in production: a mental model for operators
• uWSGI master process dead: total outage while the PID file lingers
• uWSGI monitoring checklist: the signals every production app server needs
• uWSGI monitoring maturity model: from survival to expert