$ guides / uwsgi / uwsgi-harakiri-death-spiral ▌

Operations Guides

uWSGI harakiri death spiral: workers killed and respawned while throughput collapses

Workers are being killed by harakiri and respawned in a tight loop. Every request blocks past the configured timeout. The master sends SIGKILL to the worker, forks a replacement, and the new worker immediately accepts the next queued request, which also blocks. Throughput collapses to near zero while the worker pool appears “busy” at or near 100%.

This is the harakiri death spiral: a composite failure where the root cause is almost never uWSGI itself. A downstream dependency (database, external API, DNS resolver) has become unresponsive or unreachable. Every request that touches that dependency hangs. Harakiri is working as designed, killing stuck workers, but the recycling provides no relief because the next request is equally doomed.

The signature is a tight correlation between harakiri count and respawn count (approximately 1:1), combined with busy ratio at or above 80% and a throughput collapse relative to recent baseline. Unlike a traffic spike, throughput here is low or zero despite incoming traffic.

What this means

In normal operation, harakiri is a safety mechanism. When a request exceeds the configured timeout (--harakiri N), the master sends SIGKILL to that worker and respawns it. The client gets a 502 or whatever the upstream proxy returns when the connection drops. The new worker starts fresh, ready to serve.

In a death spiral, the safety mechanism becomes the failure mechanism. The kill-respawn cycle burns CPU on fork and process initialization. Workers spend their entire lifespan blocked on the same dead dependency, never completing a single useful request. The service is effectively down from the user’s perspective, yet the master process is alive, the stats server is responsive, and health checks may still pass if they hit a lightweight endpoint.

flowchart TD
    A["Request arrives"] --> B["Worker accepts and begins processing"]
    B --> C["Request blocks on downstream dependency"]
    C --> D["Harakiri timer expires"]
    D --> E["Master sends SIGKILL to worker"]
    E --> F["Worker respawned by master"]
    F --> G["New worker accepts next queued request"]
    G --> C
    E --> H["Client receives 502/error"]
    F --> I["Throughput stays near zero"]

The key diagnostic insight: respawn rate tracks harakiri rate at approximately 1:1. Every harakiri kill produces exactly one respawn. If you subtract the harakiri rate from the respawn rate, the remainder should be consistent with normal max-requests recycling. In a death spiral, the two rates are nearly identical.

Common causes

Cause	What it looks like	First thing to check
Downstream dependency outage	All workers harakiri simultaneously, no successful completions	Check database or API health directly from the uWSGI host
Database lock or deadlock	Harakiri concentrated on specific endpoints, exceptions may precede kills	Check DB lock monitors, pg_locks, or equivalent
DNS resolution failure	Harakiri on endpoints making outbound calls, intermittent pattern	Check resolver health, `/etc/resolv.conf`, `dig` against configured resolvers
Network partition to dependency	All workers hang at same point in request lifecycle	Check connectivity from the uWSGI host to the dependency host and port
Full connection pool	Workers block waiting for a connection, not on the query itself	Check pool utilization metrics and downstream connection limits

Quick checks

These commands are safe and read-only. They assume a stats server on 127.0.0.1:9191 (adjust to match your deployment). Use uwsgi --connect-and-read <addr> for TCP stats sockets and socat - UNIX-CONNECT:<path> for UNIX sockets.

# Check harakiri count across all workers (monotonic, never resets even on respawn)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].harakiri_count] | add'

# Check respawn count to compare against harakiri count
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].respawn_count] | add'

# Check worker busy ratio (busy workers / alive non-cheaped workers)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '([.workers[] | select(.status == "busy")] | length) as $busy | ([.workers[] | select(.pid > 0 and .status != "cheap")] | length) as $alive | if $alive > 0 then ($busy / $alive * 100) else 0 end'

# Check total request throughput (sum across all workers)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].requests] | add'

# Check which URI each busy worker is processing (REQUEST_URI lives in cores[].vars)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[] | select(.status == "busy") | .id as $wid | .cores[] | select(.in_request == 1) | {id: $wid, uri: (.vars[]? | select(startswith("REQUEST_URI=")) | .[12:])}]'

# Check in-flight request age (how long each worker has been blocked)
# cores[].req_info.request_start holds the unix timestamp when the request started
uwsgi --connect-and-read 127.0.0.1:9191 | jq --argjson now "$(date +%s)" '[.workers[] | select(.pid > 0) | .id as $wid | .cores[] | select(.in_request == 1) | {worker: $wid, core: .id, age_seconds: ($now - .req_info.request_start)}]'

# Check harakiri log lines (requires harakiri-verbose for syscall detail)
journalctl -u uwsgi --since "5 minutes ago" | grep -i "HARAKIRI"

# Check kernel listen queue externally (uWSGI internal listen_queue field is unreliable)
ss -ltn 'sport = :8000'

How to diagnose it

Confirm the composite pattern. Pull harakiri count and respawn count at two timestamps 5 minutes apart. Compute the deltas. In a death spiral, harakiri delta is at least 3 and respawn delta tracks harakiri delta closely. If respawn delta is much higher than harakiri delta, some respawns are from crashes or max-requests recycling, not harakiri alone.
Confirm throughput collapse. Sum workers[].requests at two timestamps. If the delta is near zero while incoming traffic is normal, workers are not completing requests. Cross-check against proxy logs or upstream traffic data to confirm traffic is still arriving.
Identify the stuck endpoint. Read the REQUEST_URI from cores[].vars on busy workers. If all busy workers are processing the same URI pattern, that endpoint is the bottleneck. If busy workers show diverse URIs, the problem is systemic: all requests touch the same dead dependency.
Check downstream dependency health. This is almost always the root cause. Test database connectivity directly from the uWSGI host. Check API endpoint health. Run dig against the DNS resolvers the application uses. Check connection pool utilization on the downstream system.
Enable harakiri-verbose if not already set. The --harakiri-verbose flag (Linux only) logs the blocked syscall and wchan by reading /proc/<pid>/syscall and /proc/<pid>/wchan before killing the worker. Without it, the logs show *** HARAKIRI ON WORKER N (pid: XXXX, try: 1, graceful: yes/no) *** but no syscall detail.
Check for the post-buffering and threads bug. If you are running uWSGI 2.0.x with post-buffering > 0 and threads > 1, a known race condition (GitHub Issue #2706) causes spurious harakiri kills. The per-worker harakiri timer is zeroed by a fast request completing on one thread while another thread is mid-upload, producing an absolute timestamp in January 1970 that the master immediately treats as expired. This is not a downstream dependency problem. Check whether both post-buffering and threads are set in your config.
Check kernel-level listen queue. uWSGI’s internal listen_queue stats field is unreliable on standard Linux. The load field is set from the same backlog value as listen_queue (with a source TODO saying it should be more advanced), not average latency. Use ss externally instead. Look at the Recv-Q column for current queue depth and Send-Q for the backlog limit. A growing Recv-Q means connections are backing up.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Harakiri count (delta)	Each harakiri is a dropped request and a killed worker	Delta of 3 or more over 5 minutes with no prior baseline of harakiri
Respawn count (delta)	Tracks harakiri 1:1 in a death spiral	Respawn delta approximately equals harakiri delta
Worker busy ratio	Shows pool saturation	Sustained 80% or higher with throughput collapsing
Request throughput (delta)	Confirms real user impact	Near-zero delta while traffic is arriving normally
Average response time (avg_rt)	App approaching the harakiri ceiling	avg_rt approaching the configured harakiri timeout value
Stuck request age (in_request)	Shows how long each worker has been blocked	Request age approaching harakiri timeout
Exception rate	Unhandled exceptions may precede hangs	Spike in exceptions before harakiri begins
Downstream dependency latency	Root cause is almost always downstream	Latency spike or timeout on database, API, or DNS

Fixes

The downstream dependency is down or unreachable

Fix the dependency first. If you cannot fix it immediately, make the application fail fast: return a 503 or cached response instead of blocking on the dependency. Circuit breakers, request timeouts on outbound HTTP calls, and database query timeouts all prevent workers from blocking past harakiri.

Check the application’s outbound call configuration. Many HTTP client libraries default to no timeout or very long timeouts (30 seconds or more). If these exceed the harakiri value, the worker blocks until harakiri kills it. Set client-level timeouts shorter than harakiri.

A specific endpoint is the bottleneck

If the uri field shows all busy workers on the same endpoint, consider blocking that route at the proxy level. This frees workers for other traffic while the endpoint is fixed. If the endpoint requires a long-running operation such as report generation or bulk export, use a dedicated worker pool or route it to a background job system.

Spurious harakiri from post-buffering and threads

If you confirmed the Issue #2706 race condition (post-buffering greater than 0 and threads greater than 1 on uWSGI 2.0.x), the fix is to set threads = 1 or post-buffering = 0. Tuning the harakiri value does not help because the timer is corrupted, not exceeded. This bug exists because the 2.0.x branch has a per-worker harakiri timer rather than per-core timers. The master branch (2.1+) has per-core timers and is immune to this race.

Harakiri is not configured

If --harakiri is not set, workers that hang on a blocking call never get killed. They accumulate in stuck state until all workers are consumed. This is a different failure mode (worker pool starvation) but leads to the same outcome: total unavailability with no automatic recovery. Configure harakiri at 2-3x your expected maximum legitimate request duration.

Emergency: stop the bleeding

If every worker is in a kill-respawn cycle and you cannot fix the dependency immediately, consider stopping uWSGI entirely. This drops all queued connections but stops the CPU burn from constant forking. Restart after fixing the root cause or after configuring fail-fast behavior in the application.

Prevention

Configure harakiri. The default has no timeout. Without it, stuck workers are never recovered. Set it to 2-3x your expected maximum legitimate request duration.
Set client-level timeouts on all outbound calls. Database query timeouts, HTTP client timeouts, and DNS resolver timeouts should all be shorter than harakiri. Workers should never block long enough to trigger harakiri under normal dependency degradation.
Implement circuit breakers. If a dependency is failing, trip the circuit and return a fast error instead of queuing doomed requests against workers.
Enable harakiri-verbose in production. The syscall and wchan data it logs when harakiri fires is the single most useful diagnostic signal for identifying what workers are blocked on.
Correlate harakiri rate with respawn rate. If they track 1:1, workers are being killed by harakiri. If respawn rate is higher, investigate crashes or max-requests recycling separately.
Monitor downstream dependencies alongside uWSGI metrics. The root cause is almost never uWSGI. Co-display uWSGI worker metrics with database latency, API health, and DNS resolution time.

How Netdata helps

Netdata collects uWSGI stats at per-second resolution, which surfaces the four death-spiral signals as they correlate in real time:

Harakiri count delta and respawn count delta at per-second resolution confirm the kill-respawn cycle within seconds. A 1:1 correlation means respawns are harakiri-driven.
Worker busy ratio sustained above 80% with collapsing throughput distinguishes a death spiral from a traffic spike.
Average response time (avg_rt) approaching the harakiri timeout provides early warning before the cycle begins.
Anomaly detection on harakiri rate, respawn rate, and throughput flags the pattern even when individual metrics remain within static thresholds.
Co-display of uWSGI worker metrics with downstream dependency latency (database, API, DNS) puts the root cause in the same view.