uWSGI worker stuck in busy: a hung request that never returns

One worker sits in busy far longer than any legitimate request should take. The others cycle through idle and busy normally. The stuck worker is not crashing, not erroring, and not completing. It is holding a request open indefinitely and will stay that way until something kills it.

This is the single-worker poisoning pattern. A specific request triggered a pathological code path: blocking I/O without a timeout, a regex catastrophe, an unbounded database query, or a deadlock in a C extension. The worker called accept(), began processing, and never returned. Its request counter is frozen. Its running_time stopped advancing. One slot of your worker pool is permanently consumed.

The danger is silence. If you have 8 workers and one is stuck, your aggregate throughput drops by roughly 12%. Busy ratio creeps up slightly. Average response time shifts marginally. Nothing pages. The next stuck worker drops capacity by another 12%. If harakiri is not configured, there is no automatic recovery. Workers accumulate in stuck state until the pool is exhausted and the service is unresponsive.

What this means

uWSGI workers cycle through a state machine: idle to busy and back. Under normal conditions, time spent in busy tracks your application’s response latency. When a worker enters busy and never leaves, the request it accepted is hung on something the application code cannot resolve on its own.

The key diagnostic signal is stuck request age: the elapsed time since the in-flight request started. uWSGI exposes this via the stats server in workers[].cores[]. When in_request == 1 for a core, the field req_info.request_start holds the UNIX timestamp (seconds) when the request began. Verified: the path workers[].cores[].req_info.request_start is present in 2.0.x and master stats output. Current time minus that timestamp gives you the request age. A worker whose request age is measured in minutes or hours rather than milliseconds is stuck.

This is distinct from worker pool starvation, where all workers are busy because traffic exceeds capacity. In single-worker poisoning, most workers are healthy. The problem is one outlier that diverges from the rest.

flowchart TD
    A["Stats: one worker
status=busy, others=idle"] --> B["Check cores[].in_request
and req_info.request_start"]
    B --> C{"Request age
far exceeds normal?"}
    C -->|No| D["Slow but completing
check downstream latency"]
    C -->|Yes| E["Worker is stuck
inspect /proc/pid/wchan
and /proc/pid/syscall"]
    E --> F{"Harakiri
configured?"}
    F -->|Yes| G["Wait for SIGKILL
check harakiri-verbose logs"]
    F -->|No| H["Manual SIGKILL
required to free the slot"]
    G --> I["Fix root cause:
add timeout, bound query,
or fix ReDoS regex"]
    H --> I

Common causes

Quick checks

Run these read-only commands to confirm a worker is stuck and identify what it is stuck on.

# List all workers with status, URI (from cores[].vars), and running_time to spot the outlier
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[] | select(.pid > 0) | {id: .id, status: .status, pid: .pid, uri: ([.cores[]? | select(.in_request == 1) | .vars[]? | select(startswith("REQUEST_URI=")) | .[12:]] | first), avg_rt_ms: (.avg_rt / 1000), running_time_sec: (.running_time / 1000000)}]'

# Find in-flight request ages (the primary stuck-worker signal)
uwsgi --connect-and-read 127.0.0.1:9191 | jq --argjson now "$(date +%s)" '[.workers[] | select(.pid > 0) | .id as $wid | .cores[] | select(.in_request == 1) | {worker: $wid, core: .id, age_seconds: ($now - .req_info.request_start)}]'

# Check per-worker request counts and harakiri history
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[] | select(.pid > 0) | {id: .id, requests: .requests, harakiri_count: .harakiri_count, respawn_count: .respawn_count}]'

# Inspect the stuck worker's kernel state (replace PID with the stuck worker's pid)
cat /proc/<pid>/syscall
cat /proc/<pid>/wchan

# Check if the worker's request count is frozen between two polls 5 seconds apart
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[] | select(.pid > 0) | {id: .id, requests: .requests}]'
# ... wait 5 seconds, run again. A frozen request count confirms the worker is stuck.

Note: the cores[] array is suppressed if uWSGI was started with --stats-no-cores. If your stats output lacks cores[], you cannot compute stuck request age from the stats server alone. Fall back to checking whether the worker’s requests counter advances between polls.

How to diagnose it

1. Confirm the worker is truly stuck. Run the request age query above. If age_seconds for one worker is orders of magnitude higher than the others, the request is hung. Cross-check by polling requests twice with a few seconds between polls. A worker whose request count does not advance while others do is stuck.

2. Identify what the worker is stuck on. Read the REQUEST_URI from cores[].vars in the stats output to see which endpoint the busy worker is processing. This narrows the investigation to a specific code path.

3. Inspect the kernel state. Read /proc/<pid>/syscall and /proc/<pid>/wchan. The syscall number and arguments tell you what kernel operation the worker is blocked in. The wchan value tells you which kernel function it is sleeping in. Together they distinguish network I/O waits, futex waits (lock contention), and CPU-bound spins.

4. Check whether harakiri will reap it. If harakiri is configured, the master will send SIGKILL to the worker when the request exceeds the timeout. Check harakiri_count in the stats output. If it increments for that worker, harakiri fired and the worker was respawned. If harakiri-verbose is enabled, the uWSGI log will contain the blocked syscall and wchan at harakiri time, giving you the same diagnostic data without manual /proc inspection.

5. Capture a Python traceback before killing. If --py-tracebacker <socket> is configured, each worker exposes a per-worker UNIX socket that returns the current Python traceback. Connect to it to see exactly where in the Python code the worker is stuck.

6. Consider version-specific gotchas. Verified: issue #2706 documents the threads + post-buffering race (open in the uWSGI tracker). If you run uWSGI 2.0.x with threads > 1 and post-buffering > 0, a known race condition (issue #2706) can cause spurious harakiri kills. The harakiri deadline is a single per-worker field shared across all threads. A fast request completing on one thread zeroes the field while another thread is mid-upload, producing an expired timestamp that the master immediately acts on. If you see harakiri events only in threaded mode with post-buffering, suspect this bug rather than genuinely stuck requests. Workarounds are threads = 1 or post-buffering = 0. Verified in source: 2.0.x keeps a single per-worker harakiri deadline (uwsgi.workers[i].harakiri), while 2.1-dev uses per-core deadlines (uwsgi.workers[i].cores[j].harakiri); the per-core change has not been backported to the 2.0.x branch.

Metrics and signals to monitor

Fixes

If harakiri is configured

Wait for it to fire. The master will send SIGKILL to the worker and respawn it. Each kill increments harakiri_count (per-worker, monotonic, never reset even on respawn) and respawn_count. The client receives a 502 from the upstream proxy.

If harakiri-verbose is enabled, the uWSGI log will contain the blocked syscall and wchan at harakiri time. Use this output to identify the root cause without needing to catch the worker alive.

Verified: harakiri-graceful-timeout and harakiri-graceful-signal are valid options introduced in 2.0.22 (changelog; harakiri-graceful-signal defaults to SIGTERM). If you run uWSGI 2.0.22 or later, consider configuring harakiri-graceful-timeout and harakiri-graceful-signal. These enable a two-stage kill: the master first sends a configurable signal (default SIGTERM), giving the worker a chance to run cleanup handlers or log additional state, then sends SIGKILL if the graceful timeout expires. This is safer for workers holding external resources like database connections or file locks.

If harakiri is not configured

The worker stays stuck indefinitely. You must kill it manually.

# WARNING: this terminates the in-flight request. The client gets a connection
# reset or 502. The master will respawn the worker automatically.
kill -KILL <pid>

This frees the worker slot, but the root cause is unfixed. The same request pattern will poison another worker. Without harakiri, there is no safety net.

Root cause fixes by category

Blocking I/O without timeout. This is the most common cause. Audit all outbound calls: database queries, HTTP client requests, Redis operations, DNS lookups. Every external call needs an explicit timeout. The default timeout for many Python HTTP clients and database drivers is infinite or dangerously long (30 seconds or more). Set per-call timeouts that are shorter than your harakiri value.

Regex ReDoS. A catastrophic regex against adversarial input can consume 100% of a CPU core indefinitely. Identify the regex from the stuck worker’s uri and the application’s route handlers. Replace the pattern with a non-backtracking alternative, add input length limits, or use a regex engine with guaranteed linear time. The worker will show high CPU usage on one core, distinguishable from I/O-bound hangs where CPU is near zero.

Unbounded query. A SELECT * FROM large_table without LIMIT, or an ORM query that materializes an entire result set into memory. The worker’s RSS will climb steadily while the query runs. Add pagination, streaming results, or explicit LIMIT clauses. Monitor database query duration alongside worker metrics.

C extension deadlock. A native extension (database driver, image processing library, crypto module) holds a lock and never releases it. The worker blocks in futex_wait_queue_me in /proc/<pid>/wchan. This is the hardest to diagnose because Python-level tooling (py-tracebacker, tracemalloc) may not see into native code. GDB with Python extensions can sometimes extract the C-level backtrace. Fix the extension or isolate the workload.

Prevention

• Always configure harakiri. Default uWSGI has no request timeout. Without harakiri, stuck workers have no recovery mechanism. Set harakiri to 2-3x your expected maximum legitimate request duration. Use harakiri-verbose so you get diagnostic data (syscall, wchan) when it fires.
• Add per-request timeouts in application code. Harakiri is a blunt instrument: it kills the worker, which means the client gets a 502 and the worker must respawn. Application-level timeouts let you return a proper error response without losing the worker.
• Monitor per-worker metrics, not just aggregates. Aggregate throughput and average response time mask single-worker problems. One stuck worker out of 8 reduces throughput by roughly 12%, which is easy to miss in a fleet-level dashboard. Track per-worker status, running_time, requests, and harakiri_count.
• Use py-tracebacker for Python applications. It provides real-time Python tracebacks per worker without needing to attach a debugger. Combined with harakiri, it automatically logs the traceback when a worker is killed.
• Track harakiri as a rate, not a count. harakiri_count is per-worker and monotonic. It never resets, even across respawns. Alert on the delta over a time window, not the absolute value. Any sustained non-zero rate in a deployment where harakiri is normally silent indicates requests are hanging.

How Netdata helps

• Per-second collection of uWSGI stats server data, including per-worker status, running_time, avg_rt, requests, harakiri_count, and respawn_count, gives you the resolution to catch a stuck worker within seconds rather than waiting for the next polling interval.
• ML-based anomaly detection flags per-worker divergence. A single worker whose running_time or avg_rt breaks from the baseline of its peers surfaces as an anomaly even when aggregate metrics look healthy.
• Stuck request age can be computed from cores[].req_info.request_start and correlated with downstream dependency metrics (database latency, external API response time) in a single view, shortening root-cause identification.
• Harakiri rate tracking with per-worker granularity distinguishes a single poisoned worker from a systemic harakiri death spiral where all workers are being killed.
• Correlation with OS-level signals (per-process CPU, RSS, /proc state) alongside uWSGI metrics helps distinguish I/O-bound hangs (low CPU, waiting in poll) from CPU-bound spins (100% CPU, ReDoS or infinite loop).

Related guides

• uWSGI all workers busy: reading the busy ratio before the queue fills
• uWSGI HARAKIRI ON WORKER: requests killed for exceeding the timeout
• How uWSGI actually works in production: a mental model for operators
• uWSGI master process dead: total outage while the PID file lingers
• uWSGI monitoring checklist: the signals every production app server needs
• uWSGI monitoring maturity model: from survival to expert
• uWSGI thundering herd: accept() contention and the thunder-lock fix
• uWSGI worker pool starvation: the silent outage where every worker is busy