uWSGI worker pool starvation: the silent outage where every worker is busy

Every worker shows status: "busy". The master process is alive. The stats server responds instantly. Your load balancer health check returns 200. But real users are seeing timeouts, connection refused errors, or hanging pages. This is worker pool starvation.

The mechanism is a concurrency cliff. uWSGI’s pre-fork model assigns one request per worker at a time in the default configuration. When every worker is occupied with a slow or hanging request, new connections pile into the kernel listen queue (socket backlog). Once that queue fills, the kernel silently drops connections with no uWSGI log entry, no error counter, and no exception. The service is dead for real traffic while every surface-level health signal stays green.

The outage is “silent” because the master process does not serve requests. It manages worker lifecycle. The stats server runs inside the master. Both remain fully responsive during complete worker starvation. Any health check that verifies the master PID, pings the stats endpoint, or hits a lightweight /health route that completes in milliseconds will pass while real requests are being dropped.

What this means

Worker pool starvation is concurrency exhaustion with no graceful degradation. Below 100% worker utilization, additional requests are served immediately. At 100%, the next request enters the kernel socket backlog and waits. If workers do not free up before the backlog fills, the kernel refuses new connections. There is no middle ground.

The defining composite signal is three conditions occurring simultaneously:

• 100% worker busy ratio: every alive worker (pid > 0, not in cheap status) shows status: "busy".
• Growing listen queue: the kernel socket backlog depth is nonzero and climbing. Measure this externally (via ss), not from uWSGI’s listen_queue stats field, which is unreliable on standard Linux.
• Collapsing throughput: the sum of workers[].requests is flat or far below the traffic baseline.

Any single signal alone is ambiguous. Brief 100% utilization during traffic bursts is normal. A nonzero listen queue that clears in seconds is expected. Throughput dips happen for many reasons. All three together, sustained, means the worker pool is saturated and cannot recover without intervention.

Critical caveat: uWSGI’s listen_queue and load stats fields are unreliable on standard Linux. The listen_queue field depends on kernel-version-dependent TCP_INFO behavior for TCP sockets and a non-standard ioctl for UNIX sockets. The load field is identical to listen_queue in the source code (there is a TODO comment acknowledging this is wrong). The listen_queue_errors field is dead code, never incremented. All three almost always read 0 regardless of actual backlog. Measure the kernel queue externally.

flowchart TD
    A[Slow request occupies a worker] --> B[Remaining workers accept more slow requests]
    B --> C[All workers status busy]
    C --> D[New connections enter kernel backlog]
    D --> E{Backlog full?}
    E -- No --> D
    E -- Yes --> F[Kernel drops connections silently]
    C --> G[Master and stats server stay responsive]
    G --> H[Health checks pass]
    F --> I[Silent outage: green dashboard, dead service]
    H --> I

Common causes

Quick checks

All commands are read-only and safe to run during an active incident. Replace 127.0.0.1:9191 with your stats server address. If the stats server is on a UNIX socket, use the socket path instead.

# Check worker status distribution: how many busy vs idle
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[] | .status] | group_by(.) | map({status: .[0], count: length})'

# Count accepting workers (alive, not cheaped, accepting connections)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[] | select(.pid > 0 and .status != "cheap" and .accepting == 1)] | length'

# Compute worker busy ratio as percentage
uwsgi --connect-and-read 127.0.0.1:9191 | jq '([.workers[] | select(.status == "busy")] | length) as $busy | ([.workers[] | select(.pid > 0 and .status != "cheap")] | length) as $alive | if $alive > 0 then ($busy / $alive * 100) else 0 end'

# Show URI of every busy worker: which endpoint is stuck
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[] | select(.status == "busy") | {id: .id, pid: .pid, uri: .uri}]'

# Show total request count across all workers (run twice, seconds apart, to check throughput)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].requests] | add'

# Check harakiri count: are stuck workers being killed?
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].harakiri_count] | add'

# Check in-flight request age: how long has each worker been stuck?
uwsgi --connect-and-read 127.0.0.1:9191 | jq --argjson now "$(date +%s)" '[.workers[] | select(.pid > 0) | .id as $wid | .cores[] | select(.in_request == 1) | {worker: $wid, age_seconds: ($now - .req_info.request_start)}]'

# Measure kernel listen queue depth (TCP, replace :8000 with your port)
# Recv-Q = current backlog depth, Send-Q = configured backlog limit
ss -ltn 'sport = :8000'

# Check for kernel-level connection drops (system-wide counter)
nstat -az TcpExtListenOverflows TcpExtListenDrops

The cores[] array is suppressed if uWSGI is started with --stats-no-cores. If you see no cores[] data, check your configuration. The vars field inside each core also exposes request headers and URI of in-flight requests, useful when workers[].uri is empty or unavailable.

How to diagnose it

1. Confirm the composite signal. Check worker status distribution, throughput trend (run the request count command twice, a few seconds apart), and kernel listen queue depth via ss. All three must be abnormal simultaneously to distinguish starvation from a transient spike.

2. Identify which endpoints are stuck. Read workers[].uri for every busy worker. If all busy workers show the same URI, a single endpoint is the bottleneck. If URIs are diverse, the problem is systemic, likely a downstream dependency affecting all requests.

3. Check whether harakiri is configured. If harakiri_count is always 0 across all workers, harakiri may be disabled entirely. Stuck workers have no timeout and will never recover without manual intervention. This is the most common reason starvation becomes a hard outage instead of oscillating degradation. If harakiri is configured and firing, check whether respawned workers immediately get stuck again. This signals a Harakiri Death Spiral: the root cause is systemic, and recycling workers does not help.

4. Investigate what the stuck workers are blocked on. For each busy worker PID, check /proc/<pid>/syscall and /proc/<pid>/wchan (Linux). These reveal the syscall and kernel function the worker is sleeping in. Look for network I/O wait patterns (socket read), lock contention (futex), or file I/O stalls.

5. Check downstream dependencies. If workers are blocked on I/O, identify which dependency. Check database connection pool utilization, external API response times, DNS resolution latency, and network connectivity. uWSGI metrics will not tell you which dependency is slow; you need correlated visibility into downstream systems.

6. Verify connection drops at the kernel level. Run nstat -az TcpExtListenOverflows TcpExtListenDrops twice, a few seconds apart. Any nonzero rate of change means the kernel is actively refusing connections. This is definitive proof that users are experiencing failures right now.

Metrics and signals to monitor

Fixes

If harakiri is not configured

Configure --harakiri with a value 2-3x your expected maximum legitimate request duration. Without harakiri, a stuck worker is stuck forever, permanently reducing capacity. With harakiri, the master kills the stuck worker after the timeout, respawns it, and the worker can accept new requests.

Enable --harakiri-verbose to log the blocked syscall and wchan when harakiri fires (Linux only). This turns each harakiri event into diagnostic data instead of a silent kill.

Tradeoff: harakiri kills legitimate long-running requests (file exports, batch operations). Use per-route harakiri configuration to exempt known slow endpoints, or run them on a separate uWSGI instance with a higher timeout.

If a single endpoint is the bottleneck

If all busy workers show the same URI, that endpoint is consuming the entire pool. Short-term: block or rate-limit that endpoint at the load balancer to free workers for other traffic. Long-term: fix the endpoint by adding a timeout on the downstream call, optimizing the query, or adding caching.

Tradeoff: users of the blocked endpoint get errors, but the rest of the service recovers immediately.

If the listen backlog is too small

The default --listen value is 100, and the kernel default net.core.somaxconn may also be low. Set --listen 1024 or higher and ensure net.core.somaxconn is at least as high. The effective limit is the lower of the two values. A larger backlog absorbs brief traffic spikes without dropping connections.

Tradeoff: a larger backlog does not fix the underlying capacity issue. It gives you more time to detect and respond before connections are dropped, but workers still need to process the queued requests eventually.

If downstream dependencies are slow

Every outgoing call in your application (HTTP requests, database queries, cache lookups, DNS resolution) must have a timeout. A missing timeout on a single external API call can consume a worker indefinitely. Add application-level timeouts shorter than your harakiri timeout so the application fails fast instead of hanging.

Tradeoff: aggressive timeouts may cause false failures during transient downstream latency spikes. Tune per dependency based on observed behavior.

Prevention

• Configure harakiri. Set --harakiri to 2-3x expected maximum request duration. Enable --harakiri-verbose. The absence of harakiri configuration is itself a monitoring blind spot: harakiri_count will always read 0, creating a false sense of safety.
• Set adequate listen backlog. Configure --listen 1024 or higher. Ensure net.core.somaxconn is at least as high. The effective limit is the lower of the two.
• Alert on the composite signal, not individual metrics. Sustained 100% busy workers plus growing kernel listen queue plus collapsing throughput. No single metric is sufficient.
• Route health checks through the worker pool. Do not use the master PID, stats endpoint, or a dedicated lightweight route as your sole health check. The health check must experience the same queuing as real requests.
• Add application-level timeouts on all downstream calls. Every database query, HTTP request, cache lookup, and DNS resolution needs a timeout.
• Measure the listen queue externally. Do not trust listen_queue, load, or listen_queue_errors from uWSGI stats. Use ss -ltn or ss -lxn for queue depth and nstat for overflow counts.

How Netdata helps

• Per-second worker busy ratio and accepting worker count reveal saturation as it develops, before the backlog fills.
• Request throughput derived from workers[].requests at per-second granularity confirms whether workers are making progress, distinguishing real starvation from a brief spike.
• Correlated downstream metrics (database connections, external API latency, DNS resolution time) on the same dashboard as uWSGI worker metrics, since the root cause of worker starvation is almost always downstream.
• Harakiri rate as a delta over time distinguishes a death spiral from normal operation and surfaces a harakiri count that never moves, indicating it may be disabled.

Related guides

• How uWSGI actually works in production: a mental model for operators
• uWSGI master process dead: total outage while the PID file lingers
• uWSGI monitoring checklist: the signals every production app server needs
• uWSGI monitoring maturity model: from survival to expert