uWSGI all workers busy: reading the busy ratio before the queue fills

When every uWSGI worker shows status: "busy", the next incoming request does not wait in a place you can see. It lands in the kernel socket backlog, which uWSGI cannot reliably measure on standard Linux. If that backlog fills, the kernel drops connections silently. No log entry, no error counter, no uWSGI-level signal. The application looks alive but stops serving real traffic.

The worker busy ratio is the earliest internal indicator that you are approaching that cliff. It tells you what proportion of your alive, non-cheaped workers are currently processing requests. Reading it correctly requires understanding what “busy” actually means, what it does not mean, and where the visibility gap starts.

What this means

The busy ratio is straightforward to define: count workers where status == "busy" and divide by workers where pid > 0 and status != "cheap". Cheaped workers (scaled down by the cheaper subsystem, pid: 0) are excluded from both numerator and denominator.

At 100%, every alive worker is occupied. The next connection goes to the kernel listen queue (the socket backlog). uWSGI has no visibility into that queue on standard Linux: the listen_queue and load stats fields are unreliable. The listen_queue field relies on TCP_INFO behavior that varies across kernel versions, and the UNIX socket measurement requires a non-standard kernel ioctl. The load field mirrors listen_queue, not an average latency metric as its name suggests. The listen_queue_errors field exists in the JSON but is never incremented anywhere in the uWSGI 2.0.x source (it is only reported from a field nothing writes), so it reads 0 on current releases. Both fields are almost always 0 regardless of actual backlog depth.

This is the core problem: the moment you need visibility the most (all workers busy, queue filling), uWSGI goes dark. You must measure the kernel backlog externally with ss.

flowchart LR
    A[Request arrives] --> B{Worker available?}
    B -- yes --> C[Worker accepts
status = busy]
    C --> D[Request processed]
    D --> E[Worker returns to idle]
    B -- no --> F[KERNEL backlog
no uWSGI visibility]
    F --> G{Backlog full?}
    G -- no --> H[Request queues
latency increases]
    G -- yes --> I[Connection dropped
TCP RST or refused]
    H --> B

What “busy” actually means

“Busy” means the worker is not accepting new connections. It does not mean the worker is doing CPU work. A worker blocked on a database query, waiting on an external HTTP call, or sleeping in a blocking I/O call counts as busy. This is the single most common misinterpretation.

In threaded mode, “busy” means at least one thread is active. It does not mean all threads are occupied. You need per-core in_request data for thread-level concurrency visibility.

In async mode (gevent or asyncio), “busy” means the event loop is running. A worker is rarely idle because it multiplexes many concurrent requests. The busy ratio is nearly meaningless in this mode. If you are running gevent, track per-core request counts and greenlet-level concurrency instead.

Common causes

Quick checks

These commands read from the uWSGI stats server. Adjust the socket address to match your deployment. The stats server serves raw JSON on a socket by default. Use uwsgi --connect-and-read for TCP sockets and socat - UNIX-CONNECT:<path> for UNIX sockets. If --stats-http is enabled, curl http://<addr> also works.

# Compute the busy ratio (alive, non-cheaped workers only)
uwsgi --connect-and-read 127.0.0.1:9191 | jq \
  '([.workers[] | select(.status == "busy")] | length) as $busy
   | ([.workers[] | select(.pid > 0 and .status != "cheap")] | length) as $alive
   | if $alive > 0 then ($busy / $alive * 100) else 0 end'

# List each worker's status and PID (in-flight request URI lives in cores[].vars)
uwsgi --connect-and-read 127.0.0.1:9191 | jq \
  '.workers[] | select(.pid > 0) | {id, status, pid, avg_rt}'

# Count accepting workers (alive, non-cheaped, accepting == 1)
uwsgi --connect-and-read 127.0.0.1:9191 | jq \
  '[.workers[] | select(.pid > 0 and .status != "cheap" and .accepting == 1)] | length'

# Check avg_rt across alive workers
# <!-- TODO: verify whether avg_rt is reported in microseconds or milliseconds -->
uwsgi --connect-and-read 127.0.0.1:9191 | jq \
  '[.workers[] | select(.pid > 0 and .status != "cheap")]
   | if length > 0 then (map(.avg_rt) | add / length) else 0 end'

# Check harakiri count (monotonic, never reset even on respawn)
uwsgi --connect-and-read 127.0.0.1:9191 | jq \
  '[.workers[].harakiri_count] | add'

# Check request age for in-flight requests (requires cores, suppressed by --stats-no-cores)
# <!-- TODO: verify req_info.request_start field name and timestamp format across uWSGI versions -->
uwsgi --connect-and-read 127.0.0.1:9191 | jq \
  --argjson now "$(date +%s)" \
  '[.workers[] | select(.pid > 0) | .id as $wid
   | .cores[] | select(.in_request == 1)
   | {worker: $wid, core: .id, age_seconds: ($now - .req_info.request_start)}]'

# Measure kernel socket backlog externally (TCP)
ss -ltn 'sport = :8000'

# Measure kernel socket backlog externally (UNIX socket)
ss -lxn | grep uwsgi

The ss output shows Recv-Q (current queue depth) and Send-Q (backlog limit). Recv-Q should be 0 in steady state. Any sustained non-zero value means workers cannot accept fast enough.

How to diagnose it

1. Confirm the busy ratio is sustained, not a brief spike. Brief spikes to 100% during traffic bursts are normal if they self-resolve within seconds. Poll at 1-second intervals if possible. At 10-second polling, a complete starvation event can run for up to 10 seconds before detection.

2. Check what workers are busy on. Read the in-flight request URI from each busy worker’s cores[].vars (the request variables, populated only while in_request == 1). If all busy workers show the same URI, a single endpoint is the bottleneck. If URIs are diverse, the problem is systemic (downstream dependency, resource contention).

3. Check the kernel listen queue externally. Do not trust listen_queue or load in the stats JSON. Use ss -ltn for TCP or ss -lxn for UNIX sockets. If Recv-Q is non-zero and growing, the kernel backlog is filling.

4. Correlate with avg_rt. Rising avg_rt with stable or falling throughput means each request takes longer. The avg_rt field is an exponential moving average computed as (old_avg_rt + current_request_time) / 2, giving roughly 50% weight to the most recent request. This makes it responsive to recent changes but volatile. Compare it against your configured harakiri timeout. If avg_rt approaches harakiri, workers will start dying.

5. Check harakiri rate. Sum harakiri_count across all workers and track the delta between polling intervals. If harakiri is rising and respawn rate tracks it closely, you may be in a death spiral where every respawned worker immediately gets stuck again. If harakiri is not configured at all, harakiri_count is always 0 and stuck workers have no timeout. The absence of harakiri configuration is itself a risk.

6. Check accepting worker count. A worker can have status: "idle" but accepting: 0, meaning it will not take new requests. This can happen during graceful reloads. Count workers where pid > 0 and status != "cheap" and accepting == 1. If this count is zero while the master is alive, the service cannot accept any new requests.

7. Check kernel-level overflow counters. Use nstat -az TcpExtListenOverflows TcpExtListenDrops to see if the kernel has been dropping connections. These are cumulative system-wide counters since boot, not per-socket. Run the command twice with an interval between to measure the rate of change. Correlate with per-socket listen queue depth to attribute drops to uWSGI.

Metrics and signals to monitor

Fixes

Downstream dependency slowdown

If all workers are busy and avg_rt is rising, the root cause is almost always downstream. Check database connection counts, query latency, cache hit rates, and external API response times. Correlate uWSGI worker metrics with downstream dependency health on the same dashboard.

If a specific endpoint is identified, consider blocking or rate-limiting it at the load balancer to free workers for other traffic. At the application level, fail fast: return a 503 immediately instead of waiting for a downstream timeout.

Insufficient worker count

If the busy ratio is consistently above 80% during normal traffic with no downstream issue, you need more workers or faster request processing. Adding workers increases memory consumption (each worker is a full process copy). Verify that total worker RSS stays below 70-80% of system RAM, accounting for copy-on-write sharing.

If you use the cheaper subsystem, ensure cheaper minimum is at least 20% of maximum workers to maintain headroom.

Stuck workers without harakiri

If harakiri is not configured, a single hung request permanently consumes a worker slot. Workers accumulate in stuck state with no recovery. Configure --harakiri at 2-3x your expected maximum legitimate request duration, and enable --harakiri-verbose for diagnostic backtraces of the blocked syscall.

Listen backlog too small

The default --listen is 100 and the Linux kernel default net.core.somaxconn is 128 on older kernels (4096 since kernel 5.4). For high-traffic services, a brief downstream hiccup can fill this in under a second. Increase both --listen and net.core.somaxconn to absorb brief spikes without dropping connections.

Prevention

• Track busy ratio over time, not just point-in-time. A ratio that creeps from 60% to 80% over weeks is a capacity planning signal.
• Maintain at least 20% idle workers under normal traffic. The degradation curve is cliff-edge: there is no graceful degradation between “keeping up” and “dropping connections.”
• Always configure harakiri. Without it, stuck workers are permanent. The absence of harakiri configuration is a monitoring blind spot.
• Monitor the kernel listen queue externally with ss. Do not rely on uWSGI’s listen_queue or load stats fields.
• Monitor TcpExtListenOverflows at the kernel level for dropped-connection confirmation.
• Use chain reload (--chain-reload) for deployments. Standard graceful reload can briefly reduce capacity to near-zero during worker replacement, since all old workers drain before new workers finish loading; chain reload cycles workers one at a time to keep partial capacity.
• Do not use the stats endpoint as a health check. The stats server is served by the master process and remains responsive during complete worker starvation. Health checks must go through the worker pool.

How Netdata helps

• Per-second busy ratio collection from the uWSGI stats server, computed as busy workers over alive non-cheaped workers. Saturation events shorter than 10 seconds are invisible at typical polling intervals.
• Correlated timelines of busy ratio, avg_rt, harakiri rate, and request throughput on a single dashboard. When all four move together, the composite pattern points at capacity exhaustion, death spiral, or downstream failure.
• Cheaper-aware accepting worker count, distinguishing expected cheaped-down workers from workers that stopped accepting.
• Anomaly detection on busy ratio and related signals, useful where static thresholds produce false positives on small instances or during batch processing.
• External socket queue depth via ss-based collection or eBPF, providing visibility into the kernel backlog where uWSGI’s own stats go dark.

Related guides

• How uWSGI actually works in production: a mental model for operators
• uWSGI master process dead: total outage while the PID file lingers
• uWSGI monitoring checklist: the signals every production app server needs
• uWSGI monitoring maturity model: from survival to expert