$ guides / uwsgi / uwsgi-throughput-drop ▌

Operations Guides

uWSGI throughput drop: requests per second falling with traffic steady

Throughput in uWSGI is a derived metric: sum per-worker requests counters across all workers, then compute the delta between polling intervals. Anything that disrupts those counters or changes how fast workers complete requests shows up as a throughput trend.

Two caveats before you start:

Throughput can rise during failure. If the app starts returning 500s immediately without processing, requests-per-second goes up while useful throughput goes down. Always pair throughput with exception rate.
Worker respawns create artificial dips. When a worker respawns, its requests counter resets to zero, dropping the summed total by whatever that worker had accumulated. Track respawn count alongside throughput to distinguish real drops from counter artifacts.

Verified: delta_requests is a separate per-worker stats field that counts requests since the worker’s last respawn; it is reset to 0 when the worker respawns, while requests stays cumulative across respawns.

What this means

Each uWSGI worker processes requests synchronously in the default pre-fork model. When a worker finishes a request, it calls accept() to pull the next connection from the kernel listen queue. If fewer requests complete per second while incoming traffic stays flat, one of these is happening:

Workers are taking longer per request (slow dependency, CPU contention)
Workers are stuck and not completing requests at all (deadlock, blocking call without timeout)
Workers are being killed and respawned faster than they can serve (harakiri storm, crash loop)
The application is failing fast on most requests (returning errors before doing real work)

When all workers are busy or stuck, connections accumulate in the kernel backlog. Once the backlog fills, the kernel drops new connections silently. There is no uWSGI log entry for this. The only evidence is client-side errors or kernel counters like TcpExtListenOverflows.

Note: uWSGI’s internal listen_queue and load stats fields are unreliable on standard Linux. The listen_queue field relies on TCP_INFO for TCP sockets, which does not report listen queue depth consistently across kernel versions, and in current 2.0.x the master writes the same backlog value to both fields, so load is identical to listen_queue. Both almost always read 0 regardless of actual backlog. Use ss externally to measure the real queue depth.

Common causes

Cause	What it looks like	First thing to check
Workers stuck on blocking I/O	Some workers perpetually “busy”, request count frozen on specific workers, avg_rt climbing	Per-core `in_request` age and `/proc/<pid>/syscall`
Harakiri death spiral	`respawn_count` rising, throughput near zero, all workers busy	Downstream dependency health (database, external API)
Application returning fast errors	Exception rate rising sharply, avg_rt may drop (fast failures), throughput may spike before collapsing	Exception delta and application error logs
Memory pressure	Worker RSS high or growing, system swap active, avg_rt rising progressively, OOM kills in dmesg	`vmstat` for swap activity, per-worker RSS trend
Accept lock contention	Low busy ratio but high avg_rt, throughput low relative to worker count, per-core `in_request` mostly 0	Whether `thunder-lock` is enabled

Quick checks

Run these read-only commands to establish what is happening right now. Replace 127.0.0.1:9191 with your stats socket address, or use uwsgi --connect-and-read /path/to/stats.sock for UNIX sockets. Replace port 8000 in the ss example with your application port.

# Total throughput: sum of per-worker request counters
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].requests] | add'

# Per-worker snapshot: status, requests, avg_rt, exceptions
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[] | select(.pid > 0) | {id, status, requests, avg_rt, exceptions}]'

# Worker busy ratio (percentage of alive, non-cheaped workers that are busy)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '([.workers[] | select(.status == "busy")] | length) as $busy | ([.workers[] | select(.pid > 0 and .status != "cheap")] | length) as $alive | if $alive > 0 then ($busy / $alive * 100) else 0 end'

# Harakiri count (sum across all workers, track delta)
# TODO: verify whether harakiri_count is a standard per-worker stats field in your uWSGI version.
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].harakiri_count] | add'

# Exception count (sum across all workers, track delta)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].exceptions] | add'

# Respawn count (sum across all workers, track delta)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].respawn_count] | add'

# Stuck request ages: elapsed time of all in-flight requests
# TODO: verify req_info.request_start exists in your uWSGI version's stats JSON.
uwsgi --connect-and-read 127.0.0.1:9191 | jq --argjson now "$(date +%s)" '[.workers[] | select(.pid > 0) | .id as $wid | .cores[] | select(.in_request == 1) | {worker: $wid, core: .id, age_seconds: ($now - .req_info.request_start)}]'

# Accepting worker count (workers that can accept new connections right now)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[] | select(.pid > 0 and .status != "cheap" and .accepting == 1)] | length'

# External listen queue depth (Recv-Q) and backlog limit (Send-Q)
ss -ltn 'sport = :8000'

# Per-worker RSS (memory)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.workers[] | select(.pid > 0) | {id: .id, rss_mb: (.rss / 1048576)}'

Run the throughput, exception, harakiri, and respawn checks twice with a few seconds between polls. The delta between the two runs tells you the rate. A single snapshot gives you a total, not a rate.

How to diagnose it

The goal is to identify which capacity-destroying pattern is in play. The decision tree below uses signals from a single stats poll plus external checks.

flowchart TD
    A["Throughput dropping,
traffic steady"] --> B{"Exceptions
rising?"}
    B -->|Yes| C["App failing fast
Check error logs
and downstream deps"]
    B -->|No| D{"Respawn count
rising?"}
    D -->|Yes| E["Death spiral
Workers killed at timeout
Check downstream timeout"]
    D -->|No| F{"Workers stuck
in busy?"}
    F -->|Yes| G["Blocked workers
Check in_request age
and blocked syscall"]
    F -->|No| H{"RSS high or
swap active?"}
    H -->|Yes| I["Memory pressure
Check RSS growth
and vmstat"]
    H -->|No| J["Accept contention
Check thunder-lock
and listen queue via ss"]

Check exception rate first. Sum workers[].exceptions across all workers and compute the delta. If exceptions are rising, the application is failing on most requests. Throughput may be higher than baseline because error responses are fast. This rules out stuck workers and points to an application bug or downstream dependency failure. Check application logs for tracebacks.
Check respawn and harakiri counts. If respawn_count is rising, workers are being killed and replaced. If harakiri count is also incrementing (or respawn count tracks harakiri 1:1 from uWSGI logs), requests are exceeding the configured timeout. This is the death spiral pattern: every respawned worker immediately accepts a queued request that also blocks and times out. Throughput collapses because workers spend their entire lifecycle on a single doomed request. If respawn count is flat, skip to step 3.
Check for stuck workers without harakiri. If respawn count is flat but throughput is still dropping, examine per-worker status and request counts. A worker stuck in “busy” with a frozen request count is consuming a worker slot indefinitely. Use the stuck request age check to see how long current requests have been running. If harakiri is not configured, these workers will never be killed automatically. The absence of harakiri configuration is itself the problem.
Check per-worker request distribution. If one or two workers have frozen request counts while others continue processing, you have single-worker poisoning. Identify the worker PID and check /proc/<pid>/syscall and /proc/<pid>/wchan on Linux to see what it is blocked on.
Check memory pressure. Look at per-worker RSS. If RSS is high across all workers and the system is swapping, every request is slow because of page fault overhead. Check vmstat 1 for nonzero si and so columns. Check dmesg for OOM-killer activity targeting workers.
Check the external listen queue. Run ss -ltn 'sport = :PORT' and look at the Recv-Q column. A sustained nonzero Recv-Q means connections are waiting because no worker can accept them. If Recv-Q is zero but throughput is still low, the bottleneck is inside the workers, not at the socket.
Check accept lock contention. If the busy ratio is low (workers are mostly idle), avg_rt is high, and per-core in_request is 0 for most workers, workers may be contending on the accept() lock rather than processing requests. This happens when thunder-lock is not enabled and the worker count is high relative to traffic.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Request throughput (delta of summed `requests`)	Primary indicator of useful work being done	Sustained drop below baseline with steady traffic
Exception rate (delta of `exceptions`)	Distinguishes stuck workers from fast failures	Rising rate, especially when throughput appears stable or high
Worker busy ratio	Shows how close the pool is to exhaustion	Sustained 100% with rising avg_rt means queue is building
avg_rt (per worker)	Latency trend, but volatile due to EMA formula	Sustained increase above 2x recent baseline
Respawn count (delta)	Worker lifecycle churn, expected or abnormal	Rate exceeding what `max-requests` configuration predicts
Stuck request age	How long in-flight requests have been running	Request age approaching or exceeding expected maximum
Listen queue depth (via `ss`)	Connections waiting because workers cannot accept	Sustained nonzero Recv-Q
Accepting worker count	Workers actively able to accept new connections	Dropping below the cheaper minimum, or reaching zero

Fixes

Workers stuck on blocking I/O

If workers are stuck in “busy” with high request ages, the application is blocked on something without a timeout. Common culprits: database queries without statement timeouts, HTTP calls to external APIs without client-side timeouts, DNS resolution hanging, or thread-based libraries (database drivers, cache clients) initialized before fork causing GIL deadlock.

For the immediate incident, identify the stuck worker PID and check /proc/<pid>/syscall and /proc/<pid>/wchan to see what it is blocked on. If harakiri is configured, the worker will eventually be killed. If not, you need to manually kill the worker so the master respawns it:

# WARNING: terminates the worker process. Any in-flight request on that worker is lost.
kill <pid>

For prevention, ensure all downstream calls have timeouts shorter than your harakiri timeout. If thread-based libraries are the cause, initialize them per-worker after fork, or use lazy-apps so each worker loads the application independently.

Harakiri death spiral

If respawn count is rising and throughput is near zero, every request is timing out on a shared dependency. The fix is not in uWSGI configuration. The fix is the downstream dependency: a dead database, an unresponsive API, a network partition.

As an emergency measure, if one specific route is the culprit (visible in the URI field of busy workers), block that route at the load balancer to free workers for other traffic. If all traffic depends on the failed dependency, failing fast at the application level (returning a 503 immediately instead of waiting for the downstream timeout) preserves worker capacity for health checks and reduces respawn churn.

Enable harakiri-verbose to log the blocked syscall and wchan when harakiri fires.

Application returning fast errors

If exception rate is rising and throughput appears high or stable, the application is rejecting requests before doing real work. The throughput chart is misleading because error responses complete quickly. Check application logs for the specific exception types. A sudden spike across all workers simultaneously almost always indicates an external dependency failure, not an application code bug.

Memory pressure

If RSS is high and the system is swapping, performance degrades non-linearly. A graceful reload resets all workers and clears accumulated memory, but causes a brief capacity drop while workers restart:

# WARNING: all workers restart. Brief throughput drop expected.
kill -HUP <master_pid>

If reload-on-rss is configured, verify the threshold is set low enough to trigger recycling before swapping begins. If it is not configured, add it as a safety net.

For the longer term, profile memory with tracemalloc or equivalent to find the leak source. CPython’s pymalloc allocator often does not return freed memory to the OS due to fragmentation, so RSS may stabilize at a high-water mark even after the leak is fixed.

Accept lock contention

If thunder-lock is not enabled, multiple workers wake up to compete for the accept() call on the shared listening socket. The kernel burns time context-switching between workers fighting for the lock. Enable thunder-lock in the uWSGI configuration. This is a safe, well-understood directive that serializes the accept call efficiently.

Prevention

Always configure harakiri. Without it, a stuck worker stays stuck forever. Set it to 2-3x your expected maximum legitimate request duration. Enable harakiri-verbose for diagnostic backtraces.
Track throughput as a delta, not an absolute. Sum workers[].requests across all workers and compute the rate. Per-worker requests counters reset on respawn, producing noisy dips in the sum.
Pair throughput with exception rate. Rising throughput with rising exceptions means the app is failing fast, not serving more traffic.
Monitor per-worker metrics, not just aggregates. One stuck worker reduces capacity by 1/N. Aggregate throughput drops by only that fraction, which is easy to miss.
Correlate with downstream dependency health. When throughput drops, the root cause is almost always downstream. Co-display worker busy ratio alongside database connection counts, query latency, and external API response time.
Distinguish expected respawns from crash-induced respawns. If max-requests is configured, periodic respawns are healthy. If respawn rate correlates with throughput drops, workers are crashing or timing out, not recycling.

How Netdata helps

Netdata collects uWSGI stats server metrics per second and correlates throughput, exception rate, respawn count, and worker busy ratio on a single timeline. This makes it immediately visible whether a throughput drop is caused by stuck workers, fast failures, or a harakiri spiral.

ML anomaly detection on throughput and response time baselines flags deviations relative to time-of-day and day-of-week patterns, reducing false alerts from normal traffic variation.

Per-worker metric breakdowns expose single-worker poisoning that aggregate charts hide.