uWSGI response time climbing: rising avg_rt and where it comes from

When avg_rt climbs in the uWSGI stats server, the instinct is to look for slow application code. That is often right, but avg_rt is more nuanced than a simple average response time. Understanding what it actually measures is the difference between a fast diagnosis and a misleading rabbit hole.

avg_rt is an exponential moving average (EMA), not a cumulative average. When it climbs, something per-request is getting slower. The question is whether the cause is a downstream dependency (database pool exhaustion, slow external API), resource contention (CPU saturation, GC storms, GIL pressure in threaded Python), or cascading starvation (workers backing up because each request holds the worker longer). Correlating avg_rt with the worker busy ratio narrows this quickly.

What avg_rt actually is

The avg_rt field is exposed per worker in the uWSGI stats server JSON as workers[].avg_rt, an integer in microseconds. It is an EMA of per-request processing time for that specific worker, not a cluster-wide or cumulative average.

uWSGI updates it after each request as avg_rt = (old_avg_rt + current_request_time) / 2, giving roughly 50% weight to the most recent request, 25% to the one before, 12.5% to the third, and so on. After about seven requests, contributions from older values fall below 1%.

Consequences:

• Short memory: The metric tracks a short recent window, not a lifetime average.
• High volatility: The last request contributes 50% of the value. A single outlier can double avg_rt. A request that takes 2000ms on a worker whose previous EMA was 50ms pushes avg_rt past 1000ms in one update.
• No percentile information: avg_rt cannot tell you whether 99% of requests are fast and 1% are slow, or whether all requests are moderately slow. For real latency SLIs, parse per-request timing from access logs using format variables like %(msecs) (milliseconds) or %(micros) (microseconds) in your --log-format configuration.
• Per-worker, not aggregate: Each worker has its own avg_rt. A single slow worker can raise the average across all workers even when most are healthy.

Whether avg_rt resets when a worker respawns is not confirmed by the uWSGI source code. Only delta_requests is confirmed to reset on respawn. Do not assume a respawn clears avg_rt to zero.

avg_rt can also report non-null values on workers that are idle, particularly after traffic subsides. The value you see may be stale rather than current. If your polling interval is coarse, you may be reading a value from minutes ago rather than a real-time measurement.

Common causes

Quick checks

These are read-only diagnostic commands. Adjust the stats socket address (127.0.0.1:9191) to match your deployment. If your uwsgi binary is inside a virtualenv or container, use its full path.

# Check avg_rt per worker (microseconds converted to ms), filtering out cheaped workers
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.workers[] | select(.pid > 0 and .status != "cheap") | {id: .id, avg_rt_ms: (.avg_rt / 1000), status: .status}'

# Compute average avg_rt across all alive workers (ms)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[] | select(.pid > 0 and .status != "cheap")] | if length > 0 then (map(.avg_rt) | add / length / 1000) else 0 end'

# Check worker busy ratio alongside avg_rt
uwsgi --connect-and-read 127.0.0.1:9191 | jq '([.workers[] | select(.status == "busy")] | length) as $busy | ([.workers[] | select(.pid > 0 and .status != "cheap")] | length) as $alive | ([.workers[] | select(.pid > 0 and .status != "cheap")] | map(.avg_rt) | if length > 0 then add / length / 1000 else 0 end) as $avg | {busy_workers: $busy, alive_workers: $alive, busy_pct: (if $alive > 0 then ($busy / $alive * 100 | floor) else 0 end), avg_rt_ms: $avg}'

# Check harakiri count (should be zero or near-zero in healthy deployments)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].harakiri_count] | add'

# Check what URI busy workers are processing
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.workers[] | select(.status == "busy") | {id: .id, uri: .uri, avg_rt_ms: (.avg_rt / 1000)}'

# Compare EMA avg_rt against cumulative average per worker
# <!-- TODO: verify the unit of running_time in uWSGI stats JSON. The /1000 assumes microseconds. -->
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.workers[] | select(.pid > 0 and .requests > 0) | {id: .id, true_avg_ms: (.running_time / .requests / 1000), ema_avg_rt_ms: (.avg_rt / 1000), requests: .requests}'

# Check listen queue externally (uWSGI's listen_queue field is unreliable on Linux)
ss -ltn 'sport = :8000' | awk 'NR>1 {print "recv_q:", $2, "send_q(backlog):", $3}'

For the last command, replace :8000 with your uWSGI listen port. The ss Recv-Q column shows the current kernel backlog depth, and Send-Q shows the configured backlog limit.

How to diagnose it

The diagnostic flow hinges on one key correlation: is the busy ratio climbing alongside avg_rt?

flowchart TD
    A["avg_rt rising"] --> B{"Busy ratio rising?"}
    B -- "Yes, toward 100%" --> C["Cascading starvation"]
    B -- "No, stable" --> D["Per-request slowdown"]
    C --> E["Check: ss backlog, harakiri count"]
    D --> F["Check: downstream deps, GC, per-worker avg_rt"]
    A --> G{"Near harakiri timeout?"}
    G -- "Yes" --> H["URGENT: kills imminent"]
    G -- "No" --> I["Monitor and investigate"]

1. Confirm the trend is real. A single slow request can spike avg_rt by 50% in one update. Check whether the elevation persists across at least 3 to 4 polling intervals before treating it as a real trend rather than EMA noise.

2. Pull the busy ratio. This is the critical discriminator.

   • Rising avg_rt + stable busy ratio: each request is taking longer, but workers are not backing up. Points to a downstream dependency slowdown or in-application per-request cost increase. Check database query latency, external API response times, and connection pool wait times.
   • Rising avg_rt + rising busy ratio: workers are taking longer per request and accumulating. This is cascading starvation. The listen queue is filling. Check ss for backlog depth, harakiri count for kills, and whether a specific URI is consuming all workers.

3. Check for harakiri proximity. If avg_rt is approaching your configured harakiri timeout, workers are about to be killed. Each harakiri kill drops the in-flight request (client gets 502), respawns the worker, and the cycle repeats if the root cause persists. With gevent or async workers, harakiri is per-process, not per-coroutine. Every new request resets the counter, making harakiri less effective and the avg_rt-to-harakiri correlation less reliable.

4. Check per-worker divergence. If one worker has a dramatically higher avg_rt than the others, that worker hit a pathological code path: a regex catastrophe, an unbounded query, a blocking I/O call without timeout, or memory pressure specific to its process. Check that worker’s URI in the stats output and its /proc/<pid>/syscall or /proc/<pid>/wchan for the blocked state.

5. Compare avg_rt to true average. Compute running_time / requests per worker and compare it to avg_rt. If they diverge significantly, the EMA is either over-weighting recent outliers (avg_rt much higher than true average) or under-weighting them (avg_rt much lower). This tells you whether the problem is recent-onset or long-standing.

6. Check downstream dependencies. Rising avg_rt with no change in traffic, worker count, or application code almost always points downstream. Correlate avg_rt with database connection count, query latency, Redis response times, and external API health. The root cause is usually visible in the downstream system’s metrics, not in uWSGI’s.

Metrics and signals to monitor

Fixes

Downstream dependency slowdown

The most common cause. The application code has not changed, but each request waits longer on a database, cache, or external API.

• Check downstream connection pool utilization. If pools are at maximum, increase pool size or reduce worker count to lower total connection demand.
• Add or tighten timeouts on downstream calls. A database query or HTTP call without a timeout can hang indefinitely, and avg_rt will reflect the wait.
• Look for lock contention in the database (long-running transactions, advisory locks blocking queries).

Cascading starvation

Workers take longer per request, fewer workers are available, the listen queue fills, and connections start dropping.

• If a specific URI is consuming all workers (visible in the stats uri field), consider rate-limiting or circuit-breaking that endpoint at the load balancer to free workers for other traffic.
• Increase worker count if the system has CPU and memory headroom.
• Reduce per-request latency by addressing the downstream cause.

GC storms (Python)

Large heaps trigger longer GC pauses, which inflate per-request time.

• Monitor RSS growth and set --reload-on-rss to recycle workers before memory pressure becomes severe.
• Consider tuning Python GC thresholds if the application has known large-object allocation patterns.

GIL contention (threaded mode)

In Python threaded mode, CPU-bound work is serialized by the GIL despite multiple threads per worker. avg_rt rises because threads wait for the GIL, not because the application is doing more I/O.

• Move CPU-bound work to separate processes (more workers, fewer threads).
• Use --processes N --threads 1 for CPU-bound workloads instead of --threads M.

Single-worker poisoning

One worker has a dramatically higher avg_rt than others.

• Identify the worker’s PID from the stats output and check /proc/<pid>/syscall and /proc/<pid>/wchan for the blocked state.
• If harakiri is configured, wait for it to fire. If harakiri is not configured, the worker is stuck permanently. A manual kill -9 <pid> will force a respawn, but this drops the in-flight request. Use it as a last resort.
• Identify the URI that triggered the issue and add application-level protections (timeouts, result limits, input validation).

Prevention

• Do not use avg_rt as your latency SLI. Its EMA nature makes it volatile and it cannot express percentiles. Use access log percentiles (p50, p95, p99) per endpoint for SLI tracking.
• Always configure harakiri. Without it, stuck workers have no timeout and permanently consume a worker slot. Set it to 2-3x your expected maximum legitimate request duration. avg_rt approaching harakiri is an actionable early warning.
• Alert on deviation from baseline, not absolute thresholds. A REST API baselining at 50ms and an ML inference endpoint baselining at 2000ms need different thresholds. Alert when avg_rt exceeds 2x its recent baseline sustained over 5 minutes.
• Correlate avg_rt with busy ratio in dashboards. The combination is far more diagnostic than either signal alone.
• Monitor downstream dependencies alongside uWSGI metrics. When avg_rt rises, the root cause is usually visible in the downstream system first.
• Track per-worker avg_rt, not just the aggregate. A single outlier worker can be invisible in an aggregate average.

Monitoring with Netdata

• Netdata’s uWSGI collector polls the stats server at 1-second resolution, exposing per-worker avg_rt and busy ratio side by side. This granularity distinguishes a single-request spike from a sustained trend and supports the busy-ratio correlation that separates downstream slowdown from cascading starvation.
• harakiri_count is tracked as a rate (delta over time), surfacing when requests start being killed.
• Anomaly detection adapts to the EMA’s volatility without requiring per-endpoint threshold tuning.
• Database, Redis, and system metrics collected on the same host let you correlate avg_rt spikes with downstream latency or resource pressure without switching tools.

Related guides

• uWSGI all workers busy: reading the busy ratio before the queue fills
• uWSGI connection refused: clients turned away when the backlog overflows
• uWSGI reload thundering herd: capacity drops to zero during a slow restart
• uWSGI harakiri death spiral: workers killed and respawned while throughput collapses
• uWSGI harakiri not configured: stuck workers with no timeout and no recovery
• uWSGI harakiri timeout: setting it against request duration and nginx timeouts
• uWSGI harakiri-verbose: finding the blocked syscall behind a timeout
• uWSGI HARAKIRI ON WORKER: requests killed for exceeding the timeout
• How uWSGI actually works in production: a mental model for operators
• uWSGI listen backlog and net.core.somaxconn: sizing the connection queue
• uWSGI listen queue full: the backlog overflow that drops connections silently
• uWSGI listen_queue always zero: why the stats field is broken on Linux