uWSGI monitoring maturity model: from survival to expert

A four-level progression for uWSGI monitoring, from bare liveness checks to deep signal correlation. The levels are cumulative: you cannot skip to Level 3 by tracking RSS growth trends while ignoring harakiri rate. Each tier closes a specific class of blind spot that the previous tier could not see.

The model assumes the uWSGI stats server is enabled with --stats <address>. Without it, every level above survival is unreachable. HTTP access to the stats server requires the additional --stats-http flag; otherwise use uwsgi --connect-and-read <addr> for TCP sockets or socat - UNIX-CONNECT:<path> for UNIX sockets.

The four levels

flowchart TD
    L4["Level 4: Expert
per-endpoint harakiri, worker age skew,
connection-pool correlation, PSS divergence"]
    L3["Level 3: Mature
per-worker distribution, RSS trends,
serving URI, spooler/emperor, fd count"]
    L2["Level 2: Operational
busy ratio, harakiri/respawn/exception deltas,
per-worker RSS, avg_rt"]
    L1["Level 1: Survival
master alive, worker accepting,
non-zero throughput"]

    L1 --> L2 --> L3 --> L4

Level 1: survival

Goal: know whether uWSGI is alive and processing requests.

Three signals form the survival floor:

Master process alive. Check via PID file or process monitoring. If the master is dead, the entire application is down: no workers exist, no requests are processed.
At least one worker accepting. From the stats server, count workers where pid > 0 and accepting == 1 and status != "cheap". Zero accepting workers with a running master means the service cannot accept new requests. Connections queue in the kernel backlog until they time out.
Non-zero throughput during expected traffic hours. Sum workers[].requests across all workers and compute the delta between polling intervals. A sudden drop with no corresponding decrease in incoming traffic indicates workers are stuck or the application is rejecting requests.

What Level 1 catches: total outages, master process death, complete worker absence.

What Level 1 misses: worker starvation, memory leaks, harakiri storms, listen queue overflow, degraded latency. The service can pass all three checks while dropping 100% of real traffic because all workers are stuck on blocking calls. A health check that hits the stats endpoint will return a response even during complete worker starvation because the stats server is served by the master process, not by workers.

Level 2: operational

Goal: detect degradation before it becomes an outage.

Level 2 adds utilization, error rates, and resource pressure signals. This is the minimum a production deployment should have.

Worker busy ratio. Count workers where status == "busy" divided by alive workers (pid > 0 and status != "cheap"). Sustained 100% means every additional request queues in the kernel backlog with no uWSGI-level visibility. Brief spikes to 100% during traffic bursts are normal if they self-resolve within seconds.
Harakiri rate (delta). Sum workers[].harakiri_count across all workers and compute the delta between polling intervals. This counter is per-worker and monotonic, never reset even on respawn. Track the rate, not the absolute value. If harakiri is not configured, this counter is always 0, which is not a sign of health but a monitoring blind spot.
Respawn rate (delta). Sum workers[].respawn_count and compute the delta. Normal causes include max-requests recycling and reload-on-rss. To isolate crash-induced respawns from harakiri respawns, subtract the harakiri rate from the respawn rate.
Exception rate (delta). Sum workers[].exceptions and compute the delta. These are unhandled exceptions that reach the WSGI layer. Application-level error handlers that catch and return 500s do not increment this counter.
Average response time (avg_rt) per worker. The workers[].avg_rt field is an exponential moving average with factor 0.5, computed internally as (old_avg_rt + current_request_time) / 2. Each new request contributes 50% of the new value. After approximately 7 requests, older contributions fall below 1%. This makes avg_rt responsive to recent changes but volatile. A single slow request can move it significantly. Do not use it as a latency SLI; use access logs or application-level metrics for that.
Per-worker RSS. The workers[].rss field reports resident set size in bytes. The --memory-report option must be enabled for RSS and VSZ values to appear in the stats output. Consistent growth across all workers indicates a memory leak. Divergence, where one worker is much larger than others, indicates a request-specific memory issue.
Listen queue awareness. The listen_queue field in uWSGI stats is unreliable on standard Linux: TCP measurement via TCP_INFO varies across kernel versions, and UNIX socket measurement requires a non-standard kernel ioctl. The load field appears to be identical to listen_queue, not average latency despite its name. The listen_queue_errors field exists in the JSON output but may not be incremented in practice. At Level 2, the minimum requirement is awareness that these fields are unreliable and must be measured externally.

What Level 2 catches: worker pool exhaustion, coarse memory leak detection, harakiri storms, application error spikes, capacity saturation.

What Level 2 misses: which endpoint is causing the problem, whether respawns are healthy recycling or crashes, per-worker asymmetry, file descriptor exhaustion, spooler or emperor subsystem failures.

Level 3: mature

Goal: diagnose root causes, not just detect symptoms. Level 3 adds per-worker granularity, subsystem health, and external signals that compensate for uWSGI’s internal limitations.

Per-worker request distribution. Compare workers[].requests across workers. Uneven distribution suggests load imbalance, lock contention, or thundering herd effects. Enable --thunder-lock for uniform accept() distribution across workers.
RSS growth trend. Track the slope of RSS over hours, not the point-in-time value. Linear growth across all workers is a leak. A sawtooth pattern with periodic respawns is healthy recycling via max-requests or reload-on-rs.
Currently serving URI. The workers[].uri field on busy workers identifies which endpoint is consuming capacity. During starvation, if all busy workers show the same URI, that endpoint is the root cause.
Respawn classification. Expected respawn rate equals (total_requests_per_second / max_requests_per_worker) * num_workers. Respawn rate significantly above this, especially when tracking harakiri count 1:1, indicates crashes rather than recycling.
Write and read error rates per core. The workers[].cores[].write_errors and workers[].cores[].read_errors fields track socket errors during request and response handling. Sustained high write error rates indicate slow responses causing clients to disconnect. Suppressed from stats output if --stats-no-cores is enabled.
Spooler health (if configured). The spoolers[] array exposes tasks (pending count), running (0/1), and respawns. Growing tasks count means the spooler cannot keep up. Any non-zero respawn rate indicates spooler instability.
Emperor and Vassal health (if Emperor mode). Each vassal needs independent monitoring. Emperor health does not imply vassal health. A vassal can die and fail to restart due to a broken config while the Emperor continues running. The count of active vassals versus expected vassals is the primary health indicator.
Kernel overflow counters. Use nstat -az TcpExtListenOverflows TcpExtListenDrops to detect silent connection drops. These counters are system-wide, not per-socket. On multi-service hosts, correlate with per-socket queue depth via ss -ltn or ss -lxn to attribute drops to uWSGI specifically.
File descriptor count per worker. Check with ls -1 /proc/<worker_pid>/fd | wc -l against ulimit -n. File descriptor exhaustion causes silent connection failures with no uWSGI-level signal. Usage should stay below 80% of the soft limit.
Signal queue depth. Top-level signal_queue (master) and per-worker workers[].signal_queue. Should be 0 in steady state. Any sustained non-zero value means internal uWSGI signals (timers, file monitors, custom signals) are backing up.
Accepting worker count (cheaper-aware). When the cheaper subsystem is active, worker count fluctuates by design. Alert when accepting workers drop below the cheaper minimum, not against a fixed expected count. Workers with status: "cheap" have pid: 0 and are intentionally scaled down.

What Level 3 catches: root cause identification during incidents, which endpoint is slow, whether memory growth is systemic or per-worker, spooler or emperor failures, silent connection drops at the kernel level, file descriptor leaks.

What Level 3 misses: deep correlations between uWSGI metrics and downstream dependencies, precise per-endpoint failure attribution, copy-on-write memory accounting, predictive indicators.

Level 4: expert

Goal: predict failures and correlate uWSGI signals with system-level and downstream signals. Level 4 signals are typically added after the second or third major incident reveals a gap.

Per-endpoint harakiri attribution. Identify which endpoints are timing out. The stats server does not expose per-endpoint harakiri counts natively. Operators typically derive this by correlating harakiri log entries with access logs.
Worker age skew. Track workers[].last_spawn timestamps to detect asymmetric recycling. If one worker respawns far more frequently than others, it may be hitting a specific code path that causes crashes or memory bloat.
Connection-pool correlation. Correlate worker busy ratio and response time with downstream connection pool utilization (database, Redis, external APIs). uWSGI does not natively expose database connection pool metrics. This requires external instrumentation on the downstream systems, displayed alongside uWSGI metrics on the same timeline.
PSS divergence. RSS over-reports per-worker usage because shared pages (shared libraries, copy-on-write pages) are counted fully for each process. PSS (Proportional Set Size) distributes shared pages proportionally, giving better accounting. uWSGI’s stats server provides RSS and VSZ but not PSS. PSS requires reading /proc/<pid>/smaps_rollup externally. Track how quickly workers diverge from the master post-fork to measure copy-on-write efficiency over the worker’s lifetime.
Stuck request age detection. When a core has in_request == 1, compute current_time - workers[].cores[].req_info.request_start to get the elapsed time of the in-flight request. This detects stuck requests before they trigger harakiri, especially important when harakiri is not configured. The vars field in cores can expose request headers and URI of the in-flight request, useful for diagnosing which endpoint is stuck.
External socket queue monitoring. Use ss -ltn 'sport = :PORT' for TCP or ss -lxn for UNIX sockets. Check Recv-Q for current queue depth and Send-Q for the configured backlog limit. This compensates for the unreliable listen_queue stats field and provides the real saturation picture.
Running time per request. Compute workers[].running_time / workers[].requests for the true cumulative average processing time per request. This is more stable than avg_rt for trend analysis, though it does not reflect recent changes as quickly. Note that whether running_time resets on worker respawn is not confirmed by source inspection; only delta_requests is confirmed to reset.

What Level 4 catches: predictive failure indicators, downstream dependency correlations, copy-on-write memory accounting, early stuck-request detection before harakiri fires, true per-request latency trends.

Signal reference by level

Level	Signals added	Key question answered
1: Survival	Master alive, accepting worker count, throughput	Is uWSGI alive?
2: Operational	Busy ratio, harakiri/respawn/exception deltas, avg_rt, per-worker RSS	Is it degrading?
3: Mature	Per-worker distribution, RSS trends, serving URI, respawn classification, spooler/emperor health, kernel overflows, fd count, signal queue	What is broken and where?
4: Expert	Per-endpoint harakiri, worker age skew, connection-pool correlation, PSS divergence, stuck request age, external socket queue	Why is it breaking and what fails next?

Configuration prerequisites

Several signals across all levels require explicit configuration beyond --stats:

--memory-report: must be enabled for RSS and VSZ values to appear in stats output. Without it, all memory-related signals at Level 2 and above are blind.
--harakiri <seconds>: without it, stuck workers have no timeout. harakiri_count is always 0, which looks healthy but is a monitoring blind spot. A single hung request can reduce capacity by one worker indefinitely.
--harakiri-verbose: enables logging of the blocked syscall and wchan when harakiri fires (Linux only, reads /proc/<pid>/syscall and /proc/<pid>/wchan). Essential for diagnosing what workers are stuck on.
--stats-http: required if you want to access the stats server via HTTP with curl. Without it, the stats server serves raw JSON on a socket and you must use uwsgi --connect-and-read or socat.
--thunder-lock: required for uniform request distribution across workers. Without it, the kernel’s accept() thundering herd behavior can cause uneven load that masquerades as a worker-level problem.

How Netdata helps

Netdata collects uWSGI stats server output at per-second resolution, which matters for signals that change quickly during incidents. The correlations that shorten diagnosis time:

Worker busy ratio + avg_rt at per-second granularity catches capacity exhaustion before the listen queue fills. At 10-second polling intervals, a complete starvation event can run for up to 10 seconds undetected.
Harakiri rate + respawn rate correlation distinguishes a harakiri death spiral (respawns track harakiri 1:1, throughput collapses) from healthy max-requests recycling (steady respawns, zero harakiri).
Per-worker RSS trends reveal the sawtooth pattern of memory recycling versus the linear growth of a genuine leak, visible over hours rather than at a single point in time.
Kernel TCP overflow counters alongside uWSGI worker metrics surface the silent connection drops that uWSGI’s own listen_queue and listen_queue_errors fields cannot report.
Cross-layer correlation places uWSGI worker signals next to system CPU, memory, and network metrics on the same timeline. When response time increases, the correlation with downstream dependency health is immediately visible rather than requiring a separate dashboard.