uWSGI capacity planning: the leading indicators before saturation

uWSGI does not degrade gracefully. Performance looks fine until a hard limit is reached, then the service drops requests, kills workers, or refuses connections. Capacity planning is about knowing where each cliff is, measuring how close you are, and adding capacity before the edge.

Four resources saturate as cliffs: worker pool (concurrency), memory (per-worker RSS), socket backlog (kernel listen queue), and file descriptors. Each has a distinct failure mode, leading indicator, and measurement method. Some signals come from the uWSGI stats server JSON. Others require external tools because uWSGI’s internal measurement is broken or silent.

Use this reference to answer two questions: which resource will saturate first under your current growth trajectory, and how much runway remains.

The cliff-edge model

All four resources share the same degradation shape: flat performance until a threshold, then abrupt failure. The threshold, failure mode, and warning time differ for each, but none give you a gradual slope to ride while you plan.

flowchart TD
    subgraph Workers["Worker pool"]
        WA["Busy ratio > 70% at peak"] --> WB["Cliff: 100% busy, queue fills"]
    end
    subgraph Mem["Memory"]
        MA["RSS approaching 70% RAM"] --> MB["Cliff: swap begins, OOM kills"]
    end
    subgraph Backlog["Socket backlog"]
        LA["Recv-Q > 50% of backlog"] --> LB["Cliff: connections refused"]
    end
    subgraph FDs["File descriptors"]
        FA["FDs > 50% of limit"] --> FB["Cliff: EMFILE, silent drops"]
    end

You cannot wait for response time to degrade and then react. By the time latency rises measurably, you are already at or past the cliff edge for at least one resource. The leading indicators below fire before the cliff, not at it.

Worker pool (concurrency)

The worker pool is the hard limit on concurrent request processing. When all workers are busy, the next request does not get served more slowly. It queues in the kernel backlog, and if the backlog fills, it gets dropped.

Leading indicators:

• Busy ratio trending above 70% at peak. A traffic burst or downstream slowdown can push you to 100% within seconds.
• Accepting worker count declining. Workers stuck in busy or being recycled by harakiri reduce effective capacity below the configured pool size.
• First harakiri events appearing. Even isolated kills signal that some requests are approaching the timeout. Investigate before the rate climbs.
• avg_rt trending upward. avg_rt is an exponential moving average with 50% weight on the most recent request, not a cumulative average. A single slow request shifts it significantly. Use it for trend detection, not as a latency SLI.

Degradation curve: Cliff-edge at 100% busy. Below that, queuing is minimal. At 100%, Little’s Law takes over: queue depth grows non-linearly with arrival rate. A 10% traffic increase at 90% utilization can cause a 10x increase in queue depth and response time.

Runway math:

runway_seconds = (accepting_workers - busy_workers) / arrival_rate_per_second

Example: 10 accepting workers, 8 busy, 5 req/s arrival gives 0.4 seconds of headroom. That is not a planning horizon. That is a heartbeat.

For longer-term planning, track peak busy ratio at daily peak. If peak is at 85% and traffic grows 5% per week, saturation arrives in roughly 3 weeks. Bursts will hit 100% sooner.

Headroom target: Peak busy ratio below 70% for services with bursty traffic, below 80% for steady-state workloads. With the cheaper subsystem active, ensure the cheaper minimum is at least 20% of the maximum worker count so scaled-down workers are available when traffic spikes.

Measurement caveats: In threaded mode, “busy” means at least one thread is active, not that all threads are occupied. Per-core in_request is needed for thread-level visibility. In gevent or async mode, “busy” means the event loop is running, which is almost always. Busy ratio is nearly meaningless in async deployments. See uWSGI in gevent/async mode: why worker busy ratio stops meaning anything.

# Worker busy ratio from stats server
uwsgi --connect-and-read 127.0.0.1:9191 | jq '([.workers[] | select(.status == "busy")] | length) as $busy | ([.workers[] | select(.pid > 0 and .status != "cheap")] | length) as $alive | if $alive > 0 then ($busy / $alive * 100) else 0 end'

Memory (per worker)

Each worker is a full copy of the application loaded into memory. N workers times per-worker RSS is your memory floor. Python and Ruby memory allocators rarely return freed memory to the OS, so RSS tends to grow monotonically even without a true leak.

Leading indicators:

• Worker RSS growth rate (slope over hours). A sustained positive slope across all workers indicates a leak or fragmentation.
• Total worker RSS approaching 70% of system RAM. Below this, the OS page cache and kernel buffers have room. Above it, memory pressure begins.
• Major page fault rate increasing. Check /proc/vmstat for rising pgmajfault counts. These signal the kernel is doing expensive memory operations.
• Swap usage nonzero. Any swapping means you are already past the warning zone.

Degradation curve: Gradual, then cliff. Performance degrades slowly as memory pressure increases (more GC cycles, possible swap I/O). Then the OOM killer fires and workers die abruptly. With max-requests or reload-on-rss configured, the cliff is replaced by periodic recycling, producing a sawtooth RSS pattern.

Runway math:

runway_days = (reload_on_rss_mb - current_rss_mb) / rss_growth_rate_mb_per_day

Without reload-on-rss, substitute the system memory ceiling:

runway_days = (system_ram_mb * 0.7 - total_worker_rss_mb) / aggregate_growth_rate_mb_per_day

max-requests recycling resets per-worker RSS, extending the runway. But if the leak rate is high, recycling cadence may cause capacity dips during each cycle.

Headroom target: Total worker RSS, adjusted for copy-on-write sharing, should stay below 70% of system memory. Use PSS (Proportional Set Size) for accurate accounting, since RSS over-reports by counting shared pages fully for each process. PSS is available via /proc/<pid>/smaps_rollup on the host. Verified against the uWSGI source: the 2.1-dev (master) stats server adds per-worker pss and uss fields, while stable 2.0.x (up to 2.0.31) does not.

Measurement caveats: Linux copy-on-write means RSS over-reports per-worker usage. Workers start with shared pages and diverge over time as they write to memory. RSS growth is expected initially and stabilizes. The concern is growth that continues indefinitely. Workers recycled by max-requests show a sawtooth pattern, which is healthy.

# Per-worker RSS from stats server
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.workers[] | select(.pid > 0) | {id: .id, rss_mb: (.rss / 1048576)}'

Socket backlog (kernel listen queue)

The listen queue is the kernel buffer between incoming connections and available workers. When workers cannot accept connections fast enough, connections pile up here. When the queue fills, the kernel drops new connections silently.

Leading indicators:

• Recv-Q (from ss) nonzero during peaks. Any sustained nonzero value means workers cannot keep up.
• Busy worker ratio approaching 100%. This is the precursor. If workers are saturated, the backlog is filling.
• Kernel TcpExtListenOverflows incrementing. This counter tracks connections dropped because the backlog was full.

Degradation curve: Pure cliff. Below the backlog limit, requests wait but eventually get served. At the limit, connections are refused.

Runway math: If the listen queue hits N out of a configured limit of M during peaks, the cliff arrives when N consistently reaches M. If peak traffic grows at a known rate per week, estimate when N will reach M.

Headroom target: Peak listen queue depth below 50% of the configured backlog. Recv-Q should be 0 in steady state. Any sustained nonzero value is a warning sign.

Critical measurement gotcha: uWSGI’s internal listen_queue stats field is unreliable on standard Linux. The TCP measurement relies on TCP_INFO, which behaves inconsistently across kernel versions. The UNIX socket measurement requires a non-standard kernel ioctl. The load stats field is identical to listen_queue (not a latency metric, despite its name). Both fields almost always read 0 regardless of actual backlog.

The listen_queue_errors field is dead code. It exists in the JSON output but is never incremented in the uWSGI source. It is always 0 and provides no signal.

Measure externally:

# TCP listen queue depth (check Recv-Q and Send-Q columns)
ss -ltn 'sport = :8000'

# UNIX socket listen queue depth
ss -lxn | grep uwsgi

# Kernel-level overflow counters (system-wide, not per-socket)
nstat -az TcpExtListenOverflows TcpExtListenDrops

Additional gotcha: The default --listen backlog in uWSGI is 100. The kernel’s net.core.somaxconn may be 128 on older kernels. The kernel silently clamps the requested backlog to somaxconn, so setting --listen 1024 without raising somaxconn results in an effective backlog of the kernel limit. Always verify both values.

See uWSGI connection refused: clients turned away when the backlog overflows.

File descriptors

Each worker holds file descriptors for its listening socket, database connections, log files, application-opened files, and pipes. File descriptor exhaustion causes accept() and open() to fail with EMFILE. New connections are silently rejected with no uWSGI-level signal.

Leading indicators:

• Per-worker open fd count trending upward over time. A sustained upward slope indicates an fd leak (unclosed connections, file handles, or pipes).
• Total open fds for all workers approaching 50% of ulimit -n. Above 50%, plan for action.
• VSZ growing alongside fd count. When both virtual memory size and fd count climb together, the cause may be memory-mapped file handles that are not being closed.

Degradation curve: Silent cliff. There is no gradual degradation. At the limit, socket operations fail immediately. The only evidence is EMFILE errors in application logs or silent connection failures.

Runway math:

runway_hours = (fd_limit - current_fd_count) / fd_growth_rate_per_hour

File descriptor leaks often accelerate rather than growing linearly, so this projection is optimistic. If the growth rate is itself increasing, shorten the estimate.

Headroom target: Open fd count below 50% of the per-process soft limit under normal operation. Set the limit generously (65535 or higher) unless there is a specific reason not to.

Measurement: Not available in uWSGI stats. Check at the OS level. Adjust the PID file path to match your deployment:

# Per-worker fd count and limit (adjust /tmp/uwsgi.pid to your master PID file)
for pid in $(pgrep -P $(cat /tmp/uwsgi.pid)); do
    count=$(ls /proc/$pid/fd 2>/dev/null | wc -l)
    limit=$(awk '/^Max open files/ {print $4}' /proc/$pid/limits)
    echo "pid=$pid fds=$count limit=$limit"
done

Which cliff hits first

The binding constraint depends on workload pattern and configuration. Re-evaluate after any significant traffic growth, worker count change, or application code change that affects request duration.

Headroom targets summary

These targets are conservative by design. They leave room for traffic spikes, downstream slowdowns, worker restarts, and the measurement lag inherent in polling. If you operate consistently above these targets, the next burst will be the one that takes you down.

How Netdata helps

Netdata provides per-second granularity on the signals that expose each resource’s approach to its cliff:

• Worker busy ratio at per-second resolution catches brief saturation spikes that 10-60 second polling misses entirely. The listen queue can fill and overflow between coarse polls.
• Per-worker RSS trends surface the growth slope that predicts the memory cliff days or weeks before swap begins.
• ML-based anomaly detection on busy ratio, avg_rt, and request throughput flags deviations from the learned baseline before static thresholds fire.
• Host-level fd count and limit from the /proc filesystem fills the gap uWSGI does not expose natively.
• ss-derived socket metrics and kernel counters like TcpExtListenOverflows provide the listen queue signal that uWSGI’s own stats cannot reliably report.
• Correlation across worker, memory, and fd metrics in a single timeline view shortens the path from “something is wrong” to “which cliff are we approaching.”

Related guides

• uWSGI all workers busy: reading the busy ratio before the queue fills
• uWSGI avg_rt is not a real average: why the latency number lies
• uWSGI chain reload: cycling workers one at a time for zero-downtime deploys
• uWSGI connection refused: clients turned away when the backlog overflows
• uWSGI in gevent/async mode: why worker busy ratio stops meaning anything
• uWSGI threaded mode and the GIL: why more threads don’t add CPU parallelism
• uWSGI reload thundering herd: capacity drops to zero during a slow restart
• uWSGI harakiri death spiral: workers killed and respawned while throughput collapses
• uWSGI harakiri not configured: stuck workers with no timeout and no recovery
• uWSGI harakiri timeout: setting it against request duration and nginx timeouts
• uWSGI harakiri-verbose: finding the blocked syscall behind a timeout
• uWSGI HARAKIRI ON WORKER: requests killed for exceeding the timeout