$ guides / uwsgi / uwsgi-how-it-works-in-production ▌

Operations Guides

How uWSGI actually works in production: a mental model for operators

uWSGI is a pre-fork application server. A single master process spawns a pool of worker processes, each running a full copy of your application. The master never serves requests. It manages worker lifecycle, enforces timeouts, and coordinates graceful reloads. Every request flows through the same path: a client connects to a socket, the kernel queues that connection in a listen backlog, and a worker calls accept() to pull it out and process it synchronously.

The simplicity hides several cliff-edge failure modes. When all workers are busy, there is no graceful degradation. Requests pile up in the kernel backlog until it fills, and then the kernel silently drops new connections. When a worker hangs, there is no recovery unless you configured harakiri. When a reload fails, every worker can disappear at once.

This article covers the abstractions you need before the runbooks: the master-worker relationship, the kernel backlog as the primary buffer, the worker state machine, the harakiri watchdog, the cheaper subsystem, and the difference between threading and async modes.

What it is and why it matters

The core design has three components:

Master process: Loads the application, opens the listening socket, then forks workers. Does not handle HTTP requests. Its job is worker management: spawning, monitoring, enforcing per-request timeouts (harakiri), and coordinating reloads.
Worker processes: Each worker is an OS-level process with its own memory space, containing a full copy of the application. Workers compete for incoming connections by calling accept() on the shared listening socket. When a worker accepts a connection, it processes that request to completion before accepting the next one.
Kernel listen backlog: The buffer between incoming connections and available workers. When a client connects, the kernel completes the TCP handshake and places the connection in this queue. When a worker calls accept(), it pulls the next connection. If all workers are busy and the queue fills, the kernel drops new connections.

Every production failure in uWSGI traces back to one of these components. Worker exhaustion is worker pool saturation. Harakiri storms are the watchdog firing repeatedly. Memory creep is per-worker process copies growing over time. Reload failures are the fork-replace lifecycle going wrong.

How it works

The master-worker fork model

By default, uWSGI loads the application once in the master process, then forks workers. This uses copy-on-write (COW) semantics: workers share the master’s memory pages until they write to them, at which point the kernel duplicates the modified page.

The lazy-apps option changes this: each worker loads the application independently after fork. This costs more memory and increases startup time, but avoids problems with libraries that are not fork-safe: database connection pools, thread locks, file handles initialized at import time. The older lazy option is deprecated and discouraged by the uWSGI project; it changes many internal defaults and exists only for backward compatibility.

Request flow through the kernel backlog

flowchart TD
    LB[nginx / LB] -->|incoming connections| SK[Kernel listen backlog]
    SK -->|accept| W1[Worker 1: app copy]
    SK -->|accept| W2[Worker 2: app copy]
    SK -->|accept| Wn[Worker N: app copy]
    Master[Master process] -->|fork + respawn| W1
    Master -->|fork + respawn| W2
    Master -->|fork + respawn| Wn
    Master -.->|harakiri: SIGKILL| W2
    W1 -->|response| LB

The listen backlog is the primary saturation point. It has a hard limit, set by --listen (default 100) and capped by the kernel’s net.core.somaxconn. When all workers are busy, incoming connections accumulate here. Once the backlog fills, the kernel drops new connections with no application-level log entry, no error counter, and no uWSGI-level signal.

One instrumentation gap: uWSGI’s listen_queue stats field is unreliable on standard Linux. The TCP measurement via TCP_INFO is inconsistent across kernel versions, and the UNIX socket measurement requires a non-standard kernel ioctl. The load field in the stats JSON is identical to listen_queue, not average latency as its name implies. Both fields almost always read 0. Measure queue depth externally:

ss -ltn  # TCP: check Recv-Q against the socket
ss -lxn  # UNIX sockets: check Recv-Q

Worker state machine

Each worker cycles through a simple state machine:

State	Meaning
`idle`	Worker is alive and waiting to accept a new connection
`busy`	Worker is processing a request
`cheap`	Worker has been scaled down by the cheaper subsystem (pid=0)
`pause`	Worker is intentionally suspended, for example during reload
`sig<N>`	Worker is handling a signal

A worker stuck in busy indefinitely means a request has hung. This is the most common silent failure: the worker appears alive in the process table but is doing nothing useful. Without harakiri, that worker stays stuck forever, permanently reducing capacity.

Harakiri: the per-request watchdog

Harakiri is uWSGI’s safety net for stuck workers. When configured via --harakiri, the master starts a timer each time a worker accepts a request. If the worker exceeds the timeout, the master sends SIGKILL and respawns it.

Key behaviors:

Harakiri is disabled by default. Without it, a stuck worker stays stuck forever. The absence of harakiri configuration is itself a risk.
Each harakiri kill increments harakiri_count and respawn_count.
--harakiri-verbose logs the blocked syscall and wchan when harakiri fires (Linux only, reads /proc/<pid>/syscall and /proc/<pid>/wchan).
If --harakiri-graceful-timeout is set, the master sends SIGTERM first, giving the worker a chance to clean up before SIGKILL.

Harakiri is not a failure indicator. It is a safety mechanism. The problem is whatever is causing requests to hang, not the kill itself.

Reloads: re-forking the pool

A graceful reload (triggered by SIGHUP or by touching a reload-trigger file in Emperor mode) tells all workers to finish their current requests and exit. The master then forks a fresh pool from the potentially updated application code.

This creates a capacity window: old workers drain while new workers start. If the application has slow startup (heavy imports, ML model loading, cache warming), this window can stretch to seconds or minutes. --chain-reload mitigates this by cycling workers one at a time instead of all at once.

A reload with broken application code (import error, syntax error, missing dependency) kills the old workers, and the new workers fail to start. The master repeatedly tries to spawn workers that immediately die. The result is zero accepting workers.

Version note: on uWSGI 2.0.x (the current stable line), SIGTERM means “brutally reload the stack,” not “shut down.” Set die-on-term = true if you want SIGTERM to behave as convention expects. This is documented to change in the unreleased 2.1 branch.

The cheaper subsystem: dynamic worker scaling

The cheaper subsystem adjusts the worker count based on demand. Algorithms:

spare (default): Maintains a minimum number of idle workers. Spawns new workers when idle count drops below the threshold.
spare2: A variant of spare with separate scaling thresholds.
backlog (Linux TCP only): Scales based on the kernel listen queue depth. Does not work with UNIX domain sockets.
busyness (requires cheaper_busyness plugin): Scales based on actual worker utilization over time.

When workers are scaled down by cheaper, they appear in the stats JSON with status: "cheap" and pid: 0. Monitoring must account for this: alerting on a fixed expected worker count generates false positives when cheaper legitimately scales down.

Threading vs async

Threaded mode: Each worker runs N threads that share the worker’s memory space. In Python, threads share the GIL, so CPU-bound work is serialized within a worker despite multiple threads. Threads help for I/O-bound workloads where the thread is blocked on a downstream response, but they do not provide CPU parallelism. Multiple worker processes are still needed for that. enable-threads is ON by default as of uWSGI 2.0.27. On older versions, set enable-threads = true explicitly for application-generated threads to function.

Async mode (gevent, asyncio): Each worker multiplexes many concurrent I/O-bound requests via an event loop. A single worker can handle hundreds or thousands of concurrent connections, but only if the application code yields to the event loop consistently. The uWSGI documentation warns: “If you are in doubt, do not use async mode.”

In async mode, the meaning of “worker busy” changes. A worker is “busy” when the event loop is running, which is almost always. Per-core in_request counts become the true concurrency indicator, not the binary busy/idle status. Worker busy ratio, the primary utilization signal in pre-fork mode, is nearly useless in async mode.

Where it shows up in production

The mental model maps to characteristic failure patterns:

Worker exhaustion. All workers are busy, requests queue in the kernel backlog, then get dropped when the backlog fills. This manifests as rising latency followed by connection failures. The cliff is sharp: there is no gradual degradation between “keeping up” and “dropping connections.”

Harakiri death spiral. A downstream dependency (database, external API) becomes unresponsive. Every request blocks on the dependency, exceeds the harakiri timeout, and the worker is killed and respawned. The respawned worker immediately accepts a new request that also blocks. Throughput collapses to near zero while respawn rate spikes. The system burns CPU on fork cycles while serving zero useful traffic.

Memory creep. Each worker holds a full copy of the application. RSS grows over time due to memory leaks, Python memory fragmentation, or cache bloat. Workers grow until OOM-killed or recycled by max-requests or reload-on-rss. This produces a sawtooth RSS pattern.

Stuck workers without harakiri. If harakiri is not configured, a worker that hangs on blocking I/O stays stuck forever. Capacity silently degrades as workers accumulate in stuck state. By the time you notice, the service may already be unresponsive, and harakiri_count reads zero, giving a false sense of safety.

Reload blackout. A graceful reload with broken application code leaves zero running workers. Old workers are killed, new workers fail to start, and the master churns through respawn attempts while serving nothing.

Tradeoffs and when to use them

Pre-fork vs threads vs async: Pre-fork (separate processes) is the safest default. It provides true isolation and CPU parallelism. Threads add concurrency within a process but are GIL-constrained in Python and introduce shared-state bugs. Async modes offer the highest concurrency for I/O-bound workloads but require careful application design and change the meaning of monitoring signals.

lazy-apps vs default fork: Default fork saves memory via COW but can break with non-fork-safe libraries. lazy-apps is safer but costs more memory and startup time. Each respawned worker re-imports the entire application.

Cheaper vs fixed workers: Cheaper reduces resource usage during low-traffic periods but introduces monitoring variability. Fixed workers provide predictable capacity but waste resources during quiet periods. If you use cheaper, alert on utilization ratios and accepting worker count, not absolute worker counts.

reload-on-rss vs evil-reload-on-rss: Both recycle workers when RSS exceeds a threshold. reload-on-rss is graceful: the worker finishes its current request, then exits. evil-reload-on-rss sends SIGKILL mid-request, which means the client sees a broken response. Monitor write errors alongside respawn rates to detect when mid-request kills are happening.

Signals to watch in production

Signal	Why it matters	Warning sign
Accepting worker count	Primary availability metric. Zero accepting workers with a running master is critical.	Count drops to zero or fluctuates rapidly
Worker busy ratio	Current concurrency utilization. At 100%, additional requests queue in the kernel backlog with no uWSGI-level visibility.	Sustained at or near 100%
Harakiri rate	Requests are hanging or running too long. Each kill means a dropped request and a respawn cycle.	Any sustained non-zero rate
Average response time (avg_rt)	Application responsiveness trend. Approaches harakiri timeout before workers start dying.	Sustained increase or approaching configured harakiri
Worker RSS	Memory health per worker. Consistent growth across all workers indicates a leak.	Sustained positive slope over hours
Respawn rate	Worker lifecycle churn. Normal with `max-requests` recycling. Abnormal when driven by crashes or harakiri.	Rate significantly exceeds expected `max-requests` cadence
Socket backlog depth (external)	Primary saturation signal. uWSGI’s internal `listen_queue` is unreliable on Linux. Use `ss` to measure externally.	Sustained non-zero Recv-Q
Exception rate	Unhandled application errors reaching the WSGI layer.	Baseline-relative increase

How Netdata helps

Per-second worker state visibility: Netdata’s uWSGI collector pulls the stats server JSON at per-second resolution, capturing worker busy ratios, accepting worker counts, and state transitions that coarser polling intervals miss.
Correlating harakiri with downstream latency: When harakiri rate spikes, overlay uWSGI harakiri counts against database query latency, external API response times, and system-level CPU and memory metrics in the same dashboard. The root cause of a harakiri storm is almost always downstream.
Detecting memory creep before OOM: Per-worker RSS collected over time reveals the sawtooth pattern of leaks and recycling. Anomaly detection can flag RSS growth trends before they hit the reload-on-rss threshold or trigger an OOM kill.
Backlog pressure without internal metrics: Since uWSGI’s listen_queue is unreliable on standard Linux, correlate host-level TCP and socket metrics with worker busy ratios for a more accurate picture of backlog pressure.
Distinguishing recycling from crashes: Overlay respawn rate with harakiri count to immediately see whether respawns are driven by healthy max-requests recycling or harakiri-induced crashes.