Two uWSGI options recycle workers when their resident set size crosses a threshold: reload-on-rss and evil-reload-on-rss. Both produce the same effect on a monitoring dashboard. The respawn_count counter increments, RSS drops to baseline, a new worker is forked. The memory sawtooth looks healthy. From the master process perspective, the two are indistinguishable.
The difference is in what happens to the in-flight request when the kill occurs.
reload-on-rss is graceful. The master signals the worker to shut down. The worker finishes its current request, delivers the full response, and exits. The replacement worker picks up the next connection from the listen queue. No client sees anything unusual.
evil-reload-on-rss is immediate. The master sends SIGKILL to the worker, even if it is mid-response. The client’s connection breaks. The response is truncated or never sent. The write_errors counter increments. But the respawn_count ticks up exactly as it would for a graceful recycle, and the RSS drops exactly as it would for a graceful recycle.
This is the trap: operators see memory management working while clients see intermittent 502s, broken responses, or connection resets. The write_errors counter in the stats server is the signal that separates the two cases.
What it is and why it matters
Both options accept a single integer value in megabytes. The threshold is per-worker, not a global pool limit. With processes = 10 and reload-on-rss = 512, each worker is independently recycled when its own RSS exceeds 512 MB. One worker hitting 513 MB triggers a respawn of that worker only; the other nine continue serving requests.
The total memory ceiling is approximately reload-on-rss * processes. Ten workers at a 512 MB threshold means the system can hold up to 5 GB of worker RSS before recycling kicks in, assuming all workers grow at the same rate. If they do not (one worker leaks faster), that worker recycles more frequently while the others remain stable.
The companion options reload-on-as and evil-reload-on-as work identically but check virtual set size (address space) instead of RSS. RSS is the more useful signal in practice because it reflects physical memory pressure, which is what leads to swapping and OOM kills. VSZ growth without RSS growth is typically benign.
Both options remain in the current uWSGI 2.0.x stable line. Neither has been deprecated.
How it works
The master process periodically checks each worker’s RSS. When a worker crosses the configured threshold, both paths fork a replacement and increment that worker slot’s respawn_count. The divergence is in the kill itself.
flowchart TD
R[Worker RSS exceeds threshold] --> GO{Which option?}
GO -->|reload-on-rss| G1[Master signals graceful exit]
G1 --> G2[Worker finishes current request]
G2 --> G3[Client gets full response]
G3 --> G4[Worker exits, respawn_count +1]
GO -->|evil-reload-on-rss| E1[Master sends SIGKILL]
E1 --> E2[Worker dies mid-request]
E2 --> E3[write_errors +1, client gets broken response]
E3 --> E4[Worker respawns, respawn_count +1]
G1 -.->|request exceeds worker-reload-mercy| E1The dashed line captures a critical edge case. Even reload-on-rss can degrade to a mid-request kill if the current request takes longer than worker-reload-mercy (default 60 seconds). The master signals a graceful exit, starts the mercy timer, and if the worker has not exited when the timer expires, force-kills it. A worker processing a slow upload or a long-running API call at the moment RSS crosses the threshold can still produce a broken response under reload-on-rss.
The mem_collector thread
The mem_collector thread appeared in uWSGI 2.0.11 (release notes: “Evil memory monitors … are now asynchronously managed by a dedicated thread”).
Before the mem_collector thread existed, evil reload checks ran synchronously in the master’s main loop. The master could miss runaway workers if it was busy with other lifecycle work, and operators reported workers consuming unlimited memory despite having evil-reload-on-rss configured.
The evil memory monitors (evil-reload-on-rss and evil-reload-on-as) are now managed by a dedicated mem_collector thread that asynchronously checks RSS. The companion option mem-collector-freq controls the polling frequency when evil reloads are enabled (default 3 seconds). Verify your uWSGI version before relying on evil reload timing.
Where it shows up in production
The most common scenario is a slow memory leak in a Python application. Workers grow steadily as the allocator fragments memory or global caches accumulate objects. Without any recycling configured, RSS climbs until the Linux OOM killer terminates a worker, which may target the master process instead of a worker.
Teams add reload-on-rss as a defense. Workers recycle gracefully before hitting the OOM danger zone. The RSS chart shows a clean sawtooth: linear growth, sharp drop at recycle, repeat. Response times stay flat because the graceful exit does not interrupt clients. This is the healthy pattern.
The problem appears when someone also adds evil-reload-on-rss as a “safety net,” or when reload-on-rss is set so close to the OOM limit that workers do not finish their requests before memory pressure becomes critical. In both cases, the dashboard looks identical to the healthy pattern. The respawn_count goes up. The RSS drops. Memory management appears to be working.
But clients are seeing broken responses. To detect this, correlate two signals:
# Check respawn rate (delta over polling intervals)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].respawn_count] | add'
# Check write_errors (sum across all workers and cores)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].cores[].write_errors] | add'
If write_errors spikes in lockstep with respawn_count deltas, workers are being killed mid-response. If write_errors stays flat while respawn_count increments, the recycles are graceful and clients are unaffected.
Tradeoffs and when to use it
reload-on-rss: the primary defense
Use reload-on-rss as your standard memory recycling mechanism. It is predictable, does not interrupt clients, and integrates cleanly with the worker lifecycle. Pair it with max-requests for a layered defense: max-requests recycles workers after N requests regardless of RSS (catching leaks that do not manifest as RSS growth), while reload-on-rss catches memory-specific growth.
evil-reload-on-rss: the emergency backstop
The case for evil-reload-on-rss is narrow. It exists for situations where a worker’s RSS is growing so fast that waiting for request completion would risk OOM-killing the entire system. If a single worker leaks 500 MB in seconds, the graceful path may not be fast enough.
The case against it is stronger. The SIGKILL is indiscriminate: it hits regardless of where the worker is in the request lifecycle. The client has no way to distinguish a mid-response kill from a network failure or a server crash. If your reverse proxy retries the request, the retry may land on the same recycling worker pool and fail again. At scale, this creates a thundering herd of retries on top of the memory pressure that triggered the kills.
Most operators avoid evil-reload-on-rss entirely and let the Linux OOM killer handle the catastrophic case. The OOM killer is at least system-aware: it considers total memory pressure, not just a single worker’s RSS, and it logs its actions to the kernel ring buffer where you can audit them.
Complementary options
| Option | What it does | How it complements reload-on-rss |
|---|---|---|
max-requests | Worker self-exits after N requests (graceful) | Catches leaks that do not manifest as RSS growth. Predictable cadence. |
cheaper-rss-limit-soft | Prevents the cheaper subsystem from spawning new workers when total RSS exceeds a threshold | Stops the problem from getting worse by adding more memory-hungry workers. Does not recycle existing workers. |
cheaper-rss-limit-hard | Hard RSS limit for the cheaper subsystem | Triggers worker reduction when total RSS exceeds this limit. Stricter than soft. |
worker-reload-mercy | Caps graceful shutdown time before force-kill (default 60s) | If set too low, even reload-on-rss can produce mid-request kills. |
limit-as | Caps virtual address space | Different mechanism: the worker hits MemoryError on allocation failure rather than being recycled. May leave the worker unable to handle requests. Not a substitute for reload-on-rss. |
Setting the threshold
The threshold needs to sit below the OOM danger zone with enough margin that the graceful exit can complete before system memory is exhausted.
Calculate the per-worker memory budget. Total system memory minus overhead for the OS, page cache, and non-uWSGI processes, divided by worker count. Total worker RSS (adjusted for copy-on-write sharing) should stay below 70% of available system memory.
Set
reload-on-rssbelow that per-worker budget. The gap between the threshold and the OOM limit needs to accommodate the time for graceful exit, which is bounded byworker-reload-mercy.Monitor the RSS growth rate. If workers grow at R MB per hour and the threshold is T MB above baseline, each worker has approximately T/R hours before recycling. If this interval is too short (workers recycling every few minutes), the leak needs fixing, not a higher threshold.
Account for copy-on-write. Linux COW means
rssover-reports per-worker usage because shared pages (shared libraries, pre-fork application code) are counted fully for each process. The sum of worker RSS values is higher than actual memory consumption. Use/proc/<pid>/smaps_rollupfor PSS if you need precise per-worker accounting on 2.0.x; the 2.1 development line adds apssfield to worker stats.
A common mistake is setting the threshold too close to normal peak RSS. If workers normally peak at 450 MB and the threshold is 512 MB, a small traffic spike that pushes RSS to 513 MB triggers a recycle that would not have happened with the threshold at 640 MB. The threshold should reflect the OOM danger zone, not the normal operating range.
Signals to watch in production
| Signal | Why it matters | Warning sign |
|---|---|---|
Worker RSS (workers[].rss) | Shows which workers are approaching the threshold | Any worker within 10% of the configured limit is about to be recycled |
Respawn rate (workers[].respawn_count delta) | Indicates recycling frequency and cause | Rate exceeding what max-requests alone would produce signals memory-driven recycling |
Write errors (workers[].cores[].write_errors delta) | The key discriminator between graceful and mid-request kills | Spikes that track respawn deltas 1:1 indicate evil-reload-on-rss or mercy timeout kills |
Harakiri count (workers[].harakiri_count delta) | Separates timeout kills from memory kills | Harakiri increasing alongside respawns means the worker was stuck, not leaking |
Delta requests (workers[].delta_requests) | Resets on respawn, confirming the recycling happened | Sudden reset across multiple workers indicates a mass recycling event |
The most important correlation is between respawn rate and write errors. Under reload-on-rss with healthy request durations, respawns happen but write errors stay flat. Under evil-reload-on-rss, or when worker-reload-mercy expires during a graceful exit, write errors spike in lockstep with respawns. Build an alert that fires when write_error deltas are non-zero during respawn events.
Also subtract the harakiri rate from the respawn rate to isolate memory-driven recycling from timeout-driven recycling. Every harakiri kill increments respawn_count, so without this subtraction, a harakiri storm looks identical to aggressive memory recycling.
How Netdata helps
Netdata collects uWSGI stats server output at per-second resolution and surfaces the signals that distinguish graceful recycling from mid-request kills:
- Per-worker RSS trended continuously, so the sawtooth pattern is visible and the growth rate between recycles is measurable.
- Respawn rate computed as a delta, not a cumulative total, so individual recycling events appear as discrete events rather than a monotonically climbing counter.
- Write errors per core, correlated in time with respawn events. A write_error spike that coincides with a respawn delta is the signature of a mid-request kill from
evil-reload-on-rssor aworker-reload-mercytimeout. - Harakiri count tracked as a separate delta, so memory-driven kills are distinguishable from timeout-driven kills without manual subtraction.
- ML-based anomaly detection on the write_errors rate, catching the early signs of mid-request kills before the error rate becomes user-visible at scale.
Related guides
- uWSGI all workers busy: reading the busy ratio before the queue fills
- uWSGI connection refused: clients turned away when the backlog overflows
- uWSGI harakiri death spiral: workers killed and respawned while throughput collapses
- uWSGI harakiri not configured: stuck workers with no timeout and no recovery
- uWSGI harakiri timeout: setting it against request duration and nginx timeouts
- uWSGI harakiri-verbose: finding the blocked syscall behind a timeout
- uWSGI HARAKIRI ON WORKER: requests killed for exceeding the timeout
- How uWSGI actually works in production: a mental model for operators
- uWSGI listen backlog and net.core.somaxconn: sizing the connection queue
- uWSGI listen queue full: the backlog overflow that drops connections silently
- uWSGI listen_queue always zero: why the stats field is broken on Linux
- uWSGI master process dead: total outage while the PID file lingers