$ guides / uwsgi / uwsgi-worker-oom-killed ▌

Operations Guides

uWSGI worker killed by the OOM killer: mysterious respawns under memory pressure

You see worker respawns in the uWSGI log that do not match your max-requests cadence. The master reports workers dying from signal 9 (SIGKILL), and either harakiri is not configured or the harakiri count is zero. Workers come back, serve traffic for a while, then die again. The interval shrinks over time.

This is the signature of the Linux OOM killer targeting uWSGI workers. As total worker RSS grows beyond available RAM, the kernel swaps, performance degrades, and the OOM killer selects the largest process on the system. In a uWSGI deployment, that process is almost always a worker. The master respawns it, the new worker re-imports the application, RSS climbs back, and the cycle repeats.

The danger is that this looks like a uWSGI-internal problem when the root cause is system-level memory exhaustion. Without correlating per-worker RSS, swap activity, and kernel OOM events with respawn data, the diagnosis stalls.

What this means

uWSGI workers are full copies of the application loaded into memory, so they are typically the largest consumers on the host. Each OOM kill is a SIGKILL: no cleanup, no graceful shutdown, no chance for the application to release resources.

flowchart TD
    A["Worker RSS grows over hours/days"] --> B["Total worker RSS exceeds available RAM"]
    B --> C["Kernel begins swapping (si/so nonzero in vmstat)"]
    C --> D["Request latency spikes, GC pressure rises"]
    D --> E["OOM killer targets highest-RSS process (a worker)"]
    E --> F["Master logs: killed by signal 9, respawns worker"]
    F --> G["New worker re-imports app, RSS climbs back"]
    G --> B

The master sees the worker exit and logs the canonical pattern:

DAMN ! worker X (pid: Y) died, killed by signal 9 :( trying respawn ...
Respawned uWSGI worker X (new pid: Z)

The respawn is immediate. But uWSGI has no awareness of system-level memory pressure. If the system is still out of memory when the new worker starts, that worker can be killed again within seconds, creating a tight respawn loop. In this state, the master burns CPU on fork and exec while serving zero useful traffic.

This pattern is distinct from graceful recycling. max-requests and reload-on-rss produce controlled worker exits: the worker finishes its current request, then exits. An OOM kill is violent and mid-request: the client gets a broken response, and write errors may spike. The respawn_count counter increments in both cases, but only OOM kills produce killed by signal 9 in the uWSGI log and Out-Of-Memory entries in dmesg.

Common causes

Cause	What it looks like	First thing to check
Application memory leak	All workers show linear RSS growth; sawtooth if max-requests is set	Plot per-worker RSS over time
No memory recycling configured	RSS grows indefinitely, no sawtooth pattern, no periodic respawns	Check config for `reload-on-rss`, `max-requests`, `max-worker-lifetime`
`reload-on-rss` threshold set too high	Workers exceed the configured limit but system OOMs first	Compare threshold against available RAM divided by worker count
Memory spike faster than master check	Sudden OOM kill with no prior RSS warning	Look for specific endpoints that allocate large objects
Container partial cgroup OOM kill	Only one worker dies; container keeps running, degraded	Check `dmesg` on the host; verify `memory.oom.group` setting

Quick checks

These commands are read-only and safe to run during an incident.

# Confirm OOM kills in kernel log (may require root or journalctl -k on systemd hosts)
sudo dmesg | grep -i "out of memory\|oom-kill\|killed process"

# Check swap activity (si = swap-in, so = swap-out, in KB/s)
vmstat 1 5

# Per-worker RSS from uWSGI stats server (requires stats server enabled)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.workers[] | select(.pid > 0) | {id: .id, rss_mb: (.rss / 1048576)}'

# Total worker RSS
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[] | select(.pid > 0) | .rss] | add / 1048576'

# Total respawn count across all workers
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].respawn_count] | add'

# Harakiri count (to distinguish timeout kills from OOM kills)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].harakiri_count] | add'

# System memory overview
free -m

# Per-worker RSS from /proc (if stats server is unavailable; PID file path varies by deployment)
for pid in $(pgrep -P $(cat /tmp/uwsgi.pid)); do
    awk '/VmRSS/ {print "pid='$pid' rss=" $2 "kB"}' /proc/$pid/status
done

How to diagnose it

Confirm the OOM kill. Run sudo dmesg | grep -i oom or journalctl -k | grep -i oom. Look for entries that name a uWSGI worker PID. If neither shows OOM kills, the signal 9 deaths may have another cause: a monitoring agent, an external script, or a container runtime enforcing limits.
Check swap activity. Run vmstat 1 5 and look at the si and so columns. Any sustained nonzero swap activity means the system is already past the performance cliff. Swapping makes every request slower, which increases the chance that more workers accumulate in memory simultaneously.
Measure total worker RSS. Sum the rss field across all alive workers from the stats server. Compare against available RAM from free -m (the available column, not free). RSS includes shared pages from copy-on-write, so the sum over-reports actual memory usage. Use /proc/<pid>/smaps_rollup for proportional set size (PSS) if you need more accurate accounting.
Determine whether recycling is configured. Check the uWSGI config for reload-on-rss, max-requests, and max-worker-lifetime. If none are set, workers run forever and RSS never resets. That is the most common cause of gradual OOM cycles.
Correlate respawn count with harakiri count. If respawn_count is rising but harakiri_count is zero, the respawns are not timeout kills. If respawn_count tracks harakiri_count closely, you have a harakiri death spiral, not an OOM problem. See the related guide on harakiri death spiral.
Look for per-worker RSS divergence. If one worker is much larger than the others, a specific request triggered a large allocation. Check the uri field on the large worker in stats. Uniform growth across all workers points to a systematic leak or fragmentation.
Check for container partial kills. If running in a container (Docker, Kubernetes), the cgroup OOM killer may kill a single worker process rather than the entire container. The pod stays running in a degraded state. The container is not restarted. Check dmesg on the host for oom-kill entries targeting worker PIDs.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Per-worker RSS	Shows individual memory growth	Linear growth across all workers over hours
Total worker RSS	What pushes the system over the edge	Approaching available RAM
Swap usage (si/so from vmstat)	Performance cliff indicator	Any sustained nonzero swap activity
Respawn count	Counts all worker deaths	Spikes that do not correlate with harakiri count
Harakiri count	Distinguishes timeout kills from OOM kills	Zero harakiri with rising respawns means not timeout
OOM events in dmesg	Definitive confirmation	Entries naming worker PIDs
Write errors	Mid-request kills produce broken responses	Spike coinciding with respawn events

Fixes

Configure reload-on-rss

reload-on-rss triggers a graceful worker exit when that worker’s RSS exceeds the threshold (specified in MB). The worker finishes its current request, then exits and is respawned with fresh memory. This is the standard defense against Python memory fragmentation and slow leaks.

# Recycle a worker when its RSS exceeds 512 MB
reload-on-rss = 512

Set the threshold below the danger zone. With N workers, total worker RSS at the threshold is approximately N times the threshold value. That number must be well under available RAM. For example, with 8 workers and 512 MB each, you need at least 4 GB for workers alone, plus headroom for the OS, page cache, and other processes.

Tradeoff: reload-on-rss may not fire before an OOM kill. The regular check runs at the end of each request inside the worker; for evil reloads, the worker’s mem_collector thread samples RSS every mem-collector-freq (default 3 seconds) and the master compares each master cycle (default 1 second). If a single request allocates hundreds of MB in under a second, the worker can exceed system RAM before the next check.

Set max-requests and max-worker-lifetime

max-requests recycles a worker after it serves N requests. max-worker-lifetime recycles a worker after N seconds. Both produce the same effect: periodic RSS reset. Use them together or as alternatives to reload-on-rss.

max-requests = 1000
max-worker-lifetime = 3600

Note that max-worker-lifetime is disabled by default. min-worker-lifetime defaults to 10 seconds and gates recycling: max-requests (and reload-on-rss / reload-on-as) do not trigger until the worker has been alive for at least that long.

Tradeoff: If the leak rate is high, the interval between recycles may still let RSS grow too large. Calculate peak RSS as leak_rate * max_requests and verify it fits in your memory budget.

Use cgroup memory limits

Cgroup limits are enforced by the kernel and cannot be missed by timing gaps in the master loop.

If your deployment is managed by systemd, Docker, or Kubernetes, set memory limits at that layer:

# systemd unit override
[Service]
MemoryMax=4G

# Docker
docker run --memory=4g ...

# Kubernetes
resources:
  limits:
    memory: "4Gi"

When a cgroup memory limit is hit, the OOM killer activates within the cgroup. In cgroup v2, setting memory.oom.group = 1 causes the kernel to kill all processes in the cgroup together rather than picking a single victim. This avoids the partial-kill problem where the master survives but workers die one by one in a degraded state.

Tradeoff: A hard cgroup limit that is too low will cause OOM kills even without a leak. Size it based on measured peak RSS plus headroom.

Do not use evil-reload-on-rss

evil-reload-on-rss works like reload-on-rss but kills the worker with SIGKILL mid-request instead of waiting for the request to finish. The client gets a broken response. The job of killing processes under memory pressure is better left to the Linux OOM killer, which has system-wide visibility that uWSGI lacks.

If you see intermittent client errors and notice evil-reload-on-rss in the config, switch to reload-on-rss (graceful) and investigate write errors.

Do not use limit-as to prevent OOM

limit-as sets a POSIX setrlimit() on the worker’s virtual address space (VSZ), not on physical RSS. It does not prevent OOM kills. Worse, when a worker hits the VSZ limit, it can enter a state where every request fails with MemoryError rather than being cleanly recycled. The worker stays alive but serves no useful traffic.

Fix the actual leak

Recycling is a band-aid. The leak itself needs profiling and fixing. Common sources in Python applications:

Global caches (dicts, lists) that only grow and never evict
Unreleased database connections (connection pool leaks in exception handlers)
C extension memory bugs (not tracked by Python’s garbage collector)
Circular references involving __del__ methods that the GC cannot collect

Use tracemalloc (Python standard library) or objgraph to identify which objects accumulate. Compare snapshots at intervals to find the growth source.

Prevention

Monitor per-worker RSS growth rate. Alert on a sustained positive slope over hours. The slope tells you how much runway you have before OOM.
Set reload-on-rss to a value that keeps total worker RSS under 70% of available RAM.
Verify recycling is actually happening. A sawtooth RSS pattern is healthy. A flat line at a high value means either no recycling or the threshold is never reached.
In containers, enable memory.oom.group (cgroup v2) or accept that partial worker kills will degrade the pod without restarting it.
Track swap usage alongside RSS. Any nonzero si/so in vmstat means the system is past the performance cliff. Alert before OOM, not after.

How Netdata helps

Netdata surfaces the signals that distinguish an OOM cycle from other respawn causes:

Per-worker RSS collected at per-second resolution from the uWSGI stats server, showing the sawtooth or steady growth pattern that precedes OOM.
System-level swap and memory pressure from the host, correlated with uWSGI worker metrics on the same dashboard so you can see when swap activity begins relative to RSS growth.
OOM killer events from the kernel log, annotated on the timeline so you can match kill events to worker respawn spikes.
Respawn rate from the stats server, differentiated from harakiri-driven respawns so you know whether workers are dying from memory or from timeouts.
Anomaly detection on RSS growth rate, flagging slow leaks before they become an OOM incident.