$ guides / uwsgi / uwsgi-worker-memory-leak ▌

Operations Guides

uWSGI worker memory leak: RSS climbing across all workers

Per-worker RSS climbs steadily. All workers track each other in lockstep. After a restart, RSS looks stable for hours or days, then the pattern repeats. With max-requests or reload-on-rss configured, you see a sawtooth: RSS rises, a worker recycles, RSS drops, then rises again. Without those directives, workers eventually hit the OOM killer or start swapping.

The cause may be a genuine leak in application code, Python allocator fragmentation that never returns pages to the OS, a C extension bug, or Docker file-descriptor inflation. The first two are operationally identical: RSS grows monotonically and does not come back down without a process recycle.

The key diagnostic signal is whether growth is uniform across all workers or isolated to one. Uniform growth means an application-level leak or allocator fragmentation. One worker diverging means a request-specific issue: a pathological code path, a large payload, or a ReDoS. This article covers the uniform case, which is the common one.

What this means

Each uWSGI worker is a forked copy of the application. RSS grows with leaked objects, cached data, and copy-on-write page faults as workers modify previously shared pages. Normal post-fork divergence stabilizes within the first few dozen requests. Growth that continues for hours or days is pathological.

Two mechanisms produce identical symptoms:

Real leak: Python objects accumulate and are never freed. Global caches that only grow, circular references involving __del__, unreleased database connections, or C extension bugs that allocate memory outside Python’s garbage collector.
Allocator fragmentation: CPython’s pymalloc allocator requests memory from the OS in arenas but rarely returns freed pages. Even after objects are garbage-collected, RSS plateaus at a high-water mark. This is not a leak in the strict sense, but it is operationally identical: RSS does not come down without a process restart.

A third mechanism is not a leak at all but looks like one. Docker containers often ship with extremely high file descriptor limits. uWSGI pre-allocates per-fd structures, which can inflate RSS to multiple gigabytes at startup. This is a one-step jump, not gradual growth, but it catches teams off guard.

The standard defenses are worker recycling directives: max-requests (reload after N requests), max-worker-lifetime (reload after N seconds), and reload-on-rss (reload when RSS exceeds a threshold). These do not fix the leak. They bound peak memory by restarting workers before RSS reaches dangerous levels.

flowchart TD
    A[Per-worker RSS climbing] --> B{Growth uniform across all workers?}
    B -- Yes --> C{Immediate jump at startup?}
    B -- No, one diverges --> D[Request-specific issue
Check that worker URI and code path]
    C -- Yes --> E[Docker fd inflation
Check max-fd and ulimit -n]
    C -- No, gradual over hours --> F{tracemalloc shows growth?}
    F -- Yes --> G[Python-level leak
Profile with tracemalloc or objgraph]
    F -- No --> H[Native or C-extension leak
or allocator fragmentation
Use memray or accept plateau]
    G --> I[Fix leak source
Tune recycling as safety net]
    H --> I
    E --> J[Set max-fd in uwsgi.ini]

Common causes

Cause	What it looks like	First thing to check
Python global state accumulation	RSS grows linearly across all workers, sawtooth under recycling	Search for module-level dicts, lists, or caches that only append
Allocator fragmentation	RSS plateaus high after initial growth, never returns to baseline	Check if RSS stabilizes (not infinitely growing) after warmup
C extension memory bug	RSS grows but `tracemalloc` shows no Python allocation growth	Profile with memray or valgrind on native code
Docker fd inflation	RSS jumps to multiple GB immediately at startup, not gradual	Check `ulimit -n` inside the container; set `max-fd = 4096`
`uwsgi.workers()` leak (2.0.18)	RSS grows if monitoring code calls `uwsgi.workers()`	Check uWSGI version; the leak affects 2.0.18 and is fixed in 2.0.19 (issue #2056)
Unreleased downstream connections	RSS grows steadily, connection pool metrics at max	Check database or Redis connection counts per worker

Quick checks

# Per-worker RSS and VSZ from the stats server (TCP example on port 9191)
# the stats server reports rss/vsz in bytes, so dividing by 1048576 yields MiB
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.workers[] | select(.pid > 0) | {id: .id, rss_mb: (.rss / 1048576), vsz_mb: (.vsz / 1048576)}'

# Per-worker RSS from /proc (independent of stats server)
for pid in $(pgrep -P $(cat /tmp/uwsgi.pid)); do
    rss=$(awk '/VmRSS/ {print $2}' /proc/$pid/status)
    echo "pid=$pid rss=${rss}kB"
done

# Proportional set size (accounts for shared pages; more accurate than RSS)
cat /proc/<worker_pid>/smaps_rollup | grep Pss

# Check file descriptor limit inside the container
ulimit -n

# Check OOM killer history (requires root)
dmesg | grep -i "out of memory" | tail -20

# Check swap usage
grep -E "SwapTotal|SwapFree" /proc/meminfo

# Check uWSGI version for known leak bugs
uwsgi --version

# Verify recycling configuration is present
grep -E "max-requests|reload-on-rss|max-worker-lifetime" /etc/uwsgi/app.ini

How to diagnose it

Confirm the growth pattern. Poll per-worker RSS at regular intervals. Look for linear, monotonic growth across all workers that restarts (sawtooth) after recycling. If RSS stabilizes after an initial warmup period, you may be looking at fragmentation rather than a leak. Both need management, but only one has a code fix.
Check for uniformity. If all workers grow at the same rate, the leak is in application code shared across all workers. If one worker diverges significantly, that is a request-specific issue. Check the uri field on the diverging worker in the stats server to identify the problematic endpoint.
Rule out Docker fd inflation. If RSS is very high immediately at startup rather than growing gradually, check ulimit -n inside the container. Docker containers often inherit limits exceeding one billion file descriptors. uWSGI pre-allocates per-fd structures, inflating RSS. Fix this with max-fd = 4096 in your uWSGI configuration. This is a one-time cost, not ongoing growth.
Profile Python allocations. Use tracemalloc to take snapshots before and after a burst of requests. If the traced allocations grow, you have a Python-level leak. Use objgraph to identify what object types are accumulating. Common culprits: module-level lists or dicts that only append, closures holding references, circular references with __del__ methods.
Check for native leaks. If RSS grows but tracemalloc shows no growth in Python allocations, the leak is in native code: numpy, pandas, Pillow, database drivers, or other C extensions. Use memray to profile native allocations. This is harder to fix and may require upstream bug reports.
Check your uWSGI version. Version 2.0.18 introduced a memory leak in uwsgi.workers() that accumulated thousands of copies of the app dict (upstream issue #2056, reported after upgrading from 2.0.17.1 to 2.0.18). If your application or monitoring code calls uwsgi.workers() and you are on 2.0.18, upgrade: the fix landed in 2.0.19.
Verify recycling is actually firing. If reload-on-rss is configured but RSS keeps climbing past the threshold, note that the worker checks its RSS at the end of each request, so a single long-running request that allocates heavily can exceed the limit before the check fires. In async (gevent) mode this per-request end-of-request check can trigger a worker exit while other greenlets are still mid-request; verify the behavior with your concurrency model.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Worker RSS (per worker)	Primary leak indicator; uniform growth points to app leak, divergence to request issue	Sustained positive slope over hours across all workers
Respawn rate	Distinguishes expected recycling from crash-driven respawns	Rate exceeding what `max-requests` alone would produce
VSZ	Large VSZ-RSS divergence may indicate mmap or fd accumulation	VSZ growing without RSS growth
Swap usage	Memory pressure cliff edge; performance collapses non-linearly when swapping begins	Any nonzero swap usage
OOM kills	Kernel killed a worker for memory; check `dmesg`	Any new OOM kill entries
Reload-on-rss threshold distance	Runway before recycling triggers	RSS approaching threshold with high growth rate
File descriptor count	Docker fd inflation or fd leak	Per-worker fd count approaching `ulimit -n`

Fixes

All configuration changes require a uWSGI reload to take effect.

Set worker recycling as a safety net

Even before finding the root cause, configure recycling to bound peak memory. Three directives work together:

# Recycle after N requests (with jitter to avoid all workers recycling simultaneously)
max-requests = 1000
max-requests-delta = 100

# Recycle after N seconds regardless of request count
max-worker-lifetime = 3600

# Recycle when RSS exceeds threshold (in MB)
reload-on-rss = 2048

These do not fix the leak. They prevent OOM kills by restarting workers before RSS reaches dangerous levels. The sawtooth pattern they produce is healthy: it means recycling is working.

Calculate implied per-worker peak RSS: leak_rate_per_request x max_requests. If your app leaks 1MB per request and max-requests is 1000, peak worker RSS before recycling is 1000MB each. Total across all workers is that times num_workers. Set reload-on-rss below the OOM danger zone to catch workers that grow faster than expected.

Fix Docker fd inflation

If RSS jumps to multiple gigabytes at startup, add max-fd to your uWSGI configuration:

# Limit pre-allocated per-fd structures
max-fd = 4096

This is not a gradual leak. It is a one-time allocation that inflates RSS before the first request.

Profile and fix the actual leak

For Python-level leaks, use tracemalloc snapshots to identify growing allocations:

import tracemalloc

tracemalloc.start()
# ... serve a burst of requests ...
snapshot1 = tracemalloc.take_snapshot()
# ... serve more requests ...
snapshot2 = tracemalloc.take_snapshot()

for stat in snapshot2.compare_to(snapshot1, 'lineno')[:10]:
    print(stat)

Use objgraph to find what object types are accumulating. Common patterns: module-level caches without eviction, global lists that only append, circular references involving __del__.

For native leaks where tracemalloc shows no growth, use memray. It traces native allocations and can pinpoint the C extension responsible. This requires more setup but is the only way to see allocations outside Python’s allocator.

Accept fragmentation plateaus

If tracemalloc shows objects being freed but RSS does not drop, you are looking at allocator fragmentation. CPython’s pymalloc does not return arenas to the OS in most cases. After fixing a real leak, RSS may plateau at a high-water mark and never return to pre-leak levels. This is expected. Worker recycling is the only way to return that memory. Do not chase a code fix that does not exist.

Prevention

Always configure recycling. max-requests, max-worker-lifetime, and reload-on-rss should be present in every production uWSGI config. Without them, a slow leak runs unchecked until OOM.
Set max-fd in Docker. Check ulimit -n inside your container. If it is extremely high, add max-fd = 4096 to prevent per-fd structure inflation.
Monitor per-worker RSS growth rate, not just point-in-time values. A worker at 500MB that has been stable for days is healthy. A worker at 500MB that was 200MB an hour ago is not. Track the slope.
Correlate respawn rate with recycling configuration. If respawns track max-requests at the expected cadence, recycling is working. If respawns exceed what max-requests alone explains, workers are crashing or being OOM-killed. Check dmesg.
Note that mules are not workers. max-requests, reload-on-rss, and max-worker-lifetime do not apply to uWSGI mules. If you run background work in mules, implement your own recycling via signal-based exit. A leaking mule will grow without any recycling safety net.
Keep uWSGI updated. Version-specific leaks have been fixed in later releases (2.0.18 uwsgi.workers() leak fixed in 2.0.19; 2.0.17.1 fixed a memory leak in HTTPS_CLIENT_CERTIFICATE). Check the changelog for your version.

How Netdata helps

Per-second RSS tracking per worker. Netdata collects workers[].rss from the uWSGI stats server at high frequency, making the growth slope immediately visible without manual polling.
Sawtooth pattern detection. When RSS drops sharply after a recycling event and then resumes climbing, the sawtooth is visible in the same chart as the respawn rate, confirming that recycling is working.
RSS divergence detection. Netdata shows per-worker RSS side by side. Uniform growth across all workers points to an application leak; one worker diverging points to a request-specific issue.
Correlation with system memory pressure. Worker RSS charts alongside swap usage and OOM killer events let you see the cliff approaching before performance collapses.
Respawn rate context. Netdata correlates respawn count with RSS thresholds, making it clear whether respawns are healthy recycling or crash-driven churn.