uWSGI respawn rate high: telling crashes apart from max-requests recycling

You see respawn_count climbing across your uWSGI workers. The question is whether workers are crashing or whether uWSGI is doing what you configured it to do.

respawn_count is a single monotonic counter per worker slot that increments for every reason a worker can die and come back: max-requests recycling, reload-on-rss memory recycling, harakiri timeout kills, segfaults, OOM kills, and manual kill -9. The raw number tells you workers are churning. It does not tell you why.

Without knowing your max-requests configuration and subtracting the harakiri delta, the respawn signal is uninterpretable noise. This article walks through the arithmetic and the timing patterns that separate healthy recycling from real failures.

What this means

respawn_count lives on the worker slot struct, not on the process. It persists through respawns and is never reset. The stats server exposes it as workers[].respawn_count (per-worker, monotonic). There is no separate counter for crashes versus intentional recycling. You must infer the cause by combining three signals:

1. Your max-requests configuration. If recycling is configured, some respawns are expected. If it is not, every respawn is abnormal.
2. The harakiri_count delta. Every harakiri kill increments respawn_count. Subtract the harakiri rate to see what remains.
3. The timing pattern. Staggered, periodic respawns at a predictable cadence indicate healthy recycling. A simultaneous spike across all workers indicates a mass-kill event: a deploy, an OOM-killer sweep, or all workers hitting max-requests at once without a delta offset.

The diagnostic arithmetic is straightforward. Compute the expected recycling rate from your traffic volume and max-requests setting, then subtract the harakiri delta from the measured respawn rate. What remains above the expected recycling baseline is crash-driven or reload-on-rss-driven churn.

Common causes

Quick checks

# Total respawn count and harakiri count across all workers
uwsgi --connect-and-read 127.0.0.1:9191 | jq '{total_respawns: ([.workers[].respawn_count] | add), total_harakiri: ([.workers[].harakiri_count] | add)}'

# Per-worker breakdown: respawn_count, harakiri_count, requests, RSS
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.workers[] | select(.pid > 0) | {id: .id, respawns: .respawn_count, harakiri: .harakiri_count, requests: .requests, rss_mb: (.rss / 1048576)}'

# Total request count (for computing expected recycling rate)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].requests] | add'

# Check for OOM kills in kernel log
dmesg -T | grep -iE "out of memory|killed process" | tail -20

# Check uWSGI log for crash patterns (violent deaths vs graceful exits)
grep -E "DAMN ! worker|killed by signal|NO MERCY" /var/log/uwsgi/*.log | tail -20

Adjust the stats socket address (127.0.0.1:9191) to match your deployment. If your stats server uses a UNIX socket, substitute uwsgi --connect-and-read /path/to/stats.sock. If you enabled --stats-http, curl http://127.0.0.1:9191 also works.

How to diagnose it

Step 1: Establish the expected recycling rate

If max-requests is configured, compute the expected recycling rate:

expected_total_recycling_rate = total_requests_per_second / max_requests

For example, with 100 req/s fleet-wide and max-requests = 1000, expect approximately 0.1 respawns per second (one recycling event every 10 seconds). This assumes traffic is evenly distributed across workers.

Step 2: Measure the actual respawn delta

Poll respawn_count twice with a known interval (60 seconds is a good starting point) and compute the delta. Compare the measured rate to the expected recycling rate from Step 1.

Step 3: Subtract the harakiri delta

Pull harakiri_count over the same interval. Every harakiri kill increments respawn_count, so subtract the harakiri delta from the respawn delta:

non_harakiri_respawns = respawn_delta - harakiri_delta

Compare non_harakiri_respawns to the expected recycling rate from Step 1. If they are close, your respawns are healthy max-requests or reload-on-rss recycling. If non_harakiri_respawns significantly exceeds the expected recycling rate, you have crash-driven respawns.

Step 4: Classify the remainder

The following decision tree summarizes the classification logic:

flowchart TD
    A["respawn_count delta rising"] --> B{"max-requests configured?"}
    B -- No --> C["All respawns are abnormal.
Investigate crashes and OOM."]
    B -- Yes --> D["Compute expected rate:
total_rps / max_requests"]
    D --> E{"Measured rate close to expected
AND harakiri delta is zero?"}
    E -- Yes --> F["Healthy max-requests recycling.
No action needed."]
    E -- No --> G["Subtract harakiri delta
from respawn delta"]
    G --> H{"Remaining respawns > 0?"}
    H -- No --> I["Harakiri-driven respawns.
Check downstream dependencies."]
    H -- Yes --> J["Crash or mass-kill event.
Check logs and dmesg."]
    J --> K{"Simultaneous across
all worker slots?"}
    K -- Yes --> L["Mass-kill: deploy, OOM sweep,
or max-requests without delta."]
    K -- No --> M["Individual crashes.
Check signal number in uWSGI log."]

Step 5: Correlate with logs and system signals

For crash-driven respawns, the uWSGI log distinguishes violent from graceful deaths:

• Graceful exit (max-requests, reload-on-rss): worker finishes its current request, exits cleanly, master forks a replacement. Log shows Respawned uWSGI worker N (new pid: XXXX) without a preceding death message.
• Harakiri kill: worker exceeds the timeout, master sends SIGKILL. Log shows HARAKIRI ON WORKER N (and a traceback if harakiri-verbose is enabled). See uWSGI harakiri-verbose diagnosis.
• Crash (segfault, abort): log shows DAMN ! worker N died, killed by signal 11 (SIGSEGV) or signal 6 (SIGABRT). Always investigate these.
• Reload mercy timeout: worker took too long to die during a graceful reload (exceeding worker-reload-mercy, whose default is 60 seconds across 2.0.x). Log shows NO MERCY !!! as uWSGI escalates to SIGKILL.

For OOM-driven respawns, check dmesg or journalctl -k for kernel oom-killer entries targeting worker PIDs. The OOM-killer targets the highest-RSS process first, so with many uWSGI workers it picks one off at a time, creating a slow churn that looks like random respawns without any application-level signal.

Metrics and signals to monitor

Fixes

Respawn rate is at the expected recycling rate

No fix needed. This is max-requests or reload-on-rss working as designed. If the recycling cadence is too aggressive (capacity dips during peak traffic), raise max-requests or increase the reload-on-rss threshold. Each respawn briefly reduces capacity by one worker, so high recycling rates under heavy load can cause intermittent latency spikes.

Respawn rate matches harakiri delta 1:1

Workers are being killed by the timeout watchdog, not crashing. The root cause is downstream: requests are hanging on a database, external API, or lock. Do not disable harakiri to silence the respawns. Investigate the blocked syscall with harakiri-verbose and fix the downstream dependency. See uWSGI harakiri death spiral and uWSGI HARAKIRI ON WORKER.

Non-harakiri respawns exceed expected recycling rate

Workers are crashing. Check the signal number in the uWSGI log. Signal 11 (SIGSEGV) or signal 6 (SIGABRT) in production points to a C extension bug, a memory corruption issue, or a deserialization crash. Common culprits: numpy operations on malformed arrays, lxml parsing of adversarial XML, database driver segfaults on connection pool exhaustion, and pickle/yaml deserialization of malicious payloads.

If crashes started after a deployment, roll back and test in staging with the same input that triggered the crash.

All workers respawn simultaneously

This is a mass-kill event. Three common causes:

1. Deploy or reload without chain reload. A graceful SIGHUP kills all workers at once. Use --chain-reload to cycle workers one at a time and maintain capacity during deploys.
2. OOM-killer sweep. Total worker RSS exceeded available memory. The kernel kills multiple processes in rapid succession. Lower max-requests or reload-on-rss to recycle workers before they collectively exhaust memory. Check that total worker RSS stays below 70% of system RAM.
3. All workers hit max-requests simultaneously. Without max-requests-delta, workers that start together and receive balanced load all hit the limit at the same time, causing a brief total outage during respawn. This is a known problem documented in uWSGI issue #648 (unbit/uwsgi#648).

Staggering max-requests recycling

max-requests-delta adds (worker_id - 1) * delta to each worker’s max-requests value, staggering restarts so they do not all recycle at once. For example, with max-requests = 1000 and max-requests-delta = 100, worker 1 recycles at 1000 requests, worker 2 at 1100, worker 3 at 1200, and so on.

max-requests-delta was added in uWSGI 2.0.29 (backported from master). On earlier 2.0.x builds it is an unknown directive, so strict = true rejects it at startup with an “unknown config directive” error. If you use strict mode on a pre-2.0.29 build, verify whether your uWSGI version accepts this directive before relying on it.

The min-worker-lifetime directive (default 10 seconds in the uWSGI source) prevents workers from being recycled before they have run for a minimum period; max-requests and reload-on-rss recycling only take effect after this minimum uptime. Verify the default on your installed version with uwsgi --help | grep min-worker-lifetime.

Prevention

• Correlate respawns with harakiri count. If you only alert on raw respawn rate, you will page on healthy recycling and miss crash loops that happen to produce a similar count. The subtraction (respawn_delta - harakiri_delta) is the minimum viable correlation.
• Compute and document the expected recycling rate for each service based on its max-requests setting and typical traffic. Store it alongside the alert threshold so on-call engineers can distinguish expected from unexpected without doing arithmetic at 3 a.m.
• Stagger recycling with max-requests-delta to prevent simultaneous mass-recycle events. Verify compatibility with your uWSGI version and strict-mode configuration first.
• Monitor dmesg for OOM events. The kernel oom-killer is invisible in uWSGI stats. A slow OOM-kill cycle produces mysterious respawns with no application-level signal.
• Use --chain-reload for deployments to avoid mass-kill events from simultaneous worker replacement.

How Netdata helps

• Per-second respawn_count collection makes the exact timing of respawn events visible. Simultaneous spikes across all workers (mass-kill) are immediately distinguishable from staggered periodic patterns (healthy recycling).
• Correlating respawn_count with harakiri_count in the same dashboard makes the subtraction visual. If the two lines track each other, respawns are harakiri-driven. If respawn_count rises independently of harakiri_count, workers are crashing or hitting reload-on-rss.
• Per-worker RSS charts show the sawtooth pattern of memory-triggered recycling and reveal whether reload-on-rss is the driver.
• Request throughput alongside respawn rate lets you compute the expected recycling rate in real time and spot when actual respawns deviate from it.
• Anomaly detection on respawn_count can flag unusual churn patterns even when the absolute rate looks normal.
• Kernel memory metrics from the same agent let you correlate uWSGI-level respawns with system-level memory pressure and OOM conditions without switching to dmesg.

Related guides

• uWSGI all workers busy: reading the busy ratio before the queue fills
• uWSGI connection refused: clients turned away when the backlog overflows
• uWSGI harakiri death spiral: workers killed and respawned while throughput collapses
• uWSGI harakiri not configured: stuck workers with no timeout and no recovery
• uWSGI harakiri timeout: setting it against request duration and nginx timeouts
• uWSGI harakiri-verbose: finding the blocked syscall behind a timeout
• uWSGI HARAKIRI ON WORKER: requests killed for exceeding the timeout
• How uWSGI actually works in production: a mental model for operators
• uWSGI listen backlog and net.core.somaxconn: sizing the connection queue
• uWSGI listen queue full: the backlog overflow that drops connections silently
• uWSGI listen_queue always zero: why the stats field is broken on Linux
• uWSGI master process dead: total outage while the PID file lingers