$ guides / uwsgi / uwsgi-worker-respawn-loop ▌

Operations Guides

uWSGI worker respawn loop: 'DAMN ! worker died :( trying respawn'

The uWSGI master logs DAMN ! worker N (pid: XXX) died, killed by signal S :( trying respawn ... when it receives SIGCHLD for a worker that exited unexpectedly. The next line is typically Respawned uWSGI worker N (new pid: XXXX). In a respawn loop, these pairs repeat rapidly: the master forks a replacement, the replacement dies, and the cycle continues.

The signal number S is your primary diagnostic clue. Signal 9 (SIGKILL) means OOM kill or harakiri. Signal 11 (SIGSEGV) means a C extension crashed. Signal 6 (SIGABRT) means an assert failed. A worker that dies less than one second after spawn is hitting a startup failure, not a request-time problem.

Distinguish a crash loop from normal worker recycling first. With max-requests configured, workers deliberately exit after serving N requests and respawn. That is healthy. A crash loop burns CPU on repeated fork and startup cycles while serving zero traffic.

What this means

uWSGI does not throttle crash-induced respawns aggressively. If the new worker hits the same fatal condition (a bad import, a C extension segfault, memory pressure), it dies again within milliseconds. The result is a tight loop: fork, import, crash, log, fork again.

Each iteration consumes CPU for the fork syscall and application startup. In lazy-apps mode, every respawn re-imports the entire application. A 4-worker loop with a 2-second startup time burns worker-seconds on startup while serving zero requests. Accepting worker count fluctuates or stays at zero, throughput collapses, and the kernel listen queue fills with connections nobody will serve.

One non-crash case produces the same log pattern: harakiri. When harakiri fires, the master kills the worker with SIGKILL and respawns it. The log shows killed by signal 9, but you will also see HARAKIRI ON WORKER N (pid: XXXX, try: 1) !!!. If you see signal 9 without a HARAKIRI line, suspect the OOM killer.

flowchart TD
    A["DAMN ! worker died"] --> B{"Signal in log?"}
    B -->|"9 SIGKILL"| C{"HARAKIRI log present?"}
    B -->|"11 SIGSEGV"| D["Segfault in C extension or libc"]
    B -->|"6 SIGABRT"| E["Assert failure or abort"]
    C -->|Yes| F["Request exceeded harakiri timeout"]
    C -->|No| G{"Worker alive less than 1s?"}
    G -->|Yes| H["Startup crash or import error"]
    G -->|No| I["OOM killer terminated worker"]
    D --> J{"Alpine with threads > 1?"}
    J -->|Yes| K["musl libc incompatibility"]
    J -->|No| L["Check C extension versions"]

Common causes

Cause	What it looks like	First thing to check
OOM kill (signal 9, no HARAKIRI)	Worker killed mid-request, RSS near system or container limit, no startup error in app logs	`dmesg` or kernel logs for OOM-killer entries
Harakiri kill (signal 9, with HARAKIRI log)	Worker killed at exactly the harakiri timeout, `harakiri_count` rising	Downstream dependency health, harakiri-verbose traceback
Startup crash (any signal, worker alive less than 1s)	Worker dies immediately after spawn, never serves a request, tight respawn loop	Application logs for ImportError, SyntaxError, missing env vars
C extension segfault (signal 11)	Worker crashes mid-request or during init, no Python traceback	Core dump, C extension versions, application logs
Alpine musl + threads (signal 11)	Constant SIGSEGV on Alpine with threads > 1	uWSGI version, base image

Quick checks

# Read the kill signal from uWSGI logs
grep -E "DAMN ! worker .* died" /var/log/uwsgi/app.log | tail -20

# Check for HARAKIRI lines alongside the deaths
grep -E "HARAKIRI ON WORKER" /var/log/uwsgi/app.log | tail -20

# Check respawn_count and harakiri_count from the stats server
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[] | {id: .id, respawn_count: .respawn_count, harakiri_count: .harakiri_count, status: .status, pid: .pid}]'

# Count accepting workers (pid > 0, not cheap, accepting connections)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[] | select(.pid > 0 and .status != "cheap" and .accepting == 1)] | length'

# Check for OOM killer activity in kernel logs
dmesg | grep -i "oom\|killed process" | tail -20

# Check per-worker RSS from stats server
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.workers[] | select(.pid > 0) | {id: .id, rss_mb: (.rss / 1048576)}'

# Validate uWSGI options/config (loads plugins and, in non-lazy mode, the app; then exits)
uwsgi --ini /etc/uwsgi/app.ini --no-server

# To catch application import errors regardless of mode
python -c "from your_module import application"

# Check worker age from last_spawn timestamps
uwsgi --connect-and-read 127.0.0.1:9191 | jq --argjson now "$(date +%s)" '[.workers[] | select(.pid > 0) | {id: .id, pid: .pid, last_spawn_age: ($now - .last_spawn)}]'

How to diagnose it

Read the signal number from the log. The DAMN ! worker N (pid: XXX) died, killed by signal S line includes the signal. Signal 9 is SIGKILL (OOM or harakiri). Signal 11 is SIGSEGV (segfault). Signal 6 is SIGABRT (assert). This single number narrows the cause significantly.
Distinguish harakiri from OOM. If the signal is 9, check for HARAKIRI ON WORKER log lines at the same timestamp. If present, the worker was killed for exceeding the request timeout. If absent, suspect the OOM killer.
Check worker lifetime. If workers are dying within seconds of spawn, you have a startup crash loop. Look at last_spawn timestamps in the stats server. Workers that are repeatedly very young are crashing during initialization, not during request processing.
Check the OOM killer. Run dmesg | grep -i oom to see if the kernel or cgroup OOM killer terminated the worker. In containers, check the container runtime logs for OOMKilled events. If RSS was climbing before the kills, this is memory exhaustion, not a code bug.
Check respawn_count vs harakiri_count. Both are per-worker monotonic counters in the stats server. If respawn_count is increasing but harakiri_count is not, the respawns are not harakiri-driven. Factor in max-requests recycling (step 6) to isolate crash-driven respawns: crash respawns = total respawns minus harakiri kills minus max-requests recycles.
Distinguish from max-requests recycling. If max-requests is configured, some respawns are expected. Calculate the expected rate: total_requests_per_second / max_requests_per_worker * num_workers. If the actual respawn rate matches this, the respawns are normal recycling. If it exceeds this, workers are crashing.
Check application logs for startup errors. With lazy-apps enabled, each worker imports the application independently. An ImportError, SyntaxError, missing environment variable, or failed database migration will crash every worker on startup. Run uwsgi --ini /etc/uwsgi/app.ini --no-server to validate the configuration (uWSGI has no --check-config option). To catch import errors, load the callable directly in Python: python -c "from your_module import application".
Check for C extension segfaults. Signal 11 with no Python traceback usually means a C extension crashed. Check for core dumps. Alpine/musl builds of uWSGI with threads enabled have a documented history of segfaults (unbit/uwsgi issues #1514 and #2508), with no single upstream fix version covering them; test your exact uWSGI, Python, and musl versions, or switch to a glibc-based image.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`respawn_count` (per-worker, delta)	Direct measure of worker lifecycle churn	Rate exceeding expected max-requests cadence
`harakiri_count` (per-worker, delta)	Distinguishes harakiri kills from crashes	Non-zero delta means requests are timing out
Accepting worker count	If zero, service cannot serve requests	Drops to zero with master alive means total outage
Worker RSS	Memory growth leading to OOM kills	RSS approaching system or container memory limit
Worker lifetime (`last_spawn` age)	Short lifetime signals startup crash loop	Workers consistently alive less than a few seconds
Exception count (delta)	Application errors that may precede crashes	Spike in exceptions before worker deaths
System OOM events (`dmesg`)	Kernel-level evidence of memory pressure	Any OOM-kill entries referencing uWSGI worker PIDs

Fixes

OOM kill loop

If the OOM killer is targeting workers (signal 9, no HARAKIRI log, OOM entries in dmesg), the worker pool is consuming more memory than available. The respawned worker immediately grows back to the same RSS and gets killed again, creating an infinite loop.

Reduce per-worker memory by lowering reload-on-rss so workers recycle before approaching the system limit. Reduce the worker count if total RSS (workers times per-worker RSS) exceeds available memory. If running in a container, increase the memory limit. Consider max-requests as a complementary recycling mechanism.

Tradeoff: Fewer workers means less concurrency. Measure the actual per-worker RSS under load before reducing the count.

Startup crash loop (import error, missing dependency)

If workers die within seconds of spawn, the application fails to initialize. Check application logs for the specific error: ImportError, SyntaxError, missing environment variable, database migration not applied.

Run uwsgi --ini /etc/uwsgi/app.ini --no-server to validate the configuration and, in non-lazy mode, import the application. To catch import errors directly, run python -c "from your_module import application".

Roll back the deployment if the error appeared after a code change.

C extension segfault (signal 11)

If workers crash with signal 11 and no Python traceback, a C extension is faulting. Common culprits include database drivers, XML parsers, and ML libraries.

Check the C extension versions against known compatibility issues. On Alpine Linux with threads greater than 1, verify the exact uWSGI and Python versions against the reported segfault issues (unbit/uwsgi #1514 and #2508) or switch to a glibc-based image.

Enable core dumps if not already configured. A core file from the crashed worker gives you a backtrace into the C extension.

Harakiri-driven respawns

If respawns correlate 1:1 with harakiri_count increases, workers are being killed for exceeding the request timeout, not crashing. The root cause is downstream: a slow database query, an unresponsive external API, or a network partition to a dependency.

Check harakiri-verbose output for the blocked syscall. See uWSGI harakiri-verbose: finding the blocked syscall behind a timeout. Do not disable harakiri to silence the respawns. Fix the downstream dependency or tune the timeout.

Prevention

Validate before deploy. Run uwsgi --ini app.ini --no-server in your CI pipeline to catch config and (in non-lazy mode) application import errors before they reach production.
Set reload-on-rss below the OOM threshold. Workers should recycle gracefully before the kernel kills them. This converts a violent OOM kill into a graceful recycle.
Monitor respawn_count delta against expected max-requests cadence. If you know your traffic rate and max-requests value, you can compute the expected recycling rate. Alert when actual respawns exceed it.
Enable harakiri-verbose. When a harakiri fires, this logs the blocked syscall and wchan, which is the difference between a fast diagnosis and a slow one.
Pin and test uWSGI on Alpine. If using Alpine Linux, test the exact uWSGI and Python versions with threads enabled before promotion; there is no single upstream fix version for the reported Alpine segfaults (unbit/uwsgi #1514 and #2508).

How Netdata helps

Per-second respawn tracking lets you see crash loops as they form, not minutes later. Respawn rate spikes are immediately visible alongside CPU usage spikes from repeated fork cycles.
Correlating respawn_count with harakiri_count separates crash-driven respawns from harakiri-driven ones without manual log parsing. If both rise together, it is harakiri. If only respawn rises, it is a crash.
Worker RSS trending shows memory growth leading to OOM kills. You can see the sawtooth pattern of reload-on-rss recycling versus the linear climb of a worker approaching the OOM threshold.
Accepting worker count drops to zero during a startup crash loop, even while the master process appears healthy. Alerting on this catches the outage that a master-PID health check misses.
Anomaly detection on respawn rate flags deviations from the normal max-requests cadence without requiring a static threshold.