$ guides / uwsgi / uwsgi-worker-segfault ▌

Operations Guides

uWSGI worker segfault: SIGSEGV in a C extension and the respawn that follows

A worker dies with signal 11. The master respawns it. The uWSGI log records DAMN ! worker N (pid: XXXX) died, killed by signal 11 :( trying respawn ... followed by Respawned uWSGI worker N (new pid: YYYY). From the outside, the service looks like it survived a momentary blip. It did not. A SIGSEGV means something in C-level code crashed: a compiled extension (numpy, lxml, a database driver, OpenSSL, protobuf), a uWSGI internal bug, or memory corruption. Python application code cannot normally produce SIGSEGV.

The respawn masks the symptom. If the crash is systemic and a specific request triggers the same code path every time, the worker respawns, immediately accepts the next queued request, and crashes again. You get a respawn loop that burns CPU on fork and startup while serving zero useful traffic. If the crash is sporadic, you might not notice until nginx starts returning 502s when it hits a worker that is mid-respawn.

What this means

When a worker receives SIGSEGV (signal 11), uWSGI’s signal handler fires. The handler prints a message to the log and, depending on configuration, a C-level backtrace showing the stack frames at the point of crash. The master process detects that the worker exited with signal 11 and immediately forks a replacement. The new worker starts accepting requests from the kernel listen queue.

The key distinction from other respawn causes: a SIGSEGV is never a Python-level error. Unhandled Python exceptions produce 500 responses and increment the exceptions counter. A harakiri kill produces a SIGKILL (signal 9), not signal 11. max-requests recycling is a graceful self-exit. Signal 11 is a crash in compiled code.

flowchart TD
    A["Worker processes request"] --> B["C extension SIGSEGV"]
    B --> C["Segfault handler prints backtrace"]
    C --> D["Master detects signal 11 exit"]
    D --> E["Master forks replacement worker"]
    E --> F{"Same request in queue?"}
    F -->|Yes| A
    F -->|No| G["Normal operation resumes"]

The respawn loop is the dangerous failure mode. Each crash-respawn cycle has a cost: the fork, the application import (especially under lazy-apps), and the connection pool warmup. If the trigger is a queued request, the cycle repeats immediately. The master does not throttle respawns aggressively by default, so the loop can sustain high CPU burn with no useful throughput.

Common causes

Cause	What it looks like	First thing to check
C extension version mismatch or ABI break	Backtrace points to a specific extension (numpy, lxml, psycopg2, etc.); crash correlates with specific request paths	Check whether a dependency was recently upgraded or rebuilt; verify the extension matches your Python ABI
C extension bug during graceful shutdown	Worker segfaults when exiting via `max-requests`, chain reload, or SIGTERM, not during request processing	Check whether crashes coincide with recycling events; review backtrace for cleanup or destructor frames
uWSGI `catch-exceptions` bug (pre-2.0.27)	Segfault on Python 3.5+ when `catch-exceptions` is enabled	Run `uwsgi --version`; check whether `catch-exceptions = true` is in config
`limit-as` set too low on 64-bit	Workers die immediately after spawn in a tight respawn loop, no requests served	Check `limit-as` in config; try removing it or raising the value
Memory corruption from another extension	Intermittent, hard to reproduce; backtrace may point to an unrelated extension	Enable `--use-abort` for core dumps; reproduce under valgrind in staging
Security exploitation	Unusual request patterns correlate with crashes; specific payloads trigger segfaults	Review access logs and worker URIs in stats for suspicious patterns

Quick checks

These commands are read-only and safe to run in production. They assume the stats server is already enabled (e.g., stats = 127.0.0.1:9191 in your uWSGI config).

# Check uWSGI log for segfault messages
grep -E "Segmentation Fault|signal 11|SIGSEGV" /var/log/uwsgi/*.log | tail -20

# Check respawn_count from stats server (TCP socket example)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].respawn_count] | add'

# Check harakiri_count to distinguish harakiri kills from crash respawns
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].harakiri_count] | add'

# Check kernel logs for segfault evidence
dmesg -T | grep -i "segfault\|uwsgi" | tail -20

# Check uWSGI version for known bugs
uwsgi --version

# Check whether core dumps are enabled
ulimit -c

# Check where the kernel writes core dumps
cat /proc/sys/kernel/core_pattern

# Check worker status and PIDs for rapid churn
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.workers[] | {id: .id, pid: .pid, status: .status, respawn_count: .respawn_count}'

The critical check: compare respawn rate against what max-requests would explain. If you have max-requests = 1000 and 100 req/s fleet-wide, expect roughly 0.1 respawns per second. Any rate significantly above that, with no corresponding harakiri activity, points to crashes.

How to diagnose it

Read the backtrace. After the !!! uWSGI process <PID> got Segmentation Fault !!! message, uWSGI prints a C backtrace whose depth is controlled by --backtrace-depth. The top frames identify which library or extension was executing when the crash occurred. Look for function names containing recognizable library prefixes.
Distinguish crash respawns from expected recycling. Subtract harakiri_count from respawn_count to isolate non-harakiri respawns. Every harakiri kill increments both counters. If respawn_count is rising but harakiri_count is flat, workers are crashing, not timing out.
Check for recent changes. Correlate the first segfault timestamp with deploy logs, dependency upgrade logs, and OS package updates. A new version of a C extension compiled against a different Python or library version is the most common trigger.
Identify the triggering request. If the crash is reproducible, check the uWSGI stats for the URI the worker was processing. The uri field on a busy worker shows the endpoint. If all crashes happen on the same endpoint, that code path calls the crashing extension.
Enable core dumps if the backtrace is insufficient. Set --use-abort in the uWSGI config. This calls abort() on segfault instead of the default handler, which generates a core dump if ulimit -c is set to unlimited. Load the core in gdb against the uwsgi binary and run bt full for a backtrace with local variables.
Check uWSGI version for known bugs. Before 2.0.27, catch-exceptions = true caused segfaults on Python 3.5+. The logsocket plugin had a segfault fixed in 2.0.19. If you are on an older release, check the changelog for your specific crash pattern.
Reproduce in staging. Isolate the suspected extension and request path. Run uWSGI under valgrind (valgrind --tool=memcheck uwsgi --ini app.ini) to catch memory errors that do not immediately crash. Valgrind slows the worker significantly; do not run this in production.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`respawn_count` rate	Primary indicator of worker churn. Every crash produces one respawn.	Rate exceeding what `max-requests` predicts; sudden spike across multiple workers simultaneously
`harakiri_count` rate	Distinguishes harakiri kills (signal 9) from crashes (signal 11). Subtract from respawn rate to isolate crashes.	Zero change in harakiri alongside rising respawn rate
Worker RSS before crash	Memory corruption or OOM pressure can produce SIGSEGV. High RSS growth may precede a crash.	RSS spike or steady growth pattern before respawn events
Exception rate	Some crashes are preceded by exception storms in the same code path.	Exception rate rising in the window before the first segfault
Accepting worker count	During a respawn loop, accepting workers fluctuate as workers die and respawn.	Brief drops if all workers crash at once
Request throughput	A respawn loop serves no useful traffic. Throughput collapses while respawn rate spikes.	Throughput drop correlating exactly with respawn spike

Fixes

C extension version mismatch

The most common fix. If a dependency was upgraded (intentionally or transitively), the compiled extension may not match the running Python ABI or linked libraries.

Pin the extension to a known-good version.
Rebuild the extension against the current Python. Test in staging first: pip install --force-reinstall --no-binary :all: <package>.
If using psycopg2-binary, the bundled libpq may conflict with system PostgreSQL client libraries. Building from source (psycopg2 instead of psycopg2-binary) can resolve version mismatches.

C extension segfault on shutdown

Multiple C extensions crash during graceful shutdown when the interpreter tears down. OpenSSL, protobuf, PyTorch, and gevent have all had shutdown segfault issues reported.

Upgrade the extension to the latest version. Shutdown bugs are frequently fixed in releases.
If the crash only happens during max-requests recycling, consider adjusting the recycling interval or switching to reload-on-rss.
If the crash happens during SIGHUP reload, use --chain-reload to cycle workers one at a time, reducing the simultaneous teardown load.

uWSGI catch-exceptions bug

Before uWSGI 2.0.27, enabling catch-exceptions = true on Python 3.5+ could itself cause a segfault. The crash appears to come from the application but is actually a uWSGI internal bug.

Upgrade uWSGI to 2.0.27 or later.
If you cannot upgrade immediately, remove catch-exceptions = true from your config.

limit-as too low

Setting limit-as restricts the virtual address space of workers using setrlimit(RLIMIT_AS). On 64-bit systems, libraries like numpy, libssl, and others map large virtual address ranges that can exceed a modest limit-as value, causing immediate SIGSEGV on worker startup.

Remove limit-as from the configuration, or set it high enough to accommodate virtual address space usage.
Use reload-on-rss for memory limiting instead. It measures physical memory (RSS) and triggers a graceful worker reload, not a virtual address space cap.

Prevention

Always capture crash logs. The --disable-logging option (shortcut -L) suppresses request logging but does not suppress error output. Segfault messages, backtraces, and respawn notices go to stderr. Ensure stderr is captured by your log aggregation pipeline. Without the backtrace, you have no starting point for diagnosis.

Use alarm-segfault for proactive notification. The alarm subsystem can trigger a named alarm when the segfault handler fires. Define an alarm using the --alarm option with a backend such as cmd to run a script, then reference it with --alarm-segfault <name>. This gives you immediate notification rather than waiting for the next stats poll.

Ensure backtrace depth is sufficient. The --backtrace-depth option controls how many stack frames uWSGI prints after a segfault. If it is too low, you may not see the frame that identifies the offending extension. A depth of 20 to 30 frames is usually sufficient.

Pin C extension versions in production. Transitive dependency upgrades are a frequent cause of ABI mismatches. Use lockfiles and test upgrades in staging before promoting to production. A segfault that appears immediately after a pip install or a base image rebuild is almost always an ABI issue.

Monitor respawn rate against expected baseline. With max-requests configured, respawns are routine and healthy. Alert when the respawn rate exceeds what max-requests predicts by a meaningful margin and harakiri_count is not rising proportionally. This catches crash loops before they become customer-visible.

Test Python upgrades with C extensions rebuilt. Upgrading Python, even a minor version like 3.11 to 3.12, requires all C extensions to be recompiled against the new ABI. A segfault on the first request after a Python upgrade is almost always an ABI mismatch.

How Netdata helps

Per-second respawn_count tracking lets you see the exact moment a crash loop starts and correlate it with deploy timestamps, dependency changes, or traffic patterns. One-second resolution catches tight respawn loops that 15-second or 60-second polling intervals miss entirely.
Harakiri rate correlation distinguishes crash-driven respawns from harakiri-driven respawns. If respawn_count spikes but harakiri_count stays flat, the workers are crashing, not timing out. This narrows the investigation immediately.
Worker RSS history shows whether memory pressure preceded the crash. A rising RSS curve before a segfault points toward memory corruption or an OOM-adjacent condition rather than a pure logic bug.
Exception rate timing reveals whether the application was throwing errors in the same code path before the crash. A burst of exceptions followed by a segfault in the same window suggests the same request triggers both.
Anomaly detection on respawn patterns flags unusual respawn behavior even without explicit thresholds. A sudden departure from the baseline respawn rate, even below an absolute alert threshold, gets surfaced for investigation.