uWSGI master process dead: total outage while the PID file lingers

Nginx returns 502. The uWSGI PID file exists at its expected path, so monitoring reports the service as “up.” But nothing is serving traffic. The master process is dead, and the stale PID file is lying to you.

The master holds the listening socket, forks workers, enforces harakiri timeouts, handles graceful reloads, and serves the stats endpoint. When it dies, workers are gone. No connections are accepted. The outage is total.

The PID file is written at startup and cleaned up only on graceful shutdown. If the master is killed by the OOM-killer, receives SIGKILL, or segfaults, cleanup never runs. The file lingers, containing a PID that maps to no living process. Any monitoring that checks file existence rather than process liveness reports a false positive.

What this means

The PID file is a static reference for management commands (stop, reload). It contains a single integer, does not heartbeat, and does not reflect runtime state. test -f /path/to/uwsgi.pid tells you a file was written at some point, not that the process is alive.

The correct liveness test is kill -0. The kernel checks whether the PID exists in the process table and whether the caller has permission to signal it. Exit code 0 means alive; non-zero means dead or inaccessible.

Under systemd, the master typically restarts via Restart=always. But there is a window of unavailability between death and restart, and the restart may fail if the stale PID file remains or if the resource exhaustion that killed the master persists.

flowchart TD
    A["Total outage, PID file exists"] --> B{"kill -0 on PID"}
    B -->|exit 0| C["Process alive - other failure"]
    B -->|non-zero| D["Master dead - stale PID file"]
    D --> E["Check dmesg for OOM"]
    D --> F["Check journal for crashes"]
    C --> G["Check accepting workers"]
    C --> H["Check stats server"]

Common causes

Quick checks

Run these read-only commands to confirm the diagnosis. Adjust the PID file path and stats socket address to match your deployment. The paths below use /tmp/uwsgi.pid and 127.0.0.1:9191 as examples.

# THE check: test process liveness, not file existence
kill -0 "$(cat /tmp/uwsgi.pid)" 2>/dev/null && echo "alive" || echo "DEAD"

# Cross-check: is the stats server reachable? If not, the master is gone
uwsgi --connect-and-read 127.0.0.1:9191 | python3 -c "import sys,json; d=json.load(sys.stdin); print('master pid:', d['pid'])"

# Check for OOM-killer evidence in the kernel ring buffer
dmesg | grep -i oom | tail -20

# Check systemd journal for the service (adjust unit name)
journalctl -u uwsgi --since "1 hour ago" --no-pager | tail -50

# Is anything listening on the uWSGI socket?
ss -lxp | grep uwsgi

# Are any uWSGI processes running at all?
pgrep -a uwsgi

# Check available memory and swap (if OOM is suspected)
free -m

# Check disk space on the partition holding the PID file
df -h /tmp

# In Emperor mode: is the emperor alive but a vassal dead?
pgrep -a -f "uwsgi.*emperor"

How to diagnose it

1. Confirm the master is dead. Run kill -0 on the PID from the pidfile. Non-zero exit means the master is gone. Do not trust the file’s existence.

2. Check for OOM-killer evidence. Run dmesg | grep -i oom. The OOM-killer logs which process it killed and the memory state at the time. If the master PID appears, memory exhaustion killed it. Check whether workers were also being OOM-killed separately, which indicates systemic memory pressure rather than a single runaway process.

3. Check the service journal. Run journalctl -u <your-uwsgi-unit> for the period around the outage. Look for crash signals (SIGSEGV, SIGABRT), systemd restart attempts, and throttle or backoff behavior. Systemd may have already restarted the master, closing the window of unavailability but leaving the root cause unaddressed.

4. Check for orphaned workers. If no-orphans is not configured, worker processes may persist after master death as children of init (PID 1). Run pgrep -a uwsgi to check. These orphans are unmanaged: no harakiri enforcement, no respawn capability, no stats endpoint. They may continue serving traffic from inherited socket file descriptors but cannot recover from worker death and cannot be reloaded or stopped through normal uWSGI commands.

5. In Emperor mode, check the vassal independently. The Emperor being alive does not mean any specific vassal is alive. Identify the dead vassal and check its logs separately. The Emperor attempts to respawn dead vassals, but a broken config or resource exhaustion can prevent successful restart. The --emperor-throttle setting controls the respawn rate for crashing vassals, and a vassal in a tight crash loop may be effectively down for much longer than the throttle interval suggests.

6. Check for disk full conditions. If the partition holding the PID file or log directory is full, the master cannot write its PID file or logs on restart. This blocks recovery even after the original cause is addressed. Run df -h on the relevant partitions.

Metrics and signals to monitor

Fixes

Stale PID file after OOM-killer

The master was killed by the OOM-killer. The PID file is stale. Workers are gone.

1. Confirm the master is dead with kill -0 on the PID from the pidfile.
2. Remove the stale PID file. Only do this after confirming the master is truly dead, otherwise you delete a valid reference for a running process:

   rm /tmp/uwsgi.pid
   

3. Restart the service via systemd or your process manager.
4. Address the root cause. If workers grow RSS without bounds, configure reload-on-rss to recycle workers before the OOM-killer fires.

Segfault or signal kill

The master crashed due to a segfault (typically in a C extension) or was killed by an external signal.

1. Remove the stale PID file after confirming the master is dead.
2. Restart the service.
3. Investigate the segfault. Check the uWSGI log for the signal number (11 means SIGSEGV, 6 means SIGABRT). If a specific C extension is implicated, check for known bugs or version mismatches. A SIGSEGV in production always warrants investigation for memory corruption.

Emperor mode vassal death

The Emperor is alive, but a vassal is dead and not restarting.

1. Check the vassal’s config file for errors. A broken config prevents the vassal from starting.
2. Check the Emperor’s log for vassal lifecycle events such as spawned, broken, cursed, and loyal.
3. If the vassal is in a crash loop, the Emperor’s throttle mechanism limits respawn frequency. Check --emperor-throttle settings.
4. Each vassal needs independent monitoring. Emperor liveness does not imply any vassal’s liveness.

Disk full

The master cannot write its PID file or logs on restart.

1. Free disk space on the affected partition.
2. Remove the stale PID file.
3. Restart the service.
4. Investigate what filled the disk: log rotation failures, spooler backlog, or core dumps from previous crashes.

Prevention

Use safe-pidfile instead of pidfile. Writes the PID file later in startup, after the application initializes. This prevents stale PID files from failed startups where the master writes the PID file but crashes during app loading.

Use vacuum to clean up on graceful shutdown. The vacuum option removes generated files and sockets, including the PID file, on shutdown. This prevents stale files from accumulating across normal restart cycles. However, vacuum only runs on graceful shutdown. If the master is killed by SIGKILL or the OOM-killer, cleanup handlers do not execute and the PID file persists regardless.

Prefer systemd Type=notify over Type=forking. With Type=notify, uWSGI communicates readiness to systemd via a notification socket, eliminating the PID file entirely. Systemd tracks the process directly through cgroup membership. This removes the stale PID file failure mode completely.

Monitor with kill -0, not file existence. Any monitoring check, health probe, or load balancer health check that relies on the PID file existing produces false positives during this failure mode. The check must test the actual process.

Set die-on-term for clean SIGTERM handling. By default in uWSGI 2.0.x, SIGTERM triggers a brutal reload rather than a clean shutdown. The die-on-term option makes SIGTERM shut down the instance properly, allowing cleanup handlers (including vacuum) to run and producing a clean state for restart.

Consider no-orphans with caution. This option kills workers automatically when the master dies, preventing orphaned processes from lingering in an unmanaged state. The Debian manpage warns it “can be dangerous for availability,” meaning it may kill workers in scenarios where the master is temporarily unreachable but not truly dead. Evaluate this tradeoff for your deployment.

Prevent the OOM-killer from targeting the master. Configure reload-on-rss on workers to recycle them before RSS growth triggers system-wide memory pressure. The OOM-killer often targets the highest-RSS process, which in a uWSGI deployment is typically a worker rather than the master. But if the master’s RSS is inflated by shared memory regions or large configurations, it can become the victim. Monitor per-worker RSS growth and set recycling thresholds conservatively.

How Netdata helps

• Per-second process monitoring detects master process disappearance within seconds. The process check queries the kernel process table directly, not filesystem artifacts like PID files.
• Memory pressure correlation displays system memory, swap activity, and OOM-killer events alongside process state on the same timeline. When the master dies, you can immediately see whether memory exhaustion was the cause.
• Worker-level metrics including accepting worker count, respawn rate, and harakiri rate distinguish a master death from worker starvation or a reload blackout. Zero workers with a dead master looks different from zero accepting workers with a live master.
• Anomaly detection on process count and memory usage catches gradual RSS degradation before it triggers an OOM kill.
• Emperor and vassal process tracking in Emperor mode ensures that a dead vassal is detected independently of the Emperor process, closing the gap where Emperor liveness masks vassal failure.