$ guides / uwsgi / uwsgi-emperor-vassal-down ▌

Operations Guides

uWSGI Emperor healthy but vassal dead: monitoring each instance independently

The Emperor process is running, its stats endpoint responds, and it continues scanning config directories. But one application is down. Clients get connection refused or timeouts. Your monitoring says the Emperor is healthy because it is. Emperor liveness is not application liveness.

In Emperor mode, the Emperor manages one vassal (a full uWSGI instance) per config file. Each vassal is an independent process tree with its own master, workers, and stats server. The Emperor monitors config files and spawns or reloads vassals, but a vassal can die and fail to restart on a broken config while the Emperor runs unaffected. Vassal states:

Loyal: running and has served at least one request. Auto-respawned on death.
Broken: died after becoming loyal. Emperor will respawn.
Cursed: died before becoming loyal (crashed during startup). Not respawned automatically; placed in a blacklist with increasing throttle delay.

Each vassal needs its own stats endpoint and process check. Monitoring only the Emperor gives no visibility into whether individual applications are serving requests.

Vassal lifecycle and throttle behavior

When the Emperor detects a config file (via directory scan or inotify event), it spawns a vassal. The vassal goes through startup and, once it successfully handles its first request, becomes loyal. If a loyal vassal dies, the Emperor respawns it. If a non-loyal vassal dies (crashes during startup, before serving any request), it enters a blacklist and gets throttled. Repeated failures push the throttle delay higher, up to the configured maximum.

The key failure pattern: a broken config (syntax error, missing plugin, bad import path, missing environment variable) causes the vassal to crash on every spawn attempt. The Emperor retries with throttling, but the vassal never becomes loyal. In emperor-stats output, this state is reported as cursed. The Emperor process itself is unaffected.

flowchart TD
    A[Emperor scans config dir] --> B[Spawns vassal process]
    B --> C{Startup succeeds?}
    C -->|Yes| D[Serves first request]
    D --> E[Loyal: auto-respawned on death]
    C -->|No| F[Dies before first request]
    F --> G[Blacklisted and throttled]
    G --> H{Throttle below max?}
    H -->|Yes| B
    H -->|No| I[Cursed: not respawned]
    E --> J[Death triggers respawn]
    J --> D

Common causes

Cause	What it looks like	First thing to check
Broken vassal config	Vassal crashes immediately on every spawn; never becomes loyal; emperor-stats shows cursed	The vassal config file for syntax errors or invalid directives
Missing or incompatible plugin	Vassal logs inability to load a plugin at startup	The `plugins` directive and whether the plugin binary exists
Missing environment variable	Vassal starts, crashes during app initialization, retries with throttle	Application logs for import-time errors referencing env vars
File descriptor or memory limits	Vassal spawns, exhausts resources, dies; Emperor retries with throttle	`ulimit -n` for the vassal process and available system memory
Config file removed or moved	Emperor stops tracking the vassal entirely; no respawn attempts occur	Whether the config file still exists in the watched directory
Vassal running without master mode	Reloads via SIGHUP have no effect; vassal appears stuck after config change	Whether `master = true` is set in the vassal config

Quick checks

# Emperor process alive?
ps aux | grep '[u]wsgi.*emperor'

# SIGUSR1 to Emperor prints vassal status to its log
kill -USR1 $(pgrep -f 'uwsgi.*emperor' | head -1)

# Emperor log for vassal lifecycle events
grep -E "(spawned|removed|ready|loyal|cursed|broken)" /var/log/uwsgi/emperor.log | tail -30

# Emperor stats JSON (TCP socket)
uwsgi --connect-and-read 127.0.0.1:1717 | jq '.vassals[] | {id, pid, loyal, ready, accepting, cursed}'

# Emperor stats JSON (UNIX socket variant)
uwsgi --connect-and-read /tmp/emperor-stats.sock | jq '.vassals[] | {id, pid, loyal, ready, accepting}'

Each vassal object in emperor-stats carries: id (the vassal name), pid, born, last_mod, last_heartbeat, loyal, ready, accepting, last_loyal, last_ready, last_accepting, first_run, last_run, cursed (unix timestamp, 0 when not cursed), zerg, on_demand, suspended, adopted.

# List vassal config files the Emperor should be watching
ls -la /etc/uwsgi/vassals/

# Per-vassal stats socket: check worker-level health
uwsgi --connect-and-read /tmp/stats_app1.sock | jq '{pid: .pid, workers: (.workers | length), accepting: [.workers[] | select(.pid > 0 and .accepting == 1)] | length}'

# Validate a vassal config without starting the server
uwsgi --ini /etc/uwsgi/vassals/app1.ini --no-server 2>&1 | head -20

The Emperor log path varies by deployment. Check your init script, systemd unit, or --daemonize / --logto options for the actual path.

How to diagnose it

Confirm the Emperor is healthy. Check that the Emperor process is running and its stats endpoint responds. This isolates the problem to the vassal level.
Identify which vassal is down. Send SIGUSR1 to the Emperor and check its log for per-vassal status. Alternatively, read emperor-stats JSON to see which vassals have loyal: 0, ready: 0, or a cursed state.
Validate the vassal config. Run uwsgi --ini <path> --no-server to catch syntax errors, missing plugins, and invalid directives without serving traffic.
Check the vassal logs. Look for startup errors: import failures, missing modules, permission denied on sockets, address already in use.
Check whether the config file still exists. If the file was moved, renamed, or deleted from the watched directory, the Emperor stops tracking it. No respawn attempts occur.
Check the throttle state. If the vassal is blacklisted, the Emperor waits between respawn attempts. The throttle delay grows with each failure, bounded by --emperor-max-throttle (default 3 minutes). The vassal may appear down for minutes at a time even though the Emperor is trying to restart it.
Query the vassal’s own stats endpoint. Emperor stats tell you whether a vassal process exists, not whether its workers are healthy. Each vassal needs its own stats socket for worker-level visibility.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Vassal state (loyal, ready, accepting, cursed)	Directly reports whether each vassal is running and serving	Any vassal with loyal = 0 or cursed state
Active vassal count vs expected	Primary health indicator for Emperor deployments	Active count below the number of config files in the watched directory
Emperor throttle activity	Indicates crash-looping vassals consuming CPU and delaying recovery	Sustained throttle activity for a specific vassal
Per-vassal accepting worker count	Confirms the application is serving requests, not just running a master	Zero accepting workers with a live vassal master
Per-vassal worker busy ratio	Capacity utilization for each application independently	Sustained 100% busy ratio on one vassal while others are idle
Per-vassal respawn rate	Worker churn within a specific vassal	Respawn rate exceeding what max-requests recycling explains
Config file modification timestamps	In Emperor mode, config changes trigger reloads	Unexpected modification times indicating config drift

Fixes

Broken config preventing vassal startup

The most common cause. The vassal crashes during initialization due to a syntax error, a missing plugin, a bad import path, or a missing environment variable. The Emperor retries with throttling, but the vassal never becomes loyal.

Validate the config:

uwsgi --ini /etc/uwsgi/vassals/app1.ini --no-server

Fix the config error, then force a clean restart by moving the config file out of the watched directory and back in. This forces the Emperor to treat it as a new vassal, clearing blacklist state:

# Move config out to stop tracking
mv /etc/uwsgi/vassals/app1.ini /tmp/

# Wait for Emperor to detect removal (poll emperor-stats), then move back
mv /tmp/app1.ini /etc/uwsgi/vassals/

Vassal stuck in throttle or blacklist

If a vassal has crashed repeatedly, the Emperor throttles respawn attempts with increasing delays. The throttle is bounded by --emperor-max-throttle (default 3 minutes). The base throttle increment is --emperor-throttle (default 1000ms).

To clear the blacklist immediately, send SIGURG to the Emperor:

kill -URG $(pgrep -f 'uwsgi.*emperor' | head -1)

Warning: this clears the blacklist for all vassals, not just the one you are troubleshooting. Every cursed vassal will attempt an immediate respawn.

The --emperor-no-blacklist option exists and disables the blacklist subsystem entirely. The blacklist also throttles crash-looping vassals to protect against fork-bombing, so weigh that before enabling it permanently instead of an occasional SIGURG.

Vassal not reloading on config changes

If touching the vassal config file logs a reload attempt but the vassal does not actually reload, the vassal is likely not running with master = true. Without master mode, the Emperor cannot reload the vassal via SIGHUP. Add master = true to the vassal config.

No per-vassal stats endpoint

Without a per-vassal stats socket, you have no visibility into worker health within that vassal. Emperor stats only report vassal process existence, not worker-level state. Add a per-vassal stats socket using %n (vassal name without extension):

# In each vassal config file
stats = /tmp/stats_%n.sock

For HTTP access instead of raw socket, also set stats-http = true.

Emperor not respawning vassals at all

If the Emperor is running but not respawning dead vassals, verify the scan frequency and watch directory. The default scan interval is --emperor-freq (3 seconds). Check that the Emperor is watching the correct directory and that inotify is functional. On systems with many watched files, inotify watch limits can cause missed events. Check fs.inotify.max_user_watches via sysctl.

Prevention

Enable emperor-stats on the Emperor. Use --emperor-stats to expose a JSON endpoint with per-vassal status. This is the primary tool for detecting dead or cursed vassals programmatically.

Give each vassal its own stats socket. Emperor stats tell you whether a vassal process exists. Each vassal’s own stats socket tells you whether its workers are healthy. Use stats = /tmp/stats_%n.sock in each vassal config for per-instance worker metrics.

Validate configs before deployment. Run uwsgi --ini <path> --no-server in CI to catch syntax errors and missing plugins before they reach the Emperor.

Monitor vassal count against expected count. Track the number of loyal vassals against the number of config files in the watched directory. Any divergence means a vassal is down, cursed, or misconfigured.

Use the heartbeat system for stuck vassals. Vassals can send heartbeat messages to the Emperor. If no heartbeat is received within the configured window, the vassal is considered hung and reloaded. Enable --heartbeat in the vassal config and set --emperor-required-heartbeat on the Emperor to catch vassals that are alive but not making progress.

--emperor-required-heartbeat defaults to 30 seconds (the Emperor reloads a vassal that sends no heartbeat for that long); the vassal-side --heartbeat option has no default and stays disabled until you set an interval.

Do not reload the Emperor for vassal issues. Emperor reloads are drastic: they reload all vassals at once. Instead, reload individual vassals by touching their config files or sending signals to the specific vassal master process. Reserve Emperor restarts for uWSGI version upgrades.

How Netdata helps

Netdata’s per-process monitoring distinguishes the Emperor process from individual vassal processes, so you can track CPU, memory, and file descriptor usage per instance rather than aggregated into one process group. Per-second metric resolution captures vassal crash-restart cycles and throttle intervals that coarser polling misses. A vassal that crashes and respawns within a 10-second window is invisible to 15-second polling.

Anomaly detection on per-process resource usage flags unusual behavior in individual vassals (memory spikes, CPU burn from crash loops) without per-instance static thresholds. Correlation across layers lets you overlay Emperor process health, individual vassal process metrics, and system-level signals (memory pressure, FD limits, OOM events) to determine whether a vassal failure is self-contained or caused by host-level resource contention.

If each vassal exposes its own stats socket via stats = /tmp/stats_%n.sock, Netdata’s uWSGI collector can ingest worker-level metrics (busy ratio, accepting workers, respawn rate, harakiri count) per vassal, providing the independent per-instance visibility that Emperor mode requires.