uWSGI reload blackout: a broken deploy leaves zero workers running

You deployed new code to uWSGI. The graceful reload killed the old workers, but throughput is zero. The master process is alive. Every worker it forks dies before accepting a connection.

The reload mechanism worked. The new code cannot initialize. Workers crash during startup, the master respawns them, and they crash again. No worker stays alive long enough to serve a request.

The master process appears healthy. Process checks pass. The stats server responds to polling. If your health check only verifies the master PID or pings the stats endpoint, it reports green while 100% of requests are dropped at the kernel backlog.

This pattern is most visible with lazy-apps enabled, where each worker loads the application independently after fork. In preforking mode (no lazy-apps), the master loads the application itself during initialization before forking workers; a fatal import or configuration error then exits the master (or, with need-app, keeps it from starting), producing a different failure mode: the instance fails to start instead of entering a worker respawn loop.

What this means

A reload blackout is defined by three conditions occurring simultaneously:

1. The master process is alive and the stats server is reachable.
2. Zero workers are accepting connections. No worker has pid > 0 AND accepting == 1 AND status != "cheap".
3. Worker churn is visible: respawn_count is climbing or last_spawn timestamps show very recent spawn attempts.

This is distinct from several lookalike states. A dead master has an unreachable stats server. An idle server in cheaper mode has workers with status: "cheap" and pid: 0, but at least the cheaper minimum should still be alive and accepting. During a reload blackout, the master is actively spawning workers that die within seconds or milliseconds, and respawn_count climbs while throughput stays at zero.

Duration matters. A normal graceful reload may briefly reduce accepting workers while old workers drain and new ones start. If zero accepting workers persists beyond 60 seconds, startup has genuinely failed.

flowchart TD
    A["SIGHUP or touch reload trigger"] --> B["Master kills old workers"]
    B --> C["Master forks new worker"]
    C --> D{"App initializes?"}
    D -->|No: import or config error| E["Worker exits immediately"]
    E --> F["Master respawns worker"]
    F --> C
    E --> G["Zero accepting workers"]
    G --> H["Throughput at zero"]
    G --> I["Connections queue in kernel backlog"]
    D -->|Yes| J["Worker accepts requests"]

Common causes

Quick checks

All of these are safe, read-only operations. They assume the stats server is enabled and listening on 127.0.0.1:9191 (TCP). If your stats server uses a UNIX socket, replace 127.0.0.1:9191 with the socket path. If the stats server is not enabled, all stats-based diagnosis is blind: check the uWSGI master log and application logs instead.

# Check accepting worker count (should be > 0)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[] | select(.pid > 0 and .status != "cheap" and .accepting == 1)] | length'

# Check respawn churn (run twice, a few seconds apart; should be stable in healthy state)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].respawn_count] | add'

# Check total throughput (should be climbing if workers are alive)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].requests] | add'

# Check last_spawn timestamps and worker states
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.workers[] | {id: .id, pid: .pid, status: .status, last_spawn: .last_spawn, accepting: .accepting}'

# Confirm master is alive
uwsgi --connect-and-read 127.0.0.1:9191 | jq '{pid: .pid}'

# Check kernel listen queue depth (TCP socket; adjust port as needed)
ss -ltn 'sport = :8000'

If the accepting count is zero, respawn_count is climbing, the requests total is flat, and the master PID is present, you are in a reload blackout.

How to diagnose it

1. Confirm the pattern. Run the accepting worker count and respawn count checks above, separated by 5 to 10 seconds. If accepting stays at zero and respawn_count climbs, the master is spawning workers that immediately die.

2. Read the application logs. Workers that die during startup almost always write a traceback before exiting. Look for ImportError, SyntaxError, database connection failures, or configuration validation errors. The log entry should name the specific module, variable, or query that failed.

3. Check the uWSGI master log for startup errors. The master logs worker spawn and death events. In a blackout, you see rapid “spawned uWSGI worker” lines followed by worker exits, repeating. Look for messages about plugin loading failures, shared library errors, or permission problems on the socket or spool directory.

4. Determine what changed. This is almost always a deployment event. Check your deploy timestamp against when throughput dropped to zero. If using Emperor mode, a config file modification triggers the reload automatically.

5. Validate the config offline. Run uwsgi --ini app.ini --no-server to test the configuration without starting the server. This runs the normal initialization path, which in non-lazy mode includes loading the application (uwsgi_init_all_apps()), and then exits after printing “no-server mode requested. Goodbye.” — so it does catch application import errors in preforking setups. It does not exercise workers, so with --lazy-apps (where each worker loads the app) it cannot catch import errors; for that case run a Python-level import check (e.g., python -c "import your_app_module") from the deployment’s virtualenv.

Metrics and signals to monitor

Fixes

Roll back the deployment

The fastest recovery is to redeploy the last known-good code and reload again. If the rollback code loads cleanly, workers will start accepting within seconds. Do not attempt to fix the broken code in place while the service is down. Roll back first, fix forward later.

Fix the startup error and redeploy

Once the service is restored via rollback, address the root cause identified in the application logs. Common fixes:

• Apply the missing database migration before deploying.
• Add the missing environment variable to the deployment config.
• Pin the correct dependency version in the requirements lockfile.
• Install the missing uWSGI plugin or correct the protocol configuration.

After fixing, validate with uwsgi --ini app.ini --no-server before reloading.

Enable need-app for crash visibility

The need-app option causes uWSGI to exit entirely if the application cannot load, rather than silently looping with dead workers. This makes a broken deploy immediately visible to process monitoring and orchestrators (systemd, supervisor, Emperor) instead of masquerading as a healthy master with zero workers.

The interaction of need-app with lazy-apps (where each worker loads the app and reports failures to the master) is not widely documented; its exit behavior is version- and plugin-dependent. Test this configuration in staging before relying on it in production.

Prevention

Validate before every reload. Run uwsgi --ini app.ini --no-server in your deploy pipeline to catch config errors, missing plugins, and syntax problems before they reach the running server. Add a Python import check (python -c "import your_app_module") to catch application-level import errors that config validation alone may miss.

Alert on zero accepting workers with master alive. This is the composite signal that distinguishes a reload blackout from a master crash. The threshold: zero workers with pid > 0 AND accepting == 1 AND status != "cheap" for more than 60 seconds, combined with visible respawn churn. See the mental model for operators for how this fits into the broader failure pattern catalogue.

Use chain reload for safer deploys. The touch-chain-reload trigger cycles workers one at a time, waiting for each new worker to reach accepting == 1 before retiring the next old one. If the new code is broken, the chain stalls after the first worker fails to accept, and the remaining old workers keep serving traffic. This trades a longer reload window for protection against full blackouts. Chain reload provides overlap only when there is a spare worker; with a single worker the reload leaves a full serving gap while the new worker initializes, so it loses its zero-downtime property without being disabled (there is no dedicated limitation in the code).

Enable need-app. As described above, this makes broken deploys crash loudly rather than silently looping. Test under your specific configuration (preforking vs. lazy-apps) before relying on it.

Use safe-pidfile. The safe-pidfile option writes the PID file only after the server has completed its initialization phase, rather than early in startup, and never on a reload. In non-lazy mode the application is loaded during that same initialization, so the file is written only for a fully-initialized master. This prevents a stale PID file from masking a failed startup when the master never fully comes up.

Deploy behind canary or staging. Route a fraction of traffic to a canary instance with the new code before fully rolling out. A broken deploy on the canary triggers the same zero-accepting-workers pattern without affecting the full fleet.

How Netdata helps

• Per-second collection of uWSGI worker stats makes a reload blackout visible within seconds, not at the next polling interval.
• The accepting worker count metric, derived from pid > 0 AND accepting == 1 AND status != "cheap", shows whether the instance can serve traffic.
• Respawn rate and last_spawn churn make the spawn-die-respawn loop visible on a dashboard.
• Correlating uWSGI worker metrics with deployment events in a single timeline connects “throughput is zero” to “the deploy broke app startup.”
• Master process liveness distinguishes a reload blackout (master alive, zero workers) from a master crash (master dead).

Related guides

• uWSGI all workers busy: reading the busy ratio before the queue fills
• uWSGI connection refused: clients turned away when the backlog overflows
• uWSGI harakiri death spiral: workers killed and respawned while throughput collapses
• uWSGI harakiri not configured: stuck workers with no timeout and no recovery
• uWSGI harakiri timeout: setting it against request duration and nginx timeouts
• uWSGI harakiri-verbose: finding the blocked syscall behind a timeout
• uWSGI HARAKIRI ON WORKER: requests killed for exceeding the timeout
• How uWSGI actually works in production: a mental model for operators
• uWSGI listen backlog and net.core.somaxconn: sizing the connection queue
• uWSGI listen queue full: the backlog overflow that drops connections silently
• uWSGI listen_queue always zero: why the stats field is broken on Linux
• uWSGI master process dead: total outage while the PID file lingers