You deployed new code to uWSGI. The graceful reload killed the old workers, but throughput is zero. The master process is alive. Every worker it forks dies before accepting a connection.

The reload mechanism worked. The new code cannot initialize. Workers crash during startup, the master respawns them, and they crash again. No worker stays alive long enough to serve a request.

The master process appears healthy. Process checks pass. The stats server responds to polling. If your health check only verifies the master PID or pings the stats endpoint, it reports green while 100% of requests are dropped at the kernel backlog.

This pattern is most visible with lazy-apps enabled, where each worker loads the application independently after fork. In preforking mode, a failed app load may fail the master itself, producing a different failure mode.

What this means

A reload blackout is defined by three conditions occurring simultaneously:

  1. The master process is alive and the stats server is reachable.
  2. Zero workers are accepting connections. No worker has pid > 0 AND accepting == 1 AND status != "cheap".
  3. Worker churn is visible: respawn_count is climbing or last_spawn timestamps show very recent spawn attempts.

This is distinct from several lookalike states. A dead master has an unreachable stats server. An idle server in cheaper mode has workers with status: "cheap" and pid: 0, but at least the cheaper minimum should still be alive and accepting. During a reload blackout, the master is actively spawning workers that die within seconds or milliseconds, and respawn_count climbs while throughput stays at zero.

Duration matters. A normal graceful reload may briefly reduce accepting workers while old workers drain and new ones start. If zero accepting workers persists beyond 60 seconds, startup has genuinely failed.

flowchart TD
    A["SIGHUP or touch reload trigger"] --> B["Master kills old workers"]
    B --> C["Master forks new worker"]
    C --> D{"App initializes?"}
    D -->|No: import or config error| E["Worker exits immediately"]
    E --> F["Master respawns worker"]
    F --> C
    E --> G["Zero accepting workers"]
    G --> H["Throughput at zero"]
    G --> I["Connections queue in kernel backlog"]
    D -->|Yes| J["Worker accepts requests"]

Common causes

CauseWhat it looks likeFirst thing to check
Syntax or import error in new codeWorkers die in under 1 second after spawn, Python traceback in app logApplication error logs for ImportError, SyntaxError, ModuleNotFoundError
Missing environment variableWorkers start importing but fail during config initializationApp logs for KeyError, ImproperlyConfigured, or env validation errors
Database migration not appliedApp imports succeed but first schema check or query failsMigration status, DB schema version vs. expected
Incompatible dependency versionImport error mentioning version mismatch or missing attributeRequirements lockfile vs. installed packages
Missing uWSGI pluginMaster log shows plugin load failure after reload, workers spawn but cannot serveuWSGI startup log for plugin load errors

Quick checks

All of these are safe, read-only operations. They assume the stats server is enabled and listening on 127.0.0.1:9191 (TCP). If your stats server uses a UNIX socket, replace 127.0.0.1:9191 with the socket path. If the stats server is not enabled, all stats-based diagnosis is blind: check the uWSGI master log and application logs instead.

# Check accepting worker count (should be > 0)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[] | select(.pid > 0 and .status != "cheap" and .accepting == 1)] | length'

# Check respawn churn (run twice, a few seconds apart; should be stable in healthy state)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].respawn_count] | add'

# Check total throughput (should be climbing if workers are alive)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].requests] | add'

# Check last_spawn timestamps and worker states
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.workers[] | {id: .id, pid: .pid, status: .status, last_spawn: .last_spawn, accepting: .accepting}'

# Confirm master is alive
uwsgi --connect-and-read 127.0.0.1:9191 | jq '{pid: .pid}'

# Check kernel listen queue depth (TCP socket; adjust port as needed)
ss -ltn 'sport = :8000'

If the accepting count is zero, respawn_count is climbing, the requests total is flat, and the master PID is present, you are in a reload blackout.

How to diagnose it

  1. Confirm the pattern. Run the accepting worker count and respawn count checks above, separated by 5 to 10 seconds. If accepting stays at zero and respawn_count climbs, the master is spawning workers that immediately die.

  2. Read the application logs. Workers that die during startup almost always write a traceback before exiting. Look for ImportError, SyntaxError, database connection failures, or configuration validation errors. The log entry should name the specific module, variable, or query that failed.

  3. Check the uWSGI master log for startup errors. The master logs worker spawn and death events. In a blackout, you see rapid “spawned uWSGI worker” lines followed by worker exits, repeating. Look for messages about plugin loading failures, shared library errors, or permission problems on the socket or spool directory.

  4. Determine what changed. This is almost always a deployment event. Check your deploy timestamp against when throughput dropped to zero. If using Emperor mode, a config file modification triggers the reload automatically.

  5. Validate the config offline. Run uwsgi --ini app.ini --no-server to test whether the configuration parses without starting the server. For catching Python import errors specifically, also run python -c "import your_app_module" from the deployment’s virtualenv.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Accepting worker countPrimary availability metric: how many workers can serve requests right nowDrops to zero while master is alive
Respawn count (rate)Worker lifecycle churn: workers dying and being replacedRising rapidly with no corresponding throughput
Request throughput (delta)Whether requests are actually being servedFlat or zero while traffic arrives
last_spawn timestampWhen each worker was last forkedVery recent (seconds ago) on repeated polls, indicating crash loop
Listen queue (external, via ss)Whether connections are piling up in the kernel backlogNon-zero Recv-Q, connections queuing with no workers to accept them
Master process presenceWhether the master is alive at allStats server reachable (distinguishes blackout from master death)

Fixes

Roll back the deployment

The fastest recovery is to redeploy the last known-good code and reload again. If the rollback code loads cleanly, workers will start accepting within seconds. Do not attempt to fix the broken code in place while the service is down. Roll back first, fix forward later.

Fix the startup error and redeploy

Once the service is restored via rollback, address the root cause identified in the application logs. Common fixes:

  • Apply the missing database migration before deploying.
  • Add the missing environment variable to the deployment config.
  • Pin the correct dependency version in the requirements lockfile.
  • Install the missing uWSGI plugin or correct the protocol configuration.

After fixing, validate with uwsgi --ini app.ini --no-server before reloading.

Enable need-app for crash visibility

The need-app option causes uWSGI to exit entirely if the application cannot load, rather than silently looping with dead workers. This makes a broken deploy immediately visible to process monitoring and orchestrators (systemd, supervisor, Emperor) instead of masquerading as a healthy master with zero workers.

A known race condition in lazy-apps mode can cause inconsistent exit behavior when need-app is combined with lazy-apps. Test this configuration in staging before relying on it in production.

Prevention

Validate before every reload. Run uwsgi --ini app.ini --no-server in your deploy pipeline to catch config errors, missing plugins, and syntax problems before they reach the running server. Add a Python import check (python -c "import your_app_module") to catch application-level import errors that config validation alone may miss.

Alert on zero accepting workers with master alive. This is the composite signal that distinguishes a reload blackout from a master crash. The threshold: zero workers with pid > 0 AND accepting == 1 AND status != "cheap" for more than 60 seconds, combined with visible respawn churn. See the mental model for operators for how this fits into the broader failure pattern catalogue.

Use chain reload for safer deploys. The touch-chain-reload trigger cycles workers one at a time, waiting for each new worker to reach accepting == 1 before retiring the next old one. If the new code is broken, the chain stalls after the first worker fails to accept, and the remaining old workers keep serving traffic. This trades a longer reload window for protection against full blackouts. Chain reload requires at least two workers to provide overlap; with a single worker there is no old worker to keep serving while the new one initializes.

Enable need-app. As described above, this makes broken deploys crash loudly rather than silently looping. Test under your specific configuration (preforking vs. lazy-apps) before relying on it.

Use safe-pidfile. The safe-pidfile option writes the PID file only after the server has successfully initialized, rather than early in startup. This prevents a stale PID file from masking a failed startup when the master never fully comes up.

Deploy behind canary or staging. Route a fraction of traffic to a canary instance with the new code before fully rolling out. A broken deploy on the canary triggers the same zero-accepting-workers pattern without affecting the full fleet.

How Netdata helps

  • Per-second collection of uWSGI worker stats makes a reload blackout visible within seconds, not at the next polling interval.
  • The accepting worker count metric, derived from pid > 0 AND accepting == 1 AND status != "cheap", shows whether the instance can serve traffic.
  • Respawn rate and last_spawn churn make the spawn-die-respawn loop visible on a dashboard.
  • Correlating uWSGI worker metrics with deployment events in a single timeline connects “throughput is zero” to “the deploy broke app startup.”
  • Master process liveness distinguishes a reload blackout (master alive, zero workers) from a master crash (master dead).