uWSGI reload thundering herd: capacity drops to zero during a slow restart

You deploy a new version of your application. Seconds later, request throughput collapses to zero. Nginx returns 502s. The uWSGI master process is alive and the stats server responds, but no workers are accepting connections. Thirty seconds to several minutes later, throughput recovers. If this pattern aligns exactly with every deployment, you are hitting the reload thundering herd.

A standard uWSGI graceful reload kills all workers simultaneously, then forks replacements. If your application has a slow startup phase (heavy imports, ML model loading, cache warming, database connection pool initialization), there is a window where zero workers are ready to serve. The kernel listen queue fills with pending connections, overflows, and drops new connections silently. The first requests that land after workers recover hit cold caches and empty connection pools, making them abnormally slow.

This is expected behavior of the standard reload mechanism, not a bug. The fix is to change how you reload.

What this means

During a standard graceful reload (triggered by SIGHUP, touch-reload, or master FIFO r), the uWSGI master waits for running workers to finish current requests, closes all file descriptors except the listening socket, then re-executes itself. All workers are killed and respawned simultaneously.

In default prefork mode, the application is loaded once in the master before forking. Workers inherit the loaded state through copy-on-write memory. Worker startup is fast because the import already happened, and the reload window is short.

With --lazy-apps, each worker loads the application independently after fork. There is no copy-on-write sharing between workers. Every reload requires every worker to re-import the entire application from scratch. If a single cold import takes 10 seconds, the reload window is at least that long, and potentially longer if workers compete for CPU and memory during simultaneous initialization.

The following diagram contrasts the two reload strategies:

flowchart LR
    subgraph standard["Standard reload"]
        A1["SIGHUP"] --> A2["All workers exit at once"]
        A2 --> A3["Zero accepting workers"]
        A3 --> A4["Backlog overflows, connections dropped"]
        A4 --> A5["All workers re-import, slow recovery"]
    end
    subgraph chain["Chain reload"]
        C1["Trigger"] --> C2["One worker reloads"]
        C2 --> C3["Others keep serving"]
        C3 --> C4["Worker ready, next reloads"]
        C4 --> C5["Capacity never hits zero"]
    end

The key monitoring signal is accepting worker count: workers where pid > 0, status != "cheap", and accepting == 1. During the reload window, this metric drops to zero. Request throughput drops to zero in lockstep. The listen queue fills. Once it overflows (default backlog is 100 connections), the kernel silently drops new connections (SYN packets are ignored by default; RST is sent only if net.ipv4.tcp_abort_on_overflow is enabled).

This is distinct from two related patterns that operators sometimes confuse it with:

Reload blackout: new application code has a fatal import or configuration error. Workers spawn and immediately die. They never come back. The master churns through spawn-die cycles indefinitely. In the thundering herd, workers eventually recover. If they do not, you have a blackout, not a thundering herd. See uWSGI master process dead: total outage while the PID file lingers.

Accept-side thundering herd: multiple idle workers wake up to compete for a single accept() call on the shared listening socket, causing kernel lock contention. This is a performance inefficiency mitigated by --thunder-lock. It does not cause a capacity outage. Enabling --thunder-lock will not help with the reload thundering herd.

Common causes

Quick checks

All commands are read-only and safe to run during an incident.

# Check accepting worker count (zero during reload window)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[] | select(.pid > 0 and .status != "cheap" and .accepting == 1)] | length'

# Check listen queue depth externally (uWSGI's internal listen_queue field is unreliable on Linux)
ss -ltn 'sport = :8000'

# Check total requests served (run twice, seconds apart, to derive throughput)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].requests] | add'

# Check per-worker state and accepting flag
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.workers[] | {id: .id, status: .status, accepting: .accepting, pid: .pid}'

# Check kernel-level listen queue overflow counter
nstat -az TcpExtListenOverflows TcpExtListenDrops

# Check reload-related configuration
grep -E 'lazy-apps|touch-chain-reload|touch-reload|chain' /etc/uwsgi/apps-available/*.ini

# Look for recent reload or spawn events in uWSGI logs
grep -iE 'reload|spawn|worker' /var/log/uwsgi/*.log | tail -30

How to diagnose it

1. Confirm the throughput drop correlates with a deployment or reload event. Check deploy timestamps, CI/CD logs, or uWSGI logs for reload entries. The thundering herd pattern aligns exactly with reload events. If the drop does not correlate with a reload, investigate worker starvation or downstream dependency failures instead.

2. Measure the accepting-worker-count dip. Poll the accepting worker count at 1-second intervals during a reload. The duration of the zero-capacity window is your application startup time. This tells you exactly how long users are affected on every deploy.

3. Check whether --lazy-apps is enabled. If it is, each worker imports the application independently after fork. This multiplies the reload window because there is no shared import across workers. Without --lazy-apps, the master loads the application once and workers inherit it via copy-on-write, making worker startup much faster. The tradeoff: --lazy-apps is required for chain reload, and some applications are not fork-safe and need it regardless.

4. Measure cold-start time independently. Run python -c "import your_app" or equivalent outside uWSGI. If this takes more than a few seconds, you have found the bottleneck. Use python -X importtime -c "import your_app" 2>&1 | tail -30 to identify the slowest modules in the import chain. Common culprits: importing large ML frameworks, loading geo databases, initializing TLS contexts, or establishing database connection pools at import time.

5. Check listen queue behavior during the window. Use ss -ltn to check the Recv-Q column on the listening socket during a reload. A rising Recv-Q means connections are queuing. Check nstat -az TcpExtListenOverflows before and after to see if connections were dropped. The uWSGI listen_queue stats field is unreliable on Linux and should not be used for this measurement.

6. Verify the reload trigger source. In Emperor mode, modifying a vassal config file triggers a reload. Check whether a config management tool, cron job, or developer accidentally touched a touch-reload file.

Metrics and signals to monitor

Fixes

Switch to chain reload for code deploys

--touch-chain-reload <file> restarts workers one at a time. The next worker is not reloaded until the previous one is ready to accept requests. This maintains N-1 workers serving at all times, eliminating the zero-capacity window.

Requirements and tradeoffs:

• Requires --lazy-apps. Without it, the master loads the application before fork, so there is nothing to chain.
• Only useful for code updates, not configuration changes. Config changes require a full reload because the master process itself needs to re-read settings.
• You need enough workers for the overlap to matter. With only 2 workers, losing one halves capacity during each step. With 8 or more, the impact of one worker being down is minimal.
• Trigger by touching the file or sending master FIFO c.
• Overall reload takes longer (sequential, not parallel), but capacity never drops to zero.
• --lazy-apps increases per-worker memory because there is no copy-on-write sharing. Each worker holds a full independent copy of the application.

Consolidate and reduce startup initialization

The reload window is bounded by how long it takes a worker to become truly ready. If your framework defers initialization to the first request, workers report as accepting before they are fully initialized. This undermines chain reload (the next worker starts reloading before the current one is truly ready) and causes slow first requests after every spawn. Ensure all heavy initialization completes before the worker enters the accept loop.

• Move expensive imports (ML models, geo databases, large lookup tables) into the application initialization path so they happen at import time, not lazily on first request. Frameworks like Django may defer significant code loading until the first request hits a route. If the first request after each worker spawn is abnormally slow, lazy loading is the likely cause.
• Pre-warm database connection pools during worker initialization rather than on first request.
• Avoid import torch, import tensorflow, or similar heavy framework imports inside request handlers. Import them at module level so they load once per worker startup, not on the first request that triggers them.
• Profile the import chain with python -X importtime -c "import your_app" 2>&1 | sort -t '|' -k2 -rn | head -20 to find the slowest modules. Consider lazy-loading non-critical dependencies at call sites if they are only needed for specific endpoints.

Size the listen backlog for the reload window

The default --listen backlog is 100 connections. If your reload window is 15 seconds and you receive 50 requests per second, approximately 750 connections arrive during the blackout. A backlog of 100 drops most of them.

Increase --listen and the kernel net.core.somaxconn to absorb the window. The effective backlog is min(--listen, net.core.somaxconn). This does not eliminate the latency impact (queued requests still wait for workers to become ready), but it prevents silent connection drops. See uWSGI listen backlog and net.core.somaxconn: sizing the connection queue.

Tradeoff: a larger backlog means queued requests wait longer before being processed. If workers take 15 seconds to start, those connections sit in the kernel queue for 15 seconds. Some clients will time out on their end before the worker ever sees the request. A larger backlog reduces connection drops but does not reduce user-visible latency during the reload.

Deploy during low-traffic windows

If chain reload is not feasible (for example, you need to reload master-level configuration that requires a full restart), schedule deploys during the lowest-traffic period. The blackout still occurs, but fewer requests are affected. This is a mitigation, not a fix.

Prevention

• Use chain reload for all code updates. Requires --lazy-apps. Keeps N-1 workers alive at all times.
• Monitor throughput during every deploy. If the dip is measurable in your monitoring, it is affecting users. Per-second granularity matters here; polling at 10-second intervals can miss the entire reload window if startup is fast.
• Preload expensive imports at application init. Ensures workers are truly ready when they report as accepting.
• Size the listen backlog for burst absorption. Set both --listen and net.core.somaxconn; the effective backlog is the lower of the two.
• Audit reload trigger paths. In Emperor mode, any change to a vassal config file triggers a reload. touch-reload watches arbitrary files. Make sure only your deployment pipeline touches these paths.
• Consider blue-green or canary deployment. Shift traffic to a pre-warmed instance instead of reloading in place. Avoids the reload window entirely at the cost of running duplicate capacity during the switch.

How Netdata helps

• Per-second accepting worker count lets you measure the exact duration of the zero-capacity window during each reload, correlated to the second with deploy timestamps.
• Throughput at per-second resolution shows the collapse and recovery curve, letting you quantify user impact rather than guessing from coarse aggregates.
• TcpExtListenOverflows tracking catches the silent connection drops that happen when the backlog overflows. These drops produce no uWSGI log entry and no application error.
• avg_rt per worker reveals the cold-cache spike on the first requests after workers come back online, confirming the startup cost and its downstream latency impact.
• Anomaly detection on throughput can flag the reload pattern automatically, even when no explicit deploy marker exists in your monitoring.

Related guides

• uWSGI all workers busy: reading the busy ratio before the queue fills
• uWSGI connection refused: clients turned away when the backlog overflows
• uWSGI harakiri death spiral: workers killed and respawned while throughput collapses
• uWSGI harakiri not configured: stuck workers with no timeout and no recovery
• uWSGI harakiri timeout: setting it against request duration and nginx timeouts
• uWSGI harakiri-verbose: finding the blocked syscall behind a timeout
• uWSGI HARAKIRI ON WORKER: requests killed for exceeding the timeout
• How uWSGI actually works in production: a mental model for operators
• uWSGI listen backlog and net.core.somaxconn: sizing the connection queue
• uWSGI listen queue full: the backlog overflow that drops connections silently
• uWSGI listen_queue always zero: why the stats field is broken on Linux
• uWSGI master process dead: total outage while the PID file lingers