$ guides / uwsgi / uwsgi-graceful-reload-thundering-herd ▌

Operations Guides

uWSGI reload thundering herd: capacity drops to zero during a slow restart

You deploy a new version of your application. Seconds later, request throughput collapses to zero. Nginx returns 502s. The uWSGI master process is alive and the stats server responds, but no workers are accepting connections. Thirty seconds to several minutes later, throughput recovers. If this pattern aligns exactly with every deployment, you are hitting the reload thundering herd.

A standard uWSGI graceful reload kills all workers simultaneously, then forks replacements. If your application has a slow startup phase (heavy imports, ML model loading, cache warming, database connection pool initialization), there is a window where zero workers are ready to serve. The kernel listen queue fills with pending connections, overflows, and drops new connections silently. The first requests that land after workers recover hit cold caches and empty connection pools, making them abnormally slow.

This is expected behavior of the standard reload mechanism, not a bug. The fix is to change how you reload.

What this means

During a standard graceful reload (triggered by SIGHUP, touch-reload, or master FIFO r), the uWSGI master waits for running workers to finish current requests, closes all file descriptors except the listening socket, then re-executes itself. All workers are killed and respawned simultaneously.

In default prefork mode, the application is loaded once in the master before forking. Workers inherit the loaded state through copy-on-write memory. Worker startup is fast because the import already happened, and the reload window is short.

With --lazy-apps, each worker loads the application independently after fork. There is no copy-on-write sharing between workers. Every reload requires every worker to re-import the entire application from scratch. If a single cold import takes 10 seconds, the reload window is at least that long, and potentially longer if workers compete for CPU and memory during simultaneous initialization.

The following diagram contrasts the two reload strategies:

flowchart LR
    subgraph standard["Standard reload"]
        A1["SIGHUP"] --> A2["All workers exit at once"]
        A2 --> A3["Zero accepting workers"]
        A3 --> A4["Backlog overflows, connections dropped"]
        A4 --> A5["All workers re-import, slow recovery"]
    end
    subgraph chain["Chain reload"]
        C1["Trigger"] --> C2["One worker reloads"]
        C2 --> C3["Others keep serving"]
        C3 --> C4["Worker ready, next reloads"]
        C4 --> C5["Capacity never hits zero"]
    end

The key monitoring signal is accepting worker count: workers where pid > 0, status != "cheap", and accepting == 1. During the reload window, this metric drops to zero. Request throughput drops to zero in lockstep. The listen queue fills. Once it overflows (default backlog is 100 connections), the kernel silently drops new connections (SYN packets are ignored by default; RST is sent only if net.ipv4.tcp_abort_on_overflow is enabled).

This is distinct from two related patterns that operators sometimes confuse it with:

Reload blackout: new application code has a fatal import or configuration error. Workers spawn and immediately die. They never come back. The master churns through spawn-die cycles indefinitely. In the thundering herd, workers eventually recover. If they do not, you have a blackout, not a thundering herd. See uWSGI master process dead: total outage while the PID file lingers.

Accept-side thundering herd: multiple idle workers wake up to compete for a single accept() call on the shared listening socket, causing kernel lock contention. This is a performance inefficiency mitigated by --thunder-lock. It does not cause a capacity outage. Enabling --thunder-lock will not help with the reload thundering herd.

Common causes

Cause	What it looks like	First thing to check
Standard reload during deploy	Throughput drops to zero for the duration of application startup on every deploy, then recovers	Whether `--touch-chain-reload` is configured
Heavy application startup	Reload window exceeds 10 seconds. Workers peg CPU during spawn as they import large libraries or load models	Time a cold import outside uWSGI
`--lazy-apps` lengthening the window	Each worker imports independently after fork with no copy-on-write sharing. Reload window scales with worker count	Whether `--lazy-apps` is enabled and whether it is necessary
Accidental reload trigger	Unexplained throughput drop with no planned deploy. A watched file was modified, triggering `touch-reload`	uWSGI log for unexpected reload events
Deploying during peak traffic	The capacity blackout overlaps with high request volume, maximizing user impact	Traffic levels at deploy time

Quick checks

All commands are read-only and safe to run during an incident.

# Check accepting worker count (zero during reload window)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[] | select(.pid > 0 and .status != "cheap" and .accepting == 1)] | length'

# Check listen queue depth externally (uWSGI's internal listen_queue field is unreliable on Linux)
ss -ltn 'sport = :8000'

# Check total requests served (run twice, seconds apart, to derive throughput)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[].requests] | add'

# Check per-worker state and accepting flag
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.workers[] | {id: .id, status: .status, accepting: .accepting, pid: .pid}'

# Check kernel-level listen queue overflow counter
nstat -az TcpExtListenOverflows TcpExtListenDrops

# Check reload-related configuration
grep -E 'lazy-apps|touch-chain-reload|touch-reload|chain' /etc/uwsgi/apps-available/*.ini

# Look for recent reload or spawn events in uWSGI logs
grep -iE 'reload|spawn|worker' /var/log/uwsgi/*.log | tail -30

How to diagnose it

Confirm the throughput drop correlates with a deployment or reload event. Check deploy timestamps, CI/CD logs, or uWSGI logs for reload entries. The thundering herd pattern aligns exactly with reload events. If the drop does not correlate with a reload, investigate worker starvation or downstream dependency failures instead.
Measure the accepting-worker-count dip. Poll the accepting worker count at 1-second intervals during a reload. The duration of the zero-capacity window is your application startup time. This tells you exactly how long users are affected on every deploy.
Check whether --lazy-apps is enabled. If it is, each worker imports the application independently after fork. This multiplies the reload window because there is no shared import across workers. Without --lazy-apps, the master loads the application once and workers inherit it via copy-on-write, making worker startup much faster. The tradeoff: --lazy-apps is required for chain reload, and some applications are not fork-safe and need it regardless.
Measure cold-start time independently. Run python -c "import your_app" or equivalent outside uWSGI. If this takes more than a few seconds, you have found the bottleneck. Use python -X importtime -c "import your_app" 2>&1 | tail -30 to identify the slowest modules in the import chain. Common culprits: importing large ML frameworks, loading geo databases, initializing TLS contexts, or establishing database connection pools at import time.
Check listen queue behavior during the window. Use ss -ltn to check the Recv-Q column on the listening socket during a reload. A rising Recv-Q means connections are queuing. Check nstat -az TcpExtListenOverflows before and after to see if connections were dropped. The uWSGI listen_queue stats field is unreliable on Linux and should not be used for this measurement.
Verify the reload trigger source. In Emperor mode, modifying a vassal config file triggers a reload. Check whether a config management tool, cron job, or developer accidentally touched a touch-reload file.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Accepting worker count	Directly measures live serving capacity	Drops to zero during reload, stays at zero for application startup duration
Request throughput (delta)	Confirms real user impact	Collapses to zero, then recovers in a V-shaped curve
Listen queue depth (via `ss`)	Shows connections piling up during the blackout	Non-zero and rising during reload window
TcpExtListenOverflows	Kernel counter for connections dropped when backlog is full	Non-zero delta during reload
avg_rt per worker	First requests after reload hit cold caches and empty pools	Spike immediately after workers come back online
Respawn count	All workers respawned simultaneously confirms a bulk reload	Sudden uniform increment across all worker slots at the same timestamp

Fixes

Switch to chain reload for code deploys

--touch-chain-reload <file> restarts workers one at a time. The next worker is not reloaded until the previous one is ready to accept requests. This maintains N-1 workers serving at all times, eliminating the zero-capacity window.

Requirements and tradeoffs:

Requires --lazy-apps. Without it, the master loads the application before fork, so there is nothing to chain.
Only useful for code updates, not configuration changes. Config changes require a full reload because the master process itself needs to re-read settings.
You need enough workers for the overlap to matter. With only 2 workers, losing one halves capacity during each step. With 8 or more, the impact of one worker being down is minimal.
Trigger by touching the file or sending master FIFO c.
Overall reload takes longer (sequential, not parallel), but capacity never drops to zero.
--lazy-apps increases per-worker memory because there is no copy-on-write sharing. Each worker holds a full independent copy of the application.

Chain reload combined with --thunder-lock has a long-standing deadlock report (unbit/uwsgi issue #1038, opened 2015, still open); if you use both together, verify your version does not reproduce it before relying on chain reload.

Consolidate and reduce startup initialization

The reload window is bounded by how long it takes a worker to become truly ready. If your framework defers initialization to the first request, workers report as accepting before they are fully initialized. This undermines chain reload (the next worker starts reloading before the current one is truly ready) and causes slow first requests after every spawn. Ensure all heavy initialization completes before the worker enters the accept loop.

Move expensive imports (ML models, geo databases, large lookup tables) into the application initialization path so they happen at import time, not lazily on first request. Frameworks like Django may defer significant code loading until the first request hits a route. If the first request after each worker spawn is abnormally slow, lazy loading is the likely cause.
Pre-warm database connection pools during worker initialization rather than on first request.
Avoid import torch, import tensorflow, or similar heavy framework imports inside request handlers. Import them at module level so they load once per worker startup, not on the first request that triggers them.
Profile the import chain with python -X importtime -c "import your_app" 2>&1 | sort -t '|' -k2 -rn | head -20 to find the slowest modules. Consider lazy-loading non-critical dependencies at call sites if they are only needed for specific endpoints.

For Django, pre-load the URLconf during worker initialization with get_resolver(ROOT_URLCONF).url_patterns after django.setup(); this loads urls.py and the views it imports before the first request. Calling the WSGI application callable itself is not the recommended way to warm up Django routes.

Size the listen backlog for the reload window

The default --listen backlog is 100 connections. If your reload window is 15 seconds and you receive 50 requests per second, approximately 750 connections arrive during the blackout. A backlog of 100 drops most of them.

Increase --listen and the kernel net.core.somaxconn to absorb the window. The effective backlog is min(--listen, net.core.somaxconn). This does not eliminate the latency impact (queued requests still wait for workers to become ready), but it prevents silent connection drops. See uWSGI listen backlog and net.core.somaxconn: sizing the connection queue.

Tradeoff: a larger backlog means queued requests wait longer before being processed. If workers take 15 seconds to start, those connections sit in the kernel queue for 15 seconds. Some clients will time out on their end before the worker ever sees the request. A larger backlog reduces connection drops but does not reduce user-visible latency during the reload.

Deploy during low-traffic windows

If chain reload is not feasible (for example, you need to reload master-level configuration that requires a full restart), schedule deploys during the lowest-traffic period. The blackout still occurs, but fewer requests are affected. This is a mitigation, not a fix.

Prevention

Use chain reload for all code updates. Requires --lazy-apps. Keeps N-1 workers alive at all times.
Monitor throughput during every deploy. If the dip is measurable in your monitoring, it is affecting users. Per-second granularity matters here; polling at 10-second intervals can miss the entire reload window if startup is fast.
Preload expensive imports at application init. Ensures workers are truly ready when they report as accepting.
Size the listen backlog for burst absorption. Set both --listen and net.core.somaxconn; the effective backlog is the lower of the two.
Audit reload trigger paths. In Emperor mode, any change to a vassal config file triggers a reload. touch-reload watches arbitrary files. Make sure only your deployment pipeline touches these paths.
Consider blue-green or canary deployment. Shift traffic to a pre-warmed instance instead of reloading in place. Avoids the reload window entirely at the cost of running duplicate capacity during the switch.

How Netdata helps

Per-second accepting worker count lets you measure the exact duration of the zero-capacity window during each reload, correlated to the second with deploy timestamps.
Throughput at per-second resolution shows the collapse and recovery curve, letting you quantify user impact rather than guessing from coarse aggregates.
TcpExtListenOverflows tracking catches the silent connection drops that happen when the backlog overflows. These drops produce no uWSGI log entry and no application error.
avg_rt per worker reveals the cold-cache spike on the first requests after workers come back online, confirming the startup cost and its downstream latency impact.
Anomaly detection on throughput can flag the reload pattern automatically, even when no explicit deploy marker exists in your monitoring.