$ guides / uwsgi / uwsgi-chain-reload-zero-downtime ▌

Operations Guides

uWSGI chain reload: cycling workers one at a time for zero-downtime deploys

The default uWSGI graceful reload sends all workers the shutdown signal at once. Each worker finishes its current request, exits, and the master forks a replacement. During the gap between old workers dying and new workers becoming ready to accept connections, serving capacity falls to zero. For applications with fast startup, this gap is a brief hiccup. For applications that load large models, warm connection pools, or run heavy imports on startup, the gap can stretch into seconds or minutes of complete unavailability.

Chain reload is the alternative. Instead of cycling all workers simultaneously, the master reloads workers one at a time. It curses a single worker, waits for that worker’s replacement to be fully ready and accepting connections, then moves on to the next. At every point during the reload, at least N-1 workers are serving traffic. Because capacity never drops to zero, the listen backlog is not starved by the reload itself.

Why chain reload exists

A broken deploy under graceful reload is worse than a slow one. Whether triggered by SIGHUP, --touch-reload, or an Emperor vassal config change, if the new code has a fatal error (import failure, missing dependency, bad migration), the master enters a worker crash loop: workers spawn, immediately die, and no requests are served.

Even a successful reload has a capacity cost. During peak traffic, a 3-second reload window at 100 requests per second means 300 connections arrive while no worker is accepting. If your listen backlog is the default 100, the kernel starts dropping connections after the first second.

Chain reload contains both problems. By cycling workers sequentially, the old generation stays available while the new generation comes up one worker at a time. If the new code has a startup error, only one worker slot is affected. The remaining workers continue running the old code, giving you time to detect the failure and roll back.

How it works

There are two ways to trigger a chain reload:

--touch-chain-reload <file>: touch the named file to trigger the reload.
Master FIFO command c: write the character c to the master FIFO. The master FIFO must be enabled with --master-fifo <path>. Available since uWSGI 1.9.17.

When the master receives the chain reload signal, it begins the following sequence for each worker:

The master curses the worker (marks it for shutdown).
The cursed worker stops accepting new requests, finishes in-flight requests, then exits.
The master forks a new worker in its place.
The new worker loads the application and initializes.
Once the new worker is ready (its accepting flag is set to 1), the master proceeds to curse the next worker.

The critical detail is step 5. The master does not curse the next worker until the replacement for the current one is ready to accept requests. This was not always the case. In uWSGI versions before 1.9.21, the master did not check whether the newly spawned worker was actually ready before moving on. With slow-initializing applications, this meant all workers could be in startup simultaneously, defeating the purpose of chain reload. The fix in 1.9.21 added a readiness check: the master waits for the new worker to report accepting == 1 before proceeding.

flowchart TD
    T["Chain reload triggered"] --> W1["Curse worker 1"]
    W1 --> R1["New worker 1 spawns\nand loads app"]
    R1 --> A1{"Worker 1 ready\nand accepting?"}
    A1 -->|Not yet| A1
    A1 -->|Yes| W2["Curse worker 2\n(workers 1, 3..N serve)"]
    W2 --> R2["New worker 2 spawns\nand loads app"]
    R2 --> A2{"Worker 2 ready\nand accepting?"}
    A2 -->|Not yet| A2
    A2 -->|Yes| WN["Continue for\nremaining workers"]
    WN --> DONE["Chain reload complete"]

Prerequisites:

--lazy-apps: Chain reload requires lazy-apps mode, where each worker loads the application independently after fork. Without it, the application is loaded once in the master before forking, and there is no per-worker code reload to chain. The --lazy option (without -apps) is generally discouraged because it changes many internal defaults beyond just the loading behavior. Use --lazy-apps instead.
At least 2 workers: A single worker cannot chain reload without a serving gap because there is no spare worker to handle requests while the one worker recycles. The cursed worker must exit before its replacement spawns, creating a guaranteed gap with zero serving capacity.

What does NOT reload:

Chain reload only cycles request-serving workers. Mules and spoolers are not reloaded by chain reload. If your deployment relies on mules for background work or spoolers for deferred task processing, those processes continue running the old code. To reload mules and spoolers, you need a full graceful reload (SIGHUP), which sacrifices the zero-downtime property.

Worker reload mercy:

The --worker-reload-mercy option controls how long the master waits for a cursed worker to finish its current request before forcibly killing it. The default is 60 seconds. If a worker is stuck on a long-running request that exceeds this mercy period, the master sends SIGKILL and that request is lost. Set this value based on your longest legitimate request duration.

Log noise:

During a chain reload, uWSGI emits “chain is still waiting for worker N…” once per second while waiting for a new worker to become ready. For applications with slow startup, this can flood logs. Suppress it with:

log-drain = chain is still waiting for worker

uwsgi.accepting() and WorkerOverride:

If you use the --python-worker-override option (which replaces a worker with a custom Python script instead of the standard application), you must call uwsgi.accepting() from your Python code to signal readiness. Without this call, the master will wait indefinitely for the worker to become accepting. uwsgi.accepting() was added in uWSGI 2.0.14 for the chain-reload + worker-override combination.

Where it shows up in production

Scheduled deployments during peak traffic. The canonical use case. You deploy new code in the middle of the day because the release cannot wait. Without chain reload, every deploy is a mini-outage. With chain reload, users see no interruption.

Applications with slow startup. ML model loading, large GeoIP database imports, or applications that import heavy libraries (torch, pandas, tensorflow) can take 10-60 seconds per worker to initialize. An all-at-once reload means 10-60 seconds of zero capacity. Chain reload means each worker takes that time individually while N-1 workers keep serving.

Emperor mode with many vassals. Each vassal can be configured independently. Chain reload on a per-vassal basis allows zero-downtime deploys for individual applications without affecting others.

The chdir symlink gotcha. If your deployment uses a symlink for the chdir directive (common in blue-green setups where /app/current points to /app/releases/<timestamp>), chain reload will load the wrong code. The master process resolves chdir once at startup and does not re-resolve it during chain reload (the master itself does not restart). New workers fork from the master and inherit its working directory, so every replacement worker loads from the old release. Verify your symlink behavior before relying on chain reload for deploys. A full master restart (not a reload) picks up the new symlink target.

Tradeoffs and when to use it

Factor	Graceful reload (SIGHUP)	Chain reload
Capacity during reload	Drops to zero briefly	Drops by 1 worker max
Reload duration	One startup window	N x startup window
Old and new code coexist	No (hard cutover)	Yes (overlap for entire duration)
Mules/spoolers reloaded	Yes	No
Risk of full outage	High if new code is broken	Low (old workers stay up)
Minimum workers (for zero-downtime benefit)	1	2

The two tradeoffs that catch teams off guard:

Old and new code must be compatible. During a chain reload with 8 workers, there is a period where workers 1-4 run the new code and workers 5-8 run the old code. If the new code introduces a database schema change that the old code cannot handle, or changes the format of a cache entry that both generations read from the same Redis instance, you will see errors from the old workers. Plan migrations as backward-compatible first, forward-compatible second.

Total reload time is longer. If your application takes 10 seconds per worker to start, an all-at-once reload completes in roughly 10 seconds (all workers start in parallel). A chain reload with 8 workers takes roughly 80 seconds. During this entire period, you are operating at reduced capacity (7 of 8 workers at worst). For most deployments this is acceptable, but it means chain reload is not the fastest path when you need to push a critical fix and can tolerate a brief blip.

When to use graceful reload instead:

You need mules or spoolers reloaded.
Your application has no slow startup and the capacity gap is negligible.
You are deploying a breaking schema change where old and new code cannot coexist.
You have only one worker (chain reload cannot avoid a serving gap with a single worker).

Signals to watch

Signal	Why it matters	Warning sign
Accepting worker count	Should dip by exactly 1 at each step, never to 0	Drops to 0 means the new worker failed to start and the old worker already exited
Worker respawn rate	Should increment by 1 per worker cycled	Multiple simultaneous respawns mean the chain logic broke down
Request throughput	Should remain stable throughout	Drop indicates a capacity gap or the new code rejecting requests
Worker busy ratio	May briefly rise as N-1 workers absorb the load	Sustained high ratio means the reload is too slow for current traffic
Average response time (avg_rt)	Should remain within baseline	Spike indicates the new code is slower or initializing under load

The most important signal is accepting worker count. During a healthy chain reload, you should see it oscillate between N and N-1 as each worker cycles. If it drops to 0, the chain logic has failed: either the new worker crashed on startup and the old worker was already cursed, or the application has a startup error that prevents any new worker from becoming ready. Check application logs for import or startup errors immediately.

avg_rt is an exponential moving average computed as (old + new) / 2. It gives roughly 50% weight to the most recent request, which makes it responsive to latency changes during a reload. A single slow first request on a freshly spawned worker can move the average noticeably. This is expected behavior during chain reload, not necessarily a problem.

How Netdata helps

Per-second accepting worker count: Netdata polls the uWSGI stats server every second. During a chain reload, you can watch the accepting worker count oscillate between N and N-1 in real time. A drop to 0 is immediately visible and can trigger an alert before the listen queue fills.
Correlating respawn rate with throughput: A healthy chain reload shows respawns incrementing one at a time while throughput remains stable. If throughput drops while respawns spike, the new code is failing and the chain is breaking down.
Worker busy ratio during reload: When N-1 workers absorb the traffic of N, busy ratio rises. Per-second granularity shows whether this is a brief blip per worker or a sustained saturation that will overflow the listen queue.
avg_rt anomaly detection: Netdata’s ML-based anomaly detection can flag an avg_rt spike during reload even if it is below a static threshold. This catches cases where new code introduces a regression that is subtle enough to survive initial smoke tests.
Composite view: Correlating accepting worker count, respawn rate, and throughput into a single dashboard lets you distinguish a healthy chain reload from a failing one in seconds, without manually polling the stats server.