uWSGI thundering herd: accept() contention and the thunder-lock fix

You have a uWSGI deployment with multiple worker processes, and something does not add up. CPU usage is elevated. Workers toggle between idle and busy. But request throughput is low, response times are higher than expected, and adding more workers makes things worse instead of better. The system looks under load but is not actually doing much work.

This is the uWSGI thundering herd problem. When a new connection arrives on the shared listening socket, every idle worker process wakes up and races to call accept(). Only one wins. The rest burn CPU and kernel time on a context switch for nothing, then go back to sleep. Under low-to-moderate traffic with many workers, this contention consumes more CPU than the actual request processing.

The fix is a single configuration directive: --thunder-lock. But it has known interactions with certain uWSGI subsystems that every operator should understand before enabling it in production.

What this means

uWSGI is a pre-fork server. The master process creates a listening socket (UNIX domain or TCP) and forks a pool of worker processes that share the same socket fd. In the default configuration, each idle worker blocks in its event loop (epoll_wait() on Linux, kqueue() on BSD) waiting for activity on the socket.

When a connection arrives, the kernel wakes every worker blocked on that fd. Modern Linux kernels have a thundering herd mitigation, but it only applies to processes blocked directly in accept(). Because uWSGI workers are blocked in epoll_wait() internally, the kernel cannot distinguish this wakeup from any other fd event, and the herd fires.

All woken workers then call accept() on the same socket. The kernel serializes this internally (only one can succeed), but every loser pays the cost of a context switch and a failed syscall before returning to its event loop. With N workers and low traffic, most wakeups result in N-1 wasted cycles.

flowchart TD
    subgraph prob["Problem: no thunder-lock"]
        direction TB
        A1["New connection on shared socket"] --> B1["All idle workers wake from epoll_wait"]
        B1 --> C1["N workers compete for accept()"]
        C1 --> D1["One worker wins, serves request"]
        C1 --> E1["N-1 workers lose, return to idle - wasted context switches"]
    end
    subgraph fix["Fix: thunder-lock enabled"]
        direction TB
        A2["New connection on shared socket"] --> B2["Shared mutex serializes accept()"]
        B2 --> C2["Only one worker wakes per connection"]
        C2 --> D2["No contention, no wasted wakeups"]
    end

--thunder-lock adds an inter-process mutex (pthread robust mutexes on modern Linux) that serializes the accept() call across the worker pool. Instead of every worker racing, one worker acquires the lock, calls accept(), processes the connection, then releases the lock. The kernel wakes one waiter instead of all of them.

The trade-off is a small increase in minimum latency from mutex acquisition in exchange for significantly better maximum latency, more even request distribution across workers, and dramatically reduced CPU waste from context switching.

Common causes

The third cause deserves emphasis. When --gevent N is combined with --thunder-lock, the thunder-lock is silently bypassed. The startup log prints “thunder lock: enabled”, but strace shows workers calling epoll_wait() and accept4() without any intervening futex lock/unlock calls. This is a known issue (uWSGI GitHub issue #1757) with no fix committed as of uWSGI 2.0.x. If you are running gevent workers, --thunder-lock will not help you.

Quick checks

All commands below are read-only and safe to run in production.

# Check worker status and identify the busy/idle split
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[] | select(.pid > 0)] | {total: length, busy: ([.[] | select(.status == "busy")] | length), idle: ([.[] | select(.status == "idle")] | length)}'

# Check per-core in_request across all workers
# In thundering herd, most cores show in_request: 0 even when workers appear busy
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[] | select(.pid > 0) | .id as $wid | .cores[] | {worker: $wid, core: .id, in_request: .in_request}]'

# Check per-worker request counts - low counts with high CPU signals contention
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[] | select(.pid > 0) | {id: .id, requests: .requests, avg_rt_ms: (.avg_rt / 1000)}]'

# Check the listen queue externally (uWSGI internal listen_queue field is unreliable on Linux)
# Non-zero Recv-Q with idle workers signals accept contention
ss -ltn 'sport = :8000'

# Check context switch rate - elevated cs relative to throughput confirms the herd
vmstat 1 5

# Verify thunder-lock is in the config
grep -i thunder /etc/uwsgi/apps-available/*.ini

# Check startup log for thunder-lock confirmation
journalctl -u uwsgi --no-pager | grep -i "thunder lock"

# Confirm lock engine in use (pthread robust mutexes on modern Linux)
journalctl -u uwsgi --no-pager | grep -i "lock engine"

Adjust the stats socket address (127.0.0.1:9191), the service name (uwsgi), and the listening port (:8000) to match your deployment. If your stats server uses a UNIX socket, use uwsgi --connect-and-read /path/to/stats.sock instead.

How to diagnose it

1. Confirm the signal pattern. Query the stats server for worker busy ratio and per-worker request counts. In a thundering herd, the busy ratio is low or normal, but request throughput is disproportionately low relative to the number of workers. A 16-worker pool serving 50 req/s with elevated CPU is suspicious.

2. Check per-core in_request. This is the decisive signal. For each worker, examine workers[].cores[].in_request. In thundering herd contention, most cores show in_request: 0 even when the worker status fluctuates between idle and busy. The worker woke up, lost the accept() race, and went back to sleep without ever holding a request. In real starvation, in_request is 1 for most active cores because workers are genuinely processing requests that take too long.

3. Verify the listen queue externally. Use ss -ltn (TCP) or ss -lxn (UNIX socket) to check Recv-Q. A non-zero or fluctuating receive queue with mostly idle workers indicates the kernel has queued connections but workers are not efficiently accepting them. Note: uWSGI’s internal listen_queue and load stats fields are broken on standard Linux and almost always read 0 regardless of actual backlog.

4. Check context switch rate. Run vmstat 1 5 and look at the cs column. In thundering herd, context switches per second are disproportionately high relative to request throughput. Compare against a baseline taken during normal operation.

5. Distinguish from real starvation. This is the critical step. The table below contrasts the two patterns.

6. Check for gevent suppression. If you are running with --gevent, --thunder-lock is silently bypassed regardless of config. Verify with strace -f -e trace=futex,accept4,epoll_wait -p <worker_pid> for a few seconds. Attaching strace to a production worker adds overhead, so keep it brief. No futex calls between epoll_wait and accept4 means the mutex is not being acquired.

Metrics and signals to monitor

Fixes

Enable --thunder-lock

Add --thunder-lock to your uWSGI configuration. It is a boolean flag with no argument.

# uwsgi.ini
thunder-lock = true

After restarting, verify in the startup log:

thunder lock: enabled
lock engine: pthread robust mutexes

The lock engine is auto-detected at build time. On modern Linux with a recent glibc and kernel, it uses pthread robust mutexes with PROCESS_SHARED and ROBUST attributes. If a worker holding the mutex dies unexpectedly, the kernel marks the mutex as inconsistent and the next acquirer can recover it.

Trade-off: You trade a small amount of minimum latency for better maximum latency and more even request distribution across the worker pool. For most production deployments with multiple worker processes, the trade-off favors enabling thunder-lock.

Reduce worker count

If your traffic volume is low relative to your worker count, reducing workers directly reduces the size of the herd. A 4-worker pool serving 20 req/s has far less accept() contention than a 16-worker pool serving the same traffic.

If workers are mostly idle, you have more workers than necessary. The --cheaper subsystem (dynamic worker scaling) can help by scaling workers to match demand, though its specific interaction with the accept serialization mutex is not well documented.

Known interactions to watch

Three known issues affect --thunder-lock in production:

gevent bypass. As described above, --thunder-lock is silently ignored when --gevent is active. The startup log will print “thunder lock: enabled” regardless. Do not expect it to help in gevent deployments.

max-requests deadlock. When a worker holding the thunder-lock mutex reaches its --max-requests limit and exits, the mutex may not be released cleanly. The deadlock-detector thread logs [deadlock-detector] a process holding a robust mutex died. recovering... and the master waits for --worker-reload-mercy (default 60 seconds) before force-killing the worker. Under high load with frequent max-requests recycling, this can cause cascading worker deaths and brief windows of total unavailability. If you use both --thunder-lock and --max-requests, monitor for deadlock-detector log entries and consider lowering --worker-reload-mercy to shorten the recovery window.

reload stalls. Repeated uwsgi.reload() calls (via the stats API) with thunder-lock enabled can cause workers to stop accepting new connections entirely after several reloads. The root cause appears to be the master waiting for a worker blocked on acquiring the thunder lock during the reload sequence. If you trigger programmatic reloads, test thoroughly. Lowering --worker-reload-mercy mitigates but does not eliminate the issue.

Prevention

• Enable --thunder-lock by default for pre-fork (non-gevent) deployments. The uWSGI developers do not enable it by default because of potential library and kernel incompatibilities across distributions, but on modern Linux with a current glibc it is well-tested and widely used in production.
• Right-size your worker pool. More workers is not always better. Workers compete for accept() and consume memory. Match worker count to actual concurrency demand, and use the --cheaper subsystem for elastic scaling.
• Monitor the distinguishing signals. Track per-core in_request, per-worker request counts, and context switch rate alongside worker busy ratio. These signals together make thundering herd immediately distinguishable from real starvation.
• Avoid combining --thunder-lock with gevent. It does not work and gives false confidence.
• Test reload behavior. If you use --max-requests or programmatic reloads, verify that thunder-lock does not introduce stalls or deadlocks under your specific traffic patterns.

How Netdata helps

• Per-second worker metrics. Netdata collects uWSGI stats server data at per-second resolution, making the rapid busy/idle toggling of thundering herd visible in a way that 10-second or 60-second polling intervals miss entirely.
• Per-worker request distribution. Uneven or uniformly low request counts across workers are a primary indicator. Netdata surfaces per-worker request rates so you can see the imbalance immediately.
• CPU context switch correlation. Netdata’s Linux CPU collector tracks context switches per second. Placing this alongside uWSGI worker metrics lets you confirm that CPU waste correlates with accept() contention rather than request processing.
• avg_rt per worker. The exponential moving average response time is collected per worker, making latency elevation visible alongside low throughput.
• Anomaly detection on throughput. Netdata’s ML-based anomaly detection flags the throughput drop that thundering herd causes, even when worker busy ratios look normal. The anomaly fires on the mismatch between apparent busyness and actual work completed.
• Socket queue monitoring. External listen queue depth measured via ss (compensating for uWSGI’s broken internal listen_queue field) can be collected alongside uWSGI metrics for a complete picture of accept() behavior.

Related guides

• How uWSGI actually works in production: a mental model for operators
• uWSGI master process dead: total outage while the PID file lingers
• uWSGI monitoring checklist: the signals every production app server needs
• uWSGI monitoring maturity model: from survival to expert