uWSGI listen queue full: the backlog overflow that drops connections silently

Clients report connection timeouts or resets. Your load balancer shows 502s or 504s on requests to the uWSGI backend. The uWSGI master is running, workers are alive, the stats server responds, and application logs show no errors. This is listen queue overflow: the kernel accept backlog on the uWSGI listening socket is full, and the kernel is silently dropping new connections before accept().

The listen queue is the kernel-level buffer between completed TCP handshakes and available workers. When all workers are busy and this buffer fills, the kernel silently drops incoming connections. No uWSGI log entry is written. No uWSGI error counter increments. The stats server runs in the master process and stays responsive during complete worker starvation, so health checks that query the stats endpoint still pass. Health check endpoints handled by workers may also pass if they are lightweight enough to be accepted between drops. Meanwhile, real users cannot connect.

What this means

The uWSGI listen queue is the kernel-level accept() backlog on the socket the master process binds. When a connection arrives, the kernel completes the TCP handshake and places the connection in this queue. When a worker calls accept(), it pulls the next connection and processes it. If no worker calls accept() because all workers are busy, connections accumulate in the queue. Once the queue reaches its maximum size, the kernel drops new connections.

The critical problem is observability. uWSGI’s own listen_queue stats field is broken on standard Linux. For TCP sockets, it relies on TCP_INFO behavior that varies across kernel versions. For UNIX sockets, it requires a non-standard kernel ioctl. The load field in the stats JSON is identical to listen_queue (not average latency, despite its name). The listen_queue_errors field exists in the JSON output but is dead code: it is never incremented anywhere in the uWSGI source. All three fields almost always read 0 regardless of actual backlog.

The only reliable measurements come from outside uWSGI:

• ss -ltn shows Recv-Q (current queue depth) and Send-Q (the configured backlog limit) for each listening socket.
• TcpExtListenOverflows from /proc/net/netstat counts the number of times the kernel dropped a connection because the accept queue was full. This counter is system-wide, not per-socket.

flowchart TD
    A[New connection arrives] --> B{Worker idle?}
    B -- yes --> C[accept and process]
    B -- no --> D[Queue in kernel backlog]
    D --> E{Backlog full?}
    E -- no --> F[Wait for a worker]
    F --> B
    E -- yes --> G[Kernel drops connection]
    G --> H[Client sees timeout or RST]
    G --> I[No uWSGI log entry]
    G --> J[TcpExtListenOverflows increments]

When the backlog is full, the kernel does not warn uWSGI. With default kernel settings (net.ipv4.tcp_abort_on_overflow=0), the final ACK of the TCP handshake is silently dropped. The client sees a connection timeout, not an explicit error. If tcp_abort_on_overflow is set to 1, the kernel sends a RST instead and the client sees a connection reset. In both cases, uWSGI logs nothing because the connection never reached accept().

On some configurations, uWSGI may log the message *** uWSGI listen queue of socket "..." (fd: X) full !!! (N/M) *** where N is the current queue depth and M is the configured max. The message is emitted only in verbose mode (--verbose), and only when the master’s periodic queue check (roughly once per second) finds the queue full, so it is not emitted for every dropped connection and may not appear at all depending on timing and kernel measurement.

Common causes

Quick checks

Run these read-only commands during the incident. They are safe and non-disruptive.

# Check current accept queue depth and backlog limit (TCP socket)
# Recv-Q = current queue depth, Send-Q = configured backlog
# Replace :8000 with your uWSGI port
ss -ltn 'sport = :8000'

# For UNIX socket
ss -lxn | grep uwsgi

# Check kernel-level listen queue overflow counter (system-wide, not per-socket)
nstat -az TcpExtListenOverflows

# Check the kernel's somaxconn cap
cat /proc/sys/net/core/somaxconn

# Check worker busy ratio from uWSGI stats
uwsgi --connect-and-read 127.0.0.1:9191 | jq '([.workers[] | select(.status == "busy")] | length) as $busy | ([.workers[] | select(.pid > 0 and .status != "cheap")] | length) as $alive | if $alive > 0 then ($busy / $alive * 100) else 0 end'

# Check accepting worker count (alive, not cheaped, accepting connections)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '[.workers[] | select(.pid > 0 and .status != "cheap" and .accepting == 1)] | length'

# Check what busy workers are doing (which URIs are stuck)
uwsgi --connect-and-read 127.0.0.1:9191 | jq '.workers[] | select(.status == "busy") | {id, uri, avg_rt}'

# Check for CLOSE-WAIT sockets (system-wide count; filter by PID with -p, requires root)
ss -tan state close-wait | wc -l

Do not rely on the listen_queue and listen_queue_errors fields in uWSGI stats. Both are unreliable or dead on standard Linux and almost always read 0 regardless of actual backlog. Use ss and nstat instead.

How to diagnose it

1. Confirm the kernel is dropping connections. Run nstat -az TcpExtListenOverflows twice, a few seconds apart. If the counter is rising, the kernel is actively dropping connections because an accept queue is full somewhere on the host. This counter is system-wide, so if multiple services share the host, correlate with per-socket queue depth to attribute the drops to uWSGI.

2. Check the socket queue depth. Run ss -ltn 'sport = :PORT' (for TCP) or ss -lxn (for UNIX sockets). If Recv-Q is non-zero and sustained, workers are not accepting fast enough. If Recv-Q equals Send-Q, the queue is full and connections are being dropped right now.

3. Check worker saturation. Pull the stats JSON and compute the busy ratio. If 100% of non-cheaped workers are busy, the queue is filling because no worker can call accept(). Check which URIs the busy workers are processing to identify the slow endpoint.

4. Check the effective backlog limit. On Linux, uWSGI reads net.core.somaxconn at bind time and refuses to start if --listen exceeds it, logging Listen queue size is greater than the system max net.core.somaxconn (N). Keep the two aligned: raise somaxconn first, then set --listen. (The kernel itself clamps the listen() backlog to somaxconn; it does not round the accept backlog to a power of two — that rounding historically applied to the SYN queue table, not the accept queue.)

5. Check for stuck workers. If harakiri is not configured, workers that hang on blocking I/O stay stuck forever. Check whether any worker’s request count has stopped increasing while its status remains busy. Each stuck worker permanently reduces capacity by one slot.

6. Rule out the reload window. If the incident correlates with a deployment, the graceful reload may have drained all old workers before new workers finished loading the application. Check whether last_spawn timestamps in the stats JSON align with the outage window.

Metrics and signals to monitor

Fixes

Increase the backlog

Raise --listen in the uWSGI configuration and ensure net.core.somaxconn is at least as high. The default --listen is 100. The default somaxconn varies: 128 on older distributions, 4096 on kernel 5.4 and newer. Both may be too low for production traffic.

Changing --listen requires restarting uWSGI; it is a bind-time setting, not reloadable.

[uwsgi]
listen = 1024

# Set somaxconn at runtime (non-persistent across reboots; add to sysctl.conf for persistence)
sysctl -w net.core.somaxconn=1024

somaxconn changes apply only to sockets created after the change. Restart uWSGI to pick up the new value. In containers, net.core.somaxconn may not be writable without specific capabilities depending on the runtime.

A larger backlog lets the kernel buffer brief traffic spikes without dropping connections. It does not fix the underlying problem if workers are consistently saturated. It buys time.

Tradeoff: a very large backlog can mask sustained capacity problems. Connections sit in the queue for seconds, latency increases, and clients time out before a worker picks them up. Size the backlog for burst absorption, not for chronic underprovisioning.

Add worker capacity

If the busy ratio is consistently above 80% during normal traffic, the worker pool is undersized. Increase --processes or adjust the cheaper subsystem range. The degradation curve is cliff-edge: below 100% utilization, latency is roughly flat. At 100%, latency jumps non-linearly because every additional request queues in the kernel backlog.

Before adding workers, check available memory. Each worker is a full process copy. Adding workers without memory headroom triggers swapping, which makes every worker slower and can worsen the saturation.

Enable harakiri

If workers are getting stuck on blocking calls and harakiri is not configured, enable it. Without harakiri, a single hung request permanently consumes a worker slot. Over time, stuck workers accumulate until the pool is exhausted and the listen queue fills.

[uwsgi]
harakiri = 30
harakiri-verbose = true

Set the timeout to 2-3x your expected maximum legitimate request duration. The harakiri-verbose option (Linux only) logs the killed worker’s blocked syscall and wait channel by reading /proc/<pid>/syscall and /proc/<pid>/wchan at kill time, alongside uWSGI’s standard in-flight request dump.

Harakiri does not fix the underlying hang. It kills the stuck worker and respawns it, temporarily restoring capacity. Use it as a safety net while you investigate the root cause.

Fix the slow endpoint

If all busy workers show the same URI, a single endpoint is the bottleneck. Common culprits: database queries without timeouts, external API calls without client-side timeouts, DNS resolution hanging, or lock contention. Check per-worker uri and running_time in the stats JSON.

As an emergency measure during an incident, consider blocking the slow endpoint at the load balancer to free workers for other traffic while you investigate.

Use chain reload

Standard graceful reload (SIGHUP) drains all old workers simultaneously. During the transition, capacity drops while new workers initialize and load the application. Use --chain-reload to cycle workers one at a time, maintaining partial capacity throughout the reload.

Prevention

• Monitor TcpExtListenOverflows continuously. This kernel counter is the only reliable signal that connections are being dropped due to accept queue overflow. Alert on any non-zero rate of change. The counter is monotonic and system-wide; track the delta between polling intervals.
• Monitor socket queue depth with ss. Alert on sustained non-zero Recv-Q on the uWSGI listening socket. This catches the problem before the queue fills and drops begin.
• Do not trust uWSGI listen_queue, load, or listen_queue_errors stats fields. All three are unreliable or dead code on standard Linux. They almost always read 0 regardless of actual backlog.
• Keep --listen and net.core.somaxconn aligned. On Linux, uWSGI refuses to start when --listen exceeds somaxconn. Check both after configuration changes and after container image rebuilds, where somaxconn defaults may differ from the host.
• Track worker busy ratio as a leading indicator. Sustained busy ratio above 80% during normal traffic means the next traffic spike will fill the queue.
• Configure harakiri. Stuck workers without a timeout permanently reduce capacity. The absence of harakiri is itself a monitoring blind spot because harakiri_count always reads 0.
• Do not use the stats endpoint or master PID as a health check. The master and stats server remain responsive during complete worker starvation. Health checks must go through the worker pool to detect this failure mode.

How Netdata helps

Netdata surfaces the signals that matter for this failure mode and correlates them at per-second resolution:

• TcpExtListenOverflows is collected from /proc/net/netstat, giving you a direct kernel-level count of accept queue drops without manual nstat polling.
• Socket queue metrics (Recv-Q and Send-Q on listening sockets) are collected per-second, so you can see the queue filling before it overflows.
• uWSGI worker metrics (busy ratio, accepting worker count, avg_rt, per-worker status) are collected directly from the stats server, letting you correlate worker saturation with kernel-level drops on the same timeline.
• ML anomaly detection flags unusual changes in queue depth or worker busy ratio even when absolute values have not crossed a static threshold.
• Correlation across layers (kernel TCP counters, uWSGI stats, application latency) on a single timeline shortens the diagnostic path from “clients see connection timeouts” to “workers are saturated on this endpoint.”

Related guides

• uWSGI all workers busy: reading the busy ratio before the queue fills
• uWSGI harakiri death spiral: workers killed and respawned while throughput collapses
• uWSGI harakiri not configured: stuck workers with no timeout and no recovery
• uWSGI HARAKIRI ON WORKER: requests killed for exceeding the timeout
• How uWSGI actually works in production: a mental model for operators
• uWSGI master process dead: total outage while the PID file lingers
• uWSGI monitoring checklist: the signals every production app server needs
• uWSGI monitoring maturity model: from survival to expert
• uWSGI thundering herd: accept() contention and the thunder-lock fix
• uWSGI worker pool starvation: the silent outage where every worker is busy