$ guides / uwsgi / uwsgi-tcp-listen-overflows ▌

Operations Guides

uWSGI dropped connections: reading TcpExtListenOverflows and ListenDrops

Users report intermittent “connection refused” or timeouts. Nginx logs show 502s. uWSGI logs show nothing: no errors, no exceptions, no harakiri events. The master is alive, workers are accepting requests, throughput looks normal on average. But clients are being turned away.

The explanation is in two kernel counters that uWSGI cannot report on its own: TcpExtListenOverflows and TcpExtListenDrops. When all uWSGI workers are busy and the kernel’s accept queue (the listen backlog) fills, the kernel drops new connections before uWSGI’s accept() call ever runs. The application has no visibility into this event. The only evidence lives in /proc/net/netstat.

What this means

Linux maintains two queues for every LISTEN socket. The SYN queue holds half-open connections waiting for the final ACK of the handshake. The accept queue holds fully established connections waiting for the application to call accept(). When uWSGI workers are all busy, established connections accumulate in the accept queue. Once it reaches its configured limit, the kernel drops new connections silently.

flowchart LR
    A["Client SYN"] --> B["SYN queue"]
    B -->|"handshake done"| C["Accept queue"]
    C -->|"accept()"| D["uWSGI worker"]
    B -->|"SYN queue full"| E["drop: ListenDrops"]
    C -->|"accept queue full"| F["drop: ListenOverflows"]

The two counters serve different purposes:

TcpExtListenOverflows: increments when the accept queue is full and a fully established connection is dropped. This is the direct signal that the application could not call accept() fast enough.
TcpExtListenDrops: a broader counter that increments on any packet dropped on a LISTEN socket, including accept queue overflow plus other conditions such as memory allocation failure.

Both counters are cumulative and monotonically increasing. They never decrease. What matters operationally is the rate of change between samples.

Why uWSGI cannot see these drops: the kernel discards the connection before uWSGI’s accept() executes. uWSGI never receives the connection, never logs it, and never counts it in any stats field. The listen_queue field in uWSGI’s stats server is unreliable on standard Linux because it relies on a TCP_INFO ioctl that does not report actual queue depth consistently across kernel versions. The listen_queue_errors field is dead code: it is read from uwsgi.shared->backlog_errors, which nothing in the uWSGI source ever increments (confirmed in 2.0.28). The load field mirrors listen_queue: both are assigned the same sampled backlog value in the master, with a source TODO comment noting it should be something more advanced. All three are typically 0 regardless of actual backlog depth. External measurement is the only reliable approach.

Common causes

Cause	What it looks like	First thing to check
Worker pool exhaustion	All workers `busy`, Recv-Q climbing, counters ticking up	Stats server: worker busy ratio
Backlog too small for bursts	Brief counter spikes during traffic peaks, Recv-Q reaches Send-Q	`ss -tln` Recv-Q vs Send-Q on the uWSGI socket
`somaxconn` capping backlog	`--listen` set high but effective backlog is lower	`cat /proc/sys/net/core/somaxconn`
Reload blackout window	Counter spike synchronized with deploy events	Respawn timestamps in stats, deploy logs
Downstream dependency slowdown	Workers busy, avg_rt rising, no traffic increase	Application logs, downstream health checks

Quick checks

# Read kernel listen overflow/drop counters (cumulative since boot)
nstat -az TcpExtListenOverflows TcpExtListenDrops

# Alternative: parse /proc/net/netstat directly
grep -E "ListenOverflows|ListenDrops" /proc/net/netstat

# Or via netstat summary (deprecated on some distributions, prefer nstat)
netstat -s | grep -iE "listen|overflow"

# Check per-socket accept queue depth on the uWSGI port
# Recv-Q = current connections queued, Send-Q = configured backlog
ss -tlnp sport = :8000

# For UNIX domain sockets
ss -lxnp | grep uwsgi

# Check the effective somaxconn limit
cat /proc/sys/net/core/somaxconn

# Check uWSGI worker busy ratio (requires stats server enabled)
uwsgi --connect-and-read 127.0.0.1:9191 | \
  jq '([.workers[] | select(.status == "busy")] | length) as $busy | ([.workers[] | select(.pid > 0 and .status != "cheap")] | length) as $alive | if $alive > 0 then ($busy / $alive * 100) else 0 end'

The nstat -az flags show all counters including those currently at zero. Run it twice a few seconds apart during the incident. Any increase between samples means connections are being dropped right now.

How to diagnose it

Confirm the rate is non-zero. Run nstat -az TcpExtListenOverflows TcpExtListenDrops, wait 5 to 10 seconds, run it again. Any delta means active drops. A zero reading does not prove safety: the queue can fill and drain between samples if your polling interval is coarse.
Attribute the drops to uWSGI. These counters are host-wide, not per-socket. If the host runs other TCP services (another uWSGI instance, a database, Redis, nginx), the drops may originate from a different process entirely. Check ss -tlnp on the specific uWSGI port. If Recv-Q is climbing toward Send-Q while the kernel counters increment, uWSGI is the source. If Recv-Q is 0 on the uWSGI socket but counters are rising, investigate other listeners on the same host.
Check the effective backlog. Compare the Send-Q value from ss (the actual backlog the kernel honored at listen() time) against your configured --listen value. If Send-Q is lower than --listen, somaxconn is capping it. On kernel 5.4 and later, somaxconn defaults to 4096; on older kernels it defaults to 128, which silently limits even a --listen 1024 configuration.
Check worker utilization. If all workers report status: "busy", the root cause is capacity: workers cannot call accept() fast enough because they are occupied with in-flight requests. If workers are idle but Recv-Q is still climbing, the problem is in the accept path itself. Potential causes include missing --thunder-lock causing thundering herd contention, or workers stuck in a slow startup phase.
Check for reload correlation. If counter spikes coincide with deployment timestamps, the reload window may be saturating the queue. During a graceful reload (SIGHUP), old workers drain current requests while new workers start. If application startup is slow (heavy imports, model loading, cache warming), there is a window where few or zero workers are accepting connections and the listen queue fills.
Cross-reference with downstream health. Worker exhaustion is frequently caused by downstream slowdown rather than insufficient worker count. If avg_rt is rising and workers are persistently busy, check database connection counts, external API response times, and any blocking calls that lack timeouts. The fix for connection drops may be in a completely different system.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
TcpExtListenOverflows (rate)	Direct evidence of accept queue overflow	Any non-zero rate in production
TcpExtListenDrops (rate)	Broader drop signal, catches non-overflow drops	Any non-zero rate
ss Recv-Q on uWSGI socket	Per-socket queue depth for attribution	Sustained non-zero, or approaching Send-Q
Worker busy ratio	Capacity utilization of the pool	Sustained above 80%
Accepting worker count	Workers available to call accept()	Dropping below expected minimum
Average response time (avg_rt)	Downstream slowdown indicator	Rising trend approaching harakiri threshold
somaxconn vs –listen	Effective backlog ceiling	somaxconn < –listen

Fixes

Backlog too small

Increase both --listen and somaxconn together. The effective backlog is min(--listen, somaxconn). Raising one without the other has no effect.

# Check current somaxconn
cat /proc/sys/net/core/somaxconn

# Set a higher value (requires root, non-persistent)
echo 4096 | sudo tee /proc/sys/net/core/somaxconn

# Persist via sysctl
# Add to /etc/sysctl.d/99-listen.conf:
# net.core.somaxconn = 4096
# Then: sudo sysctl --system

In uWSGI configuration:

listen = 1024

uWSGI checks somaxconn at socket creation: if --listen exceeds it, uWSGI logs “Listen queue size is greater than the system max net.core.somaxconn” and exits (behavior present since at least uWSGI 1.9.17). Increase somaxconn first, then restart uWSGI with the higher --listen value.

Worker pool exhaustion

If workers are genuinely saturated, increasing the backlog only delays drops. The backlog buys time during brief bursts, but sustained saturation requires more capacity:

Add workers via --processes N, constrained by available memory (each worker is a full process copy) and CPU cores.
Reduce per-request latency. Profile slow endpoints, optimize database queries, add caching for repeated expensive computations.
Enable --threads N for I/O-bound workloads. In Python, the GIL serializes CPU-bound work within a worker, so threads help with I/O wait but not computation.
Consider async mode (gevent or asyncio) for workloads dominated by concurrent I/O-bound requests. This changes the concurrency model and the meaning of “busy worker.”

Reload window saturation

If drops correlate with deployment events:

Use --touch-chain-reload (or the master FIFO c command) to cycle workers one at a time instead of killing and respawning all workers simultaneously.
Be cautious with --lazy-apps: each worker imports the application independently, which increases per-worker startup time and extends the reload window.
Deploy outside peak traffic windows until the startup time is understood and bounded.

tcp_abort_on_overflow (diagnostic only)

Setting net.ipv4.tcp_abort_on_overflow = 1 causes the kernel to send a TCP RST instead of silently dropping when the accept queue is full. This makes drops immediately visible to clients as “connection refused” rather than a silent timeout. It is useful for confirming that drops are happening during diagnosis, but it changes the client-visible failure mode. Leave it at 0 (the default) in steady-state production.

Prevention

Monitor kernel counter rates continuously. Any non-zero rate of TcpExtListenOverflows means users are experiencing connection failures. Alert on rate of change, not absolute value.
Set somaxconn greater than or equal to –listen. Audit this relationship whenever either value changes. On shared hosts, somaxconn is system-wide and affects all services.
Monitor ss Recv-Q on the uWSGI socket. A non-zero Recv-Q means workers are not keeping up. This is an earlier signal than the kernel drop counters because it shows the queue building before overflow occurs.
Track worker busy ratio as a capacity leading indicator. Sustained ratios above 80% leave limited headroom for traffic bursts. The degradation curve is cliff-edge: once workers hit 100%, there is no graceful degradation, only queuing then dropping.
Do not rely on uWSGI’s listen_queue, load, or listen_queue_errors stats fields. All three are unreliable or dead code on standard Linux. Use ss and kernel counters instead.

How Netdata helps

Netdata collects kernel TCP extension counters every second, including TcpExtListenOverflows and TcpExtListenDrops, directly from /proc/net/netstat. This provides several diagnostic advantages:

Per-second rate detection catches brief overflow spikes that 10-second or 60-second polling intervals miss. The accept queue can fill, overflow, and drain between coarse samples.
Host-wide counter correlation with per-process metrics. On multi-service hosts, Netdata’s process-level CPU, memory, and file descriptor metrics help attribute host-wide drops to the specific service saturating its accept queue.
Anomaly detection on counter rates flags unusual rates of change before static thresholds fire, catching the early stages of worker pool exhaustion.
Correlation with uWSGI stats. When the Netdata uWSGI collector is enabled, worker busy ratio, average response time, and respawn counts are visualized alongside the kernel TCP counters. A rising ListenOverflows rate next to 100% worker busy ratio confirms the diagnosis.
System pressure context. CPU saturation, memory pressure, and swap activity are tracked alongside TCP counters. When drops correlate with resource pressure, the root cause is visible without switching tools.