Apache listen queue overflow: Recv-Q growth, ListenBacklog, and refused connections

Users report “connection refused” or intermittent timeouts, but Apache is running, the port is open, and a TCP connect from localhost sometimes works. The load balancer flaps the backend in and out of rotation. Nothing in the access log explains it, because the failing connections never got far enough to be logged.

This is the signature of listen queue overflow. Connections complete the TCP handshake in the kernel and sit in the accept queue waiting for an Apache worker to call accept(). When workers cannot keep up, the queue fills, and the kernel starts ignoring or resetting new connection attempts. The server looks up. It is effectively down for a slice of arriving connections.

The useful property of this failure mode is that it is the earliest saturation signal Apache produces. Recv-Q growth appears before request latency rises, because the queued connections have not reached a worker yet. If you catch it here, you catch worker exhaustion before users do.

What this means

Between the network and an Apache worker there is a kernel-maintained queue: the accept queue (TCP listen backlog). A connection that has completed the three-way handshake waits there until Apache accepts it. The queue depth is bounded by Apache’s ListenBacklog directive (default 511), further capped by the kernel’s net.core.somaxconn. The effective limit is the lower of the two, and the kernel does not warn you when it truncates.

For LISTEN sockets, the counters you read mean something different than on established sockets:

• Recv-Q in ss -ltn output is the current number of connections sitting in the accept queue. Not bytes. Connections.
• Send-Q is the maximum queue depth: the effective backlog after the somaxconn cap is applied.

When Recv-Q approaches Send-Q, the kernel drops incoming connection attempts. Depending on kernel version and configuration, the client sees a timeout (the SYN or the final handshake ACK is silently ignored, and the server retransmits SYN/ACK until it gives up) or a fast RST (connection refused). Either way: the port answers, health probes may or may not squeeze through, and real users get intermittent failures. This is the “server up but unreachable” state.

flowchart LR
  client[Client SYN] --> synq[SYN queue]
  synq -->|handshake done| acceptq[Accept queue = Recv-Q]
  acceptq -->|accept| worker[Apache worker]
  acceptq -.->|full: drop or RST| refused[Connection refused / timeout]
  worker -.->|all workers busy| acceptq
  somaxconn[somaxconn cap] -.->|truncates ListenBacklog| acceptq

One important nuance: a brief non-zero Recv-Q during a traffic burst is normal. The queue exists to absorb bursts. The failure is a sustained backlog, which means the accept side is structurally slower than the arrival side. The way you tell them apart is worker utilization: burst spike with idle workers available is noise; sustained Recv-Q with BusyWorkers near MaxRequestWorkers is saturation.

Common causes

The first two dominate. Listen queue overflow is almost never a kernel tuning problem in isolation; it is the visible tail of workers not accepting fast enough.

Quick checks

All read-only. Run these before touching any configuration.

# 1. Current accept queue depth and effective max for Apache's listeners
ss -ltn | grep -E ':80\s|:443\s'
# Recv-Q = connections waiting for accept(); Send-Q = effective backlog

# 2. Same, with owning process (needs root for -p)
ss -tlnp '( sport = :80 or sport = :443 )'

# 3. Kernel overflow counters (cumulative since boot)
nstat -az 2>/dev/null | grep -iE 'ListenOverflows|ListenDrops' || \
  netstat -s | grep -i 'listen'

# 4. The kernel cap on the backlog
sysctl net.core.somaxconn

# 5. Worker utilization right now
curl -s http://localhost/server-status?auto | grep -E 'BusyWorkers|IdleWorkers|Scoreboard'

# 6. Has Apache explicitly hit its worker limit?
grep 'AH00484' /var/log/apache2/error.log 2>/dev/null | tail -5 || \
  grep 'AH00484' /var/log/httpd/error_log | tail -5

Reading the results:

• Check 1: Recv-Q of 0 is healthy. Sustained values above 10 mean workers are not keeping up. Recv-Q within sight of Send-Q means drops are happening or imminent.
• Check 3: ListenOverflows incrementing is the authoritative proof that the accept queue has overflowed. Take two samples 60 seconds apart; the delta is what matters, not the lifetime counter.
• Check 4: if somaxconn is lower than your ListenBacklog, your configured backlog is a fiction. The kernel silently caps it. Kernels before 5.4 defaulted to 128, which quietly reduced Apache’s default 511 to 128; newer kernels default to 4096.
• Check 6: AH00484: server reached MaxRequestWorkers setting confirms the backlog growth is caused by worker exhaustion, not by an arrival spike.

How to diagnose it

1. Confirm the overflow is real and current. Sample ss -ltn for the Apache ports a few times over 30 to 60 seconds. A single non-zero Recv-Q sample means nothing. Sustained non-zero values, or a growing trend, do. Cross-check with the ListenOverflows delta from nstat. If the counter is not increasing, you are looking at queuing, not dropping yet.

2. Check what the queue limit actually is. Read Send-Q from ss -ltn. That number is the truth: it already reflects min(ListenBacklog, somaxconn). If Send-Q is 128 on a server where someone set ListenBacklog 511, you have found a somaxconn truncation.

3. Determine whether the accept side is saturated. Pull BusyWorkers and IdleWorkers from server-status?auto. If IdleWorkers is zero and BusyWorkers equals MaxRequestWorkers from your config, Apache cannot accept faster no matter how big the backlog is. The backlog is just a buffer in front of a full pool. This is the saturation case.

4. Find out why workers are busy. Expand the scoreboard state distribution. Many W states with normal CPU points at slow backends holding workers (the classic slow backend cascade). Many R states points at slow clients or a Slowloris pattern. Many K states on prefork or worker MPM points at keepalive hoarding. The fix for each is completely different, and none of them is “raise the backlog.”

5. Rule out the transient case. If BusyWorkers is well below the limit and idle workers exist, a brief Recv-Q spike is burst absorption, possibly combined with a slow child spawn ramp after a restart. Note it as thin headroom and move on. Alerting on this pattern will train everyone to ignore the signal.

6. Check for a SYN flood only after ruling out the above. High SYN-RECV counts concentrated from few sources, with syncookies active, is a different incident. ListenOverflows during a flood means the queue is full of half-open garbage, not queued legitimate clients.

Metrics and signals to monitor

Fixes

If workers are exhausted

The backlog is downstream of the real problem. Enlarging it buys seconds, not capacity.

• Find what holds workers. Scoreboard states first. Slow backends: fix the backend or lower ProxyTimeout to fail fast and release workers. Keepalive hoarding on prefork/worker: lower KeepAliveTimeout or move to event MPM, where keepalive connections are handled by the listener thread instead of occupying workers.
• Raise MaxRequestWorkers only if memory allows. The ceiling is available_memory / per_child_RSS. Raising the worker limit past that trades connection refused for OOM kills, which is worse. See the sizing guide linked below.

If the backlog itself is too small

Justified when workers have headroom but bursts still overflow a small queue.

• Raise ListenBacklog in the Apache config (default 511).
• Raise net.core.somaxconn to at least the same value, or the kernel silently truncates your setting. Set it persistently in /etc/sysctl.d/ and apply it. Prefer sysctl -p /etc/sysctl.d/<your-file>.conf over sysctl --system, which reapplies every sysctl file on the box and can change unrelated live settings. Verify afterwards with ss -ltn: Send-Q should now show the new effective value. Note that Apache only passes ListenBacklog to listen() when it creates its listening sockets at startup, so the new value takes effect on restart, not on a plain sysctl change.
• Tradeoff: a deeper queue converts fast refusals into slow accepts. Clients hang in the handshake or wait longer for a first byte instead of failing immediately and retrying elsewhere. Behind a load balancer, a moderately sized queue with fast failure is often better than a huge queue that makes the LB hold connections to a struggling node.

If overflow events are invisible until users complain

That is a monitoring gap, not a tuning gap. The fixes above do not recur if you alert on the leading indicators: Recv-Q trend, ListenOverflows delta, and worker utilization together.

One more knob to know about rather than reach for: net.ipv4.tcp_abort_on_overflow=1 makes the kernel send RST instead of silently dropping when the queue is full. It turns timeouts into fast refusals, which sounds appealing, but it also hides the symptom from clients and monitoring and can confuse retry logic. The default (drop, retransmit SYN/ACK) is usually the right behavior.

Prevention

• Alert on the combination, not the single metric. Sustained Recv-Q > 0 AND worker utilization > 80% is a ticket. Recv-Q approaching Send-Q or ListenOverflows increasing is urgent. Recv-Q alone during a burst is not.
• Verify the effective backlog after any change. ss -ltn Send-Q is the ground truth. Config files lie when somaxconn truncates.
• Keep worker headroom. At least 25% of MaxRequestWorkers idle at daily peak. When that erodes, capacity planning, not backlog tuning, is the answer.
• Use event MPM unless a module forces prefork. It removes keepalive hoarding as a backlog driver.
• Baseline ListenOverflows. A counter that has been flat for months and starts climbing is one of the cleanest leading indicators you will get from this stack.

How Netdata helps

• Per-second sampling of listening socket queue depth catches Recv-Q spikes that a 60-second poll averages away, which matters because this signal fluctuates fast.
• Kernel TCP counters including ListenOverflows are collected continuously, so you can alert on the delta instead of eyeballing cumulative counters during an incident.
• Apache scoreboard metrics (BusyWorkers, IdleWorkers, state distribution via mod_status) sit on the same timeline as socket-level data, so you can see in one view whether backlog growth coincides with worker exhaustion or with idle capacity.
• Anomaly detection on these signals flags the transition from “occasional burst queuing” to “sustained backlog,” which is exactly the distinction that is painful to encode as a static threshold.
• Correlating listen queue depth with 5xx rate and LB health check failures shortens the path from “users report timeouts” to “accept queue full because workers are stuck on a slow backend.”

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion
• Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status
• How Apache HTTPD actually works in production: a mental model for operators
• Apache AH00484: server reached MaxRequestWorkers setting - worker pool exhausted
• Apache MaxRequestWorkers tuning: sizing the worker pool against memory
• Apache HTTPD monitoring checklist: the signals every production web server needs
• Apache HTTPD monitoring maturity model: from survival to expert