Apache ListenBacklog vs net.core.somaxconn: the silently truncated accept queue

You tuned Apache for burst absorption. You set ListenBacklog 2048 (or left the default 511, reasoning it was generous). Then a traffic spike arrived, workers saturated, and connections were refused far earlier than your capacity model predicted. The scoreboard and MaxRequestWorkers get the blame, but the real culprit is often one layer down: the kernel quietly rewrote your backlog at listen() time and never told anyone.

Apache passes ListenBacklog as the backlog argument to listen(2). Per the listen(2) man page, if the backlog argument is greater than the value in /proc/sys/net/core/somaxconn, the kernel silently caps it to that value. No log line, no warning from Apache, no error return. The effective maximum accept queue on any listening socket is always min(ListenBacklog, net.core.somaxconn).

What follows: the mechanism, how to read the truncated backlog from ss, how to raise both values together, and why the default 511 is too small on busy servers even when Recv-Q reads zero most of the time. For where the accept queue sits in Apache’s wider saturation model, see How Apache HTTPD actually works in production.

What the accept queue is and why it matters

When a client completes the TCP three-way handshake with Apache’s listening socket, the connection lands in the kernel’s accept queue. It stays there until an Apache worker calls accept() and takes it. This queue is the last buffer before service denial: while it has room, a momentarily saturated Apache still absorbs connection bursts; when it fills, the kernel starts dropping or resetting incoming connections and clients see connection refused or timeouts.

The queue exists because accept rate and arrival rate are never perfectly matched. Workers are busy finishing requests, children are spawning, a graceful restart just happened. A deep enough queue smooths those gaps. A shallow one turns every transient stall into refused connections, and a load balancer watching TCP connect behavior may pull the server from rotation before Apache has logged anything at all.

Two failure modes follow:

1. Queue overflow under saturation. All workers busy, queue fills, SYNs get dropped or RST. The server appears up (the port is open, the process runs) but is unreachable.
2. Silent truncation at configuration time. You believe the queue holds 511 or 2048 connections. It actually holds 128, because net.core.somaxconn on your kernel defaults lower than your ListenBacklog, and the kernel never mentioned it.

How the two limits interact

Apache httpd 2.4 defaults ListenBacklog to 511. The odd number is deliberate: a comment in the Apache source explains it defaults to 511 instead of 512 because some systems store the backlog as an 8-bit datatype, and 512 truncated to 8 bits is 0 while 511 survives as 255. If you set the directive to a non-zero value, httpd 2.4 requires it to be at least 512.

The kernel side has its own default, and it changed at a version boundary that still matters:

• Linux before 5.4: net.core.somaxconn defaults to 128.
• Linux 5.4 and later: defaults to 4096.

The practical consequences by platform:

So on a stock RHEL 7 or RHEL 8 system, Apache’s default 511 is silently truncated to 128. The server you think absorbs a 511-connection burst actually absorbs 128. On kernel 5.4 and later, no truncation happens by default, but the moment you raise ListenBacklog above 4096 the same trap reappears.

One more layer: tuning tooling can change somaxconn out from under you. The throughput-performance profile in RHEL’s tuned service sets net.core.somaxconn to 2048. Set ListenBacklog 4096 on such a system without checking and you get 2048.

flowchart TD
  A[ListenBacklog in Apache config
default 511] --> C[listen backlog argument]
  B[net.core.somaxconn
128 pre-5.4, 4096 since 5.4] --> C
  C --> D{Kernel compares at listen time}
  D -->|backlog greater than somaxconn| E[Silently capped
no warning anywhere]
  D -->|backlog within somaxconn| F[Used as configured]
  E --> G[Effective accept queue
= min of the two]
  F --> G
  G --> H{Queue full under load?}
  H -->|yes| I[Connections dropped or RST
clients see refused or timeout]

Reading the effective backlog with ss

You cannot detect truncation from Apache. Apache logs nothing about it, and apachectl configtest validates syntax, not kernel behavior. The only reliable check is to ask the kernel what the listening socket actually got.

# Show current and maximum accept queue for Apache listeners
ss -ltn | grep -E ':80\s|:443\s'

For LISTEN sockets, the column meanings are inverted from what most people expect:

• Recv-Q: current number of connections sitting in the accept queue, waiting for accept().
• Send-Q: the maximum backlog the socket actually has. This is the effective, post-truncation value.

If your config says ListenBacklog 511 and ss shows Send-Q of 128, the kernel capped you. Compare Send-Q against your configured value on every Apache host; a mismatch is the truncation, and the fix is on the kernel side.

For a live view of queue depth with more detail:

# Detailed view; current queue depth appears as unacked
ss -lti sport = :80

The current accept queue depth shows up as unacked:N in this output, not in a field named “backlog”. The kernel maps the accept queue counter to the unacked field for listening sockets, which confuses people who go looking for a backlog label.

Finally, check whether connections are already being dropped:

# Listen overflow counters
nstat -a | grep -i listen
netstat -s | grep -i "listen"

Rising ListenOverflows (and ListenDrops) means the queue has already filled and the kernel has already refused connections. That is the trailing indicator; by the time it moves, users felt it.

Raising both values in tandem

The rule: never tune one side without the other, and always verify with ss afterward.

1. Decide the target queue depth. Base it on burst absorption, not steady state. How many connections can arrive during the worst-case window where all workers are busy: a slow backend stall of a few seconds, a graceful restart, a TLS handshake burst? For most busy servers, 1024 to 4096 is a reasonable range. Note that kernels cap the listen() backlog at a hard upper bound regardless of somaxconn.

2. Set the kernel side first.

# Apply immediately
sysctl -w net.core.somaxconn=4096

# Persist across reboot
echo 'net.core.somaxconn = 4096' > /etc/sysctl.d/90-apache-backlog.conf
sysctl --system

3. Set the Apache side. Add or adjust the directive in the server config:

ListenBacklog 4096

Remember the constraint: a non-zero value must be at least 512 in httpd 2.4.

4. Restart Apache fully. This step is easy to get wrong. A graceful reload is not enough to apply a new backlog to existing listening sockets; httpd must be restarted after the sysctl change for the new backlog to take effect. A hard restart drops active connections, so do it during a low-traffic window or behind a load balancer with connection draining. Run apachectl configtest first, then restart.

5. Verify.

# Confirm the socket got what you configured
ss -ltn | grep -E ':80\s|:443\s'

Send-Q should now equal your ListenBacklog. If it shows the old value, either the restart did not happen or something re-applied a lower somaxconn (check for competing sysctl files and tuned profiles).

Containers and Kubernetes

Changing somaxconn on the host does not affect running containers. Each container gets its own network namespace, and the namespace initializes somaxconn from the kernel’s build-time constant (128 pre-5.4, 4096 since 5.4), not from the host’s current value. If Apache runs in a container, set the sysctl inside the container’s namespace:

# Docker: set the sysctl for the container's net namespace
docker run --sysctl net.core.somaxconn=4096 ...

On Kubernetes, use securityContext.sysctls on the pod. net.core.somaxconn is treated as an unsafe sysctl, so the kubelet must be started with --allowed-unsafe-sysctls=net.core.somaxconn before pods can set it. If you skip this, the pod sets ListenBacklog into a namespace capped at the kernel default and you are back to silent truncation.

Why 511 is too low on busy servers even when Recv-Q reads zero

Operators look at ss, see Recv-Q at zero, and conclude the backlog is fine. That confuses the snapshot with the risk. Recv-Q is a point-in-time sample of a queue that fluctuates in milliseconds; a zero reading only means the queue was empty at that instant. It says nothing about the burst you have not had yet.

The queue exists for the bad windows, not the good ones:

• Worker saturation events. When all workers are busy (a slow backend holding threads, a Slowloris wave, a genuine traffic spike), new connections stack in the accept queue. The queue is the only buffer; after it, connections are refused. On the worker pool’s cliff-edge degradation curve, queue depth is the difference between “degraded for a few seconds” and “load balancer pulled the node”.
• Graceful restarts. Old children drain while new ones spawn. Accept rate dips. A shallow queue turns a routine reload into refused connections.
• Cold start bursts. After a crash restart, SSL session caches are empty and full TLS handshakes spike CPU. Accept slows exactly when reconnecting clients hammer the listener.

A server doing thousands of connections per second with a 511-deep queue has sub-second burst absorption. The operational guidance: Recv-Q consistently zero but Send-Q still at the default 511 on a high-traffic server is a planning item, not an all-clear. Raise it before the incident, not after. The tradeoff is modest: a longer queue means saturated servers hold connections longer before refusing them, so clients wait instead of failing fast. Behind a load balancer with aggressive health checks, that is usually the right trade; for fail-fast architectures, keep the queue shorter deliberately, but know you chose it.

Signals to watch in production

Correlate queue depth with worker utilization before touching ListenBacklog. If Recv-Q grows while workers are saturated, more backlog buys seconds, but the real fix is worker capacity or backend latency. If Recv-Q grows while workers are idle, something else is blocking accept(), and a bigger queue will only hide it longer.

How Netdata helps

• Netdata collects Apache scoreboard and worker metrics from mod_status, so you can watch BusyWorkers and IdleWorkers trending toward saturation, the condition that starts filling the accept queue.
• TCP listener and connection state metrics from the host let you track queue behavior and connection states alongside Apache’s own view, on the same dashboard and timeline.
• AH00484 MaxRequestWorkers events and 5xx rates can be correlated against queue growth to distinguish “backlog too small” from “workers exhausted”.
• Anomaly detection on worker utilization surfaces the slow drift toward saturation that makes a 511-deep queue insufficient long before the first refused connection.
• Per-second granularity catches the brief saturation windows, graceful restarts and burst arrivals, that minute-resolution polling misses entirely.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion
• Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status
• Apache AH00484: server reached MaxRequestWorkers setting - worker pool exhausted
• Apache MaxRequestWorkers tuning: sizing the worker pool against memory
• Apache HTTPD monitoring checklist: the signals every production web server needs
• How Apache HTTPD actually works in production: a mental model for operators