PHP-FPM listen backlog overflow: the kernel silently dropping connections

The signature: a PHP-FPM pool where every status page field looks healthy yet the web server sporadically returns 502s. Active processes are not pinned at pm.max_children, the listen queue field reads zero, the slow log is quiet, opcache hit rate is normal. The 502s cluster into short bursts that may or may not line up with known load events.

The kernel is dropping incoming FastCGI connections because the listen backlog has filled. PHP-FPM has no visibility into these drops. Once the backlog is at capacity, the kernel refuses to enqueue new connections, and the status page cannot report a queue depth above the configured listen.backlog value because those connections never enter a queue FPM could observe. The failure happens entirely below PHP-FPM’s instrumentation layer.

The fix is not to enlarge the backlog. A deeper queue only delays drops; it does not make the pool drain faster. The real fixes are more workers, faster requests, or fewer arrivals.

What this means

PHP-FPM workers process one request at a time. When all workers are busy, the kernel parks incoming FastCGI connections in the socket backlog, sized by listen.backlog in the pool configuration. The master process eventually calls accept() and hands each connection to an idle worker.

The backlog’s effective size is min(listen.backlog, net.core.somaxconn). PHP’s built-in default is version- and platform-dependent: PHP < 8.2 uses 511 on Linux/macOS; PHP 8.2+ uses -1 on Linux, FreeBSD, OpenBSD, DragonFly BSD, and macOS, which Linux clamps to net.core.somaxconn. Other platforms use 511. net.core.somaxconn itself varies by distribution and configuration (older hosts commonly used 128, many modern hosts use 4096). Some hardened or cloud-optimized images ship net.core.somaxconn = 128, which silently caps even a large listen.backlog. Always confirm with sysctl net.core.somaxconn before reasoning about effective backlog depth.

Once the effective backlog fills, kernel behavior depends on socket type:

• TCP listener: the kernel increments TcpExtListenOverflows and TcpExtListenDrops in /proc/net/netstat and drops the new connection. Visible via nstat or netstat -s.
• Unix domain socket (the common case for nginx plus PHP-FPM on the same host): no SYN retry, no TCP counter increments. The kernel refuses the connect() attempt. nginx logs an upstream connect failure and returns 502. Linux does not expose a portable cumulative Unix-socket accept-queue drop counter in procfs; dynamic tracing can inspect connect and backlog state, but it is not a standard drop counter.

The PHP-FPM side sees nothing of this. The listen queue field in the status page is a snapshot of current backlog depth, and by the time you read it the burst may have passed, so it reads 0. The max listen queue high-water mark can show that the pool touched its limit, but not how many connections were rejected.

flowchart TD
    A[nginx opens FastCGI conn] --> B{Worker free?}
    B -- yes --> C[accept and dispatch]
    B -- no --> D{Backlog full?}
    D -- no --> E[enqueue in kernel]
    E --> C
    D -- yes --> F[kernel drops connection]
    F --> G[nginx logs 502]
    F -. invisible to .-> H[FPM status page]
    F --> I[TcpExtListenDrops on TCP; no counter on Unix sockets]

Common causes

The most insidious row is the fourth. With a 10-second polling interval, a 200-millisecond burst that fills and drains the backlog produces zero visible samples in the FPM status page but plenty of kernel counter increments and 502s in the nginx error log.

Quick checks

# TCP listener: kernel drop counters (cumulative since boot)
nstat -az | grep -E 'ListenDrops|ListenOverflows'

# Same counters in a friendlier form
netstat -s | grep -i listen

# Effective backlog vs current queue depth (TCP)
ss -tlnp | grep 9000

# Effective backlog vs current queue depth (Unix socket)
ss -xlnp | grep php

# Confirm sysctl cap
sysctl net.core.somaxconn

# PHP-FPM status page fields
curl -s http://127.0.0.1/fpm-status | grep -E '^(listen queue|max listen queue|listen queue len)'

Interpretation rules:

• On ss -tlnp and ss -xlnp LISTEN lines, Recv-Q is the current accept queue depth and Send-Q is the effective maximum backlog after somaxconn clamping. If Recv-Q approaches Send-Q, the kernel is about to start dropping. If Recv-Q equals Send-Q for any sustained period, drops are almost certainly happening.
• nstat values are cumulative since boot. To detect active drops, sample, wait a minute, sample again. Any delta on ListenDrops or ListenOverflows is a problem.
• The status page field listen queue len reflects what PHP-FPM asked for in listen.backlog, not what the kernel actually granted. The only source of truth for the effective cap is Send-Q on the ss LISTEN line.
• For Unix sockets there is no portable cumulative drop counter exposed in procfs. Dynamic tracing with bpftrace or perf can inspect connect and backlog state, but it is not a drop counter. Use ss -xlnp Recv-Q vs Send-Q as the real-time signal, and correlate with nginx error log timestamps.

How to diagnose it

1. Confirm kernel drops are happening. On a TCP listener, watch nstat -az | grep -E 'ListenDrops|ListenOverflows' over a short window. Any non-zero delta is conclusive. On a Unix socket, monitor Recv-Q via ss -xlnp and correlate with nginx error timestamps; there is no equivalent portable cumulative drop counter in procfs.
2. Cross-reference with nginx error log timestamps. Filter for upstream connect failures during the same window: grep -E "connect.*failed|Connection refused|no live upstreams" /var/log/nginx/error.log. The 502s should line up with periods when drops incremented.
3. Confirm workers are actually busy at the drop moments. Poll the FPM full status page at 1-second intervals during a burst and count workers in Running state. If running worker count equals pm.max_children at those moments, you have confirmed the precondition.
4. Check max listen queue historical peak. A non-zero value on a freshly restarted pool tells you the pool has already touched its backlog since the last restart. If max listen queue is close to listen queue len, you have been at the edge.
5. Inspect per-worker request URIs. A small number of slow endpoints holding workers hostage is the most common pattern. Pull curl -s 'http://127.0.0.1/fpm-status?full&json' and look for repeated URIs with high request duration. If a handful of endpoints account for most of the worker time, fixing those is faster than adding workers.
6. Verify the effective backlog matches intent. Compare sysctl net.core.somaxconn against the Send-Q column in ss -xlnp for the FPM socket. If they differ, the pool config is not doing what you think.

Metrics and signals to monitor

The first two rows are the only ones that prove the failure. Everything else is a leading indicator or corroborating signal.

Fixes

Add workers

If memory headroom permits, raise pm.max_children. The capacity formula is max_children = (available_memory * 0.7 - OS_overhead) / avg_worker_PSS, using PSS rather than RSS to avoid overstating memory by 30-50% on shared opcache pages. More workers means faster accept() drain, which means the backlog never fills.

This is the right fix when per-worker request duration is healthy (matches baseline p95) and the problem is purely traffic volume exceeding concurrency.

Make requests faster

If workers are busy because a small set of endpoints is slow, fix those endpoints first. Adding workers to compensate for slow database queries or external API timeouts only delays the next saturation event. Inspect the slow log and per-worker request URIs to find the offenders, then optimize the code, add timeouts, or circuit-break against the slow backend.

This is the right fix when slow log entries cluster on specific endpoints, or when active processes are high but CPU is low (workers are I/O-blocked, not computing).

Raise net.core.somaxconn only if it is the real cap

If sysctl net.core.somaxconn is below your configured listen.backlog, the kernel is silently clamping the queue. Raising somaxconn is reasonable as a one-time system tuning step. But raising it without addressing the underlying saturation only buys more queue capacity before drops start. It does not increase request throughput.

Do not just raise listen.backlog

Increasing listen.backlog is the most tempting and least effective fix. A deeper queue means longer waits for queued requests, which means higher latency for the requests that do get served. It also means larger bursts of work arriving at the workers once the queue drains, which can deepen the original saturation. The default listen.backlog (511 on PHP < 8.2, -1 on PHP 8.2+) is sufficient for almost all workloads once the pool is correctly sized.

Prevention

• Kernel drop counters, continuously. TcpExtListenDrops on TCP listeners is the only signal that catches this failure mode before users report 502s. For Unix sockets, monitor Recv-Q / Send-Q ratio instead. A 1-second polling interval is appropriate; 10 seconds will miss transient bursts.
• Recv-Q relative to Send-Q on the FPM listening socket. A trendline of Recv-Q / Send-Q gives early warning before drops begin.
• Status page and worker count in the same view. A non-zero listen queue while active processes is below max_children suggests a worker spawn or socket issue rather than pure saturation.
• Set request_slowlog_timeout in production. Without slow log traces, you cannot tell the difference between “lots of traffic” and “a few slow endpoints eating the pool.”
• Set request_terminate_timeout to a finite value. A stuck request permanently removes a worker, silently reducing capacity and pushing the pool toward backlog overflow.
• Set pm.max_requests to 500-1000. Memory leaks reduce effective worker count over time, narrowing the gap between normal load and saturation.
• Coordinate timeout hierarchies. If nginx fastcgi_read_timeout is longer than FPM request_terminate_timeout, FPM kills the worker while nginx keeps waiting; if shorter, the worker keeps running for a response nobody will read.
• Verify reload behavior. SIGUSR2 produces a brief no-worker window during which the backlog can fill quickly. If logrotate fires SIGUSR2 at midnight and 502 bursts cluster around that time, see PHP-FPM graceful reload: the brief no-worker window on SIGUSR2.

How Netdata helps

Netdata collects the signals that prove or rule out backlog overflow at per-second resolution, which matters because the failure mode is often sub-10-seconds and invisible to slower pollers.

• TCP listener drops and overflows (TcpExtListenDrops, TcpExtListenOverflows) are collected as first-class metrics from /proc/net/netstat, so you can alert on rate-of-change rather than eyeballing cumulative counters.
• Recv-Q and Send-Q on listening sockets give you the effective backlog vs current queue depth ratio without a custom collector. This works for both TCP and Unix socket listeners.
• PHP-FPM status page fields (active processes, listen queue, max listen queue, max children reached) appear in the same dashboard as the kernel counters, so you can correlate the inside-pool view with the outside-pool drop view.
• Composite alerts combining “active processes near max_children” with “kernel ListenDrops incrementing” give you continuous confirmation of the diagnostic steps above, without manual nstat polling.

Related guides

• PHP-FPM 502 Bad Gateway: the web server cannot reach the pool
• PHP-FPM 504 Gateway Timeout: requests accepted but never finishing in time
• PHP-FPM active processes near max_children: reading pool utilization
• PHP-FPM in containers: cgroup limits and the silent OOM kill
• PHP-FPM “child N exited on signal 11 (SIGSEGV)”: worker segfaults
• PHP-FPM crash loop and fork storm: workers dying faster than they serve
• PHP-FPM dynamic mode scaling lag: why the pool cannot keep up with bursts
• PHP-FPM emergency restart: “failed processes threshold reached, initiating reload”
• PHP-FPM graceful reload: the brief no-worker window on SIGUSR2
• How PHP-FPM actually works in production: a mental model for operators
• PHP-FPM idle processes at zero: no burst headroom left
• PHP-FPM listen queue growing: the earliest signal of saturation