PHP-FPM listen queue growing: the earliest signal of saturation

The listen queue is the kernel-managed socket backlog between the web server and PHP-FPM workers. When it carries a sustained non-zero depth, requests are arriving faster than workers can drain them. By the time the web server logs a 502, the queue has already overflowed and the kernel has started dropping connections.

The signal is easy to miss for two reasons. First, the FPM status page field is a point-in-time snapshot: a 10-second poller can report 0 even while bursts spill into the queue and drain between polls. Second, on Unix domain sockets (the most common production transport), the status page always reports 0 for all three queue fields due to a long-standing PHP bug. You have to read the kernel’s Recv-Q via ss, or poll at 1-second intervals, to see the real depth.

What this means

The listen queue (listen queue on the FPM status page, Recv-Q on the LISTEN line of ss) is the number of FastCGI connections parked in the kernel socket backlog, each waiting for an idle worker. Every connection in the queue is a real user request that has already been forwarded to PHP-FPM and is on hold.

A sustained non-zero value means the pool cannot keep up with current demand. Each queued request accumulates latency equal to its queue wait on top of normal processing time. When the queue depth reaches listen.backlog, the kernel stops accepting new connections and silently drops them. From the web server’s perspective that looks like ECONNREFUSED or a connect timeout, surfacing to users as 502 Bad Gateway.

By the time you see 502s in the web server logs, the queue has already overflowed. The queue depth itself is the early warning, often minutes ahead of actual failures.

flowchart LR
  A[Web server receives request] --> B[Connects to FPM socket]
  B --> C{Idle worker available?}
  C -- yes --> D[Worker handles request]
  C -- no --> E[Connection enters listen queue]
  E --> F{Queue full at listen.backlog?}
  F -- yes --> G[Kernel drops connection]
  G --> H[Web server logs 502]
  F -- no --> I[Waits for worker]
  I --> D

Common causes

Quick checks

All commands below are read-only and safe on a production host.

# FPM status page queue fields (adjust path to match pm.status_path)
curl -s http://127.0.0.1/fpm-status | grep "^listen queue\|^max listen queue\|^listen queue len\|^active processes\|^idle processes\|^max children reached"

# Kernel-level queue depth on the FPM listening socket (Unix socket)
# Recv-Q = current backlog depth, Send-Q = configured backlog
ss -lxn | grep php

# TCP transport variant (adjust port)
ss -tlnp | grep 9000

# Kernel counters for connections dropped because the backlog was full
nstat -az | grep -iE "ListenOverflows|ListenDrops"

# Effective somaxconn (caps listen.backlog)
cat /proc/sys/net/core/somaxconn

# Pool config: confirm pm mode, max_children, listen.backlog
# Path varies by distribution; this covers Debian/Ubuntu
grep -E "^(pm\.|listen\.)" /etc/php/*/fpm/pool.d/*.conf

# Full per-worker view: which scripts are running and for how long
curl -s 'http://127.0.0.1/fpm-status?full' | grep -E "request duration|request URI|state"

On Unix domain sockets, ss is the most reliable signal because the status page reports 0 for queue depth (see diagnosis step 2 below).

How to diagnose it

1. Confirm the queue is actually growing, not a single noisy sample. Poll the status page or ss at 1-second intervals for at least 30 seconds. A single non-zero reading may be a transient burst that drains on its own. A sustained or climbing value is real saturation.

2. Verify the signal source. If PHP-FPM listens on a Unix domain socket, the status page fields listen queue, max listen queue, and listen queue len always report 0 due to a PHP bug in the status code (it uses tcp_info, which only works for INET sockets). Use ss -lxn | grep php and read Recv-Q instead. For TCP, both the status page and ss work.

3. Check whether workers are all busy. If active processes equals pm.max_children and idle processes is 0, the pool is the bottleneck. If active is low but the queue grows anyway, the master is not dispatching connections to workers, which usually points to socket permission issues or a master process problem.

4. Distinguish demand from slow requests. If accepted conn rate has spiked, the cause is demand. If accepted conn rate is normal but active workers are pinned high, requests are taking longer than usual and tying up workers.

5. Read the slow log. If request_slowlog_timeout is set, the slow log shows where workers are blocked: PDO calls, curl_exec, session_start, file I/O. If the slow log is empty, the directive is probably 0 (disabled), which is the most common monitoring gap.

6. Check kernel drop counters. nstat -az | grep -i listen. Non-zero ListenOverflows means connections have already been dropped at the backlog. Users have already seen failures.

7. Inspect per-worker request URIs. The ?full status output shows what each active worker is executing. A single endpoint dominating the active set points to a slow or hot path in the application.

Metrics and signals to monitor

Fixes

Either give the pool more capacity (more workers, or faster workers), or reduce the demand arriving at it.

Worker pool exhaustion

If active processes is at pm.max_children and the queue is growing, the pool is undersized.

• Raise pm.max_children if memory headroom exists. Estimate the safe ceiling first: (available_RAM - OS_overhead) / avg_worker_PSS. Use PSS, not RSS, because forked workers share opcache pages. Raising pm.max_children without checking memory leads to OOM.
• Reduce per-request duration if workers are I/O-bound. The slow log tells you where they are stuck.
• Consider static mode if bursts outrun the dynamic scaler. Dynamic PM forks workers based on spare-server thresholds, which introduces fork latency under sudden bursts. Static mode keeps all workers resident at the cost of constant memory.

Apply changes with systemctl reload php-fpm (SIGUSR2 to the master). Note that the service name varies by distribution (php-fpm, php8.2-fpm, etc.).

Slow dependency contagion

The most common trigger. Workers that normally complete in 50ms now block for seconds on a database, external API, or NFS mount. They hold their slot the entire time, so a few slow requests can drain the whole pool.

• Fix the dependency. The slow log stack trace shows which call is blocked. Check the database slow query log, external API latency, or NFS stats.
• Set request_terminate_timeout (for example, 30s) so a single stuck request cannot hold a worker indefinitely.
• Add circuit breakers or shorter timeouts in application code. Default PHP cURL timeouts are often very high.
• Do not just raise pm.max_children. New workers hit the same slow dependency and also block. The queue drains briefly, then refills.

Traffic spike

If accepted conn rate has spiked well above baseline and the queue grows in step, demand has outstripped capacity.

• Enable rate limiting at the web server (nginx limit_req, Apache mod_qos) to protect FPM from burst traffic it cannot absorb.
• Scale horizontally if this is sustained growth rather than a transient burst.
• Pre-warm opcache after any restart so cold-start compilation does not compound the spike.

Backlog too small

If Recv-Q is hitting Send-Q on ss, or max listen queue approaches listen queue len, connections are being dropped at the kernel.

• Raise listen.backlog in the pool config. The effective backlog is capped by net.core.somaxconn.

• Check net.core.somaxconn because it caps the effective backlog: cat /proc/sys/net/core/somaxconn. Raise it if your configured listen.backlog exceeds the current value.
• Restart, do not reload. Changing listen.backlog requires a full systemctl restart php-fpm because the listening socket must be recreated. SIGUSR2 reload will not pick up the new value.

A larger backlog buys time but does not fix the underlying capacity gap. It is a buffer, not a cure.

Prevention

• Poll at 1-second intervals. The status page is a point-in-time snapshot. A 10-second poller will miss transient queue build-ups entirely.
• Monitor the kernel socket, not just the status page. On Unix domain sockets, the status page queue fields are always 0. Read ss Recv-Q for the actual depth.
• Track max listen queue between restarts. It is a high-water mark that reveals past saturation even when current depth is 0. It resets on restart, so frequent deploys erase this history.
• Alert on any sustained non-zero queue depth during normal traffic hours, not just when it approaches listen.backlog. By then, connections are already being dropped.
• Track kernel ListenOverflows and ListenDrops as the definitive signal that connections have been refused at the kernel.
• Keep 20-30% of workers idle during peak. Below that, one slow dependency or one burst pushes you into the queue.
• Restrict the status page to localhost or trusted IPs. The full status output exposes request URIs and script paths.

How Netdata helps

• Per-second collection of listen queue, max listen queue, active processes, idle processes, and max children reached from the FPM status page, so transient queue build-ups are visible rather than averaged away by a 10-second poller.
• Kernel socket metrics from ss and /proc/net/netstat (ListenOverflows, ListenDrops) alongside the FPM status fields, which is the only reliable way to see drops on a Unix domain socket where the status page reports 0.
• Anomaly detection on queue depth and worker utilization, surfacing a slow climb before it crosses a static threshold.
• Correlation across the request path: FPM queue depth against web server 502/504 rates, accepted conn rate, slow request counter, and per-worker request duration. A queue growing while accepted conn rate holds steady tells a different story than a queue growing alongside a traffic spike.
• Per-pool dashboards for multi-pool deployments, because a single saturated pool behind a round-robin load balancer means every Nth request is slow.

Related guides

• How PHP-FPM actually works in production: a mental model for operators
• PHP-FPM monitoring checklist: the signals every production pool needs
• PHP-FPM monitoring maturity model: from survival to expert
• PHP-FPM worker exhaustion: all workers busy and requests piling into the backlog