PHP-FPM slow request cascade: one slow dependency drains the whole pool

Your PHP-FPM pool is pinned at max_children. The listen queue is climbing. Users see 502 Bad Gateway or 504 Gateway Timeout. CPU on the box is flat and the application error log is quiet. The PHP code is fine; it is waiting on a slow dependency.

This is the slow request cascade. A database query regresses, an external API starts timing out, an NFS mount stalls, or DNS resolution hangs. A request that normally completes in 50ms now takes 5 to 30 seconds. Each slow request holds a worker for the full duration. The pool drains in seconds, the socket backlog fills, and the kernel starts refusing connections. This is the most common PHP-FPM failure mode in production, and the fix is almost never inside PHP-FPM itself.

The tell is low CPU with a saturated pool. Compute-bound work pins CPU. An I/O-bound stall pins workers while CPU sits idle. The slow log, when configured, points directly at the blocking call. Without it, you are correlating FPM worker counts with downstream logs after the fact.

What this means

PHP-FPM uses a process-based concurrency model. Each worker handles one request at a time, no in-process concurrency. Maximum concurrent request capacity equals the number of active worker processes. When all workers are occupied, the next request queues in the socket backlog. Once the backlog overflows, the kernel drops connections at the socket layer. PHP-FPM has no visibility into those drops.

A slow dependency turns this model into a bottleneck. Consider a pool of 50 workers averaging 50ms per request, serving 200 req/s. If a database query regresses to 5 seconds, each affected request occupies a worker 100 times longer. Effective capacity collapses to a handful of workers within seconds, even though inbound traffic has not changed.

flowchart TD
    A[Slow dependency
DB / API / NFS / DNS] --> B[Subset of requests
slow from 50ms to 5-30s]
    B --> C[Each slow request
holds a worker hostage]
    C --> D[Pool drains:
no idle workers]
    D --> E[New connections
queue in socket backlog]
    E --> F[Backlog fills
listen.backlog limit]
    F --> G[Kernel drops
new connections]
    G --> H[nginx 502 / 504
to users]

Raising max_children does not help if new workers will also block on the same dependency. Adding workers burns RAM and buys seconds at best. The fix has to be upstream: repair, isolate, or fail fast on the slow dependency.

Common causes

Quick checks

These are read-only. Run them before changing anything. The status path (/fpm-status) depends on your nginx and pm.status_path configuration; adjust accordingly.

# Confirm slow log is configured and capturing
php-fpm -tt 2>&1 | grep -E 'slowlog|request_slowlog_timeout'
tail -50 /var/log/php-fpm/slow.log

# Pool snapshot
curl -s http://127.0.0.1/fpm-status

# Active vs idle vs queue depth vs max-children-reached counter
curl -s http://127.0.0.1/fpm-status | grep -E 'active|idle|listen queue|max children|slow requests'

# Per-worker view - find stuck workers and their URIs (request duration is microseconds)
curl -s 'http://127.0.0.1/fpm-status?full' | grep -E 'pid|state|request duration|request uri|script'

# Kernel-level listen queue depth on the FPM socket (Recv-Q column)
ss -xlnp | grep php     # Unix socket
ss -tlnp | grep 9000    # TCP

# Kernel drop counters - confirm overflow when status page is unresponsive
nstat | grep -i listen

# CPU on FPM workers - low CPU confirms I/O-bound stall, not compute
ps -C php-fpm -o pid,pcpu,pmem,etime,args --sort=-pcpu | head -20

# Web server upstream errors
grep -c "connect.*failed\|Connection refused\|upstream timed out" /var/log/nginx/error.log

If the status page itself does not respond, the main pool is fully busy serving the status request behind real traffic. On PHP 8.0+ you can configure pm.status_listen on a separate socket so the status page is served from a side-pool even when the main pool is saturated. Without it, you are blind exactly when you need the data most.

How to diagnose it

1. Confirm the failure mode before touching anything. Three signals together confirm slow-dependency drain: active processes at or near max_children, listen queue non-zero or growing, and CPU usage lower than the stall would suggest.

2. Locate the slow log. If request_slowlog_timeout is 0 (the default in many distros), there is no slow log data and you must enable it before the next incident. In the moment, fall back to per-worker ?full status and the dependency’s own metrics.

3. Read the stack traces. The top frame is the function running when the timeout fired; earlier frames are below. A trace whose top frame is curl_exec, PDO::query, mysqli_query, pg_query, stream_socket_client, file_get_contents on an http(s) path, or session_start is the signature of a slow external dependency or a session lock. Multiple traces pointing at the same call confirm the source.

4. Cross-reference against per-worker status. Match long request duration values in the ?full output with the same scripts appearing in slow log entries. Note that request duration is in microseconds, not milliseconds. 1000000 is one second.

5. Verify against the dependency’s own metrics. If the slow log points at PDO, check the DB slow query log, lock waits, connection count. If curl_exec, check the upstream API latency from the app host directly. If NFS, check nfsstat and whether df hangs. The dependency should show corresponding stress. If it does not, suspect DNS, network partition, or a connection pool at the application layer.

6. Decide whether the stall is single-endpoint or systemic. Slow log entries concentrated on one script path point at a specific slow endpoint, often a missing index or an unbatched loop. Entries spread across many scripts point at systemic backend trouble: DB failover, DNS outage, NFS hang, network partition.

7. Check kernel overflow counters. If users see 502 but the status page shows listen queue = 0, you may be sampling between drops. nstat TcpExtListenOverflows and TcpExtListenDrops give the cumulative view of connections the kernel refused while you were polling.

Metrics and signals to monitor

Fixes

Immediate relief: clear blocked workers

If the dependency has recovered but workers are still draining stuck requests, free them by hand. This is relief, not a fix. If the dependency is still slow, the freed worker will be re-acquired by another slow request within seconds.

# Identify long-running workers from full status
curl -s 'http://127.0.0.1/fpm-status?full' | grep -B1 -A3 'request duration'

# Recycle a specific stuck worker
# WARNING: this targets a single worker process. Verify the PID belongs to
# an FPM worker before sending the signal.
# SIGQUIT to an individual worker PID gracefully terminates just that worker
# (community-documented; the master respawns it).
kill -SIGQUIT <pid>

Short-term: protect the pool

• Block the offending endpoint at the web server layer to protect the rest of the site. If the slow log shows all slow requests hitting one URI, deny that path in nginx and let the rest of the application breathe.
• Add or tighten timeouts in application code. PHP cURL has no default execution timeout. Set explicit CURLOPT_TIMEOUT and CURLOPT_CONNECTTIMEOUT so blocked calls fail fast instead of holding the worker for 30 seconds.
• Temporarily raise pm.max_children only if memory headroom exists and the dependency is recovering. New workers will block too if the root cause persists. Capacity check: avg_worker_RSS * new_max_children + OS_overhead must fit comfortably under available RAM. If you guess wrong, the OOM killer will make the outage worse.
• Enable request_terminate_timeout as a safety net (30-60s, application-dependent). This hard-kills any single request that exceeds the limit, preventing one stuck dependency call from holding a worker indefinitely.

Root cause: fix or isolate the dependency

• Database: add the missing index, fix the lock contention, scale the connection pool with pgBouncer or ProxySQL, tune the slow query.
• External API: negotiate tighter SLAs, add a circuit breaker in application code, cache responses where possible.
• NFS: switch to a local cache, fail fast on stale mounts, or move the workload off NFS.
• DNS: tighten resolver timeouts, run a local caching resolver, eliminate cross-region DNS lookups in the hot path.
• Session locks: call session_write_close() early, or move sessions to Redis or Memcached to change the locking semantics.

Prevention

• Enable request_slowlog_timeout on every production pool. 3 to 5 seconds is a reasonable starting point. It must be lower than request_terminate_timeout, or the worker is killed before a backtrace can be captured.
• Configure request_terminate_timeout (30-60s) to bound the worst case. Left at 0, a single hung request permanently removes a worker until pool restart.
• Configure pm.status_listen (PHP 8.0+) on a separate socket. The status page then remains reachable when the main pool is saturated, eliminating the monitoring blind spot during the cascade itself.
• Align nginx fastcgi_read_timeout and PHP-FPM request_terminate_timeout. If nginx times out first, it closes the upstream connection but the FPM worker keeps running until it finishes or request_terminate_timeout fires, producing phantom workers that occupy pool slots serving responses no client will read. Note that PHP max_execution_time measures CPU time on Unix, not wall clock, so it will not fire for requests blocked on I/O (database queries, cURL, NFS, DNS). For this failure mode, request_terminate_timeout is the timeout that actually protects the pool.
• Instrument dependencies with their own monitoring. The cascade becomes diagnosable in seconds when you can correlate FPM slow log bursts against DB latency, API response times, and NFS operation latency.
• In Docker, add --cap-add=SYS_PTRACE. PHP-FPM uses ptrace to capture worker backtraces for the slow log. Without the capability, the slow log file stays empty even though PHP-FPM logs “executing too slow, logging” warnings.
• Consider raising request_slowlog_trace_depth for applications with deep framework call stacks, so the trace reaches the blocking call rather than stopping inside framework bootstrap frames.

How Netdata helps

• Per-second collection of PHP-FPM status metrics catches the active-to-idle-to-queue transition that 10 or 30 second polling misses entirely. The cascade can unfold in seconds.
• ML anomaly detection flags the moment the active/idle ratio breaks pattern, before the listen queue starts growing.
• Correlation panels pair PHP-FPM saturation with co-conspirators on the same host: MySQL or PostgreSQL query latency, Redis operation timing, NFS RPC latency, network connection states, CPU steal and iowait. A spike in any of those at the moment workers pin is your root cause.
• The PHP-FPM slow request counter, per-worker request duration from ?full status, and kernel listen drop counters can be charted together to confirm the failure pattern rather than inferring it from a single signal.
• Anomaly advisors surface TcpExtListenOverflows and TcpExtListenDrops so you see kernel-level drops that the FPM status page cannot report.

Related guides

• PHP-FPM active processes near max_children: reading pool utilization
• How PHP-FPM actually works in production: a mental model for operators
• PHP-FPM idle processes at zero: no burst headroom left
• PHP-FPM “server reached pm.max_children setting (N), consider raising it”
• PHP-FPM monitoring checklist: the signals every production pool needs
• PHP-FPM monitoring maturity model: from survival to expert
• PHP-FPM worker exhaustion: all workers busy and requests piling into the backlog