PHP-FPM worker exhaustion: all workers busy and requests piling into the backlog

PHP-FPM worker exhaustion is the most common PHP-FPM failure mode in production. The signature: active processes equals pm.max_children, the listen queue is growing, and the web server starts returning 502 and 504 responses. In dynamic or ondemand mode the max children reached counter is also incrementing.

The counterintuitive part: CPU is often LOW. Workers spend most of their time waiting on I/O, not computing. When all workers block on a slow database query or an unresponsive external API, CPU drops while requests pile up. If you are judging saturation by CPU alone, you will miss the problem.

The correct first response is not to restart or blindly raise pm.max_children. Read the slow log, determine whether a slow dependency is draining your workers, and then decide whether adding workers (if memory allows) helps or simply creates more stuck workers.

What this means

PHP-FPM uses a master-worker architecture. Each worker handles exactly one request at a time. There is no in-process concurrency. Maximum concurrent request capacity equals the number of active worker processes.

When all workers are busy, new FastCGI connections from the web server enter the socket backlog queue (kernel-managed, sized by listen.backlog). Workers accept from that listening socket themselves; if none is available, a queued connection remains there until a worker finishes and calls accept() again. If the backlog fills, the kernel refuses new connections at the OS level. The web server sees connection refused or timeout errors and returns 502 (connection failure) or 504 (gateway timeout) to users.

flowchart TD
    A[All workers busy] --> B[New requests enter socket backlog]
    B --> C[max children reached increments]
    B --> D{Backlog fills?}
    D -- No --> E[Requests wait, latency rises]
    D -- Yes --> F[Kernel refuses new connections]
    F --> G[502 and 504 errors reach users]
    E --> H{Workers drain backlog?}
    H -- Yes --> I[System recovers]
    H -- No --> F

A critical detail: PHP-FPM has no visibility into connections the kernel drops when the backlog overflows. The status page shows listen queue at its configured maximum but cannot tell you how many connections were refused. You need kernel-level monitoring (the ListenOverflows and ListenDrops counters in /proc/net/netstat, or the Recv-Q column in ss) to see those drops.

One more trap: if the listen queue appears empty while all workers are busy and users are seeing 502s, the backlog may have already overflowed. When the kernel backlog is completely full, new connections are rejected without entering the queue. The queue shows zero depth while connections are silently dropped.

Common causes

Quick checks

Run these read-only checks to confirm the exhaustion pattern and narrow the cause:

# FPM status: look for active = max_children, listen queue > 0
curl -s http://127.0.0.1/fpm-status

# Per-worker details: which scripts are running and for how long
curl -s 'http://127.0.0.1/fpm-status?full'

# Listen queue depth from the kernel (Recv-Q on the LISTEN line)
ss -xlnp | grep php
ss -tlnp | grep 9000

# Recent slow log entries (requires request_slowlog_timeout to be configured)
tail -50 /var/log/php-fpm/slow.log

# Worker process count and memory (excludes master process)
ps -eo pid,rss,cmd | grep '[p]hp-fpm' | grep -v master

# OOM kills targeting PHP-FPM workers
dmesg | grep php-fpm

# Web server upstream errors and 502/504 response counts
grep -c "connect.*failed\|no live upstreams\|Connection refused" /var/log/nginx/error.log
grep -c " 502 \| 504 " /var/log/nginx/access.log

# Kernel-level socket overflow counters (alternative: netstat -s | grep -i listen)
nstat | grep -i listen

The status path is configurable via pm.status_path and must be passed through by your web server. By default, the status path is handled by a worker from the pool, so total exhaustion can make it unreachable. Since PHP 8.0, pm.status_listen configures a separate status listener that operates independently of pool workers.

How to diagnose it

1. Confirm the exhaustion pattern. Check the status page: active processes equals pm.max_children, listen queue is positive, and max children reached is non-zero (in dynamic or ondemand mode). If the status page is unreachable, check the web server error log for upstream connection failures and use ss to check backlog depth.

2. Read the slow log. This is the single most important step. If request_slowlog_timeout is set (it defaults to 0, meaning disabled), the slow log captures stack traces showing exactly where each slow worker was blocked. Database functions (PDO, mysqli), cURL functions, or session_start in the trace tell you the blocking source. If the slow log is empty because the timeout was never configured, set it now. You are diagnosing blind without it.

3. Check per-worker request URIs and durations. The full status page (?full) shows each worker’s current request URI, script path, request duration in microseconds, and last request CPU and memory. Look for workers with durations orders of magnitude above the median. The script field identifies the specific endpoint. Note: request duration is in microseconds, not milliseconds. 1,000,000 equals 1 second.

4. Determine: traffic overload or slow dependency drain?

   Signal	Traffic overload	Slow dependency drain
   CPU usage	High (workers computing)	Low (workers blocked on I/O)
   Slow log	May not show specific blocking	Fills before listen queue grows
   Request duration distribution	Uniformly elevated	Bimodal: some fast, some extremely slow
   Killing blocked workers	No lasting effect	Immediate but temporary relief
   Adding more workers	Helps (more capacity)	New workers also get stuck
   Accepted connections rate	Spikes or stays high	May plateau or drop

5. Check memory before considering more workers. Run the RSS check above. The formula for memory-safe max_children is: (total_RAM * 0.7 - OS_overhead) / avg_worker_RSS. If you are already near that ceiling, adding workers will trigger the OOM killer rather than improving throughput.

Metrics and signals to monitor

Fixes

Slow dependency drain

This is the most common root cause. Do not raise pm.max_children first. The new workers will also get stuck on the same dependency.

1. Read the slow log to identify the blocking call (database query, external API, NFS, DNS, session lock).
2. If a specific endpoint is the culprit, block it at the web server level to protect the rest of the site.
3. Send SIGQUIT to blocked workers for immediate relief (kill -SIGQUIT <worker_pid>). PHP-FPM installs that signal in workers, but it closes the listen socket and starts shutdown; a worker blocked in an uninterruptible syscall may not exit until the syscall returns. The in-flight request is abandoned and the master respawns a replacement.
4. Address the root cause in the dependency: fix the slow query, reduce the external API timeout, resolve the DNS issue, or fix the NFS mount.
5. If the dependency cannot be fixed quickly, add circuit breakers or fail-fast timeouts in application code so workers are not held hostage indefinitely.

Traffic spike exceeding capacity

1. Enable rate limiting at the web server level to protect PHP-FPM from burst traffic.
2. If memory allows, increase pm.max_children. Calculate the memory-safe ceiling first using the formula above.
3. Reload with kill -SIGUSR2 <master_pid> or systemctl reload php-fpm. Note: the graceful reload drains old workers before spawning new ones, unlike nginx which starts new workers first. There is a brief window with zero workers available during the reload.

pm.max_children set too low

The upstream PHP-FPM code falls back to pm.max_children = 2 when pm = dynamic and the value is not set; the official Docker PHP image commonly ships 5. This is far too low for production. If you run a default image or package without overriding it, a small burst can saturate the pool.

1. Calculate the memory-safe ceiling: (total_RAM * 0.7 - OS_overhead) / avg_worker_RSS.
2. Set pm.max_children to that value in the pool configuration.
3. Use PSS (Proportional Set Size from smem or /proc/<pid>/smaps_rollup) for the calculation instead of RSS. Shared opcache pages inflate RSS by 30 to 50 percent in naive ps output, leading to an unnecessarily low max_children.

Session lock contention

File-based PHP sessions use an exclusive lock (flock(LOCK_EX)) acquired at session_start(). Concurrent requests from the same session ID serialize completely. This looks like worker exhaustion but is actually lock contention on a single session file.

1. Check the slow log for blocking at session_start().
2. Call session_write_close() early in the request, after session data is no longer needed.
3. For AJAX-heavy applications, switch to Redis or Memcached session handlers, which use different locking semantics.

Prevention

• Enable the slow log in every production pool. Without request_slowlog_timeout configured, you see busy workers but cannot identify the blocking call.
• Set pm.max_requests to 500-1000. This forces periodic worker recycling and is the primary defense against unbounded memory growth from leaks in code or extensions.
• Set request_terminate_timeout to 30-60 seconds. This prevents a single stuck request from permanently removing a worker from the pool.
• Put the status page on a separate listener with pm.status_listen, available since PHP 8.0. This keeps it reachable when all pool workers are busy.
• Monitor the listen queue, not just 502 rates. The listen queue provides minutes of advance warning. By the time 502s appear, the backlog has already overflowed.
• Calculate max_children from memory, not CPU cores. Workers spend most of their time on I/O. A 4-core machine can run 50-200 workers if memory allows.
• Verify listen.backlog against net.core.somaxconn. Setting listen.backlog higher than the kernel’s somaxconn has no effect. The kernel silently caps it.
• Poll status metrics at 1-second intervals. Saturation events unfold in seconds. A 10-second poll interval will miss transient queue buildups.

How Netdata helps

• Per-second polling of PHP-FPM status metrics (active processes, idle processes, listen queue, max children reached, slow requests) catches transient saturation events that 10-second intervals miss.
• Correlation across the request path: when active processes spike and listen queue grows, you can overlay web server 502/504 rates, database connection counts, and upstream dependency latency in a single view. The low-CPU, high-saturation pattern that characterizes a slow-dependency drain is immediately visible.
• ML anomaly detection on worker counts and listen queue depth surfaces the slow drift toward exhaustion before the backlog overflows.
• Per-worker memory tracking (RSS) alongside system-wide memory, so you can verify that adding workers will not trigger the OOM killer.
• Kernel-level socket monitoring (ListenOverflows, ListenDrops) catches connection drops that the PHP-FPM status page fundamentally cannot see.
• Composite alerting on conditions like “active equals max_children AND listen queue greater than 0 for more than 60 seconds” turns two correlated signals into a single actionable alert rather than two independent thresholds.

Related guides

• PHP-FPM “server reached pm.max_children setting (N), consider raising it”
• PHP-FPM active processes near max_children: reading pool utilization
• PHP-FPM idle processes at zero: no burst headroom left
• PHP-FPM listen queue growing: the earliest signal of saturation
• PHP-FPM sizing pm.max_children: by memory, not by CPU cores
• PHP-FPM monitoring checklist: the signals every production pool needs
• How PHP-FPM actually works in production: a mental model for operators
• PHP-FPM monitoring maturity model: from survival to expert
• PHP-FPM slow request cascade: one slow dependency drains the whole pool
• PHP-FPM 504 Gateway Timeout: requests accepted but never finishing in time
• PHP-FPM slow log: turning on request_slowlog_timeout to see what is slow
• PHP-FPM request duration climbing: spotting stuck and outlier workers