$ guides / php-fpm / php-fpm-workers-blocked-on-io ▌

Operations Guides

PHP-FPM workers busy at low CPU: blocked on the database, API, or filesystem

PHP-FPM active processes is climbing toward pm.max_children, the listen queue is building, users are seeing slow responses or intermittent 502s, and yet system CPU is flat at 10 to 20 percent. The pool looks saturated in terms of worker slots, but the CPU headroom suggests the box is barely working.

Each PHP-FPM worker handles exactly one request at a time. When a worker blocks on I/O (a database query, an external API response, a DNS lookup, or a filesystem operation), it holds its worker slot for the entire duration of that wait while consuming essentially zero CPU. It is “active” from FPM’s scoreboard perspective but idle from the kernel scheduler’s perspective.

Raising pm.max_children in response rarely helps and often makes things worse: new workers will also block on the same dependency, multiplying the downstream connection load without improving throughput. The fix is to identify what the workers are waiting on and address that dependency, or add a timeout that lets workers fail fast.

What this means

The diagnostic signature is the divergence between active processes and system CPU utilization. When active processes approaches pm.max_children and CPU stays low, the workers are not computing. They are parked in the kernel’s I/O wait state, holding slots that no amount of CPU can free.

This contrasts with the CPU-bound case, where high active processes correlates with high CPU because workers are genuinely executing PHP. In the I/O-bound case, the PHP runtime has issued a blocking syscall (a recvfrom on a database socket, a curl_easy_perform on an HTTP connection, a read on an NFS mount) and the process is descheduled until data arrives or the call times out.

The downstream consequence is the same as any worker exhaustion: as blocked workers accumulate, fewer slots remain for normal traffic. The listen queue starts filling. Once it overflows, the kernel drops connections and the web server returns 502. The PHP application code is often completely healthy. It is patiently waiting for a dependency that is slow, unreachable, or contended.

The key field on the status page that confirms this is last request cpu, available in full mode (?full). For workers in Idle state, this shows the CPU percentage consumed by their last completed request. A value near zero for a request whose request duration was several seconds is definitive proof the worker spent that time blocked on I/O, not executing PHP. Note that for workers still in Running state, last request cpu is always zero because CPU accounting only completes when the request finishes.

flowchart TD
    A[Active processes climbing] --> B{System CPU low?}
    B -- Yes --> C[I/O-bound worker drain confirmed]
    B -- No --> D[CPU-bound: check opcache, compute paths]
    C --> E[Read slow log for stack traces]
    E --> F{Top of stack shows}
    F -- PDO / mysqli --> G[Database lock or slow query]
    F -- curl / stream --> H[External API or DNS latency]
    F -- file I/O functions --> I[NFS stall or disk saturation]
    F -- session_start --> J[Session lock contention]

Common causes

Cause	What it looks like	First thing to check
Database lock contention or slow query	Slow log stack traces show PDO or mysqli functions; database shows lock waits or long-running queries	Slow query log; `pg_locks` / `information_schema.INNODB_TRX`
Database connection limit reached	Workers block acquiring a connection; DB shows connection count at `max_connections`	`SHOW PROCESSLIST` or `pg_stat_activity` vs DB max connections
External API or DNS latency	Slow log shows curl or stream functions at top of stack	Call the API endpoint directly and measure latency
NFS or shared filesystem stall	Workers enter D state (uninterruptible sleep); `dmesg` shows `rpc_wait_bit_killable`	`nfsstat` retransmits, `mountstats`, `dmesg`
Session lock contention	Slow log shows blocking at `session_start`; affects concurrent requests from same session ID	Check `session.save_handler` and concurrent same-session requests
cgroup I/O throttling	Workers block on file I/O despite healthy storage; cgroup IO limits set	Check systemd slice `IOWeight` or cgroup v2 I/O limits

Quick checks

# Check active processes vs max_children ratio
curl -s http://127.0.0.1/fpm-status | grep -E "active processes|max children|idle processes"

# Check for I/O-bound workers: low last request cpu with long request duration
curl -s 'http://127.0.0.1/fpm-status?full' | grep -E "request duration|last request cpu|state"

# Check slow request counter rate (run twice, 10 seconds apart, compare values)
curl -s http://127.0.0.1/fpm-status | grep "slow requests"

# Read recent slow log entries (path varies by distro and pool config)
tail -100 /var/log/php-fpm/slow.log

# Check if slow log is even configured
php-fpm -tt 2>&1 | grep -E "slowlog|request_slowlog_timeout"

# Check for workers in D state (uninterruptible sleep, typical of NFS stalls)
ps -eo pid,stat,wchan,cmd | grep '[p]hp-fpm' | grep -v master | awk '$2 ~ /^D/'

# Check database connection count from this host
# PostgreSQL:
psql -c "SELECT count(*) FROM pg_stat_activity WHERE client_addr = '<this_host>';"
# MySQL:
mysql -e "SELECT COUNT(*) FROM information_schema.processlist WHERE HOST LIKE '<this_host>%';"

# Check kernel-level listen queue overflow counter
nstat | grep -i ListenOverflows

# Check what outbound connections FPM workers are holding
ss -tnp | grep php-fpm | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head

How to diagnose it

Confirm the divergence. Pull the status page and compare active processes against your system CPU metric. If active is above 80 percent of pm.max_children and CPU is below 30 percent, you are in I/O-bound territory. Pull ?full and check workers in Running state with high request duration values.
Verify the slow log is configured. request_slowlog_timeout defaults to 0, which disables it entirely. If php-fpm -tt shows it unset or 0, you are flying blind. Set it to 5 seconds (or whatever threshold fits your baseline latency) and reload. The slow log is the single most important diagnostic tool for this pattern. When a worker exceeds the threshold, FPM sends SIGSTOP to the process, captures a backtrace via ptrace, then sends SIGCONT. The resulting stack trace shows exactly where the worker was parked.
Read the slow log stack traces. Group entries by the top of the call stack. If you see PDO methods (PDO::query, PDOStatement::execute) or mysqli calls at the top, the database is the bottleneck. If you see curl_exec or stream wrappers, an external API is slow. If you see fopen, fread, or session_start, the filesystem or session locking is the issue. If you see rpc_ functions, NFS is involved.
Check the dependency directly. Once the slow log identifies the category, measure the dependency in isolation. Run the slow query against the database with EXPLAIN ANALYZE. Curl the external API endpoint from the FPM host. Check nfsstat or mountstats for NFS retransmits and timeouts. The dependency’s own metrics will confirm the diagnosis.
Check the connection multiplier. Count how many database connections originate from this FPM host. Each FPM worker that opens a database connection during request processing holds one connection. N active workers means up to N concurrent database connections from this host alone. If you have multiple app servers, multiply accordingly. Compare against the database’s max_connections or connection pool size.
Check for D-state workers. If ps shows workers in state D (uninterruptible sleep), they are stuck in a kernel I/O syscall, typically NFS. These workers cannot be killed with SIGTERM. The only recovery is fixing the NFS server or rebooting. dmesg will often show the WCHAN as rpc_wait_bit_killable.
Check for self-request deadlocks. If a PHP worker makes an HTTP request back to the same application (e.g., calling its own API endpoint), it consumes a second worker slot. Under load, workers end up waiting for each other. Use ss -tnp | grep php-fpm to check whether FPM workers are connecting to the application’s own gateway address.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`active processes` / `max_children` ratio	Primary saturation indicator	Sustained above 0.8 with low CPU
System CPU utilization vs active processes	Divergence reveals I/O-bound pattern	Active climbing, CPU flat
`slow requests` counter (rate of change)	Quantifies how many requests exceed slowlog threshold	Sustained non-zero rate
Per-worker `request duration` and `last request cpu`	Identifies individual blocked workers	Long duration with near-zero CPU
`listen queue` depth	Leading indicator before connection drops	Any sustained non-zero value
Database connection count from FPM host	Validates the N-workers-equals-N-connections math	Approaching DB max_connections
Kernel `ListenOverflows` counter	Detects drops invisible to FPM	Any increment
Worker state distribution (D vs R vs S)	D-state workers indicate kernel I/O stall	Any D-state workers

Fixes

If the database is the bottleneck

The slow log will show PDO or mysqli at the top of the stack. Check for lock contention, missing indexes causing full table scans, or queries that regressed after a schema change. If the issue is connection exhaustion rather than query slowness, the fix is a connection pooler. At scale, N FPM workers each opening a connection means N connections from this host. Put PgBouncer (PostgreSQL) or ProxySQL (MySQL) between FPM and the database. This lets hundreds of PHP workers share a smaller pool of database connections.

Persistent connections (PDO::ATTR_PERSISTENT) are a simpler mitigation that caps database connections at pm.max_children, since each worker reuses one persistent connection. This is adequate for single-host deployments but does not multiplex the way an external pooler does.

If an external API is slow

The slow log will show curl_exec or stream functions. PHP’s libcurl has no default timeout, so a worker calling a dead or slow API will wait indefinitely. Set explicit CURLOPT_TIMEOUT and CURLOPT_CONNECTTIMEOUT in application code. A typical safe pattern is 2 seconds for connect, 5 to 10 seconds for total request time. Add circuit breaker logic so repeated failures short-circuit instead of queuing more workers behind the same dead dependency.

If NFS or the filesystem is stalled

Workers accessing files on NFS will enter D state when the NFS server stalls. They cannot be killed or recycled until the syscall completes. The kernel call stack will show rpc_wait_bit_killable. The only fix is to restore the NFS server. To prevent recurrence, consider whether the application truly needs NFS for the code paths that are blocking, or whether those files can be served from local storage. Check cgroup I/O limits if running under systemd slices with IOWeight or IODeviceWeight, as these can throttle filesystem access even when underlying storage is healthy.

If session lock contention is the cause

The slow log will show blocking at session_start. With file-based sessions, concurrent requests from the same session ID serialize via flock(LOCK_EX) on the session file. AJAX-heavy pages firing multiple parallel requests for the same user will all queue behind one lock. The fix is to call session_write_close() as early as possible in the request lifecycle, after the last session write. For a structural fix, switch to Redis or Memcached session handlers, which have different locking semantics.

If the dependency cannot be fixed immediately

Set or lower request_terminate_timeout so stuck workers are killed after a hard limit rather than holding their slot forever. This trades a 502 response for the stuck request (better than a hung connection) and frees the worker slot for subsequent traffic. The timeout should be coherent with the web server’s fastcgi_read_timeout to avoid phantom workers that continue executing after nginx has already returned a 504 to the user.

You can also selectively kill blocked workers to free capacity temporarily:

# Gracefully terminate a specific blocked worker (it finishes current request first)
kill -SIGQUIT <worker_pid>

Do not kill D-state workers. SIGQUIT will not interrupt a kernel I/O wait.

Prevention

Enable the slow log on every production pool. Set request_slowlog_timeout to a value appropriate for your baseline latency (5 seconds is a reasonable starting point). This is the single highest-value configuration change for diagnosing I/O-bound workers.
Set explicit timeouts on all outbound calls. Database queries, HTTP API calls, and cache connections should all have fail-fast timeouts. Default PHP cURL timeout is effectively infinite.
Set request_terminate_timeout. A hard ceiling prevents stuck workers from permanently consuming slots. 30 to 60 seconds is typical, adjusted for your application’s legitimate long-running endpoints.
Install a connection pooler at scale. If your FPM worker count times the number of app servers exceeds the database’s max_connections, you need PgBouncer or ProxySQL. Do not rely on each worker managing its own connection.
Monitor the active-vs-CPU divergence as a composite signal. Alerting on active processes alone will fire during legitimate traffic spikes. Alerting on the divergence (high active, low CPU, slow requests incrementing) catches the I/O-bound drain pattern specifically.
Avoid NFS for hot code paths. If the application reads templates, configs, or user uploads from NFS, any server stall will pin workers in D state. Prefer local storage or a CDN for read-heavy paths.

How Netdata helps

Netdata collects PHP-FPM status page metrics at per-second resolution, which matters because saturation events unfold in seconds. The divergence between active processes and system CPU is visible in real time when both metrics share the same collection cadence.
The slow request counter rate is collected alongside active and idle process counts, so you can see whether worker saturation correlates with slow log activity in the same time window.
Per-worker request duration data from full status mode lets you distinguish bimodal distributions (some fast, some extremely slow) from uniform slowdown, which narrows the diagnosis between endpoint-specific blocking and systemic backend failure.
Database connection metrics from Netdata’s PostgreSQL, MySQL, and generic database collectors can be overlaid against FPM active process counts. When both climb together, the database connection multiplier is the likely cause.
ML anomaly detection flags the specific pattern of rising active processes with flat CPU as anomalous, even before it crosses a static threshold. This is useful because the absolute values are workload-dependent, but the divergence shape is a reliable signal.
Kernel-level socket metrics, including listen queue depth and overflow counters, provide the layer below FPM’s own visibility. These catch connection drops that the status page cannot report.