$ guides / php-fpm / php-fpm-dynamic-mode-scaling-lag ▌

Operations Guides

PHP-FPM dynamic mode scaling lag: why the pool cannot keep up with bursts

During a traffic burst, your PHP-FPM pool reports total processes well below pm.max_children, yet the listen queue fills and latency spikes. The status page shows idle workers hitting zero, then climbing back a few seconds later as new workers come online. By then, requests have already queued and some users have already seen 502s or 504s.

This is dynamic mode scaling lag. The dynamic process manager is reactive, not predictive. It checks idle-worker counts once per second on a fixed internal timer and forks replacement workers after the deficit is already visible. When a burst arrives faster than the check-and-fork cycle can respond, the idle pool drains to zero and the listen backlog absorbs the overflow until new workers are ready.

The enabling condition is almost always pm.min_spare_servers set too low for the burst pattern. With min_spare at 1 or 2, a burst of 20 concurrent requests consumes the entire idle buffer before the master process has completed a single check cycle.

What this means

In dynamic mode (pm = dynamic), the master maintains workers between pm.min_spare_servers and pm.max_children. When idle count drops below min_spare_servers, it forks new workers. When idle exceeds max_spare_servers, it kills excess workers.

The critical limitation: this check runs on a fixed one-second internal timer, not on every incoming request. The master’s event loop processes signals and reaps children between checks, but the spare-server evaluation is not triggered by connection arrival. A burst of 50 requests in 100 milliseconds hits the pool before the next check cycle fires.

On PHP 8.1+, the master can fork up to pm.max_spawn_rate workers (default 32) per check cycle when it detects an idle deficit. If the deficit exceeds that cap, the master needs multiple check cycles to catch up. During those cycles, new requests queue in the socket backlog. On PHP versions before 8.1, pm.max_spawn_rate does not exist.

The interval is separate from systemd_interval, and current upstream PHP-FPM does not expose it as a pool directive.

During the gap between demand arrival and capacity readiness, the listen queue fills. If it reaches listen.backlog, the kernel drops connections and the web server returns 502.

flowchart TD
    A["Traffic burst arrives"] --> B["Idle workers drain to zero"]
    B --> C["One-second maintenance timer not yet fired"]
    C --> D["New requests enter socket backlog"]
    D --> E["Maintenance timer fires: master forks workers"]
    E --> F{"Deficit covered in one cycle?"}
    F -->|"No"| G["More cycles needed, queue grows"]
    F -->|"Yes"| H["Workers accept queued requests"]
    G --> H
    H --> I["Queue drains, latency normalizes"]
    D --> J{"Queue reaches listen.backlog?"}
    J -->|"Yes"| K["Kernel drops connections, 502s"]
    J -->|"No"| I

Common causes

Cause	What it looks like	First thing to check
`pm.min_spare_servers` too low	Idle hits 0 during bursts, listen queue spikes, total processes well below `max_children`	Pool config: `grep min_spare` in pool .conf
`pm.start_servers` too low	Cold pool after restart or reload takes too long to reach operating capacity	Compare `start_servers` to typical peak active count
`pm.max_spawn_rate` too low (PHP 8.1+)	Master forks workers but queue keeps growing for multiple check cycles	PHP version and `pm.max_spawn_rate` setting
Burst pattern exceeds fork rate	“seems busy” warnings in error log, idle oscillates between 0 and `min_spare`	Error log for “seems busy” entries
`listen.backlog` too small	Queue fills quickly, 502s appear before workers are exhausted	Status page: `listen queue len` vs `max listen queue`

Quick checks

# Check current pool state (idle, active, total, queue).
# Adjust URL to match your pm.status_path and web server config.
curl -s http://127.0.0.1/fpm-status?json | python3 -c "
import sys,json
d=json.load(sys.stdin)
for k in ['active processes','idle processes','total processes','listen queue','max listen queue','listen queue len','max children reached']:
    print(f'{k}: {d[k]}')"

# Check kernel-level socket backlog (Unix socket)
ss -xlnp | grep php

# Check kernel-level socket backlog (TCP)
ss -tlnp | grep 9000

# Check pm settings in pool config (path varies by distribution)
grep -E "^pm\.|^pm =|^listen\.backlog" /etc/php/*/fpm/pool.d/*.conf

# Check PHP version (determines pm.max_spawn_rate availability)
php-fpm -v 2>/dev/null | head -1 || php -v | head -1

# Look for "seems busy" warnings in the error log (path varies by distribution)
grep "seems busy" /var/log/php-fpm/error.log 2>/dev/null | tail -20
# or via journalctl (unit name varies: php-fpm, php8.1-fpm, etc.)
journalctl -u php-fpm --since "1 hour ago" | grep "seems busy" | tail -20

# Check if max_children was reached (counter since last restart)
curl -s http://127.0.0.1/fpm-status | grep "max children reached"

# Watch idle vs listen queue in real time (1-second poll)
watch -n 1 'curl -s http://127.0.0.1/fpm-status | grep -E "idle processes|listen queue|active processes"'

# Check kernel listen overflow counters
nstat | grep -i listen

How to diagnose it

Confirm the pool is not at max_children. If active processes equals pm.max_children and the listen queue is growing, you have worker exhaustion, not scaling lag. The max children reached counter will be incrementing. See the related guide on active processes near max_children.
Check whether idle processes hit zero during the burst. Poll the status page at 1-second intervals during a burst. If idle drops to 0 while total processes is still below max_children, the process manager has not yet forked enough workers. This is the signature of scaling lag.
Compare the timing of idle depletion and listen queue growth. If the listen queue starts growing at the same moment idle hits zero, the pool had no buffer. If the queue grows a few seconds after idle hits zero, the master’s check cycle has not fired yet. Both point to min_spare_servers being too low, but the delay tells you whether the timer or the fork rate is the bottleneck.
Check the error log for “seems busy” warnings. The master emits this warning when the idle count is below pm.min_spare_servers and it must spawn children. Frequent warnings during normal traffic confirm the pool is repeatedly caught without spare capacity.
Verify the PHP version and pm.max_spawn_rate. On PHP 8.1+, pm.max_spawn_rate (default 32) caps how many workers the master can fork per check cycle. If your burst requires 100 new workers and max_spawn_rate is 32, the master needs at least 4 cycles to catch up. On PHP versions before 8.1, pm.max_spawn_rate does not exist.
Rule out slow requests. If workers are stuck on a slow backend (database, external API), they are not returning to idle, which looks like scaling lag but is actually a slow-dependency drain. Check the slow log and per-worker request durations. See the related guide on 504 gateway timeout.
Check kernel listen overflow counters. If ListenOverflows or ListenDrops are incrementing, the backlog has filled and the kernel is dropping connections. This is the end state of scaling lag left unaddressed.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Idle processes	The buffer that absorbs bursts before the master reacts	Drops to 0 while total < `max_children`
Listen queue	Requests waiting for a worker; earliest user-facing degradation signal	Non-zero and growing while idle is 0
Max listen queue	High water mark since restart; shows past saturation even if current queue is 0	Approaching `listen queue len`
Total processes	Whether the master is actually scaling the pool	Stuck at or near `min_spare + active`, not climbing toward `max_children`
Max children reached	Counter of times the master wanted to fork but hit the ceiling	Incrementing (rules out `max_children` as the bottleneck if zero)
“seems busy” log entries	Master warning that idle workers are below `pm.min_spare_servers`	Any occurrence during normal traffic
Kernel ListenOverflows	Connections dropped at the kernel level when backlog is full	Counter incrementing

Fixes

Warning: All config changes require a graceful reload (SIGUSR2). The reload drains existing workers before the master re-execs, creating a brief window where no workers accept new connections. Plan config changes during low-traffic periods.

Raise pm.min_spare_servers

The primary fix. Set min_spare_servers high enough to absorb your typical burst before the master’s check cycle fires. If your burst pattern is 20 concurrent requests arriving within 200ms, min_spare should be at least 20, plus headroom.

The tradeoff: each idle worker consumes memory (typically 30-200MB RSS depending on the application). More spare workers means more memory reserved at all times, even during low traffic. Verify that min_spare_servers times average worker RSS fits within your memory budget. Use PSS (Proportional Set Size) from /proc/<pid>/smaps_rollup for accurate per-worker memory accounting, since RSS overestimates by counting shared opcache pages in each worker.

Raise pm.max_spare_servers

If you raise min_spare, also raise max_spare. The master kills idle workers above max_spare during low-traffic periods, so if max_spare is close to min_spare, the pool will oscillate: fork workers during a burst, then immediately kill them when traffic subsides, then fork again at the next burst. Set max_spare comfortably above min_spare. PHP requires max_spare to be greater than or equal to min_spare.

Raise pm.start_servers

start_servers controls how many workers exist immediately after a restart or reload. If start_servers is low, the pool starts with a deficit and must scale up under live traffic. The default formula is (min_spare_servers + max_spare_servers) / 2. If you raise min and max spare, start_servers should follow.

Raise pm.max_spawn_rate (PHP 8.1+)

If the master is forking workers but the queue keeps growing for multiple check cycles, the per-cycle fork cap may be the bottleneck. Raising pm.max_spawn_rate above the default 32 allows the master to fork more workers per check cycle, reducing the number of cycles needed to cover a large deficit.

The tradeoff: forking many workers simultaneously spikes CPU (page table copying, PHP runtime initialization) and memory. On memory-constrained hosts, a high spawn rate can trigger cgroup OOM kills in containers. See the related guide on cgroup OOM in containers.

Switch to static mode

If bursts are frequent and predictable, and memory is sufficient, pm = static eliminates scaling lag entirely. All workers are pre-forked at startup and always running. There is no timer check, no fork delay, no reaction gap. The cost is constant memory usage at peak levels even during idle periods.

Static mode is the right choice when traffic is bursty, memory headroom exists, and the application’s per-worker RSS is stable. If per-worker RSS grows over time, static mode still works but requires pm.max_requests for recycling, which reintroduces brief spawn delays during worker replacement.

Do not just raise listen.backlog

A larger listen.backlog gives the kernel more queue space, which buys time during the reaction delay. But it does not make workers appear faster. If the burst is sustained, a bigger backlog delays the 502s by a few seconds but does not fix the underlying spawn lag. Use this only as a stopgap while tuning spare servers or switching to static.

Prevention

Size min_spare_servers to your burst pattern. Measure the maximum concurrent request arrival rate during your peak traffic window. Set min_spare to at least that number plus 20% headroom.
Monitor idle-vs-listen-queue timing. The earliest signal of scaling lag is idle hitting zero followed by the listen queue growing. If you see this sequence repeatedly, the pool is under-provisioned for spare capacity, even if total processes never reaches max_children.
Poll at 1-second intervals. Scaling lag events unfold in seconds. A 10-second poll interval will miss the idle-to-zero spike and the brief queue buildup entirely. You will see a clean status page and wonder why users reported latency.
Test with realistic burst patterns. Load tests that ramp traffic gradually will not reproduce scaling lag. Use step-load or spike tests that simulate sudden concurrent request arrival.
Consider static mode for production. If your traffic is bursty and memory allows, static mode removes the reaction delay entirely. Dynamic mode is better suited for environments where memory pressure is the binding constraint and traffic is steady.

How Netdata helps

Per-second polling of PHP-FPM status page metrics catches the idle-to-zero spike and the listen queue growth that 10-second pollers miss entirely.
Correlation of idle processes with listen queue depth on the same timeline shows the reaction delay: idle drops, queue grows, workers appear seconds later. The gap between those events is your scaling lag.
The “max children reached” counter as a rate metric distinguishes scaling lag (counter is zero, total is below max) from true worker exhaustion (counter is incrementing, total is at max).
Kernel-level socket monitoring (Recv-Q on the listening socket, ListenOverflows counters) reveals backlog overflow that the FPM status page cannot see.
Per-worker RSS tracking helps you calculate the memory cost of raising min_spare_servers, so you can size the spare buffer without risking OOM.