PHP-FPM OPcache hit rate below 99%: silent CPU and latency tax

A PHP-FPM pool with a healthy OPcache sits above 99% hit rate after warmup. Every request that misses recompiles PHP source into bytecode: parse, compile, optimize, and store. That work costs CPU on the worker handling the request and adds latency. When the cache is full and cannot admit new scripts, every worker handling those uncached scripts pays the compile tax on every request.

OPcache hit rate is easy to misread. It is cumulative since the shared memory segment was last allocated, so a long history of healthy hits can paper over a live problem. A pool that served ten million hits then started missing every other request minutes ago will still report a hit rate of 99.99%. Track the miss rate live, not the cumulative percentage.

A sub-99% hit rate after warmup points at one of a small set of conditions: the cache is full and cannot admit new scripts, the cache is too small for the codebase, opcache.validate_timestamps is re-stat-ing source files, or your deployment strategy multiplies file paths so the same scripts get cached under different keys. Each has a different signature in opcache_get_status().

What this means

A miss means the worker falls back to disk read, lex, parse, and compile. On a framework request that pulls in dozens or hundreds of files, even a small miss rate translates into a substantial CPU and latency bill spread across every worker in the pool.

The cumulative hit rate from opcache_get_status(false)['opcache_statistics']['opcache_hit_rate'] is computed from hits and misses counters that reset only when OPcache restarts. To detect a current problem you must compute the miss rate from counter deltas. A static 99.99% reading can hide a regression that started in the last minute.

Two ranges matter operationally:

• Below 99% after warmup is a yellow flag. Something is causing more misses than baseline.
• Below 95% means a meaningful fraction of the codebase is being recompiled on a recurring basis. PHP is paying the compile tax on a large share of requests.

Both conditions are silent in the FPM status page. There is no misses field there. You only see the symptoms downstream: elevated CPU across all workers, uniformly elevated per-worker request duration, and elevated request latency that does not correlate with any single slow endpoint.

flowchart TD
    A[Request needs script] --> B{In OPcache SHM?}
    B -- yes --> C[Serve bytecode, hit++]
    B -- no --> D[Read source from disk]
    D --> E[Lex, parse, compile]
    E --> F{Slot available?}
    F -- yes --> G[Store bytecode, miss++]
    F -- no --> H[No LRU eviction]
    H --> I[Recompile next request too]
    F -- cache_full, wasted below
max_wasted_percentage --> I

The right-hand path is the trap. OPcache uses first-come, first-serve admission with no LRU eviction. When the cache is full but wasted memory is below opcache.max_wasted_percentage (default 5%), no restart is triggered. Uncached scripts are recompiled every request as if OPcache were absent. The hit rate quietly degrades and nothing in the PHP-FPM status page tells you why.

Common causes

Quick checks

These are read-only. The first one assumes you have a web-accessible PHP script that calls opcache_get_status(false) and returns JSON. Do not call opcache_get_status() from the CLI to inspect the FPM pool: OPcache uses a separate shared memory segment per SAPI, so a CLI probe reads a different cache (or none at all).

# Check OPcache statistics from the FPM pool's own SAPI
curl -s http://127.0.0.1/opcache-status.php | python3 -c "
import sys,json
d=json.load(sys.stdin)
s=d['opcache_statistics']; m=d['memory_usage']
total=m['used_memory']+m['free_memory']+m['wasted_memory']
print(f\"Hit rate: {s['opcache_hit_rate']:.2f}%\")
print(f\"Hits: {s['hits']}, Misses: {s['misses']}\")
print(f\"OOM restarts: {s['oom_restarts']}, Hash restarts: {s['hash_restarts']}\")
print(f\"Scripts: {s['num_cached_scripts']} / keys: {s['num_cached_keys']} / max keys: {s['max_cached_keys']}\")
print(f\"cache_full: {s['cache_full']}, restart_pending: {s['restart_pending']}, restart_in_progress: {s['restart_in_progress']}\")
print(f\"Used: {m['used_memory']/1048576:.0f}MB, Free: {m['free_memory']/1048576:.0f}MB, Wasted: {m['wasted_memory']/1048576:.0f}MB ({100*m['wasted_memory']/total:.1f}%)\")"

# Compute a live miss rate from two samples 10s apart
for i in 1 2; do
  curl -s http://127.0.0.1/opcache-status.php | python3 -c "import sys,json; print(json.load(sys.stdin)['opcache_statistics']['misses'])"
  sleep 10
done
# Divide the delta by 10 to get misses per second

# Verify opcache is loaded for the FPM SAPI
# opcache_get_status() returns false, not an empty array, when disabled
curl -s http://127.0.0.1/opcache-status.php | python3 -c "
import sys,json
d=json.load(sys.stdin)
print('opcache_enabled:', d.get('opcache_enabled'))"

# Count PHP files the application could ask OPcache to cache
find /path/to/app -name '*.php' -type f | wc -l

# Check INI settings from the FPM SAPI, not CLI
# CLI and FPM may load different php.ini files; check from the web SAPI
curl -s http://127.0.0.1/opcache-status.php | python3 -c "
import sys,json
# Extend opcache-status.php to also return ini_get() output for these keys
" 2>/dev/null || echo 'Add ini_get() calls for opcache.memory_consumption, opcache.max_accelerated_files, opcache.max_wasted_percentage, opcache.validate_timestamps, opcache.revalidate_freq to your status script'

# Fallback: check FPM binary directly (path varies by distribution)
php-fpm -i 2>/dev/null | grep -E 'opcache\.(memory_consumption|max_accelerated_files|validate_timestamps)' || echo 'php-fpm binary not found in PATH; check via phpinfo() page instead'

# Confirm where opcache.* settings are actually applied
# The CLI php --ini output shows the CLI SAPI's php.ini, not necessarily FPM's
# Settings applied via php_admin_value in pool config are too late:
# the shared memory segment is allocated before pool config is read
grep -r 'opcache' /etc/php/*/fpm/ 2>/dev/null || grep -r 'opcache' /etc/php-fpm* 2>/dev/null

How to diagnose it

1. Confirm the pool is warmed up. If uptime since the last FPM restart is under 5-10 minutes, a low hit rate is expected cold-start behavior. Wait for warmup before treating this as an incident.
2. Pull the full opcache_get_status(false) payload. Use the web-accessible script, not the CLI. The CLI SAPI has its own cache and opcache.enable_cli defaults to off.
3. Compute the live miss rate, not the cumulative percentage. Sample misses twice with a known interval and divide by the interval. Anything above a few misses per second on a steady-traffic pool is real.
4. Check cache_full, restart_pending, restart_in_progress. The combination cache_full=true with both restart flags false is the silent-cache-full trap. The cache cannot admit new scripts and will not restart because wasted memory is below max_wasted_percentage.
5. Check num_cached_scripts against max_cached_keys. If they are equal or nearly equal, you are hitting the script-count ceiling. The configured max_accelerated_files is rounded up internally to a prime number; the effective limit is reported in max_cached_keys.
6. Check free_memory and wasted_memory as fractions of total. free_memory below 10% of total means you are near the memory ceiling. wasted_memory above 30% means fragmentation from invalidated scripts is reclaimable only by a full OPcache reset.
7. Check oom_restarts and hash_restarts. Non-zero values mean OPcache has forcefully cleared itself. A single oom_restart event resets the hit rate and explains a recent drop.
8. Inspect validate_timestamps and revalidate_freq. If validate_timestamps=1, every include triggers a stat() call. With revalidate_freq=2 the recheck happens at most every 2 seconds per file, but the syscall volume is still meaningful on a large codebase.
9. Correlate with deployment timing. If hit rate drops after every deploy and never fully recovers, suspect that the deploy writes to a new path and the old bytecode is still in the cache consuming memory.
10. Correlate with CPU and per-worker duration. OPcache thrash shows uniformly elevated CPU across all workers and uniformly elevated per-worker request duration, with no single slow endpoint. That distinguishes it from a slow-dependency worker drain.

Metrics and signals to monitor

Fixes

Cache full without restart

Increase opcache.memory_consumption and opcache.max_accelerated_files together. Memory alone does not help if you are also at the script-count ceiling, and a higher file limit does not help if you are out of memory. After changing either, apply a full FPM restart or a graceful SIGUSR2 reload. The reload re-execs the master and allocates a fresh shared memory segment; OPcache cannot be resized in place.

Tradeoff: every megabyte given to OPcache is a megabyte not available for worker RSS or OS page cache. Use PSS, not RSS, when sizing worker memory so the OPcache shared segment is not double-counted across workers.

memory_consumption too small

Default opcache.memory_consumption is 128 MB. For a modern framework with a large vendor tree, that is often undersized. Size it from observed used_memory + free_memory + wasted_memory after a representative warm window, then add 20-30% headroom.

Tradeoff: on hosts where FPM workers are the binding memory constraint, growing OPcache may force a lower pm.max_children. Run the capacity math first.

max_accelerated_files too low

Default is 10000, rounded up internally to a prime number. For applications with a large vendor tree, count actual .php files and set max_accelerated_files to roughly 1.5x that count.

The configured value is rounded up to the next value in the upstream table; read max_cached_keys from opcache_get_status() for the effective limit. Tradeoff: a higher limit slightly increases hash table memory overhead.

validate_timestamps=1 left on in production

opcache.validate_timestamps defaults to 1 (development-friendly). With it on, PHP re-stats source files according to opcache.revalidate_freq (default 2 seconds). On a large codebase that is a measurable syscall volume, and any file change invalidates and recompiles.

For production where deploys are explicit events, set opcache.validate_timestamps=0 and reset OPcache as the last step of the deploy. Call opcache_reset() from a web script, or do a full FPM restart. A graceful SIGUSR2 reload clears the shared-memory OPcache by re-execing the master. Tradeoff: you lose automatic detection of source file changes, so any deploy that does not also reset OPcache will continue serving stale bytecode.

Deploy strategy multiplying file paths

A “new checkout per release” pattern causes OPcache to key the same script under a new path after every deploy. The old bytecode remains in the cache consuming memory until a reset or restart, and num_cached_scripts climbs with each release.

Mitigations, in order of preference: use a stable deployment path with atomic symlink swaps and reset OPcache after the swap; or call opcache_reset() shortly after each deploy to clear old entries. A full FPM restart also works but has higher overhead than a targeted reset. The opcache.file_cache mechanism can pre-warm a cache baked into the image, but it does not eliminate the path-multiplication problem by itself. opcache.file_cache_read_only is available in PHP 8.5 and later for a read-only cache directory.

Settings applied in the wrong place

OPcache memory settings must be in the php.ini loaded at SAPI startup, not in PHP-FPM pool config via php_admin_value. The shared memory segment is allocated before pool config is applied. phpinfo() may show the changed value, but the allocated memory uses the default. PHP 8.5 emits a startup warning for a too-late opcache.memory_consumption change in pool configuration; older releases may fail silently.

Prevention

• Track miss rate, not just hit rate. The cumulative percentage lags reality. Alert on the rate of change of misses.
• Size from observation, not from defaults. After a representative warm window, read used_memory + free_memory + wasted_memory and num_cached_scripts, then set both memory_consumption and max_accelerated_files with 20-30% headroom.
• Disable validate_timestamps in production and make OPcache reset an explicit deploy step.
• Monitor cache_full combined with the restart flags. The silent-cache-full trap is invisible if you only watch hit rate.
• Watch oom_restarts and hash_restarts as monotonic counters. Any increment is an event worth correlating with deploy time.
• Use a stable deploy path or reset OPcache immediately after every release.
• Confirm where opcache. settings live.* Pool config is too late; the php.ini loaded at SAPI startup is what counts.
• Pre-warm after full restarts. Hit critical endpoints before returning the pool to full traffic, especially after a full restart that recreates the shared memory segment empty.

How Netdata helps

• Per-second OPcache signals sit next to FPM pool metrics in the same dashboard, so cache regressions line up with worker utilization, listen queue depth, and per-worker duration in the same time window.
• The miss rate is computed from misses counter deltas, so you see the live problem instead of waiting for the cumulative hit rate to catch up.
• cache_full, restart_pending, and restart_in_progress are surfaced as discrete signals. The silent-cache-full trap (cache full, no restart) is detectable as a single composite condition rather than a reading you have to interpret by hand.
• CPU per worker, per-worker request duration, and OPcache signals share the same per-second resolution. A recompilation event shows up as a simultaneous step in CPU and miss rate across the whole pool, distinct from an endpoint-specific slow request.
• Anomaly detection on the OPcache miss rate flags deviations against the pool’s own baseline. A 99.5% hit rate at 10,000 requests per second is still a meaningful tax that a static threshold will not catch.
• Restart counters (oom_restarts, hash_restarts) are tracked as monotonic counters with rate-of-change alerts, so a single self-clear event is correlated with the deploy or traffic event that preceded it.

Related guides

• PHP-FPM 504 Gateway Timeout: requests accepted but never finishing in time
• PHP-FPM active processes near max_children: reading pool utilization
• PHP-FPM in containers: cgroup limits and the silent OOM kill
• PHP-FPM “child N exited on signal 11 (SIGSEGV)”: worker segfaults
• PHP-FPM crash loop and fork storm: workers dying faster than they serve
• PHP-FPM dynamic mode scaling lag: why the pool cannot keep up with bursts
• PHP-FPM emergency restart: “failed processes threshold reached, initiating reload”
• PHP-FPM graceful reload: the brief no-worker window on SIGUSR2
• How PHP-FPM actually works in production: a mental model for operators
• PHP-FPM idle processes at zero: no burst headroom left
• PHP-FPM listen queue growing: the earliest signal of saturation
• PHP-FPM “server reached pm.max_children setting (N), consider raising it”