$ guides / apache-httpd / apache-httpd-swap-thrashing ▌

Operations Guides

Apache swap thrashing: the latency cliff before the OOM kill

The symptom arrives before the failure. P95 latency doubles, then triples, across every endpoint at once, including static files that should be served from page cache. No 5xx spike yet. No OOM kills in dmesg yet. The Apache parent process is fine, the scoreboard shows workers in normal states, and every request is suddenly slow. This is the swap window: the period between “memory is tight” and “the kernel starts killing children,” where Apache is technically up but effectively degraded.

Apache’s own documentation is blunt about this: “A webserver should never ever have to swap, as swapping increases the latency of each request beyond a point that users consider ‘fast enough’” (Apache 2.4 performance tuning). An Apache child that has been swapped out must be paged back in before it can serve a request, and that child responds 10 to 100 times slower than normal. Because any request can land on any child, a fraction of swapped children raises tail latency for everyone.

This guide covers how the progression works, how to confirm it in minutes, and how to fix the memory math that causes it. The trigger is usually a worker pool sized without regard for per-child RSS; see Apache MaxRequestWorkers tuning for the full sizing procedure.

What this means

Linux does not go from “plenty of memory” to “OOM kill” in one step. As Apache’s total RSS approaches physical RAM, the kernel reclaims memory in a predictable order, and each stage degrades Apache differently:

Page cache reclaim. The kernel drops clean file-backed pages first. Apache still works, but static files that were previously served from cache now hit disk. Read I/O rises, latency creeps up, and nothing looks wrong in free output because the freed cache shows up as available.
Swapping. With page cache exhausted, the kernel starts moving anonymous pages (Apache child process memory) to swap. Children that get swapped out respond 10 to 100 times slower when paged back in. Tail latency spikes fleet-wide.
Thrashing. If the working set genuinely does not fit in RAM, pages are swapped in, used briefly, and swapped out again. The system spends its time moving pages instead of serving requests. Allocations technically succeed, just extremely slowly, so the OOM killer may not fire for a long time while the server is unresponsive.
OOM kill cascade. Finally the kernel OOM killer starts terminating processes. It usually selects the highest-badness task—often an httpd child—so you can have a “running” Apache with few or no functional children. The parent normally respawns replacements, which allocate memory and can be killed again.

flowchart TD
  A[Apache RSS grows toward RAM] --> B[Kernel reclaims page cache]
  B --> C[Static file reads hit disk - latency creeps]
  C --> D[Anonymous pages swapped out]
  D --> E[Swapped children 10-100x slower - tail latency spikes]
  E --> F{Working set fits in RAM?}
  F -->|yes| G[Recovers when pressure eases]
  F -->|no| H[Swap thrashing - paging dominates, OOM does not fire]
  H --> I[OOM killer kills children]
  I --> J[Parent respawns children]
  J --> A

The operational point: any sustained swap usage by Apache children is a warning, not a curiosity. By the time the OOM killer fires, you have already been serving degraded latency for minutes or hours. The goal is to catch stages 1 and 2, not stage 4.

Common causes

Cause	What it looks like	First thing to check
MaxRequestWorkers oversized for RAM	Swap usage climbs with traffic peaks; worst case at full worker count	`MaxRequestWorkers x avg child RSS` vs total RAM
Module memory leak (mod_php, mod_perl)	Per-child RSS grows monotonically over hours or days; swap climbs even at steady traffic	Per-PID RSS trend; `MaxConnectionsPerChild` set to 0
Graceful restart pile-up	Memory jumps correlate with deploys or log rotation; old and new generations overlap	Repeated “resuming normal operations” in error log; many `G` states in scoreboard
Traffic growth past capacity	Slow daytime trend: MemAvailable declining week over week	Peak total Apache RSS trend vs RAM
Another process eating memory	Apache RSS stable but system MemAvailable falling	`ps` sorted by RSS across all processes, not just httpd

The first cause is the most common and the most self-inflicted. MaxRequestWorkers set to 1000 on a 4 GB server with mod_php children at 50 MB each implies 50 GB of worst-case memory. The server has 4 GB. Under a real burst, the system swaps, then thrashes, then OOM kills.

Quick checks

All read-only. Run these during the latency spike.

# 1. Available memory and swap totals
free -m
grep -E "MemAvailable|SwapTotal|SwapFree" /proc/meminfo

MemAvailable is the number that matters. Low MemFree with high MemAvailable is normal Linux cache behavior, not a problem. Declining MemAvailable is the leading indicator.

# 2. Swap in/out rates right now (si = swap in, so = swap out, KB/s)
vmstat 1 5

Sustained non-zero si is the smoking gun: the kernel is paging memory back in to run processes, which means those processes stall on disk before they can serve requests. Brief so bursts with zero si are less alarming; sustained si is what kills latency.

# 3. Total and per-child Apache RSS
ps -C httpd -o pid,rss,cmd --sort=-rss 2>/dev/null || \
  ps -C apache2 -o pid,rss,cmd --sort=-rss

# Average RSS and process count (braces matter: without them the pipe
# only attaches to the second ps)
{ ps -C httpd -o rss --no-headers 2>/dev/null || \
  ps -C apache2 -o rss --no-headers; } | \
  awk '{sum+=$1; count++} END {if (count) printf "Avg RSS (KB): %d  Count: %d\n", sum/count, count; else print "No Apache processes found"}'

Multiply the average by MaxRequestWorkers (from your config, not server-status) and compare against total RAM. If the product exceeds RAM, the pool can never be fully busy without swapping.

# 4. OOM kills and kernel memory pressure history
dmesg -T | grep -i -E "oom|killed process" | tail -20

If OOM kills are already present, you are past the warning stage. Note which PIDs were killed; Apache children killed and respawned repeatedly is the cascade pattern.

# 5. Which children are actually swapped
for pid in $(pgrep 'httpd|apache2'); do
  swap=$(awk '/VmSwap/{print $2}' /proc/$pid/status 2>/dev/null)
  [ -n "$swap" ] && [ "$swap" != "0" ] && echo "PID $pid: ${swap} kB swapped"
done

VmSwap in /proc/[pid]/status shows per-process swap usage. Any Apache child with non-zero swap during a latency event confirms the mechanism.

# 6. Memory pressure stall information (kernel 4.20+)
cat /proc/pressure/memory

The full line shows the share of time all non-idle tasks were simultaneously stalled on memory; kernel documentation calls extended time in that state thrashing. No universal web-workload avg10 threshold exists. Track the trend, register a PSI trigger sized to your latency objective, and investigate any sustained rise above that local baseline.

How to diagnose it

Confirm the latency is memory-correlated, not backend-correlated. Check the scoreboard state distribution: curl -s http://localhost/server-status?auto | grep Scoreboard. Swap thrashing looks different from a slow backend cascade. In a backend cascade, workers pile into W state while CPU and memory look normal. In swap thrashing, latency rises across all request types including static files, vmstat shows swap-in activity, and per-child VmSwap is non-zero. If you proxy and only proxied paths are slow, read Apache backend response time instead.
Establish whether swap usage is sustained or a one-off. Sample vmstat 1 for 30 to 60 seconds and check SwapFree twice, a few minutes apart. A one-time swap-out from an overnight cron job or backup is noise. Continuous si traffic during business hours is the incident.
Determine which cause applies. Compare per-child RSS now against a baseline from days ago. Growing RSS per PID points at a leak: monotonic per-child growth, MaxConnectionsPerChild 0, swap rising before OOM kills. Stable RSS with swap rising only at peak traffic points at an oversized worker pool or simple capacity exhaustion.
Do the memory math. Worst case = MaxRequestWorkers x max observed child RSS. For prefork with mod_php, 50 to 100 MB per child is a reasonable starting estimate, but measure it. Apache’s maximum theoretical memory should not exceed about 70% of RAM; the rest belongs to the OS, page cache (which static file serving depends on), and everything else on the box.
Check for restart pile-ups. grep "resuming normal operations" /var/log/apache2/error.log | tail -20 (or /var/log/httpd/error_log). Restarts minutes apart combined with many G states in the scoreboard mean multiple generations of children are coexisting, each holding memory. Graceful restarts during high load briefly double Apache memory.
Rule out the neighbors. ps -eo pid,rss,cmd --sort=-rss | head -20. If the top consumers are not Apache, the fix is not in Apache config.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`MemAvailable` trend	The leading indicator; falls before any swap activity	Sustained downward trend across days
Swap in rate (`si`)	Direct measure of processes stalling on paged-in memory	Any sustained non-zero value during traffic
Swap out rate (`so`)	Kernel pushing pages out; precursor to swap-in stalls	Rising at traffic peaks
Per-child `VmSwap`	Shows which Apache children are actually swapped	Any non-zero value in production
Total Apache RSS vs RAM	The sizing equation behind everything	Approaching 70% of RAM
Per-child RSS trend per PID	Detects leaks before they become swap	Monotonic growth over days
PSI memory `full` stall	Time all tasks stalled on memory simultaneously	Sustained non-zero `avg10`
P95/P99 request latency	The user-visible symptom	Jump affecting all endpoints at once, static included

Fixes

Worker pool oversized for RAM

Recalculate MaxRequestWorkers from memory, not from traffic aspiration: MaxRequestWorkers = (usable RAM for Apache) / (max observed child RSS), with Apache’s total capped near 70% of RAM. This usually means lowering the number, which feels wrong during a traffic spike, but a queue in the listen backlog beats a swapped worker pool. Queued connections wait hundreds of milliseconds; swapped-out children stall for seconds. The full sizing procedure is in Apache MaxRequestWorkers tuning.

Tradeoff: a lower worker ceiling means earlier MPM worker-limit messages and earlier 503s under genuine overload. That is the correct failure mode. It is fast, visible, and recoverable, unlike thrashing.

Memory leak in a module

Set MaxConnectionsPerChild to a finite value (5000 to 10000) to force periodic child recycling. This bounds leak growth and is a band-aid, not a fix. Then find the leak: compare RSS growth across children serving different URL patterns. If mod_php is involved, set PHP’s memory_limit, check for known extension leaks, and consider moving to PHP-FPM, which isolates PHP memory from Apache children entirely.

Tradeoff: child recycling costs a brief spawn burst. Tune MinSpareServers/MinSpareThreads so recycling does not cause cold-child latency.

Graceful restart pile-up

Reduce restart frequency (deploy batching, less aggressive config management). GracefulShutdownTimeout applies to graceful-stop, not a graceful reload; tune request, keepalive, and proxy timeouts to limit lingering work.

Genuine capacity exhaustion

If RSS is stable, the pool is correctly sized, and MemAvailable still declines month over month, the workload has outgrown the host. Add RAM or split the workload. No Apache directive fixes physics.

What not to do

Do not treat “add more swap” as a fix. Swap does not make the working set fit; it only changes how slowly the system fails. Swap is not useless in general: on systemd systems, systemd-oomd uses PSI to kill runaway workloads proactively, and it needs some swap present to have time to react before the system livelocks. But for Apache latency specifically, the correct target is zero sustained swap usage by children, and the fix is always the memory math or the leak. Similarly, do not restart Apache as the first response: it clears the symptom for hours while the leak or sizing error reloads.

Prevention

Size from memory. Derive MaxRequestWorkers from measured child RSS. Re-derive after any module change, especially adding mod_php or a large framework.
Cap child lifetime. Non-zero MaxConnectionsPerChild in any deployment with embedded interpreters. The default of 0 is wrong for mod_php and mod_perl.
Watch the leading indicators, not the kill. Alert on MemAvailable trend and any sustained swap-in rate. OOM kills in dmesg are the postmortem, not the alarm.
Keep headroom. Apache’s worst-case memory at or below 70% of RAM, leaving page cache for static file serving.
Restart hygiene. Batch config changes, avoid automated restarts more often than necessary, and review request/keepalive/proxy timeout settings.

How Netdata helps

Available memory trend, not free memory. Netdata charts MemAvailable per second, so the slow reclaim phase (page cache being eaten) is visible days before swap starts, which is when the fix is cheapest.
Swap I/O rates separated from swap usage. Swap used staying flat is meaningless if pages are churning; the swap in/out rate charts show the thrash itself, which is what correlates with the latency spike.
Per-process RSS and swap. Per-application memory breakdowns let you see Apache’s aggregate RSS climbing and identify which processes hold swapped pages, without writing /proc scraping loops.
Apache scoreboard correlation. The Apache collector charts BusyWorkers, idle workers, and request rates alongside system memory, so you can tell “workers full because traffic is high” from “workers slow because they are being paged in” in one view.
Latency overlaid on memory pressure. Request latency percentiles next to swap-in rate turns the mechanism into a picture: latency steps up exactly when swap-in starts, not when RAM runs out.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

The Netdata solution

Apache HTTP Server monitoring with Netdata

Netdata monitors Apache HTTP Server with per-second metrics from mod_status, pre-built dashboards, and ML-powered anomaly detection. Watch busy versus idle workers and the scoreboard state mix, requests per second, bytes served per second, and request processing duration alongside the rest of your stack, so you catch the worker-exhaustion, slow-backend, and memory incidents in these runbooks before they page anyone.

See web server monitoring → Start monitoring free