Your Apache server has been up for three weeks. Nothing is erroring. Request rate is normal, latency is fine, the scoreboard looks healthy. But free memory keeps shrinking, swap usage is creeping up, and last night the OOM killer took out two httpd children. The parent respawned them, and the cycle started over.

This is the memory-leak slow death: a module loaded into Apache (most commonly mod_php, mod_perl, or a custom module) leaks a small amount of memory per request. Each child’s RSS grows monotonically over hours or days. Nothing looks broken until the system runs out of memory, the OOM killer starts shooting children, and the parent respawns fresh ones that also leak. You are now in a kill-and-respawn spiral.

The signal is not a sudden jump. It is a steady, per-PID upward trend in RSS. If you only watch total system memory or alert on absolute thresholds, you will not see it until the OOM kills start. This article covers how to confirm the pattern, find the leaking module, contain the damage, and fix it properly.

What this means

Apache children (on prefork especially, but also worker and event) are long-lived processes. With the default MaxConnectionsPerChild 0, a child handles requests forever. Every request allocates memory through the module pipeline and APR pools. Well-behaved modules return that memory when the request ends. A leaking module does not, or leaks into interpreter state (PHP, Perl) that survives across requests inside the same process.

There is a second, compounding behavior: Apache children generally do not return memory to the OS even when it is freed internally. Once a child’s allocator has grown the process to hold a big request, that high-water mark tends to stick. RSS ratchets upward and rarely comes back down.

The cascade:

flowchart TD
  A[Module leaks memory per request] --> B[Child RSS grows monotonically]
  B --> C[Total Apache RSS approaches RAM]
  C --> D[Page cache reclaimed, then swap]
  D --> E[OOM killer targets largest children]
  E --> F[Parent respawns fresh children]
  F --> B

MaxConnectionsPerChild 0 is almost always present when this pattern bites. A finite value recycles children periodically and bounds how large the leak can grow before the process is destroyed. The default is wrong for any deployment running mod_php or mod_perl.

One measurement caveat before you start: RSS overcounts. RSS includes shared library pages that are loaded once in physical RAM but counted against every child. Summing RSS across all children can exceed total RAM by a wide margin. For the true per-child cost, use PSS (Proportional Set Size) from /proc/<pid>/smaps_rollup. PSS divides shared pages among the processes sharing them.

Common causes

CauseWhat it looks likeFirst thing to check
Leaking application code under mod_phpRSS grows fastest on children serving PHP URLs; growth tracks request countCompare RSS growth of children handling PHP vs static URLs
Leaking mod_perl or custom C moduleSame monotonic growth, tied to specific handlers or vhostsCorrelate RSS growth with the URL patterns that child serves
MaxConnectionsPerChild 0 (the default)Any small leak becomes fatal because children never recycleGrep the config for the directive; unset means 0
Large request or response bodies bufferedIndividual children spike to 2x+ average RSS after specific requestsLook for the biggest-RSS child and what it serves
Vulnerable httpd version (known leak CVE)Growth tied to HTTP/2 traffic, not application URLsCheck httpd version against the security page

The last row matters: not every leak is your code. CVE-2023-45802 was an HTTP/2 stream reset memory leak fixed in 2.4.58, and later mod_http2 memory issues were fixed in 2.4.67 and 2.4.68. If you run HTTP/2 on an older 2.4.x and the growth correlates with HTTP/2 connections rather than application URLs, upgrading is the fix. See the Apache 2.4 vulnerabilities list at https://httpd.apache.org/security/vulnerabilities_24.html.

Quick checks

These are all read-only and safe to run during an incident.

# Per-child RSS, biggest first (Debian uses apache2, RHEL uses httpd)
ps -C httpd -o pid,rss,vsz,cmd --sort=-rss 2>/dev/null || \
  ps -C apache2 -o pid,rss,vsz,cmd --sort=-rss

# Average and max child RSS
( ps -C httpd -o rss --no-headers 2>/dev/null || \
  ps -C apache2 -o rss --no-headers ) | \
  awk '{sum+=$1; count++; if($1>max)max=$1} END {print "Avg (KB):", sum/count, "Max (KB):", max, "Count:", count}'

# True per-child cost via PSS (kernel 4.14+; not present on RHEL 7's 3.10)
grep -E '^(Pss|Rss):' /proc/<pid>/smaps_rollup

# Per-child lifetime and request counts (requires mod_status).
# The full HTML page has a per-slot table: "Acc" is requests served,
# "SS" is seconds since that slot's child started. High values on the
# biggest-RSS children confirm old, never-recycled processes.
curl -s http://localhost/server-status | grep -A40 "Srv PID"

# Check whether children are being recycled at all
grep -ri "MaxConnectionsPerChild\|MaxRequestsPerChild" /etc/httpd/ /etc/apache2/ 2>/dev/null

# Evidence the endgame has started: OOM kills
dmesg | grep -i "oom\|killed process" | tail -20

# Memory pressure context
grep -E "MemAvailable|SwapFree|SwapTotal" /proc/meminfo

What to look for: a wide spread between average and max RSS, the oldest children (highest per-slot “SS”, most requests served) sitting at the top of the RSS list, and any swap usage at all. Sustained swap usage on Apache children is a warning sign on its own.

How to diagnose it

  1. Confirm the trend is per-PID and monotonic. Take RSS snapshots per PID now and again in an hour. A leak shows each PID’s line sloping upward at a similar rate. A one-time spike (deploy, cache warm, big request) shows a step, not a slope.

  2. Rule out “freed but not returned.” If RSS jumped during a traffic event and has been flat since, the allocator is holding freed pages at a high-water mark, not leaking. Only continued growth under continued traffic confirms a leak.

  3. Quantify with PSS, not RSS. Pull Pss: from /proc/<pid>/smaps_rollup for a few children. This is the real memory each child costs, and the number you need for sizing and bounding the damage.

  4. Identify the leaking module by URL pattern. Compare RSS growth across children serving different URL families: PHP application URLs vs static files, one vhost vs another, one endpoint vs the rest. The children whose RSS climbs fastest point at the handler responsible. On prefork this is straightforward since one child serves one request at a time; sample the child’s current request from the full server-status page alongside its RSS.

  5. Check the configuration for the amplifier. If MaxConnectionsPerChild is 0 or unset, the leak has unlimited runway. That does not cause the leak, but it is why the leak becomes an outage.

  6. Check the version. If HTTP/2 is enabled and httpd is older than 2.4.58 (or older than 2.4.68 for the later mod_http2 issues), treat a version-locked leak as a candidate cause before blaming application code.

  7. Estimate the blast radius. Compute MaxRequestWorkers x max_observed_child_RSS. If that exceeds roughly 70% of RAM, the leak is not the only problem: your worker pool is oversized against real per-child memory. See Apache MaxRequestWorkers tuning: sizing the worker pool against memory.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Per-PID RSS trendThe primary leak signal; slope, not levelMonotonic growth per PID over hours or days
Per-PID PSS (smaps_rollup)True per-child cost without shared-page double countingPSS climbing in step with RSS confirms real growth
Avg vs max child RSS spreadDetects heavy requests and uneven leak exposureAny child >2x the average
Total Apache RSS vs RAMPositions you on the degradation curveMaxRequestWorkers x Avg RSS approaching 70% of RAM
MemAvailable trendLeading indicator before swap and OOMSteady decline at constant traffic
Swap usageSwapped children respond far slower; page-in latency dominatesAny sustained swap usage by httpd children
OOM kills in dmesgThe terminal phase has begunAny httpd child killed by the OOM killer
Requests served per childExplains RSS differences between childrenHighest-RSS children are also the oldest

Fixes

Contain first: bound the leak with child recycling

Set MaxConnectionsPerChild to a finite value so every child is destroyed and replaced after serving that many connections:

MaxConnectionsPerChild 5000

Start around 5000-10000, then watch per-child RSS over days. If children still grow large before hitting the limit, lower it. Do not go very low (hundreds): child creation is not free, and on prefork each new child is a full process fork. This is a band-aid. It converts an OOM outage into steady, bounded process churn while you find the real leak. The old directive name MaxRequestsPerChild still works; it was renamed in the 2.4 development line.

Fix the leaking code

Once the URL pattern implicates a module: for mod_php, set and enforce PHP’s memory_limit (note it does not cap leaks inside C extensions that allocate outside PHP’s allocator), and audit the application and extensions for known leaks. The durable structural fix for PHP is moving from mod_php to PHP-FPM, which isolates PHP memory into its own pools with independent worker recycling and removes the PHP runtime from every Apache child entirely. The same logic applies to mod_perl: an embedded interpreter’s leaks become Apache’s problem as long as it runs in-process.

Resize the pool against reality

If MaxRequestWorkers x real per-child PSS exceeds 70% of RAM, lower MaxRequestWorkers. A leaky child at 200MB RSS with 256 workers is a 50GB promise on a machine that cannot keep it.

Upgrade httpd when the leak is in the server

If the evidence points at HTTP/2 handling rather than application code, upgrade to the current 2.4.x and re-measure. This is the one case where “patch it” is the correct first fix.

Do not lead with a full restart. It resets RSS to baseline and destroys your evidence, and the leak restarts with the first request.

Prevention

  • Set MaxConnectionsPerChild on every deployment with embedded interpreters. The unlimited default is wrong for mod_php and mod_perl. Treat 5000-10000 as the starting band, tuned by observed RSS growth.
  • Trend per-child RSS and PSS over days, not just current levels. Leaks are invisible to threshold alerts until the terminal phase. The slope is the signal.
  • Size MaxRequestWorkers from measured PSS. Recompute after every significant application or module change, since per-child cost changes with your code.
  • Watch swap as an early warning. Any sustained swap usage by httpd children means you are already on the degradation curve, before OOM kills begin.
  • Keep httpd current on the 2.4 line. Memory-leak CVEs in mod_http2 are recurring. Pinning an old minor version re-introduces known leaks.

How Netdata helps

  • Per-process memory charts track each httpd child’s RSS individually over time, so the monotonic per-PID slope that defines this pattern is visible days before the OOM kills start, not after.
  • Correlating per-child memory with system-level MemAvailable and swap usage shows where you are on the degradation curve: page cache reclamation, swapping, or imminent OOM.
  • Apache worker utilization (BusyWorkers/IdleWorkers from mod_status) alongside memory tells you whether growing memory tracks real load or grows at constant traffic, which is the leak signature.
  • Child process respawn events and uptime resets surface the kill-and-respawn spiral once OOM begins, and distinguish “children recycling by design” from “children dying.”
  • Long retention on per-PID trends lets you compare this week’s growth rate to last week’s after a config or code change, which is how you confirm a fix actually stopped the leak.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.