Apache killed by the OOM killer: the memory exhaustion cascade

Your monitoring says Apache is up. The parent process exists, the port is listening, but the site is down or intermittently dead. When you finally check the kernel log, there it is: Out of memory: Killed process ... (httpd). Not once. Dozens of times, at regular intervals, going back hours.

This is the Apache OOM death spiral. The kernel OOM killer terminates httpd child processes because total memory demand exceeded RAM. The Apache parent survives, respawns the children, the new children immediately start serving queued requests and allocating memory, and the OOM killer kills them again. The service looks “running” while serving little or nothing.

The defining operational fact of this failure: the kill never appears in the Apache error log. The process was killed externally by the kernel, so the evidence lives in dmesg and the kernel journal, not in /var/log/apache2/error.log or /var/log/httpd/error_log. Operators who only watch Apache’s own logs can stare at a dying server for a long time without seeing a single error line.

This guide covers how to confirm the pattern, how to tell the two root causes apart, and how to fix the sizing so it does not come back.

What this means

Apache’s memory exhaustion archetype is simple arithmetic: MaxRequestWorkers x per-child RSS > available RAM. When every worker slot is occupied and each child process holds its real resident memory, total Apache memory exceeds physical memory. The degradation curve is gradual, then a cliff. First the kernel reclaims page cache, then it starts swapping (a swapped-out child takes 10-100x longer to respond when paged back in), and finally the OOM killer activates.

Two details make this worse than a generic OOM:

1. The OOM killer usually targets children, not the parent. Its heuristic favors the task using the most allowed memory, adjusted by oom_score_adj and eligibility; the largest consumer is usually an httpd child. The parent normally survives, but selection is not guaranteed. You can have a “running” Apache with zero functional children.
2. The parent respawns what the kernel kills. Each respawned child picks up queued work, allocates memory, and becomes the next OOM candidate. Without intervention this loop runs indefinitely.

flowchart TD
  A[MaxRequestWorkers x per-child RSS exceeds RAM] --> B[Kernel reclaims page cache]
  B --> C[Swapping begins, latency spikes 10-100x]
  C --> D[OOM killer terminates httpd children]
  D --> E[Parent respawns children]
  E --> F[New children serve queued requests, allocate memory]
  F --> D
  D -.->|evidence in dmesg / journalctl -k, not Apache error log| G[Operator sees running parent, dead service]

There are two root causes, and the fix differs for each:

• Sizing error. MaxRequestWorkers is set higher than memory can support. This is the most common Apache misconfiguration: 1000 workers on a 4GB box with mod_php children at 50MB each implies 50GB of demand. Under a traffic burst, the math catches up.
• Per-child leak. A module (commonly mod_php, mod_perl, or a custom module) leaks memory per request. Child RSS grows monotonically over hours or days until the OOM kills start. MaxConnectionsPerChild 0 (the default, meaning children never recycle) is almost always present in this pattern.

Common causes

If Apache shares the host with other large consumers (a database, a JVM), the OOM badness heuristic tends to select the task using the most allowed memory, adjusted by oom_score_adj and eligibility; that task may not be the process that caused the pressure. Check what was actually growing before assuming Apache is the culprit.

Quick checks

All read-only. Run these first.

# 1. Confirm OOM kills happened (the kill is NOT in the Apache error log)
dmesg -T | grep -i -E "out of memory|oom-kill|killed process" | tail -20

# 2. Kernel journal equivalent, useful when dmesg has rotated
journalctl -k | grep -i oom | tail -20

# 3. Current memory state
grep -E "MemTotal|MemAvailable|SwapTotal|SwapFree" /proc/meminfo

# 4. Per-child RSS, sorted by largest (use apache2 on Debian/Ubuntu)
ps -C httpd -o pid,rss,vsz,cmd --sort=-rss 2>/dev/null || \
  ps -C apache2 -o pid,rss,vsz,cmd --sort=-rss

# 5. Average and count
ps -C httpd -o rss --no-headers 2>/dev/null || ps -C apache2 -o rss --no-headers | \
  awk '{sum+=$1; count++} END {print "Avg RSS (KB):", sum/count, "Count:", count}'

# 6. Which MPM is active (prefork vs worker vs event changes the whole interpretation)
apachectl -V 2>/dev/null | grep -i mpm || httpd -V | grep -i mpm

# 7. Parent alive but children missing? The classic "zombie healthy" state
pgrep -o 'httpd|apache2'          # oldest PID = parent
pgrep -c 'httpd|apache2'          # total process count

# 8. Configured limits actually in effect
grep -rE "MaxRequestWorkers|MaxConnectionsPerChild|ServerLimit|MaxClients" \
  /etc/apache2/ /etc/httpd/ 2>/dev/null | grep -v "^.*#"

# 9. Recent graceful restarts (pile-up suspect)
grep -c "resuming normal operations" /var/log/apache2/error.log 2>/dev/null || \
  grep -c "resuming normal operations" /var/log/httpd/error_log

# 10. Worker saturation right now
curl -s http://localhost/server-status?auto | grep -E "BusyWorkers|IdleWorkers"

Note that check 3 uses MemAvailable, not MemFree. MemFree ignores reclaimable cache and will make a healthy system look closer to OOM than it is.

How to diagnose it

1. Confirm the kill. Step 1 or 2 above must show httpd/apache2 children being killed. If there are no OOM lines, this is a different incident; stop here. If the killed PIDs are Apache children and the parent PID never appears in the kill list, you have the standard pattern.

2. Reconstruct the timeline from the kernel log. Look at kill frequency. Kills only during traffic peaks point to a sizing error. Kills arriving at a steady cadence regardless of load, with kill timestamps getting closer together over days, point to a leak.

3. Snapshot per-child RSS now and again in an hour. A sizing error shows children clustered around a stable RSS. A leak shows per-PID RSS climbing monotonically. The playbook pattern is unambiguous: plot RSS per PID over time, and a leak is a rising line per process.

4. Do the arithmetic. Worst case Apache memory = MaxRequestWorkers x max_observed_child_RSS. Compare against RAM. If the result exceeds roughly 70% of total RAM, the configuration cannot survive a full worker pool. For prefork with mod_php, 50-100MB per child is a common starting range; validate with your own measurements, not the estimate.

5. Check MaxConnectionsPerChild. If it is 0 (unlimited, the default), children never recycle and any leak compounds forever. Its presence at 0 does not prove a leak, but its absence as a bound is what turns a small leak into a terminal event.

6. Rule out restart pile-up. If memory spikes line up with deploys, config management runs, or log rotation, and the scoreboard shows many G (gracefully finishing) states, old and new generations of children are overlapping. Each overlapping generation multiplies memory. Frequent graceful restarts plus slow requests can OOM a correctly sized server.

7. Check systemd-level limits. If httpd runs under systemd, MemoryMax in the unit file caps the service regardless of Apache config, and TasksMax caps total processes and threads. A cgroup limit below Apache’s theoretical demand produces OOM kills with apparently free system memory. Check the unit file before blaming the Apache config.

Metrics and signals to monitor

Fixes

Right-size MaxRequestWorkers against memory

Derive the limit from memory, not from a guess: MaxRequestWorkers = memory_available_to_Apache / per_child_RSS, using the largest observed child RSS, not the average. Keep the theoretical maximum at or below 70% of total RAM on a dedicated host, leaving room for the OS, page cache (which matters for static file performance), and anything else on the box. This is covered in depth in the MaxRequestWorkers tuning guide.

Tradeoff: a lower worker ceiling means traffic bursts queue in the listen backlog instead of being served. That is the correct failure mode. Queued connections are recoverable; an OOM death spiral is not.

Bound the leak with MaxConnectionsPerChild

Set MaxConnectionsPerChild to a finite value (5000-10000 is the playbook range) so children recycle periodically and leaked memory is reclaimed. This bounds any leak but does not fix it.

Tradeoff: recycling children costs a small amount of churn (fork/startup work) and briefly raises process count during replacement. It is cheap insurance. Leaving it at the default of 0 is wrong for any deployment running mod_php or mod_perl.

Actually find the leak

Compare RSS growth across children serving different URL patterns to isolate which module or application path leaks. If PHP is involved, enforce a memory_limit in PHP itself, and seriously consider moving PHP out of the httpd process to PHP-FPM, which both shrinks per-child RSS dramatically and lets you recycle PHP workers independently of Apache.

Reduce per-child memory footprint

If you are on prefork only because of mod_php, moving PHP to PHP-FPM unlocks the event MPM, the default MPM in Apache 2.4 on systems supporting threads and thread-safe polling. Event handles keepalive connections in a listener thread instead of tying up workers, and threaded children amortize memory across many connections. Prefork children carrying an embedded interpreter (10-50MB+ each, 50-100MB with mod_php) are the most common way servers end up sized into OOM territory.

Stop restart pile-ups

Reduce whatever is triggering frequent graceful restarts (over-eager config management, aggressive log rotation), and set GracefulShutdownTimeout (for example 30 seconds) so old-generation children cannot linger indefinitely holding memory alongside the new generation.

Prevention

• Treat MaxRequestWorkers as a derived value. Recompute it whenever per-child RSS changes materially: new application version, new module, PHP framework upgrade.
• Never run with MaxConnectionsPerChild 0 when embedded interpreters are loaded.
• Trend per-child RSS and MemAvailable. A leak gives you days of warning if you are watching; the OOM kill should never be the first signal you see.
• Watch swap. Any sustained swap usage by Apache children is a pre-OOM warning, not a curiosity.
• Alert on kernel OOM lines. Grep-class monitoring of dmesg/journalctl for oom-kill and Killed process catches this even when Apache’s own logs are silent.
• Deploy discipline. Avoid back-to-back graceful restarts under load; the old-plus-new overlap briefly multiplies memory.

How Netdata helps

• Netdata charts per-process RSS for httpd/apache2 children over time, which is exactly the trend that separates a slow leak from a sizing error, without you having to snapshot ps by hand.
• System memory context (MemAvailable, swap in/out) is collected on the same second-level timeline as Apache’s worker metrics, so you can see swap growth and worker saturation converge before the first kill.
• The Apache collector tracks BusyWorkers and IdleWorkers from mod_status, letting you correlate a full worker pool with the memory climb that follows.
• Netdata’s logs and alerting can watch the kernel journal for OOM kill events, closing the blind spot where the Apache error log stays clean while children die.
• Restart events and scoreboard state changes are visible alongside memory, which makes graceful restart pile-ups (many G states plus a memory spike) easy to spot after the fact.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• Apache MaxRequestWorkers tuning: sizing the worker pool against memory
• Apache AH00484: server reached MaxRequestWorkers setting - worker pool exhausted
• Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status
• Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion
• Apache keepalive consuming workers: KeepAliveTimeout, the K state, and MPM choice
• How Apache HTTPD actually works in production: a mental model for operators
• Apache HTTPD monitoring checklist: the signals every production web server needs