Apache graceful restart pile-up: G states, stacked generations, and doubled memory

Memory on an Apache host climbs in steps, each step landing a few minutes after a deploy, a config push, or a log rotation. The scoreboard shows an unusual number of G (gracefully finishing) workers. ps shows far more httpd children than MaxRequestWorkers should allow. Nothing is erroring yet, but the host is drifting toward swap, and the next restart will make it worse.

This is the graceful restart pile-up pattern. Each graceful restart spawns a new generation of children while the old generation drains. Old children do not exit until their in-flight requests complete. If requests are slow (large downloads, slow proxied backends, long-lived connections), the old generation lingers. If another graceful restart arrives before the previous generation has drained, you now have three generations of children, each holding memory. Stack enough generations and the host OOMs.

The pattern is usually self-inflicted: CI/CD pipelines that reload on every deploy, configuration management agents that restart Apache every run, or log rotation that signals Apache too aggressively. The server itself is healthy. The restart cadence is the bug.

What this means

A graceful restart (apachectl graceful, which sends SIGUSR1 to the parent) tells Apache to re-read its configuration, re-open its logs, and replace its children without dropping connections. The parent starts new children on the new configuration and advises old children to exit after their current request finishes. In the scoreboard, old-generation workers still serving requests show as G, gracefully finishing.

The key property: there is no hard deadline by default. GracefulShutdownTimeout defaults to 0, which means wait indefinitely. An old child serving a slow request, a large file transfer, or a long-lived proxied connection stays alive, and keeps its memory, for as long as that request lasts.

One graceful restart with slow requests gives you a brief overlap of old and new children. That is normal: G states during a graceful restart are expected, and process count may temporarily exceed MaxRequestWorkers. The failure mode starts when restarts arrive faster than generations drain. Memory then scales with the number of stacked generations, not with the configured worker limit.

flowchart TD
  A[CI/CD deploy, config run, or logrotate] -->|SIGUSR1 graceful restart| B[Parent spawns new generation children]
  B --> C[Old generation children enter G state]
  C --> D{In-flight requests complete?}
  D -->|Yes| E[Old children exit, memory freed]
  D -->|No: slow requests, large transfers, long-lived connections| F[Old children linger with full RSS]
  F -->|Another graceful restart arrives| B
  F -->|Generations stack| G[Process count exceeds MaxRequestWorkers]
  G --> H[Memory = generations x children x per-child RSS]
  H --> I[Swap, then OOM kills]

Common causes

Quick checks

All of these are read-only.

# Count scoreboard states; a pile-up shows many G entries
curl -s http://localhost/server-status?auto | grep "Scoreboard:" | \
  awk '{print $2}' | fold -w1 | sort | uniq -c | sort -nr

# Busy and idle workers; G states count toward BusyWorkers
curl -s http://localhost/server-status?auto | grep -E "BusyWorkers|IdleWorkers|ServerUptimeSeconds"

# Restart frequency: repeated entries close together confirm the pile-up trigger
grep -E "resuming normal operations|caught SIGTERM|graceful restart" \
  /var/log/apache2/error.log /var/log/httpd/error_log 2>/dev/null | tail -20

# Process count and memory per child; count exceeding MaxRequestWorkers
# (as child processes or threads, depending on MPM) indicates stacked generations
ps -C httpd -o pid,ppid,rss,cmd --sort=-rss 2>/dev/null || \
  ps -C apache2 -o pid,ppid,rss,cmd --sort=-rss

# Total Apache memory footprint; -C accepts both names so this works
# on Debian (apache2) and RHEL (httpd) families
ps -o rss --no-headers -C httpd -C apache2 2>/dev/null | \
  awk '{sum+=$1; count++} END {printf "Total RSS (MB): %.0f  Processes: %d\n", sum/1024, count}'

# Confirm the last reloads actually applied cleanly
apachectl configtest 2>&1

Two things to note while reading output. First, BusyWorkers includes all non-idle states, so G workers inflate it; do not mistake a G-heavy BusyWorkers reading for genuine traffic saturation. Second, during a graceful restart the process count can legitimately exceed MaxRequestWorkers for a short window. The alarm condition is that state persisting, not its existence.

How to diagnose it

1. Confirm the restart cadence. Pull the “resuming normal operations” lines from the error log and look at the gaps. Restarts minutes apart, or restarts aligned with deploy or cron schedules, are the trigger. One restart per day with lingering G states is a slow-request problem, not a cadence problem.

2. Quantify the G population. Run the scoreboard histogram above a few times over several minutes. G workers that appear, drain, and disappear are normal. G workers that persist across samples, or a G count that grows with each restart, are the pile-up.

3. Identify what the G workers are doing. The full (non-auto) server-status page shows, per slot, the request each worker is handling. Look for patterns: large file downloads to slow clients, proxied requests to a slow backend, or long-lived connections. Remember that %D in the access log includes client transfer time, so a 100MB download to a 1Mbps client legitimately holds a worker for a long time. If your storage or MPM differs from what server-status assumes, the detail columns may differ; the state column is the reliable part.

4. Measure the memory cost. Sum RSS across all httpd processes and compare against the host. A reasonable capacity rule: Apache’s maximum theoretical memory should stay under 70% of RAM. With stacked generations, actual usage can exceed the MaxRequestWorkers x per-child RSS bound you planned for, because that bound assumes one generation.

5. Find the trigger. Match restart timestamps to the automation: deploy pipeline logs, configuration management run logs, cron entries for logrotate. The restart source is almost never Apache itself.

6. Check for OOM pressure. If the host has already tipped over, dmesg will show OOM kills against httpd children. That confirms the pile-up reached its terminal stage and tells you the fix is urgent, not cosmetic.

Metrics and signals to monitor

Fixes

Reduce restart frequency

This is the actual fix. Everything else is a bound on the damage.

• Deploy pipeline batching. Reload once at the end of a deploy, not once per configuration fragment. Each reload spawns a generation, and generations are the cost.
• Configuration management on-change only. Configure the agent to reload Apache only when a managed file actually changed, not on every run. Unconditional restarts at agent intervals are the most common cause of stacked generations.
• Log rotation cadence and method. Rotate less aggressively, and make the postrotate action send SIGUSR1 (graceful), never SIGHUP, which is a hard restart that drops all connections. Avoid copytruncate, which races with Apache’s writes and can lose log lines; use rename plus graceful restart, or piped logging with rotatelogs. Rotation is a scheduled restart, so it sets the floor on restart frequency.

Bound old-generation linger time

Set GracefulShutdownTimeout to a finite value (for example, 30 seconds) so old children cannot wait forever:

GracefulShutdownTimeout 30

The default of 0 means wait indefinitely. One important caveat: operator reports indicate this directive is enforced for graceful-stop but may not be honored during a graceful restart (SIGUSR1) on all versions. Test it against your specific version before relying on it.

Tradeoff: when the timeout does apply, any request still in flight when it fires is cut off. For a host serving large downloads or long-polling clients, 30 seconds may be too short. Set it from your observed P99 request duration, not from a guess.

Remove the reason old requests are slow

If generations linger because requests genuinely take minutes, bounding the restart side only goes so far. Slow proxied backends hold workers in W and hold old generations in G; see the slow backend cascade pattern in Apache backend response time and Apache 504 Gateway Timeout. Large transfers to slow clients are legitimate holds; those argue for a longer drain window and fewer restarts, not for killing requests.

Right-size memory headroom

Re-run the capacity math with the pile-up in mind: worst case is not MaxRequestWorkers x per-child RSS, it is that figure times the number of generations you can plausibly stack. If your restart automation can produce three overlapping generations, either the host must absorb three generations of memory or the restart cadence must change. Usually the cadence changes, because the memory cost is multiplicative.

Prevention

• Alert on restart frequency. More than a handful of graceful restarts per day, outside a known deploy window, is worth a ticket.
• Track G-state duration, not just count. A G worker that drains in seconds is healthy. A G worker alive for ten minutes tells you exactly how long your restart interval must exceed.
• Watch total Apache RSS against the 70% headroom rule. Step increases aligned with restart times catch the pile-up before swap does.
• Gate deploys on drain. If the pipeline must reload, wait for the previous generation’s G count to reach zero before issuing the next graceful restart.
• Validate config before every reload. Run apachectl configtest first. A failed graceful reload leaves the old configuration running silently, and the retry loop some tools enter makes the pile-up worse.

How Netdata helps

• Scoreboard state distribution over time, including the G state as its own series, so a growing or non-draining G population is visible as a trend instead of a snapshot you happened to catch.
• Apache process count and total RSS per host, which makes the multi-generation memory multiplication directly visible and correlatable with restart timestamps.
• Uptime and restart events, so the restart cadence (the trigger) sits on the same timeline as the memory and G-state curves (the effect).
• BusyWorkers and IdleWorkers, which let you separate a G-inflated busy count from genuine traffic saturation when the two look similar on a single metric.
• System memory, swap, and OOM events, so you can see the pile-up approaching its terminal stage and confirm when it has tipped over.

Correlating restart events, G-state duration, and total RSS on one dashboard turns this from “memory is weird after deploys” into a named pattern with a known fix. Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status
• Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion
• Apache 504 Gateway Timeout: slow backends, ProxyTimeout, and worker pile-up
• Apache backend response time: telling ‘Apache is slow’ from ’the backend is slow’
• Apache 5xx error rate: 500 vs 502 vs 503 vs 504 and what each one means
• Apache CLOSE_WAIT and TIME_WAIT: connection leaks versus normal churn
• Apache CPU saturation: TLS handshakes, mod_deflate, mod_rewrite, and mod_security