$ guides / apache-httpd / apache-httpd-graceful-reload-stale-config ▌

Operations Guides

Apache graceful reload ran the old config: the silent stale-configuration trap

You edited a vhost, ran apachectl graceful, saw no error, and moved on. Hours later someone notices the new TLS certificate is not being served, the new redirect is missing, or the old ProxyPass target is still receiving traffic. Apache never went down and no alert fired. The reload failed and the old configuration kept running.

This is one of Apache’s worst failure modes because nothing looks broken. The server keeps serving and request rates stay normal. The only thing wrong is that the running configuration is not the configuration on disk, which quietly invalidates every assumption you make while debugging the next incident.

This guide covers why graceful reloads silently fail, how to confirm whether the running config is stale, and how to make reloads verifiable instead of assumed.

What this means

apachectl graceful sends SIGUSR1 to the parent process. The parent re-reads the configuration, starts new children with the new config, and lets old children finish in-flight requests before exiting. That is the design. The trap is in the failure handling:

On some distributions, apachectl graceful does not run a configuration check before signalling the parent. This is distribution-dependent, not universal.
Even when a syntax check runs, it does not catch everything. Runtime errors, such as a certificate file that does not exist, a module that fails to load, or a directive that is syntactically valid but semantically broken, only surface when the new children try to start.
When new children fail to start, the old children keep serving with the old configuration. The reload is effectively ignored, and the operator who ran the command sees a clean exit and believes the change is live.

There is a second trap in the opposite direction: on some Ubuntu/Debian releases, apachectl graceful was remapped to delegate to systemctl in ways that changed the semantics entirely (in some versions it triggered a full restart rather than a graceful one). The apachectl wrapper is patched per distribution, so what graceful does on your host is a property of your distro’s packaging, not of upstream Apache. Verify locally instead of assuming.

flowchart TD
  A[Operator runs apachectl graceful] --> B{Config check run?}
  B -->|Yes, passes| C[Parent signals new children to start]
  B -->|No - distro dependent| C
  C --> D{New children start OK?}
  D -->|Yes| E[Old children drain, new config live]
  D -->|Runtime error| F[New children fail to start]
  F --> G[Old children keep serving OLD config]
  G --> H[Operator believes change is live - silent staleness]

Common causes

Cause	What it looks like	First thing to check
Runtime error not caught by syntax check	`configtest` says “Syntax OK” but new children fail on start	Error log between “configured” and “resuming normal operations”
Distro `apachectl` wrapper skips the pre-check	Reload returns success regardless of config validity	Read the wrapper: `less $(which apachectl)`
Include wildcard pulls in a stray .conf file	Unexpected directives active; or expected config silently missing	Enumerate what is actually included on disk
SSL certificate or key problem in new config	HTTPS vhost fails in new generation, old one keeps serving	Error log for SSL module errors after reload time
Graceful used where a full restart is required	New setting (e.g. certain limits) never takes effect	Know which changes need restart, not reload
Overlapping reloads pile up old generations	Multiple “resuming normal operations” close together, memory climbing	Restart frequency in the error log

Quick checks

All read-only except check 8, which is called out below.

# 1. Validate the on-disk config right now
apachectl configtest 2>&1

# 2. Look at what the last reload actually did
grep -E "resuming normal operations|Syntax error|configuration error" \
  /var/log/apache2/error.log 2>/dev/null | tail -10
# RHEL path: /var/log/httpd/error_log

# 3. Show the window leading up to the most recent reload completing
grep -B30 "resuming normal operations" /var/log/apache2/error.log | tail -40

# 4. Check restart/reload cadence - pile-ups hide failures
grep -E "resuming normal operations|caught SIGTERM" \
  /var/log/apache2/error.log | tail -20

# 5. How long has this generation been running?
curl -s http://localhost/server-status?auto | grep -E "ServerUptimeSeconds|RestartTime"

# 6. How many Apache generations are actually alive?
pgrep -c 'httpd|apache2'
ps -C httpd -o pid,ppid,lstart,cmd --sort=lstart 2>/dev/null || \
  ps -C apache2 -o pid,ppid,lstart,cmd --sort=lstart

# 7. Scoreboard: lingering G states mean old generation still draining
curl -s http://localhost/server-status?auto | grep Scoreboard | \
  sed 's/Scoreboard: //' | fold -w1 | sort | uniq -c | sort -rn

# 8. See exactly what your distro's wrapper does on graceful
sudo sh -x $(which apachectl) graceful 2>&1 | head -40

Warning on check 8: it performs a real graceful reload because it executes the wrapper. Run it during a low-risk window, or read the script with less $(which apachectl) instead if you only want to inspect the logic.

How to diagnose it

Establish ground truth: does the on-disk config parse? Run apachectl configtest. If it fails, the on-disk config is broken and the running config is definitely stale. Fix the config before anything else.
Establish ground truth: is the running config the on-disk config? “Syntax OK” only proves the file parses. It does not prove the running processes loaded it. Find the most recent “resuming normal operations” entry in the error log and read every line between the preceding “configured”/signal messages and it. Any module load failure, certificate error, or child spawn failure in that window means the new generation failed.
Check process start times against your reload time. A graceful reload does not reset ServerUptimeSeconds; only a full restart does. What proves a new generation started is child process start times (check 6) that postdate your reload. If all children predate your reload, no new generation ever started.
Inspect the wrapper. On systemd-based distributions, apachectl is often a patched shell script. Confirm whether graceful maps to a real httpd -k graceful, to systemctl reload, or to something else. Behavior differs between Debian/Ubuntu and RHEL families, and has changed across releases within the same distro.
Enumerate includes. Include and wildcard include directives mean “the config” is whatever files exist on disk at reload time. A stray file dropped into an included directory by a package, a colleague, or an old deployment script becomes part of the config silently. List what the wildcards actually match and diff against what you expect.
Verify behaviorally, not just structurally. After any reload, probe the thing you changed: request the new vhost, check the served certificate (openssl s_client against the live listener), hit the new redirect. This is the only check that proves the running config is the intended config.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Error log between “configured” and “resuming normal operations”	This is where silent reload failures are recorded	Any [error] or [crit] lines in that window
“resuming normal operations” frequency	Reload cadence and pile-up detection	Repeated entries close together; entries you did not trigger
`ServerUptimeSeconds` and child process start times	Proves whether a new generation actually started	Children older than the last intended reload
Scoreboard `G` states	Old generation draining after reload	`G` states persisting long after the reload
Process count vs. expected children	Multiple generations overlapping multiplies memory	Process count climbing across successive reloads
Behavioral probe of the changed directive	The only proof the change is live	Probe returns old behavior after reload reported success

Fixes

The config on disk is broken

Fix the config, then reload. Never reload on hope. Make apachectl configtest a hard gate:

# Gate the reload on configtest - safe pattern for scripts and humans
apachectl configtest && apachectl graceful

If configtest passes but the reload still silently fails (runtime error), the error log window from the reload is your diagnostic. Fix the underlying issue: missing certificate file, module load failure, a listen address already bound, or similar.

The distro wrapper behaves unexpectedly

Bypass the wrapper and signal Apache directly:

# Direct graceful, bypassing distro wrapper logic
apachectl -k graceful
# or equivalently, signal the parent yourself
sudo kill -USR1 $(cat /var/run/apache2/apache2.pid)   # RHEL: /var/run/httpd/httpd.pid

Check your distribution’s documentation for what systemctl reload apache2 (or httpd) actually executes; on systemd hosts the unit’s ExecReload is the authoritative definition.

The change requires a full restart, not a reload

Some changes do not take effect through a graceful reload. Directives tied to process limits (for example ServerLimit, and raising file-descriptor limits via systemd LimitNOFILE) require a full restart. Certificate files are read by mod_ssl at server startup and are not re-read during normal operation; for certificate changes, use the restart required by your TLS configuration and verify the live certificate rather than assuming every graceful reload reloads it. A full restart drops active connections, so schedule it:

# Disruptive: drops all in-flight connections
apachectl configtest && systemctl restart apache2   # or: httpd on RHEL

Old generations piling up

If you see many G states and memory climbing across reloads, reduce reload frequency and bound the drain time with GracefulShutdownTimeout (for example 30 seconds) so old workers cannot linger indefinitely.

Prevention

Gate every reload on configtest. apachectl configtest && apachectl graceful, in scripts, CI/CD, and config management. Non-negotiable.
Read the error log window after every reload. Automate it: capture the log position before the reload, print everything up to and including “resuming normal operations”, and fail the deploy on any [error] or [crit] lines.
Verify behaviorally. After the reload, probe the specific change you made. Treat “command exited 0” as no evidence at all.
Know your wrapper. Once per host image, inspect what apachectl graceful and systemctl reload actually do. Record it in the runbook. Do not assume upstream behavior on a packaged system.
Audit include directories. Wildcard includes mean any file in the directory becomes configuration. Restrict write access, and alert on unexpected files appearing in included paths.
Bound old-generation lifetime. Set GracefulShutdownTimeout so stuck requests cannot keep stale generations alive forever.
Alert on reload staleness. If a config file’s mtime is newer than the last “resuming normal operations” entry plus a margin, someone changed config without a verified successful reload.

How Netdata helps

Uptime and restart tracking: Netdata charts Apache uptime from mod_status over time, so an unexpected restart, or a reload that never produced a new generation, is visible as a discontinuity instead of a log line nobody read.
Scoreboard state history: persistent G states and generation overlap show up as trends, catching graceful-restart pile-ups and drains that never finish.
Log correlation: Netdata’s web log monitoring puts request-level signals and error cadence on the same timeline as the reload event, which is exactly the window this failure hides in.
Process and memory per generation: per-process RSS and process counts expose overlapping generations multiplying memory after repeated reloads.
Behavioral verification: correlating response codes and latency against the reload timestamp helps confirm the new config is actually serving, not just that the reload command exited cleanly.

Netdata’s Apache HTTP Server monitoring brings these signals together with per-second metrics and anomaly detection so a silently stale configuration surfaces as an observable event rather than a 3 a.m. discovery.

The Netdata solution

Apache HTTP Server monitoring with Netdata

Netdata monitors Apache HTTP Server with per-second metrics from mod_status, pre-built dashboards, and ML-powered anomaly detection. Watch busy versus idle workers and the scoreboard state mix, requests per second, bytes served per second, and request processing duration alongside the rest of your stack, so you catch the worker-exhaustion, slow-backend, and memory incidents in these runbooks before they page anyone.

See web server monitoring → Start monitoring free