Apache unexpected restarts: SIGTERM vs SIGUSR1, crashes, and OOM respawns

mod_status shows ServerUptimeSeconds at 340 again, and it was 340 an hour ago too. Something is restarting Apache, and unless you deployed at that exact moment, it is not you. The error log says resuming normal operations every few minutes, or caught SIGTERM, or nothing at all between restarts. Each signature points at a different failure mode.

Apache restarts fall into three buckets: intentional graceful reloads (SIGUSR1), hard restarts (SIGTERM or SIGHUP, often from logrotate or config management), and crash respawns (child segfaults, OOM kills, or the parent dying and being restarted by systemd or a wrapper script). The first is usually harmless. The second drops every in-flight connection. The third means something is actively wrong and will keep happening until you find it.

What this means

Apache’s parent process never serves requests. It reads configuration, opens log files, binds ports, and manages a pool of child processes (prefork) or process/thread hybrids (worker, event). When the parent receives a signal, what happens to your traffic depends entirely on which signal:

• SIGUSR1 (apachectl graceful, systemctl reload httpd): the parent re-reads configuration and log files, then tells existing children to exit after finishing their current request. New-generation children spawn alongside the old ones. No connections are dropped. Old children show as G (gracefully finishing) in the scoreboard, and mod_status counters are not reset.
• SIGHUP (apachectl restart): the parent kills all children like SIGTERM but does not exit itself. It re-reads configuration and starts a fresh generation. All in-flight connections die. mod_status statistics are reset to zero.
• SIGTERM (apachectl stop, or anything running kill <pid>): the parent kills all children immediately, terminating in-progress requests, then exits. If something restarts the service afterward (systemd, a container entrypoint, a cron job), you see this as an “unexpected restart.”
• Crash / OOM kill: no signal at all. The kernel OOM killer or a segfault takes out a child (logged as child pid NNNN exit signal Segmentation fault (11)) or, worse, the parent. A dead parent means full service loss until whatever supervises it starts a new one.

Two consequences follow hard restarts. First, the mod_ssl shared-memory session cache is wiped, so the next few minutes bring a burst of full TLS handshakes and a CPU spike, which is expensive on RSA key exchange. (The shmcb cache survives a graceful reload; clients using session tickets may also resume without server state, so the spike’s size depends on your TLS setup.) Second, every lifetime counter in mod_status (ReqPerSec, DurationPerReq, CPULoad) restarts its averaging window, so post-restart numbers are not comparable to pre-restart ones. A graceful reload resets neither, which is one more reason to prefer it.

flowchart TD
  A[Uptime reset detected] --> B{Error log at restart time}
  B -->|"caught SIGTERM"| C[Hard restart
who sent it? logrotate, cron, config mgmt]
  B -->|"resuming normal operations
no SIGTERM"| D[Graceful reload SIGUSR1
deploy, logrotate, config mgmt]
  B -->|"exit signal Segmentation fault"| E[Child crash
module bug, often mod_php]
  B -->|"nothing in Apache log"| F[External kill
check dmesg for OOM killer]
  C --> G[Fix the trigger]
  D --> H[Check frequency
pile-up risk if overlapping]
  E --> I[Identify module
bound with MaxConnectionsPerChild]
  F --> J[Fix memory sizing
MaxRequestWorkers x RSS vs RAM]

Common causes

Quick checks

# 1. Current uptime: has the parent restarted recently?
curl -s http://localhost/server-status?auto | grep -E "ServerUptimeSeconds|RestartTime"

# 2. Restart history from the error log (both distro paths)
grep -E "resuming normal operations|caught SIGTERM|graceful restart|Segmentation fault" \
  /var/log/apache2/error.log /var/log/httpd/error_log 2>/dev/null | tail -30

# 3. OOM kills: Apache cannot log its own death, the kernel does
dmesg -T | grep -i -E "oom|killed process" | tail -20

# 4. Segfaults attributed to httpd children at the kernel level
dmesg -T | grep -i "segfault" | tail -20

# 5. Who supervises the service and what does it do on failure?
systemctl status httpd 2>/dev/null || systemctl status apache2
systemctl show httpd -p Restart -p OOMPolicy 2>/dev/null

# 6. What is logrotate configured to do after rotating?
cat /etc/logrotate.d/httpd 2>/dev/null; cat /etc/logrotate.d/apache2 2>/dev/null

# 7. Multiple generations piling up? Count running children vs expectations
ps -C httpd -o pid,ppid,rss,etime,cmd --sort=-rss 2>/dev/null | head -20 || \
  ps -C apache2 -o pid,ppid,rss,etime,cmd --sort=-rss | head -20

# 8. Scoreboard state distribution: many G states = overlapping graceful restarts
curl -s http://localhost/server-status?auto | grep "Scoreboard:" | \
  awk '{print $2}' | fold -w1 | sort | uniq -c | sort -nr

All of these are read-only and safe during an incident.

How to diagnose it

1. Establish the restart timeline. Pull ServerUptimeSeconds from mod_status and the restart messages from the error log. Build a list of timestamps. If restarts cluster at fixed times (02:00, every hour on the hour), suspect cron, logrotate, or a config management run before you suspect crashes.

2. Classify each restart by its log signature. resuming normal operations alone means a graceful reload (SIGUSR1) or a SIGHUP restart; caught SIGTERM means a hard stop; exit signal Segmentation fault before a restart means a crash; silence in the Apache log combined with an uptime reset means an external kill, and your next stop is dmesg for the OOM killer.

3. If graceful reloads: check frequency and overlap. More than a handful of resuming normal operations entries per day, with no matching deploys, means an automated trigger is firing too often. Check whether old generations drain: many G states in the scoreboard plus elevated memory means slow requests are holding old children alive across restarts. That is the graceful restart pile-up pattern, and it multiplies memory by the number of overlapping generations.

4. If segfaults: identify the module. On prefork, a segfault kills only that child and the parent respawns it; on worker/event, a crashing thread can take the whole process and its threads with it. Segfaults in Apache children are almost always a loaded module, not Apache core. mod_php, mod_perl, and mod_security are the usual suspects. Some segfaults during shutdown are benign; segfaults mid-traffic are not.

5. If OOM kills: do the memory math. The OOM killer targets children first, so you can have a “running” Apache whose children die in cycles. Compute MaxRequestWorkers x average child RSS and compare to RAM. If the product exceeds roughly 70% of total RAM, the configuration guarantees OOM under full load regardless of what else you tune. Then check MaxConnectionsPerChild: at its default of 0 (unlimited), a leaky module grows each child until the kernel intervenes.

6. If nothing matches: check the supervisor. A systemd unit, container entrypoint, or monitoring script may be killing and restarting the service. On systemd, systemctl status shows recent restarts and the unit’s Restart= policy. The stock RHEL httpd unit sets OOMPolicy=continue, so a single OOM-killed child does not terminate the whole service; the behavior when the parent itself dies depends on the distro’s Restart= setting.

Metrics and signals to monitor

Fixes

Hard restarts from logrotate or scripts

The correct logrotate pattern for Apache rotates the file and then triggers a graceful reload so Apache reopens its logs, not a hard restart. If your logrotate config runs kill -HUP, apachectl restart, or /bin/systemctl restart, replace it with a graceful reload, or move to piped logging with rotatelogs, which needs no restart at all. Avoid copytruncate: it can lose log lines racing Apache’s writes.

Excessive graceful reloads

Find the trigger and slow it down. Config management agents running every few minutes, CI pipelines reloading per-change, and multiple logrotate configs each issuing a reload are the common offenders. Batch config changes so one reload applies many edits. If slow requests keep old children alive across reloads, set GracefulShutdownTimeout (default is 0, meaning wait indefinitely) to bound how long old generations linger. This forcibly drops requests still in flight when the timeout hits, so size it against your longest legitimate request.

Child segfaults

• Isolate the module. On prefork with mod_php, the strongest structural fix is moving PHP to PHP-FPM, which removes the PHP runtime from Apache children entirely. The same logic applies to mod_perl.
• Update everything in lockstep. Segfaults frequently appear after an Apache or module package update where a third-party module was not rebuilt against the new version. Rebuild or reinstall the module.
• Confirm it is mid-request. Segfaults logged during shutdown are often benign. Correlate crash timestamps with request volume before treating it as your incident cause.

OOM respawns

• Size MaxRequestWorkers from memory, not hope. MaxRequestWorkers = (RAM available to Apache) / (average child RSS). Measure average RSS under real load with ps, do not guess. For prefork with mod_php, children commonly land in the 50-100MB range.
• Bound the leak. Set MaxConnectionsPerChild to a finite value (5000-10000 is a common range) so children recycle before a leaky module grows them into OOM territory. This masks the leak; it does not fix it. Track down the leaking module in parallel.
• Check the systemd cage. If httpd runs under systemd, MemoryMax and TasksMax in the unit can impose limits tighter than your Apache math assumes. A cgroup OOM kill looks identical to a system-wide one in its effect on Apache.

Failed reloads with stale config

Always run apachectl configtest before reloading in production. A graceful reload that fails the config check leaves the old configuration running, and depending on the distribution the reload attempt may not validate first. Worse, errors that only surface at runtime (not caught by the syntax check) can make new children fail to start while old ones keep serving a shrinking pool. Watch the error log between the “configured” and “resuming normal operations” lines after every reload.

Prevention

• Alert on uptime resets. ServerUptimeSeconds resetting is the cheapest possible restart detector. More than one unexpected (non-graceful) restart per day is a ticket; zero is the expectation.
• Gate restarts behind change windows. Graceful reloads should correlate 1:1 with deploys or config changes. A reload with no matching change is an investigation, not noise.
• Set MaxConnectionsPerChild on any deployment running embedded interpreters (mod_php, mod_perl). The default of 0 is the wrong default there.
• Set GracefulShutdownTimeout to a finite value so stuck old generations cannot accumulate memory across reloads.
• Fix logrotate to use graceful reloads or piped logging, and never copytruncate.
• Keep the memory budget written down. MaxRequestWorkers x measured child RSS < 70% of RAM. Recheck it after every module or PHP version change, because per-child RSS changes with them.
• Expect the post-restart handshake burst. After a hard restart, the SSL session cache is empty and CPU spikes from full TLS handshakes. If restarts are frequent, this recurring spike is a real capacity cost, not a curiosity.

How Netdata helps

Netdata’s Apache collector scrapes mod_status continuously, which turns restart detection from a log-grep exercise into a timeline you can correlate:

• ServerUptimeSeconds over time shows every restart as a drop to zero, timestamped precisely enough to line up with deploy pipelines, cron, and logrotate windows.
• BusyWorkers, IdleWorkers, and the scoreboard are sampled at high resolution, so you can see G states from old generations draining, or piling up, across overlapping graceful reloads.
• Per-process memory for httpd/apache2 children exposes the monotonic RSS growth of a leaky module days before the OOM killer gets involved, and shows the transient double-memory footprint of old plus new generations during reloads.
• CPU per process makes the post-restart TLS handshake burst visible and lets you quantify what frequent restarts actually cost.
• Correlating uptime resets against memory, CPU, and worker states in one view is what separates “logrotate did it” from “a leaky module did it” in minutes instead of an afternoon of grep.

Netdata’s Apache HTTP Server monitoring brings these signals together with per-second metrics and anomaly detection.

Related guides

• Apache 500 Internal Server Error: modules, handlers, and misconfiguration
• Apache 502 Bad Gateway: a backend that returned an invalid response
• Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion
• Apache 504 Gateway Timeout: slow backends, ProxyTimeout, and worker pile-up
• Apache 5xx error rate: 500 vs 502 vs 503 vs 504 and what each one means
• Apache AH00558: Could not reliably determine the server’s fully qualified domain name
• Apache backend response time: telling ‘Apache is slow’ from ’the backend is slow’
• Apache balancer member in error state: reading balancer-manager and failover
• Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status
• Apache SSL certificate expired: the total, preventable HTTPS outage
• Apache CLOSE_WAIT and TIME_WAIT: connection leaks versus normal churn
• Apache CPU saturation: TLS handshakes, mod_deflate, mod_rewrite, and mod_security