You found this line in the error log:

[core:notice] [pid 1234] AH00052: child pid 5678 exit signal Segmentation fault (11)

A child process died on a memory access violation. The parent logged the exit status and spawned a replacement. That is Apache’s designed recovery path, and a single occurrence with no user impact is that mechanism working. But a segfault is never “normal noise”: a bug executed in Apache, a loaded module, or a shared library, and any segfault in production deserves root-cause analysis. Multiple per minute means the service is actively degrading.

The message tells you almost nothing on its own. No module name, no request, no stack trace. The work is extracting that missing context: which MPM you run, what the blast radius of one crash is, which module was in the process, and whether the crash correlates with a deploy, a reload, or a specific request pattern.

What this means

Signal 11 (SIGSEGV) is the kernel killing a process that touched memory it does not own: a null dereference, use-after-free, buffer overrun, or bad pointer in C code. Apache core is heavily audited, so in practice the fault usually sits in a loaded module or a library the module pulls in: mod_php and its extensions are the classic offender, but mod_perl, mod_wsgi, third-party modules, and TLS or compression libraries are all candidates.

The blast radius depends on your MPM, which is the first thing to establish:

  • prefork: each child is one process handling one connection. A segfault kills exactly that process and the one request it was serving. The client sees a dropped connection; everyone else is unaffected.
  • worker / event: children are multi-threaded. A segfault in any thread takes down the entire child process and every thread in it. On event with ThreadsPerChild 25, one bad pointer can kill up to 25 in-flight requests at once. Repeated crashes on a threaded MPM show up as capacity loss, not just log noise.

Some segfaults are benign: crashes logged during shutdown or graceful restart, where a module’s cleanup hook dereferences something already freed, are common and harmless. The timestamp relative to “resuming normal operations” or “caught SIGTERM” entries tells you which case you are in.

Common causes

CauseWhat it looks likeFirst thing to check
mod_php or a PHP extensionCrashes correlate with specific PHP endpoints; children grow large before dyingRSS trend per child; which URLs precede crashes in the access log
Buggy third-party/custom moduleCrashes start right after a module was added or upgradedConfig diff and package history; disable the module on a test node
Known httpd CVE in your versionCrashes under specific protocol pressure (e.g., HTTP/2)Compare httpd -v against the Apache 2.4 vulnerabilities list
Graceful reload interactionCrashes cluster right after logrotate or config reloadsError log timestamps vs “resuming normal operations” entries
Library version mismatchCrashes begin after an OS package update, not an Apache changeldd on loaded modules; package update history
Shutdown/restart noiseOne crash per child only during stop or gracefulTimestamp alignment with SIGTERM/SIGUSR1; benign if so

Quick checks

All read-only and safe on a live server.

# 1. How many crashes, and when
grep -i "segmentation" /var/log/apache2/error.log /var/log/httpd/error_log 2>/dev/null | tail -30

# 2. Crash rate right now (per minute over recent history)
grep -i "segmentation" /var/log/apache2/error.log 2>/dev/null | \
  awk '{print substr($0,1,17)}' | uniq -c | tail -20

# 3. Kernel view: which binary and which library faulted
dmesg -T | grep -i "segfault" | tail -20
# journalctl equivalent on systemd hosts
journalctl --since "1 hour ago" | grep -i segfault | tail -20

# 4. Which MPM is running (decides blast radius)
apachectl -V 2>/dev/null | grep MPM || httpd -V | grep MPM

# 5. Do crashes cluster around restarts or reloads?
grep -E "resuming normal operations|caught SIGTERM|graceful restart|Segmentation" \
  /var/log/apache2/error.log 2>/dev/null | tail -30

# 6. Are children ballooning before they die (leaky module)?
ps -C httpd -o pid,rss,etime,cmd --sort=-rss 2>/dev/null | head -15 || \
  ps -C apache2 -o pid,rss,etime,cmd --sort=-rss | head -15

# 7. Apache and module versions
httpd -v 2>/dev/null || apache2 -v

The dmesg line is the most underrated check here. The kernel logs the faulting instruction pointer and often the library, e.g. segfault at 0 ip ... error 4 in libphp.so[...]. That one field frequently names the culprit library without any further work.

How to diagnose it

flowchart TD
  A[Segfault in error log] --> B{Crash rate?}
  B -->|Multiple per minute| C[Service degrading: mitigate first]
  B -->|Occasional| D{Correlated with restart or reload?}
  D -->|Yes, shutdown only| E[Benign cleanup crash: note and move on]
  D -->|No| F[dmesg: which library faulted]
  C --> G[Capture core dump]
  F --> G
  G --> H[gdb backtrace: faulting module and frame]
  H --> I{Module identified?}
  I -->|mod_php / extension| J[Plan PHP-FPM migration]
  I -->|httpd core| K[Check CVE list, upgrade]
  I -->|Third-party| L[Disable, upgrade, or report upstream]

  1. Quantify. Get the crash count and rate from the error log (checks 1 and 2 above). One crash per day is a ticket. Multiple per minute is service degradation: on a threaded MPM, each crash drops a slice of worker capacity, and the respawned child starts cold.

  2. Classify by timing. Line up crash timestamps against graceful restarts, logrotate runs, and deploys. Crashes only during shutdown are benign. Crashes right after every graceful reload on worker/event are a known bad interaction; repeated reloads (for example, logrotate firing SIGUSR1 frequently) are the trigger to examine. Crashes that start exactly at a deploy or package update point at what changed.

  3. Get the faulting library from the kernel. dmesg -T | grep -i segfault often names the library directly. If it says libphp.so, libcurl.so, or a module .so, you have your suspect list.

  4. Enable core dumps for the next occurrence. Set a dump location and allow cores:

    CoreDumpDirectory /var/crash/apache

    plus ulimit -c unlimited for the Apache process (on systemd, set LimitCORE=infinity in the unit). Caveats: the directory must be writable by the child user; SELinux can block the write even when everything else is correct (check the audit log for denials); and on many modern distros kernel.core_pattern pipes cores to systemd-coredump instead, so your file lands under /var/lib/systemd/coredump/ rather than CoreDumpDirectory. Check cat /proc/sys/kernel/core_pattern and use coredumpctl list / coredumpctl debug in that case.

  5. Backtrace the core. gdb /usr/sbin/httpd /path/to/core then bt full. The top frames name the module and function. A core that resolves only to ?? frames means missing debug symbols; install the -debuginfo/-dbgsym packages for httpd and the suspect modules and wait for the next core.

  6. Reproduce under one child if safe. On a staging host, running a single child and replaying traffic against the suspect URL pattern makes the crash deterministic and the backtrace unambiguous. Never do this on production.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Segfault count/rate in error logPrimary incident signal; rate distinguishes noise from degradationAny occurrence; multiple per minute = degrading
Child process count and respawn rateEach crash forces a respawn; churn costs CPU and cold childrenChild PID turnover accelerating
Per-child RSS trendLeaky modules often grow before they faultRSS climbing over hours, then child disappears
5xx / dropped connections at crash timeOn threaded MPMs one crash kills many in-flight requests5xx bursts aligned with segfault timestamps
Uptime and restart eventsCorrelates crashes with reloads and restartsCrash timestamps adjacent to “resuming normal operations”
Server load / CPU during respawn stormsMass child replacement is expensiveCPU spike following crash clusters

Fixes

If mod_php or a PHP extension is the culprit

The durable fix is to move PHP out of the Apache process: switch to PHP-FPM behind mod_proxy_fcgi. The PHP runtime then crashes (if it crashes) inside the FPM pool, where it cannot take Apache workers down with it, and Apache can run the event MPM. Short of migration, disable the faulting extension and set MaxConnectionsPerChild to a finite value (5000-10000) so children recycle before accumulated corruption or leaks trip the fault. Recycling is a containment measure, not a fix.

If the fault is in Apache core

Check your exact version against the official 2.4 vulnerability list. Several fixed CVEs are crash bugs (including use-after-free and double-free issues in mod_http2 and other modules), so a child that segfaults under HTTP/2 or file-descriptor pressure on an older 2.4.x may simply need an upgrade. If you are current and have a clean backtrace, report it upstream with the core.

If crashes follow graceful reloads

Reduce reload frequency (logrotate and config management are the usual triggers) and make sure only one reload happens at a time. On threaded MPMs, repeated rapid graceful restarts are a known crash trigger.

If a third-party module faults

Disable it on a canary node and confirm the crashes stop. Upgrade to the module’s latest build against your httpd version; modules compiled against a different httpd ABI are a classic source of exactly this crash.

Do not treat “Apache recovered on its own” as resolution. The parent respawning children keeps the service up while the underlying bug keeps firing, and on event/worker each firing drops real requests.

Prevention

  • Migrate off mod_php. PHP-FPM removes the most common segfault source from the Apache address space.
  • Set MaxConnectionsPerChild to a finite value on any deployment with embedded interpreters. It bounds leak-driven corruption and recycles children on a schedule.
  • Keep core dumps permanently enabled on at least a canary subset of the fleet, so the next crash is diagnosable without a reproduction window.
  • Alert on the string. Any occurrence of “Segmentation fault” in the error log should open a ticket; rate-based escalation should page at sustained multiple-per-minute.
  • Limit graceful reload frequency in logrotate and config management, and never reload in tight loops.
  • Track versions. Pin httpd and module versions, watch the 2.4 vulnerability list, and stage upgrades so crash-introducing versions are caught before fleet-wide rollout.

How Netdata helps

  • Netdata’s Apache collector scrapes mod_status continuously, so worker counts, idle/busy workers, and request rates are time-series rather than the point-in-time snapshots server-status gives you by hand. Child churn from repeated segfaults shows up directly in worker utilization and request-rate dips.
  • Correlating the error-log segfault timestamps against BusyWorkers, 5xx rate, and per-process memory in one dashboard tells you immediately whether a crash had user impact (threaded MPM capacity drop) or was contained (prefork, single process).
  • Per-process RSS trends make the leak-then-fault pattern visible days before children start dying.
  • Restart and uptime tracking catches the crash-reload correlation, which is the quickest discriminator between benign shutdown noise and a real fault.
  • Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.