Apache error log monitoring: severity levels, AH codes, and what to alert on

Metrics tell you that something is wrong with Apache. The error log tells you what. When BusyWorkers pegs at MaxRequestWorkers, the scoreboard shows saturation, but the error log hands you AH00484: “server reached MaxRequestWorkers setting.” When a child dies, the process table shows a respawn; the error log shows “Segmentation fault.”

This guide covers how to read the log efficiently: the severity hierarchy, the AH code scheme introduced in 2.4, the patterns worth alerting on, and the configuration gotchas that silently suppress the messages you need.

How Apache decides what to log

Every error log line carries a severity. LogLevel sets a floor: messages at the configured level and everything more severe are written, everything below is dropped.

The default is warn. Two consequences follow. First, [notice] messages like “resuming normal operations” and “caught SIGTERM” are below the floor, which is what you want for restart forensics but easy to forget. Second, anything a module logs at info is invisible by default, which matters more than most operators realize (see the gotchas below).

In 2.4 there are also trace1 through trace8 levels below debug.

The practical reading rule: [error] is the working level for request and module failures, [crit] and above should be near zero on a healthy server, and any [emerg] or [alert] is a stop-and-look event. As a rough baseline, a sustained rate above 10 [error] lines per minute is abnormal for most deployments; baseline your own.

The AH code scheme

Apache 2.4 tags nearly every log message with a unique code of the form AHxxxxx, for example AH00484. Each code maps to one specific message at one specific location in the source tree. This matters operationally for two reasons:

• Codes are stable greppable anchors. Message text varies with arguments (paths, PIDs, client addresses); the code does not. Alert rules should match the code, not the full message.
• Codes disambiguate similar messages. 503 in the access log can mean worker exhaustion or a balancer member in error state. The error log distinguishes them because they carry different codes.

Apache 2.2 logs have no codes; the format there was fixed and message matching is the only option. The error log line format became customizable in 2.4 via ErrorLogFormat, so on a mixed fleet do not assume your parsing regexes port across versions.

What to alert on

Not everything in the error log is signal. Crawler 404s and client-side noise dominate volume; the codes below are the ones that correlate with real production failure modes.

Two nuances on the headliners:

• AH00484 fires once per exhaustion event, not per rejected request. A single line can represent thousands of queued connections. Never alert on “more than N occurrences” of it; any occurrence during production traffic is worth investigating. Corroborate with BusyWorkers, listen queue depth, and 503s before paging. See Apache AH00484: server reached MaxRequestWorkers setting.
• Segfaults on threaded MPMs are worse than on prefork. On prefork a segfault kills one child. On worker or event, a crashing thread can take the entire process and all of its threads with it. The log line alone does not tell you which; check the scoreboard and process count around the event.

Some [error] lines are client-side noise: broken pipes when clients disconnect, “File does not exist” for scanner probes. Baseline these and filter them from alert rules, or you will train the team to ignore the log.

LogLevel gotchas that hide real incidents

Setting the level too high. A LogLevel of crit or alert feels like noise reduction. It also hides every proxy failure, every permission problem, and every module error, because those log at [error]. If your error log looks suspiciously clean during an incident, check the level first.

Per-module levels are the right tool for targeted verbosity. Instead of raising the global level to debug and drowning, raise one module: LogLevel warn ssl:debug keeps the floor at warn but turns on debug for mod_ssl. Conversely, LogLevel ssl:warn quiets one noisy module without touching the rest.

2.4 moved 404s to info. In 2.4, “File does not exist” messages were downgraded from error to info. With the default warn floor they vanish entirely. This has silently broken fail2ban rules and security monitoring that scraped for 404s after upgrades from 2.2. To restore them for the core module only: LogLevel warn core:info.

Debug volume is explosive. A single request at LogLevel debug generates many lines per module hook. Use per-module scoping, and never leave global debug on in production. Besides the noise, every one of those lines is a synchronous write (next section).

Virtual hosts can have their own error logs. A per-VHost ErrorLog directive redirects that vhost’s messages away from the main log. If you alert only on the main log, vhost-scoped failures are invisible. Either aggregate all vhost logs into your collection pipeline or consolidate them.

Failure modes of the logging pipeline itself

The error log is on the request hot path, and it has its own failure modes.

flowchart TD
  E[Event in a worker] --> W{Write path}
  W --> F[ErrorLog file - synchronous write]
  W --> P[Piped log program]
  W --> V[Per-VHost error log]
  F -->|disk slow or full| L[Worker blocks in scoreboard L state]
  P -->|program dies| S[SIGPIPE - child may crash]
  K[Kernel OOM killer] -->|never reaches Apache| D[dmesg / kernel log only]

Synchronous writes block workers. File-based logging is synchronous: the worker blocks until the write completes. On slow or saturated storage, workers accumulate in the L (Logging) scoreboard state, which is indistinguishable from worker exhaustion from the outside. A sustained L count above a few percent is a storage or log-pipeline problem, not a traffic problem.

Piped logs can fail silently. ErrorLog "|/path/to/program" hands log lines to a child process. If that program dies or stalls, writes block or fail, and children can take SIGPIPE and crash. Piped logging with rotatelogs avoids copytruncate races and unbounded file growth, but you trade one failure mode for another: monitor the pipe process itself.

A full log disk is a service outage, not a logging outage. When the log filesystem fills, workers finish requests but cannot log them and hang in L state. The port stays open; the server serves nothing. The signature: scoreboard dominated by L, disk at 100%, error log stops updating. Keep logs on their own filesystem and alert on it before it fills.

Some fatal events never appear here. When the kernel OOM killer terminates httpd children, nothing is written to the Apache error log, because the process was killed externally. The evidence is in dmesg or the journal: dmesg | grep -i oom. If you see unexplained child respawns with a clean error log, check the kernel log before assuming an Apache bug. The same applies to nf_conntrack table saturation and SELinux denials, which surface here as generic “Permission denied” or dropped connections but are explained only in system logs.

Quick reference: reading the log during an incident

# Errors and above, most recent first
grep -E "\[error\]|\[crit\]|\[alert\]|\[emerg\]" /var/log/apache2/error.log | tail -20
# RHEL path: /var/log/httpd/error_log

# Worker exhaustion events
grep "AH00484" /var/log/apache2/error.log | tail -20

# Child crashes (check both Apache and kernel logs)
grep -i "segfault\|segmentation" /var/log/apache2/error.log | tail -20
dmesg | grep -i "segfault.*apache\|segfault.*httpd" | tail -20

# Restart history: frequency and type
grep -E "resuming normal operations|caught SIGTERM|graceful restart" /var/log/apache2/error.log | tail -20

# The things the error log will never tell you
dmesg | grep -i oom | tail -20

Pair each pattern with its metric corroborator: AH00484 with BusyWorkers/IdleWorkers and listen queue depth, segfaults with child process churn, L-state buildup with disk space and I/O latency on the log filesystem, proxy AH codes with 502/503/504 rates in the access log. The error log gives you the cause; the metrics tell you the blast radius.

How Netdata helps

• Error-level log rate as a first-class signal. Netdata can track lines per minute at [error] and above, so a silent ramp in proxy or permission errors shows up as a trend, not a surprise during the postmortem.
• AH00484 correlated with saturation metrics. A MaxRequestWorkers event next to BusyWorkers, listen queue depth, and 503 rate on one dashboard is the difference between “page now” and “watch it.”
• Scoreboard L state next to disk metrics. Workers blocking on log writes show up as Logging-state growth alongside log filesystem utilization and I/O latency, which pinpoints a log stall without log diving.
• Restart and crash context. Uptime resets and child churn correlated with system-level events (including OOM kills from the kernel side) close the gap where the error log itself is silent.
• Per-second granularity. Error log bursts during deploys and graceful-restart pile-ups are visible at the resolution they actually happen at, rather than averaged away.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• Apache 502 Bad Gateway: a backend that returned an invalid response
• Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion
• Apache 504 Gateway Timeout: slow backends, ProxyTimeout, and worker pile-up
• Apache backend response time: telling ‘Apache is slow’ from ’the backend is slow’
• Apache balancer member in error state: reading balancer-manager and failover
• Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status
• Apache CLOSE_WAIT and TIME_WAIT: connection leaks versus normal churn
• How Apache HTTPD actually works in production: a mental model for operators
• Apache keepalive consuming workers: KeepAliveTimeout, the K state, and MPM choice
• Apache listen queue overflow: Recv-Q growth, ListenBacklog, and refused connections
• Apache AH00484: server reached MaxRequestWorkers setting - worker pool exhausted
• Apache MaxConnectionsPerChild: bounding leaky modules by recycling children