Apache HTTPD monitoring maturity model: from survival to expert

Most teams monitor Apache at whatever level their last outage forced them to reach. A disk fills up, the site goes down, and “check log disk space” appears in the runbook. A backend hangs, workers exhaust, and “watch BusyWorkers” gets added. This model organizes that organic growth into four deliberate levels so you can see what you have, what you are missing, and which blind spot will produce your next incident.

One prerequisite before any of this is interpretable: know your MPM. Apache 2.4 defaults to the event MPM, where keepalive connections are handled by a dedicated listener thread instead of worker threads. On prefork or worker, a keepalive connection holds a worker slot hostage; on event, it does not. Signals like scoreboard K states mean opposite things depending on which MPM is running, so check first (apachectl -V | grep MPM or your config) and keep it in mind for everything below.

Use the model in order. Each level assumes the previous one is automated and alerting, not just documented. Skipping ahead gives you expert dashboards on top of a server that can still die from a full log disk.

The four levels at a glance

flowchart TD
  L1["Level 1: survival
process, HTTP probe, log disk, fatal log keywords"]
  L2["Level 2: operational
workers, 5xx, RPS, latency, RSS, FDs, backlog"]
  L3["Level 3: mature
scoreboard states, per-child RSS, backend time, per-vhost"]
  L4["Level 4: expert
R-state ratio, connect vs response, OCSP, conntrack"]
  L1 --> L2 --> L3 --> L4

Each level answers a different question. Level 1: is it up? Level 2: is it healthy under current load? Level 3: where is worker time actually going? Level 4: what rare or silent failure will hurt next?

Level 1: survival

The absolute minimum. These four checks catch catastrophic failures and nothing else.

Process existence. A missing httpd parent process means the service is down: crashed, OOM-killed, or stopped. Page if the parent is absent for more than 60 seconds; brief absence during a restart is normal.
HTTP health check. A process can be alive with zero functional children, so probe the actual critical path and expect a 2xx within 5 seconds. A check that only fetches a static file misses backend failures, module crashes, and vhost-specific config errors.
Log partition disk space. When the log filesystem fills, workers finish requests but block in the Logging state; the server stays “up” and serves nothing. This failure is silent until it is total.
Error log keywords. Grep for MaxRequestWorkers (message AH00484), Segmentation fault, and No space left on device. Each of these is definitive on its own; none requires interpretation.

# Parent process (oldest match is the parent)
pgrep -o 'httpd|apache2'

# Critical-path probe, 5s timeout
curl -sf -o /dev/null -w "%{http_code}\n" --max-time 5 http://localhost/health

# Log filesystem
df -h /var/log/apache2/ 2>/dev/null || df -h /var/log/httpd/

# Fatal keywords
grep -E "AH00484|Segmentation fault|No space left" /var/log/apache2/error.log | tail

What this level misses: everything short of total failure. Worker exhaustion, slow backends, memory leaks, and listen queue overflow all develop for minutes or hours before Level 1 notices, and it notices only because users are already timing out.

Level 2: operational

This is what a competent team runs day to day. It answers “is Apache healthy right now” and detects saturation as it develops, not after it completes.

BusyWorkers and IdleWorkers. The primary saturation gauge, from server-status?auto. Ticket above 80 percent of MaxRequestWorkers sustained; page when utilization exceeds 95 percent with zero idle workers, corroborated by listen queue growth, AH00484 in the error log, or 503s. MaxRequestWorkers is not exposed in server-status, so record it from config.
5xx rate, with 503 broken out. 5xx above 1 percent sustained is a ticket; healthy is under 0.1 percent. A 503 specifically means worker or proxy pool exhaustion. 502 and 504 mean backend trouble, not Apache trouble.
Request rate. Compute from the delta of Total Accesses over an interval. The ReqPerSec field is a lifetime average since restart and is nearly useless for real-time decisions. A drop toward zero while the load balancer still sends traffic is critical.
Latency, p50 and p99, from %D in the access log. Remember that %D includes client transfer time: a 100MB download to a slow client shows a huge %D without Apache being slow. Filter by URL pattern and response size before alerting, and never use %T, which rounds to whole seconds.
Total Apache memory. Sum of all children RSS. Compare MaxRequestWorkers x average child RSS against RAM; above roughly 70 percent you are one traffic spike from an OOM cascade.
File descriptor count per child. Each connection, log file, and backend socket costs an FD. Ticket above 70 percent of the per-process limit; FD exhaustion is a cliff edge, not a slope.
Listen backlog depth. Recv-Q on the listening socket is the earliest saturation signal, appearing before latency moves. Default ListenBacklog is 511 and the kernel’s net.core.somaxconn can silently cap it lower.
Error log rate. Lines per minute at [error] and above. Sustained increase from baseline is a ticket even before you read the content.
Proxy 502/504 rate, if proxying. 502 means the backend refused or answered garbage; 504 means it connected but exceeded ProxyTimeout. Track separately from generic 5xx.
Restart events. ServerUptimeSeconds plus resuming normal operations and caught SIGTERM in the error log. More than one unexpected restart per day means instability somewhere.
Certificate validity. Days to expiry on every vhost, not just the default. Expired certs are among the most deterministic and most preventable outage causes.

# Worker utilization, throughput counters, uptime
curl -s http://localhost/server-status?auto | grep -E "BusyWorkers|IdleWorkers|Total Accesses|ServerUptimeSeconds"

# Listen queue depth
ss -ltn | grep -E ':80\s|:443\s'

# 5xx rate over recent requests (assumes common/combined log format, status in field 9)
tail -1000 /var/log/apache2/access.log | awk '$9 ~ /^5/ {e++} END {print "5xx:", (e+0)/NR*100 "%"}'

What this level misses: causation. Level 2 tells you workers are saturating, not why. A scoreboard full of W states from a slow backend looks identical to a genuine traffic overload until you add the next level.

Level 3: mature

Full coverage of internals. This level answers “where is worker time going” and “which component is the bottleneck.”

Full scoreboard state distribution. The single most diagnostic structure Apache exposes. Many W means slow clients or slow backends; many R means slow request bodies or Slowloris; many L means log stall; many G means a graceful restart is lingering; many D means DNS lookups are blocking workers. On event MPM, significant K in the scoreboard is abnormal, because keepalive should be offloaded to the listener thread and counted in ConnsAsyncKeepAlive instead.
Per-child RSS trend. Average RSS hides leaks; a monotonically climbing per-PID line finds them. If MaxConnectionsPerChild is 0 (the default), nothing bounds that growth.
Backend response time, per backend. Slow backends are the number one cause of worker exhaustion in proxy deployments. Without this signal you will debug Apache while the backend burns.
Proxy pool utilization. The default pool max equals ThreadsPerChild (1 on prefork), which is far too small for production, and pools are per child process. Pool exhaustion returns 503 immediately, with no queuing.
TLS session resumption. Full handshakes cost an order of magnitude more CPU than resumed sessions. The shmcb session cache can silently fill and evict, dropping resumption and raising CPU with no error logged. Test directly: openssl s_client -connect localhost:443 -reconnect 2>/dev/null | grep -c Reused.
Connection-state breakdown. ESTABLISHED, TIME_WAIT, and CLOSE_WAIT counts from ss. Persistent CLOSE_WAIT means Apache failed to close connections after the peer did: a leak.
Per-vhost error rate. Apache exposes no per-vhost request counters natively, so this comes from log analysis. On vhost-heavy servers, fleet-wide averages routinely hide one vhost that is failing completely.

# Scoreboard state histogram
curl -s http://localhost/server-status?auto | grep "Scoreboard:" | \
  awk '{print $2}' | fold -w1 | sort | uniq -c | sort -nr

# Per-child RSS, heaviest first
ps -C httpd -o pid,rss,vsz,cmd --sort=-rss 2>/dev/null || \
  ps -C apache2 -o pid,rss,vsz,cmd --sort=-rss

# Connection states on Apache ports
ss -tnH '( sport = :80 or sport = :443 )' | awk '{print $1}' | sort | uniq -c | sort -rn

What this level misses: rare events and silent failures. Level 3 runs well for months, then a Slowloris attack, a conntrack table overflow, or a broken OCSP stapling config produces symptoms none of these dashboards explain.

Level 4: expert

Signals teams add after the incident that taught them. Each one exists because something hurt once.

Scoreboard R-state ratio. Normal traffic rarely puts more than a few percent of workers in R. Sustained R above 20 percent means slow request bodies, bad networks, or a slow-read attack holding workers hostage. Combine with source-IP concentration before calling it an attack.
Backend connect time versus response time. This distinguishes “backend down” from “backend slow”: connection refusal fails fast (AH01114 in the error log, and the request returns 503), while a connected-but-slow backend burns a worker until ProxyTimeout and returns 504. Apache exposes no single timing metric that splits connect time from response time, so the split comes from the error-log pattern plus the status code; direct backend probes supply the timing view from outside.
OCSP stapling health. Stapling failures are silent: Apache serves handshakes without a staple, each client fetches OCSP itself, and latency rises with nothing in the default logs. Verify with openssl s_client -connect host:443 -status and watch for AH01929 and AH02217.
GracefulShutdownTimeout effectiveness. Default 0 means old-generation workers linger indefinitely after a graceful restart. Frequent restarts plus slow requests stack generations of children and multiply memory. Track G states over time and bound the drain (for example 30 seconds).
mod_reqtimeout rejections. mod_reqtimeout is loaded by default in 2.4 with header=20-40,MinRate=500 body=20,MinRate=500. Count the resulting 408s as their own class: they are your slow-client defense working, not ordinary errors. Note that when an AcceptFilter is in use (the default on Linux), the handshake and header timeouts only start once the kernel hands the socket to a worker.
nf_conntrack utilization. The kernel connection tracking table fills under heavy connection rates and then drops packets invisibly; Apache looks fine while connections vanish. Check dmesg | grep "nf_conntrack: table full" and track table usage against its maximum.
Async connection metrics, event MPM only. ConnsTotal, ConnsAsyncKeepAlive, ConnsAsyncWriting, ConnsAsyncClosing. High ConnsAsyncKeepAlive is normal and healthy; high ConnsAsyncWriting points at slow clients.

What pushes you up a level

Movement between levels is almost always incident-driven. The composite failure patterns map cleanly onto the earliest level that catches them:

Failure pattern	Level 1-2 sees	Earliest level that explains it
Log stall deadlock	Disk full, probe failing	Level 3: `L` states dominate the scoreboard
Slow backend cascade	503/504, workers saturated	Level 3: backend response time plus `W` growth
Memory leak slow death	Total RSS rising, OOM kills	Level 3: per-child RSS trend
Slowloris / slow read	Nothing distinctive	Level 4: R-state ratio, reqtimeout 408s
Graceful restart pile-up	Memory spikes after reloads	Level 4: `G` states vs GracefulShutdownTimeout
conntrack overflow	Unexplained packet drops	Level 4: nf_conntrack utilization
OCSP stapling failure	Nothing; clients just slower	Level 4: stapling validation

If a pattern in the left column has bitten you and your current level sits below the right column, that gap is your next work item.

How Netdata helps

The Apache collector polls server-status?auto continuously, turning point-in-time snapshots into time series for BusyWorkers, IdleWorkers, full scoreboard state distribution, request and byte rates, uptime, and the event-MPM async connection counters. That automates most of Levels 2 and 3.
System collectors supply the surrounding saturation signals in the same view: per-process RSS, FD counts against limits, listen queue depth, and nf_conntrack utilization, which covers the Level 2 and Level 4 signals Apache does not expose itself.
Correlating W-state growth against backend latency and 504 rate is what separates a slow-backend cascade from a genuine traffic overload; having both on one dashboard collapses that diagnosis from hours to minutes.
Access log parsing turns 5xx, 503, 502, and 504 rates plus per-vhost breakdowns into metrics, covering the Level 2 error signals and the Level 3 per-vhost gap that mod_status cannot fill.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

The Netdata solution

Apache HTTP Server monitoring with Netdata

Netdata monitors Apache HTTP Server with per-second metrics from mod_status, pre-built dashboards, and ML-powered anomaly detection. Watch busy versus idle workers and the scoreboard state mix, requests per second, bytes served per second, and request processing duration alongside the rest of your stack, so you catch the worker-exhaustion, slow-backend, and memory incidents in these runbooks before they page anyone.

See web server monitoring → Start monitoring free