$ guides / apache-httpd / apache-httpd-monitoring-checklist ▌

Operations Guides

Apache HTTPD monitoring checklist: the signals every production web server needs

This is a working checklist for engineers running Apache HTTPD in production. It organizes the signals worth collecting into four maturity levels, from “is the process alive” through “which worker is stuck and why.” Use it to audit an existing setup for gaps or to build one without over-instrumenting on day one.

Two things before the list. First, every saturation signal in Apache is interpreted through the active Multi-Processing Module (MPM), so the checklist starts there. Second, the levels are cumulative: Level 2 assumes Level 1 is in place. Do not skip ahead. The scoreboard state distribution is useless if nobody is watching the error log for AH00484.

Prerequisite: know your MPM

The MPM determines how connections map to execution units, which determines what “saturated” looks like. Before interpreting any worker or scoreboard signal, confirm which MPM is loaded:

apachectl -V 2>/dev/null | grep -i mpm
# or
httpd -V 2>/dev/null | grep -i MPM

MPM	Model	What you run out of	Key monitoring implication
prefork	One process per connection, no threads	Memory. Each connection costs a full process, typically 10-50MB+ with modules loaded	Keepalive connections hold a full process hostage. `K` states on the scoreboard directly consume capacity
worker	Multiple processes, each with multiple threads	Threads. A stuck backend can hold a thread indefinitely and starve the pool in that child	Keepalive still consumes a worker thread
event	Like worker, plus a listener thread handling keepalive asynchronously via epoll/kqueue	Threads for active requests only	Default in 2.4.x. Keepalive is offloaded to the listener thread and tracked via `ConnsAsyncKeepAlive`. Significant `K` states in the scoreboard on event MPM are abnormal

The classic mistake: applying prefork intuition (keepalive hoards workers) to an event MPM server, or vice versa. On event, high ConnsAsyncKeepAlive is healthy. On prefork, the equivalent connection count means worker exhaustion is imminent.

The four levels build on each other:

flowchart TD
  L1["Level 1 - Survival: process, HTTP probe, log disk, error keywords"]
  L2["Level 2 - Operational: workers, 5xx, RPS, latency, memory, cert"]
  L3["Level 3 - Mature: scoreboard states, per-child RSS, FDs, backlog"]
  L4["Level 4 - Expert: R-state ratio, TLS resumption, conntrack, async conns"]
  L1 --> L2
  L2 --> L3
  L3 --> L4

Level 1: survival

The minimum. These four checks catch the catastrophic failures: server down, server unreachable, and the two most common silent killers.

Process presence. The httpd/apache2 parent process must exist, the PID file must point at it, and there should be exactly one parent. A stale PID file means an unclean shutdown; multiple parents mean failed graceful restarts are accumulating. Page if the parent is absent for more than 60 seconds. The trap: the parent can be alive with zero functional children (the OOM killer targets children first), so process presence is necessary but not sufficient.
Critical-path HTTP probe. A GET against the real service path (not a static file) returning 2xx within about 5 seconds. A TCP port check only proves the kernel is listening; an HTTP probe proves a worker can complete a request. A check that fetches static content misses backend failures, module crashes, and vhost-specific config errors.
Log filesystem space. When the log disk fills, workers finish requests but block in the Logging state, and the server goes effectively dead while the process looks fine. Alert at 80% full on the filesystem hosting /var/log/apache2 or /var/log/httpd.
Error-log keywords. Grep the error log for three strings: AH00484 (server reached MaxRequestWorkers), Segmentation fault, and No space left on device. Any AH00484 means worker saturation actually happened. Any segfault in production needs root cause analysis. Any “No space left” is already an incident.

Level 2: operational

Everything in Level 1, plus the signals that tell you whether the server is healthy under current load, not just alive.

BusyWorkers and IdleWorkers. From http://localhost/server-status?auto. BusyWorkers / MaxRequestWorkers is your primary saturation ratio. Ticket at sustained >80% for 10 minutes. Page when utilization exceeds 95%, IdleWorkers is zero, uptime is over 600 seconds (rules out cold start), and at least one corroborator is present: listen queue Recv-Q sustained above zero, AH00484 in the error log, or 503s appearing. MaxRequestWorkers is not exposed in server-status; know it from config.
HTTP 5xx rate. From the access log. Healthy is under 0.1%. Ticket on anything sustained above 1%. Read the codes: 502 is a backend returning garbage or refusing connections, 503 is worker or proxy-pool exhaustion, 504 is backend timeout. In proxy deployments, 502/503/504 rates are backend failure indicators, not Apache problems.
Request rate. Compute rate from the Total Accesses counter delta, not the ReqPerSec field, which is a lifetime average since restart and useless for real-time work. Compare against time-of-day baselines. A drop toward zero while the load balancer reports sending traffic is critical.
Request latency, P50 and P95. From the access log %D field (microseconds). Use %D or %{ms}T, never %T: rounding to whole seconds makes sub-second latency monitoring useless. Caveat: %D includes client transfer time, detailed in the gotchas section.
Total Apache memory. Sum of RSS across all children. The ceiling check is arithmetic: MaxRequestWorkers x average child RSS must stay under about 70% of RAM. If the product exceeds physical memory, you have a configured OOM waiting for a traffic peak.
Error log rate. Lines per minute at [error] severity and above. Sustained increase from baseline is a ticket; any [emerg] or [alert] needs immediate investigation.
Uptime and restart events. ServerUptimeSeconds from server-status, plus resuming normal operations, caught SIGTERM, and graceful restart entries in the error log. More than one unexpected (non-graceful) restart per day indicates instability.
Certificate validity. Days to expiry from the live certificate via openssl s_client, for every vhost, not just the default. Ticket at 30 days, escalate at 7. Auto-renewal failures are the usual culprit.

echo | openssl s_client -connect localhost:443 2>/dev/null | \
  openssl x509 -noout -enddate

Level 3: mature

Everything in Level 2, plus the signals that explain why a saturation or latency event is happening while it is happening.

Full scoreboard state distribution. The scoreboard is the single most diagnostic structure Apache exposes. Collect it over time, not just on demand:

curl -s http://localhost/server-status?auto | grep "Scoreboard:" | \
  awk '{print $2}' | fold -w1 | sort | uniq -c | sort -nr

The characters: _ waiting, S starting, R reading request, W sending reply, K keepalive, D DNS lookup, C closing, L logging, G gracefully finishing, I idle cleanup, . open slot. The distribution tells you where time is going: many W means slow clients or slow backends, many R means slow request bodies or Slowloris, many L means a log stall, many G means a graceful restart pile-up, many D means hostname-based access control is blocking workers on DNS lookups.

Per-child RSS trend. Track RSS per PID over days, not just the fleet sum. Monotonic per-child growth is the memory-leak signature, and it almost always coexists with MaxConnectionsPerChild 0 (the default). Setting a finite value (5000-10000) bounds leaks by forcing periodic child recycling; it is a band-aid, not a fix.
File descriptor usage per child. Count entries in /proc/[pid]/fd for each child and compare against the limit in /proc/[pid]/limits. Each client connection, backend socket, and log file consumes an FD; vhost-heavy servers with separate logs multiply usage. Ticket above 70% of the limit in any child. FD exhaustion is a cliff: at the limit, every new connection, file open, and backend connect fails immediately with “Too many open files.”
Listen backlog depth. ss -ltn on ports 80/443: Recv-Q is the current queue, Send-Q is the maximum (ListenBacklog, default 511, capped by net.core.somaxconn). Sustained Recv-Q above zero means connections arrive faster than Apache accepts them. This is the last buffer before connections get refused, and it fires before users see failures. Also track the kernel’s listen overflow counters (nstat -az | grep -i listen).
Backend response time (if proxying). Slow backends are the most common cause of Apache worker exhaustion in proxy deployments: workers pile up in W state waiting, IdleWorkers drains, the backlog fills, and from outside “Apache is down” while Apache itself is fine. Monitor backend health separately from Apache health, or you will debug the wrong process for hours. Also watch proxy pool sizing: the default max per proxy worker equals ThreadsPerChild (1 on prefork), which is far too small for most production workloads.
Backend member health (if proxying with a balancer). balancer-manager shows whether individual backends are marked errored or disabled. Any errored backend with active traffic is a ticket.
Connection count by state. ESTABLISHED, TIME_WAIT, CLOSE_WAIT from ss. High TIME_WAIT is normal and the kernel’s problem. Persistent CLOSE_WAIT is not: it means Apache failed to close connections the remote end already closed.
Configuration reload success. A failed graceful reload is silently ignored: the old config keeps running and nobody knows the new one never applied. Check apachectl configtest and grep the error log for syntax errors after every reload.
Log filesystem I/O health. Beyond space: sustained high I/O latency on the log filesystem blocks workers in L state the same way a full disk does.

Level 4: expert

Signals most teams add after the incident that taught them the lesson. You can add them earlier and skip the incident.

Scoreboard R-state ratio. Normal traffic rarely puts more than about 5% of workers in R. Sustained R above 20% is the Slowloris / slow-read signature: connections sending data byte-by-byte, holding workers indefinitely. Corroborate with source-IP concentration and low throughput relative to connection count before calling it an attack; slow legitimate uploads look similar. The defense is mod_reqtimeout; confirm it is loaded and RequestReadTimeout is configured for your distribution before relying on it. Counting its 408 rejections gives you a slow-client metric separate from real errors.
Backend connect time versus response time. Distinguishes “backend is down” (fast connection refusal or connect timeout) from “backend is slow” (connected, then waiting). Different fixes; conflating them wastes response time.
TLS session resumption rate. Full handshakes cost far more CPU than resumed sessions. If the session cache is misconfigured, undersized, or not shared across children (SSLSessionCache shmcb:), resumption silently degrades and CPU climbs with no error logged. The only symptoms are higher CPU and higher TTFB.
OCSP stapling success. Stapling failures are silent: Apache serves handshakes without the staple, clients make their own OCSP requests, and everyone gets extra latency with nothing in the error log at default levels. Test with openssl s_client -connect host:443 -status and watch for stapling errors in the error log; AH01929 (response-cache store failure) and AH02217 (issuer-certificate initialization failure) are useful examples, not an exhaustive list.
GracefulShutdownTimeout effectiveness. During graceful restarts, old-generation children linger until their requests finish. Overlapping restarts stack generations and multiply memory. Many G states persisting after a restart means old workers are stuck; GracefulShutdownTimeout (default 0, wait indefinitely) puts a hard deadline on the drain.
Kernel nf_conntrack utilization. Under heavy connection rates the conntrack table fills and the kernel drops packets invisibly. dmesg | grep "nf_conntrack: table full". This looks like a network problem and is actually firewall-subsystem saturation.
Event MPM async connection metrics. ConnsTotal, ConnsAsyncWriting, ConnsAsyncKeepAlive, ConnsAsyncClosing from server-status (event MPM only). High ConnsAsyncKeepAlive is normal. High ConnsAsyncWriting points at slow clients.
AH00484 as a counted event. Most teams grep for it during incidents. Level 4 counts occurrences over time. Brief hits during bursts that the backlog absorbs are tolerable; recurring hits mean capacity, config, or a slow dependency needs attention.

Gotchas that apply at every level

%D includes client transfer time. A 100MB download to a 1Mbps client shows %D of roughly 800 seconds. That is not Apache being slow. Filter by response size or URL pattern before alerting on latency, or your tail-latency alerts will be noise.

A scoreboard full of W with normal request rate is a red flag, not a green one. Workers stuck waiting on a backend all show W. The request rate looks normal only because requests that started before the backend hung are still completing. This is the early stage of the slow-backend cascade.

200 OK does not mean success. A proxied application returning error pages with status 200 is invisible to status-code monitoring. Synthetic checks need content assertions, not just status codes.

K states mean opposite things depending on MPM. On prefork/worker, many K states mean keepalive is consuming workers; reduce KeepAliveTimeout. On event, keepalive is handled by the listener thread, so significant K in the scoreboard indicates the async offload is misbehaving.

Health check traffic inflates your baselines. Behind a load balancer, LB health checks can be a meaningful share of total requests. Account for them before alarming on request-rate deviations.

systemd limits override Apache config. TasksMax, MemoryMax, and LimitNOFILE in the unit file win over anything in httpd.conf. TasksMax is the sneaky one: it caps total processes plus threads and can hold Apache below its configured MaxRequestWorkers with no Apache-side error.

mod_status is a snapshot, not a time series. Every signal above that comes from server-status needs periodic sampling and rate/delta computation to be useful. Polling once per second is sufficient; the values fluctuate faster than that on their own.

How Netdata helps

Most of these signals live in different places (server-status, access log, error log, /proc, ss, openssl) and only become diagnostic when correlated.

Netdata’s Apache collector polls server-status?auto continuously, turning BusyWorkers, IdleWorkers, the scoreboard state distribution, and the event-MPM ConnsAsync* counters into per-second time series instead of point-in-time snapshots.
Scoreboard states tracked over time make the composite patterns visible: W climbing while throughput falls (slow backend), R climbing with flat throughput (slow clients), L dominating (log stall).
Web log parsing surfaces 5xx rates broken down by status code, request-rate trends, and latency percentiles from %D, so the Level 2 error and latency signals do not require awk pipelines.
System-level collection covers the Level 3 resource signals in the same view: per-process RSS, file descriptor usage versus limits, listen backlog and overflow counters, disk space and I/O on the log filesystem, and certificate expiry.
Because all of these land on one timeline, the correlations the checklist depends on, such as worker saturation plus backlog growth plus AH00484, can be checked in seconds instead of assembled from five terminals.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

The Netdata solution

Apache HTTP Server monitoring with Netdata

Netdata monitors Apache HTTP Server with per-second metrics from mod_status, pre-built dashboards, and ML-powered anomaly detection. Watch busy versus idle workers and the scoreboard state mix, requests per second, bytes served per second, and request processing duration alongside the rest of your stack, so you catch the worker-exhaustion, slow-backend, and memory incidents in these runbooks before they page anyone.

See web server monitoring → Start monitoring free