$ guides / apache-httpd ▌

APACHE HTTPD · OPERATIONS PLAYBOOK

Apache's real limit isn't traffic — it's a bounded worker pool, a kernel accept queue, and whatever the slowest backend is doing

A process- or thread-based HTTP server where every connection maps to a worker slot, a bounded pool (MaxRequestWorkers) sits behind a fixed-size kernel listen queue, and — in reverse-proxy mode — a single slow backend can hold every worker hostage. We trace how that design behaves under load, where it turns from serving into queuing into refusing connections, and what to do when it does.

> Start with the monitoring checklist → # Jump to the full guide list

Apache's defaults get you serving in minutes, then hand you a set of cliff-edges that most teams only meet during an incident.

The defaults work. Until every worker is busy and Apache logs AH00484: server reached MaxRequestWorkers setting — at which point new connections pile into the kernel listen backlog and then get refused, a cliff-edge rather than a slowdown. Until a proxied backend slows down and every worker stacks up in the W state waiting on it, so Apache looks broken while its own CPU and memory sit idle. Until a leaky mod_php child grows unbounded because MaxConnectionsPerChild was left at 0, and the OOM killer starts a respawn spiral. Until the log disk fills and workers hang in the Logging (L) state — port open, nothing served. Until a certificate quietly expires and every HTTPS client is locked out at once.

These guides are written for engineers who already run Apache, not for people learning what a web server is. The first thing they insist on is knowing your MPM — prefork, worker, or event — because it changes how every saturation signal is read. The goal is the mental model of how the server actually behaves under load, the failure patterns that keep recurring, the monitoring story that catches them before they page anyone, and the runbooks you wish someone had handed you before your last incident.

How Apache HTTPD actually runs in production

Apache is not just a request handler. It is a multi-process (or multi-threaded) server where every connection maps to a worker slot in a fixed-size scoreboard, a bounded worker pool sits behind a kernel accept queue, and TLS, the module pipeline, the handler, and — in proxy mode — backend connection pools each sit on the request path. Most production failures live between these layers, not inside any one of them.

clients + connections

Each TCP connection costs a file descriptor and, on prefork/worker, a worker slot for its whole lifetime — including idle keepalive time. On the event MPM a listener thread parks idle keepalive connections asynchronously (<code>ConnsAsyncKeepAlive</code>) so they do not tie up workers. Knowing your MPM decides how you read every connection signal.

CLIENT

kernel listen queue

Connections that completed the TCP handshake wait here until a worker calls <code>accept()</code>. The depth is <code>ListenBacklog</code> (default 511), itself capped by <code>net.core.somaxconn</code>. A non-zero, growing <code>Recv-Q</code> is the earliest saturation signal — it shows before latency rises. When the queue overflows, the kernel drops SYNs or sends RST: the port is open but the server is unreachable.

MPM workers + scoreboard

The bounded pool that does the work. <code>MaxRequestWorkers</code> caps concurrency; <code>ServerLimit × ThreadsPerChild</code> caps the scoreboard, a shared-memory segment that records every slot's state. This is the single most diagnostic structure Apache exposes — and the resource you run out of first. Full pool means instant queuing, with no graceful degradation.

WORKER

TLS termination

Full TLS handshakes are the most CPU-intensive thing Apache does; session resumption cuts that roughly tenfold. A too-small or unshared <code>shmcb</code> session cache, disabled tickets, or RSA-only key exchange turns HTTPS into a CPU wall. An expired certificate is a total, instant outage — and it is not a broker metric, so it needs an external check.

TLS

module + filter pipeline

Every request runs an ordered chain of phases and filters — auth, access control, <code>mod_rewrite</code>, <code>mod_security</code>, <code>mod_deflate</code>, the handler, logging. Each hook can block, fail, add latency, or allocate memory from the request's APR pool. A single complex regex or heavy rule set can peg one core per request.

MODULES

content: static + dynamic

Static files served from page cache are cheap; <code>mod_php</code>, <code>mod_perl</code>, and friends embed an interpreter into every child, multiplying per-worker memory 3–10× and forcing prefork. This layer sets the memory ceiling that determines how many workers you can safely run.

HANDLER

reverse proxy + backends

When Apache proxies, each backend connection comes from a per-child pool sized by <code>max=</code> — whose default is far too small for production. A slow backend holds workers in the <code>W</code> state; a full pool returns 503; a refused connect returns 502. This is the number-one cause of "Apache is slow" in modern deployments, and it lives entirely outside Apache.

PROXY

logging + disk

Every request writes the access log; problems write the error log. When the log filesystem fills or a piped log program dies, workers finish the request but block writing the log, hanging in the Logging (<code>L</code>) state. The server stays "up" while serving nothing — and the error log itself may stop updating.

LOG

Why this matters: "Apache is slow" or "the site is down" can come from a full worker pool, an overflowing listen queue, a slow backend starving every worker, a leaking mod_php child heading for OOM, a full log disk freezing workers in Logging, a TLS handshake CPU wall, or a kernel conntrack table dropping packets Apache never sees. The symptom rhymes but each layer has a different signal — and a different fix.

The failures you'll actually see

Most Apache incidents fall into a small set of recurring patterns. Recognise the shape, and triage gets dramatically faster.

CRITICAL

Worker pool exhaustion

Every worker is busy, IdleWorkers hits zero, and Apache logs AH00484: server reached MaxRequestWorkers setting. New connections queue in the kernel listen backlog until it too fills, then get refused with an RST. There is no graceful degradation — the jump from serving normally to queuing everything is nearly instantaneous, and the only buffer is ListenBacklog.

AH00484 in the error log (worker pool at its configured ceiling)
BusyWorkers = MaxRequestWorkers, IdleWorkers = 0
Listen-queue Recv-Q non-zero and climbing
503s appearing while request rate paradoxically drops

Investigate →

ACTIVE

The slow backend cascade

A proxied backend slows down. Each request holds its worker longer, workers pile up in the W state, idle slots drain to zero, and the listen backlog fills — yet Apache's own CPU and memory are perfectly normal, because the workers are waiting, not working. Requests time out as 504s, then 503s once the pool is fully consumed. The site is down; Apache is healthy.

Scoreboard dominated by W states with normal Apache CPU/memory
Backend response time elevated (>5× baseline)
504 Gateway Timeout, then 503, on proxied paths
Direct static requests may still succeed

Investigate →

CRITICAL

Memory exhaustion and the OOM cascade

MaxRequestWorkers × per-child RSS exceeds RAM — often because the pool was sized without doing the memory math, or a mod_php child leaks with MaxConnectionsPerChild 0. The kernel OOM killer targets children (never the parent first), the parent respawns them, and they get killed again. The kill lands in dmesg, not Apache's error log, so you can have a "running" parent with zero functional children.

OOM kill lines in dmesg / journalctl for httpd children
Per-child RSS climbing monotonically over hours/days
Sustained swap usage before the first kill
Parent alive but children absent or constantly respawning

Investigate →

ACTIVE

File descriptor exhaustion

Each connection, log file, and backend socket costs a file descriptor. At the per-process limit a child can no longer accept connections, open files, or reach backends — a cliff-edge that often hits before MaxRequestWorkers because the default ulimit -n of 1024 was never raised. Errors can be sporadic, affecting only the child that peaked, and are routinely misdiagnosed as disk or network faults.

(24)Too many open files repeating in the error log
Per-process FD count near the ulimit in /proc/PID/limits
Intermittent 5xx or connection refused under load
Proxy connect failures (AH01114) alongside the FD errors

Investigate →

CRITICAL

The log stall deadlock

The log filesystem fills, or a piped log program dies. Workers finish their requests but block writing the log and hang in the Logging (L) state. The scoreboard fills with L, throughput collapses, latency goes extreme — and yet the process is up, the port is open, and the error log may stop updating entirely. The server appears alive while serving nothing.

(28)No space left on device (may be the last error logged)
Scoreboard dominated by the L state, sustained > 2 minutes
df shows 100% on the log filesystem
Access log timestamps stop advancing

Investigate →

IMMINENT

The expired certificate outage

A TLS certificate quietly reaches its expiry date and every HTTPS client is locked out at once — browsers show NET::ERR_CERT_DATE_INVALID. It is one of the most deterministic, most preventable outages there is, and because expiry is not an Apache metric it is invisible to server-side monitoring without an explicit external check. Let's Encrypt renews every 90 days, so a stalled renewal is the usual cause.

Days-to-expiry crossing your alert threshold on any vhost
Auto-renewal (certbot) last-run failures
TLS handshake failures appearing only in the error log
Certificate/SNI hostname mismatch failing client-side

Investigate →

The Netdata solution

Apache HTTP Server monitoring with Netdata

Netdata monitors Apache HTTP Server with per-second metrics from mod_status, pre-built dashboards, and ML-powered anomaly detection. Watch busy versus idle workers and the scoreboard state mix, requests per second, bytes served per second, and request processing duration alongside the rest of your stack, so you catch the worker-exhaustion, slow-backend, and memory incidents in these runbooks before they page anyone.

See web server monitoring → Start monitoring free

Choosing a tool

Best Apache Monitoring Tools: 11 Picks Ranked for 2026

A ranked review of the tools teams actually shortlist here, what each one is genuinely good at, and how the pricing behaves as you scale.

Read the buyer's guide →

Apache HTTPD monitoring maturity levels

Apache observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your web tier matters. Most production servers should land at the second level.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor. With these signals you can answer one question: is Apache alive and can it complete a request? You will not learn what broke, but you will learn that something broke before users do. Survival is enough for dev servers and non-critical sites.

Parent process presence Is the httpd/apache2 parent alive, not just a stale PID file?
HTTP critical-path check Does GET / return 2xx within a few seconds, not just a TCP connect?
Log filesystem free space A full log disk freezes workers in the Logging state.
Error-log keyword watch AH00484, Segmentation fault, No space left, Too many open files.

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production servers should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident on its own: worker saturation, backend failures, error spikes, resource pressure, restarts.

BusyWorkers / IdleWorkers The primary saturation gauge; keep ≥25% idle at peak.
5xx rate, 503 specifically 503 is the worker/proxy-pool exhaustion signal; healthy is <0.1%.
Request rate vs baseline A drop while demand holds means silent load shedding.
Request latency p50 / p99 (%D) The distribution, not the average; filter client transfer time.
Total Apache RSS Sum of all children against RAM; the OOM runway.
Per-process FD count vs limit Cliff-edge: at the limit new connections are refused.
Listen-queue depth (Recv-Q) The earliest saturation signal, before latency moves.
Proxy 502/504 rate + cert expiry Backend health surfaced through Apache, plus days-to-expiry.

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone up. A slowly leaking child, a proxy pool creeping toward full, resumption quietly falling off, a backend drifting slower. None of these page you on day one. They become page-out incidents on day thirty.

Full scoreboard state distribution Where time is spent: W (backend), R (slow read), K, L, G.
Per-child RSS trend A monotonically climbing PID is a module leak.
Backend response time per backend Separates "Apache slow" from "backend slow".
Proxy connection-pool utilisation Busy vs max per backend; the default max is too small.
TLS session resumption rate Below ~80% means the handshake CPU wall is near.
Connection states (CLOSE_WAIT) Sustained CLOSE_WAIT is an Apache-side connection leak.
Per-vhost request + error rate One bad vhost hides in the aggregate.
Config-reload validation A failed graceful reload silently keeps the old config.

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. Slowloris detection, the invisible conntrack wall, OCSP stapling, draining behaviour on restart. Most teams never need every signal here. Add the ones your incident history says you do.

Scoreboard R-state ratio Slowloris / slow-read detection; normal is a few percent.
Backend connect vs response time Tells "backend down" (fast refuse) from "backend slow".
OCSP stapling success rate Silent failure makes every client pay OCSP latency.
GracefulShutdownTimeout effectiveness Are old-generation (G) workers actually draining?
mod_reqtimeout rejections Slow-client disconnects as a class apart from real errors.
Kernel nf_conntrack utilisation The table-full wall that drops packets Apache never sees.
Per-request memory sampling Which endpoints bloat a child's RSS.
Event MPM async connection metrics ConnsTotal / ConnsAsyncKeepAlive baseline and drift.

Operating mistakes worth avoiding

The traps Apache teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

Never looking at the scoreboard

Most teams watch CPU, memory, and request rate but never read the scoreboard — the one structure that tells you exactly what every worker is doing. "Why is Apache slow?" always starts there: many <code>W</code> means a backend or slow clients, many <code>R</code> means slow reads or Slowloris, many <code>L</code> means the log disk. Without it you are debugging blind.

Setting MaxRequestWorkers without doing the memory math

The classic footgun: <code>MaxRequestWorkers 1000</code> on a 4GB server with 50MB <code>mod_php</code> children implies 50GB of demand and a catastrophic OOM under load. The setting must be derived from <code>available_memory / per_worker_RSS</code>, and it is bounded by <code>ServerLimit</code> — which needs a full restart, not a graceful reload, to raise.

Not separating backend health from Apache health

When Apache proxies, "Apache is down" and "the backend is down" look identical from outside, and <code>%D</code> bundles backend time with Apache and client-transfer time so it cannot tell them apart. Teams that do not monitor backend response time separately burn hours debugging a perfectly healthy Apache.

Leaving proxy connection pools at their tiny defaults

The default <code>max=</code> for proxy workers is absurdly small for production (often <code>ThreadsPerChild</code>, which is 1 on prefork). Teams hit 503s under moderate load, raise <code>MaxRequestWorkers</code> — the wrong bottleneck — and spiral into misdiagnosis. Size the pool to expected concurrent proxied requests and enable <code>keepalive=</code>.

Interpreting signals without knowing the MPM

Monitoring that is right for prefork — worry when keepalive (<code>K</code>) states consume workers — is counterproductive on event, where keepalive is offloaded to the listener thread and many <code>K</code> connections are normal and efficient. Teams that do not know which MPM they run read every saturation signal wrong.

Alerting on %D without filtering client transfer time

<code>%D</code> includes the time to stream the response to the client, so a 100MB download to a slow client logs as hundreds of seconds even though Apache served it instantly. Alerting on raw <code>%D</code> buries real server-side slowness under client-induced noise. Filter by response size or measure TTFB, and never use <code>%T</code> — it rounds to whole seconds.

Leaving MaxConnectionsPerChild at 0 with leaky modules

The default of <code>0</code> means children never recycle, so a leaking <code>mod_php</code> or <code>mod_perl</code> child grows without bound until the OOM killer intervenes. A finite value (5000–10000) forces periodic recycling and reclaims the leak. It is a band-aid, not a fix, but the right default for any embedded-interpreter deployment.

Putting logs on the same filesystem as everything else

When the log filesystem fills, workers hang in the Logging state and the server serves nothing — the log stall deadlock. If that filesystem is shared with the application or the OS, a log explosion takes down the whole host, not just logging. Give logs their own filesystem and monitor its free space as a first-class signal.

Apache HTTPD runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

Worker pool and the scoreboard

▸

Connections, keepalive, and the listen queue

▸

Reverse proxy and backends

▸

Memory, RSS, and OOM

▸

HTTP errors and the error log

▸

File descriptors and system limits

▸

Throughput and latency

▸

Logging and disk

▸

TLS, certificates, and handshakes

▸

Restarts, reload, and configuration

▸

Security and integrity

WHERE TO GO NEXT

Setting up Apache monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.

> Start with the checklist > Back to Operations Guides