How Apache HTTPD actually works in production: a mental model for operators

Most Apache incidents are misdiagnosed for the same reason: the operator is reading signals without knowing which execution model produced them. A scoreboard full of K states is a crisis on prefork and a rounding error on event. “Apache is down” and “the backend is down” look identical from outside a reverse proxy. MaxRequestWorkers means something different depending on whether your concurrency unit is a process or a thread.

This article is the model layer. It covers the abstractions every Apache runbook, alert, and tuning decision depends on: the three MPMs, the scoreboard, worker pool arithmetic, the kernel listen backlog, the module pipeline, mod_proxy connection pools, and APR memory pools. Almost every Apache outage is one of a small set of resource exhaustion patterns, and each maps directly onto a structure described here.

This is grounded in Apache 2.4 behavior. Where other versions differ in a way that matters operationally, it is called out.

The MPM decides everything

The Multi-Processing Module determines how an incoming connection maps to an execution unit. It is compiled in or loaded once, and it shapes the meaning of nearly every metric you will collect. Check which one you are running before interpreting anything:

# Identify the active MPM
apachectl -V 2>/dev/null | grep MPM
# or
httpd -V 2>/dev/null | grep MPM

prefork. One process per connection. Each child handles exactly one request at a time. No threading, which makes it safe for non-thread-safe modules (classic mod_php is the usual reason it still exists). Per-process RSS is typically 10-50 MB or more depending on loaded modules. The resource you run out of is memory, because every concurrent connection costs a full process.

worker. Hybrid. Multiple child processes, each running multiple threads, each thread handling one connection. Far more memory-efficient than prefork. The failure mode to internalize: a single stuck backend can hold a thread indefinitely, starving that child’s thread pool.

event. An evolution of worker, and the default in Apache 2.4. A dedicated listener thread manages keepalive connections asynchronously (epoll on Linux, kqueue on BSD) and hands a connection to a worker thread only when an actual request arrives. This decouples keepalive connection count from worker thread consumption. On event, keepalive connections show up in ConnsAsyncKeepAlive in server-status, not as K states in the scoreboard. Significant K in the scoreboard on event is abnormal and worth investigating; the same picture on prefork is routine.

Two event-MPM caveats from the 2.4 documentation that matter in production: it falls back to worker-like behavior for connection filters that declare themselves incompatible with async operation, and for output filters that must buffer the whole response body (CGI, FCGI, and some proxied content paths). And the event MPM’s well-known “scoreboard is full, not at MaxRequestWorkers” error means all active-process slots are occupied even though active requests are below MaxRequestWorkers, often because graceful generations linger. Since 2.4.24, event may use allocated slots up to ServerLimit for graceful processes while MaxRequestWorkers continues to bound active processes; the error remains possible at that higher boundary.

How a request moves through the server

The event MPM path, since that is what most production 2.4 installs run:

flowchart LR
  C[Client] -->|SYN| KB[Kernel listen backlog]
  KB -->|accept| L[Listener thread]
  L -->|new request| W[Worker thread pool]
  L -.->|idle keepalive| KA[Async keepalive via epoll]
  KA -->|request arrives| W
  W --> P[Module pipeline phases]
  P --> H[Handler or mod_proxy]
  H --> B[Backend connection pool]
  H --> R[Response to client]

On prefork there is no listener thread or async keepalive: a child process accepts a connection and holds it for its entire lifetime, including keepalive idle time. That single difference explains why prefork melts under connection floods that event handles trivially.

The scoreboard: Apache’s most diagnostic structure

The scoreboard is a shared-memory segment with one slot per possible worker, each slot recording the worker’s current state. Its size is fixed at startup as ServerLimit times ThreadsPerChild and cannot grow dynamically. mod_status is just a reader of this segment.

The states that drive diagnosis:

Two structural facts matter. First, W is ambiguous: writing bytes to the client, waiting on a backend, and doing internal processing all render as W, so a scoreboard full of W tells you workers are held, not why. Second, a slot in . state with load present means the scoreboard has room but children are not spawning, which points at memory or process limits rather than Apache config.

Worker pool arithmetic

Three directives bound the pool, and they interact:

• ThreadsPerChild: threads per child process. Default 25 on worker/event.
• ServerLimit: ceiling on child processes. Default 16 on worker/event, 256 on prefork.
• MaxRequestWorkers: the actual cap on simultaneous requests, renamed from MaxClients in 2.3.13.

On worker/event, the defaults multiply out to 16 x 25 = 400. On prefork, where each process is one worker, the default MaxRequestWorkers is 256. The sizing rule that prevents the most common catastrophic misconfiguration:

MaxRequestWorkers <= available_memory_for_apache / per_worker_memory

Setting MaxRequestWorkers to 1000 on a 4 GB host with mod_php children at 50 MB is a 50 GB commitment against 4 GB of RAM. The result is swap thrash and an OOM kill cascade under load. Derive the limit from measured per-child RSS, and keep Apache’s theoretical maximum (MaxRequestWorkers x worst-case child RSS) under roughly 70% of RAM.

One operational trap from the 2.4 docs: changes to ServerLimit (and ThreadLimit) are ignored during a graceful restart. They only take effect on a full stop and start. If you raised ServerLimit, did a graceful, and saw no effect, that is why.

The kernel listen backlog

When all workers are busy, new connections do not fail immediately. They queue in the kernel’s TCP listen backlog, sized by ListenBacklog (default 511) and silently capped by net.core.somaxconn if that is lower. When the backlog fills, clients get RST or timeouts: the port is open, the process is alive, and the service is effectively down.

This is the leading indicator of worker exhaustion. ss -ltn shows Recv-Q (current backlog depth) against Send-Q (the configured maximum) for listening sockets. Brief non-zero Recv-Q during bursts is normal; sustained Recv-Q means connections arrive faster than Apache accepts them, and it appears before user-visible latency because those connections have not reached a worker yet.

The module and filter pipeline

Every request passes through ordered phases: URI translation, access control, authentication, content generation by a handler, then logging. Every module hook can block, fail, or add latency, and input/output filters (mod_ssl, mod_deflate, mod_headers) sit in a chain around the handler, each allocating from the request’s memory pool.

Operationally, this explains three things. A worker in W state may be stuck inside any module in the chain, not “sending a reply.” mod_deflate buffers before it can send the first byte, which inflates time-to-first-byte without anything being wrong. And complex filter chains inflate per-request memory, which feeds back into the worker pool sizing math above.

mod_proxy backend pools

When Apache reverse-proxies, each child process maintains its own pool of backend connections. Two facts cause most proxy-side incidents:

• The default pool max equals ThreadsPerChild (so 1 on prefork). Total backend concurrency is max times the number of children, and the defaults are far too small for production load.
• Pool exhaustion is a cliff. Without acquire, an unavailable worker returns HTTP 503 immediately; with acquire, Apache waits only for the configured milliseconds, then returns SERVER_BUSY. There is no unbounded queue.

This produces the signature misdiagnosis: 503s under moderate load, operator raises MaxRequestWorkers (the wrong bottleneck), nothing improves. Separately, a slow backend holds frontend workers in W state for the full backend latency, so backend slowness consumes your entire worker pool from the inside while Apache’s own CPU and memory look normal. Proxy-specific error codes decode as: 502 (backend refused or answered garbage), 503 (pool exhausted or all balancer members errored), 504 (backend exceeded ProxyTimeout).

APR memory pools

Apache allocates through Apache Portable Runtime pools rather than per-object malloc. Memory for a connection or request comes from a pool that is destroyed wholesale when that connection or request ends. This is fast and prevents most classic leaks, but freed memory returns to the process’s allocator, not necessarily to the OS, so a child that has served many requests can hold a large RSS indefinitely. MaxMemFree (default 2048 KB) caps how much free memory the allocator retains, but it limits the free list, not total process size. The reliable bound is MaxConnectionsPerChild: the default 0 means children never recycle, which is the wrong default for anything running mod_php or mod_perl. A finite value (commonly 5000-10000) forces periodic recycling and turns unbounded leak growth into a sawtooth.

Where the model pays off

Each characteristic Apache failure is one of these structures saturating:

• Worker exhaustion: all slots in W, R, D, or K; backlog fills; the MPM worker-limit message (for example AH00484) appears once in the error log.
• Slow backend cascade: W states climb, IdleWorkers falls, 504s then 503s, while request rate paradoxically drops because queued requests never complete.
• Memory exhaustion: MaxRequestWorkers x per-child RSS exceeds RAM; swap, then OOM kills, then respawn-and-leak-again.
• Slowloris: many connections trickling bytes hold workers in R indefinitely; mod_reqtimeout is the defense.
• Log stall: disk full or dead log pipe; workers finish requests but block in L; the server looks alive and serves nothing.
• Graceful restart pile-up: G states accumulate, multiple child generations coexist, and memory multiplies. Reduce reload frequency; GracefulShutdownTimeout applies to graceful-stop, not a graceful reload.

Signals to watch in production

How Netdata helps

• The Apache collector polls server-status?auto and charts BusyWorkers, IdleWorkers, request rate, and the full scoreboard state distribution over time, so a drift toward W or R dominance is visible before the pool empties.
• Async connection counters (ConnsTotal, ConnsAsyncKeepAlive, ConnsAsyncWriting, ConnsAsyncClosing) are charted on event MPM, which makes the “why are there K states on event” question answerable at a glance.
• Because Netdata also collects per-process RSS, system memory, and TCP listen socket stats from the same host, you can correlate worker utilization with the backlog filling and with per-child memory growth in one view, instead of sampling ss and ps by hand during an incident.
• Error rate and latency signals from log-based collection sit on the same dashboard as the scoreboard, which is exactly the join you need to distinguish a slow backend cascade (workers held, Apache CPU idle, 504s rising) from genuine overload.

Apache monitoring in Netdata brings these signals together with per-second metrics and anomaly detection.

Related guides

• Apache HTTPD monitoring checklist: the signals every production web server needs
• Apache HTTPD monitoring maturity model: from survival to expert
• Apache AH00484: server reached MaxRequestWorkers setting — worker pool exhausted
• Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion
• Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status
• Apache scoreboard states explained: what _ S R W K D C L G tell you
• Apache MaxRequestWorkers tuning: sizing the worker pool against memory
• Apache listen queue overflow: Recv-Q growth, ListenBacklog, and refused connections
• Apache keepalive consuming workers: KeepAliveTimeout, the K state, and MPM choice
• Apache ListenBacklog vs net.core.somaxconn: the silently truncated accept queue
• Apache CLOSE_WAIT and TIME_WAIT: connection leaks versus normal churn
• Apache 502 Bad Gateway: a backend that returned an invalid response