Apache keepalive consuming workers: KeepAliveTimeout, the K state, and MPM choice

The symptom looks like a capacity problem: Apache logs its MPM worker-limit code (for example event-MPM AH00484), new connections start queuing, and users see slow responses or 503s. But the request rate is low, and nothing in CPU, memory, or bandwidth explains it. Then you open the scoreboard and see it: a wall of K states. Most of your workers are not serving requests. They are parked on idle keepalive connections, waiting for a next request that may never come.

On the prefork and worker MPMs, each persistent connection holds a worker slot for the full KeepAliveTimeout, even though no work is happening. With a timeout of 15 to 60 seconds and enough concurrent clients, the pool drains at a fraction of the request rate you would expect. On the event MPM this problem largely disappears, because keepalive handling is offloaded to a listener thread.

What this means

HTTP keepalive lets a client reuse one TCP connection for multiple requests, avoiding the cost of reconnecting (and re-doing the TLS handshake) per request. The tradeoff is that Apache must decide what to do with the connection between requests. The answer depends entirely on the MPM:

• prefork: one child process per connection. A keepalive connection pins an entire process in the K (keepalive read) state until the next request arrives or KeepAliveTimeout expires.
• worker: one thread per connection. Same problem, one level cheaper: a keepalive connection pins a thread.
• event: a dedicated listener thread watches idle keepalive sockets asynchronously (epoll/kqueue) and hands the connection to a worker thread only when a request actually arrives. Idle keepalive connections show up in the ConnsAsyncKeepAlive counter in server-status, not in worker slots.

So on prefork and worker, effective capacity is not MaxRequestWorkers divided by request rate. It is MaxRequestWorkers divided by (request rate x total worker hold time), where hold time includes the keepalive wait. A client that makes one request and then holds the connection for a 30-second KeepAliveTimeout occupies a worker 100 times longer than a request that takes 300 ms.

flowchart LR
  subgraph PF["prefork / worker MPM"]
    C1[client connection] --> W1[worker process/thread]
    W1 -->|request done| K1["K state: waits KeepAliveTimeout"]
    K1 -->|timeout or next request| C1
  end
  subgraph EV["event MPM"]
    C2[client connection] --> L[listener thread]
    L -->|request arrives| W2[worker thread]
    W2 -->|response flushed| L
    L -->|idle| K2["ConnsAsyncKeepAlive (no worker held)"]
  end

Common causes

Quick checks

All of these are read-only.

# 1. Confirm which MPM is active
apachectl -V 2>/dev/null | grep -i mpm || httpd -V | grep -i mpm

# 2. Pull the machine-readable status page
curl -s http://localhost/server-status?auto

# 3. Count scoreboard states - look for a dominant K population
curl -s http://localhost/server-status?auto | grep "Scoreboard:" | \
  awk '{print $2}' | fold -w1 | sort | uniq -c | sort -nr

# 4. Busy vs idle workers
curl -s http://localhost/server-status?auto | grep -E "BusyWorkers|IdleWorkers"

# 5. On event MPM: check async connection counters
curl -s http://localhost/server-status?auto | grep -E "^Conns"

# 6. Confirm worker exhaustion events in the error log
grep -hE 'AH(00161|00286|00484)' /var/log/apache2/error.log /var/log/httpd/error_log 2>/dev/null | tail -5

# 7. Compare connection count to request throughput
ss -tn 'state established and ( sport = :80 or sport = :443 )' | wc -l

The diagnostic signature: K count is a large fraction of all slots, BusyWorkers is near MaxRequestWorkers, IdleWorkers is zero, but the Total Accesses delta over a minute is modest. High hold time, low work.

How to diagnose it

1. Confirm the MPM. Everything downstream depends on this: apachectl -V | grep -i mpm, or check the loaded mpm_* module in your config. If you are on prefork because of mod_php or another non-thread-safe module, keepalive tuning matters more, not less.

2. Read the scoreboard. Take the state distribution (check 3 above). On prefork/worker, K above roughly 30% of slots is the pattern. On event, keepalive should barely appear in the scoreboard at all; a significant K population on event is itself a finding, because it means connections are not being offloaded to the listener thread. Apache falls back to worker-style handling for connection filters that declare themselves incompatible with event, in which case one worker thread is reserved per connection again.

3. Separate K from W. If the scoreboard is dominated by W (sending reply) instead of K, you have a different problem: slow clients or a slow backend holding workers during response generation. Keepalive tuning will not fix that. See Apache scoreboard states explained for the full state-by-state reading.

4. Check the queue and the error log. Run ss -ltn on the listening ports: a growing Recv-Q while workers are stuck in K confirms new connections are queuing behind idle keepalive holds. The MPM worker-limit code in the error log (prefork AH00161, worker AH00286, event AH00484) confirms the pool hit its ceiling.

5. Quantify the hold time. Pull the current KeepAliveTimeout, KeepAlive, and MaxKeepAliveRequests values from the running config. Defaults in 2.4.x are KeepAlive On, KeepAliveTimeout 5, MaxKeepAliveRequests 100. If someone raised the timeout to 15, 30, or 60 seconds “to reduce handshake overhead,” that is your smoking gun on prefork/worker.

6. Check who is holding the connections. ss -tn sport = :80 | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head shows whether a few source IPs (a load balancer, a monitoring system, a single misbehaving client) account for most of the held connections.

Metrics and signals to monitor

Fixes

Reduce KeepAliveTimeout (prefork/worker)

This is the direct fix. Apache’s own performance guidance recommends keeping the timeout low and discourages raising it above roughly 60 seconds; the 5-second default exists precisely to bound this effect. For high-traffic prefork or worker deployments, 1 to 2 seconds is a common choice. Most browsers fire their follow-up requests (CSS, JS, images) within a second or two of the first response, so a short timeout preserves the practical benefit of keepalive while freeing workers quickly.

KeepAlive On
KeepAliveTimeout 2

Tradeoff: shorter timeouts mean clients reconnect more often. On HTTPS that means more TLS handshakes, which cost CPU. If you cut the timeout and see CPU rise on TLS-heavy sites, that is the cost of the trade, and it is usually still cheaper than exhausted workers.

Bound MaxKeepAliveRequests

MaxKeepAliveRequests caps how many requests one connection can serve before Apache closes it (default 100). Lowering it forces periodic connection turnover, which recycles workers and also helps bound per-child memory growth. It does not fix the idle-hold problem on its own; combine it with the timeout change.

Move to the event MPM

If nothing in your module set requires prefork (classic mod_php is the usual reason), event removes the problem structurally: keepalive connections stop consuming worker threads at all. Two caveats:

• Switching MPM means disabling one mpm_* module and enabling another, then a full Apache restart, so plan it as a disruptive change. Validate module compatibility (mod_php in particular) on a staging host first.
• Some connection filters are incompatible with event and force it back into worker-style one-thread-per-connection behavior for those connections, which quietly reintroduces the problem for the affected traffic.

After switching, verify with the scoreboard: K should nearly vanish and ConnsAsyncKeepAlive should carry the idle load instead.

Raise MaxRequestWorkers only after fixing the hold time

Adding workers to absorb keepalive holds is buying memory to park idle connections. On prefork especially, each worker is a full process, and MaxRequestWorkers x per-child RSS must stay within RAM. Fix KeepAliveTimeout first; then re-evaluate pool size against real concurrency. See Apache MaxRequestWorkers tuning for the sizing math.

Put a keepalive-friendly layer in front

If you cannot leave prefork (mod_php) and cannot tolerate short timeouts, a reverse proxy or load balancer that handles keepalive efficiently in front of Apache absorbs the idle connections and speaks to Apache over a small, well-behaved connection pool. This moves the problem off the scarce resource (Apache workers) onto a component built for cheap connection handling.

Prevention

• Alert on the K fraction, not just BusyWorkers. BusyWorkers at 95% tells you the pool is full; the K fraction tells you why, and it rises before the pool fills.
• Alert on K appearing on event MPM. It means offloading has failed and you are silently back to worker-style connection costs.
• Pin KeepAliveTimeout in config management and treat increases as a reviewed change. A well-meaning bump from 5 to 30 seconds is how this incident usually starts.
• Watch connections-per-request ratio over time. A rising ratio at constant RPS is an early warning that clients or intermediaries are holding connections longer.
• Re-validate after any MPM or module change. Switching MPMs, adding a connection filter, or fronting Apache with a new LB all change keepalive economics.

How Netdata helps

• Netdata collects the Apache scoreboard continuously via server-status, so you see the K population as a time series rather than a point-in-time snapshot during the incident.
• BusyWorkers and IdleWorkers are charted together, making the “full pool, low throughput” divergence visible at a glance.
• On event MPM, ConnsAsyncKeepAlive and the other async connection counters are charted alongside worker states, so offloading failures (scoreboard K rising while async keepalive stays flat) stand out.
• Correlating the scoreboard breakdown with requests per second on the same dashboard separates keepalive hoarding (K-dominant, low RPS) from slow-backend starvation (W-dominant) in seconds.
• Error log alerting catches AH00484 the moment the pool ceiling is hit, instead of after users report 503s.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion
• Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status
• How Apache HTTPD actually works in production: a mental model for operators
• Apache AH00484: server reached MaxRequestWorkers setting - worker pool exhausted
• Apache MaxRequestWorkers tuning: sizing the worker pool against memory
• Apache HTTPD monitoring checklist: the signals every production web server needs
• Apache HTTPD monitoring maturity model: from survival to expert
• Apache scoreboard states explained: what _ S R W K D C L G tell you