Apache scoreboard states explained: what _ S R W K D C L G tell you

When someone asks “why is Apache slow?”, the first place to look is not CPU, not memory, not the access log. It is the scoreboard. The scoreboard is a shared-memory segment where every worker slot records what it is doing right now, one character per slot. It is what mod_status reads, and it is the most diagnostic structure Apache exposes.

A scoreboard snapshot answers the question metrics alone cannot: not “is Apache busy?” but “busy doing what?” A server with 200 workers stuck in W is a different incident from 200 workers stuck in R or L, even though all three look identical from the outside: the site is down.

This article decodes every scoreboard character, explains what a distribution skewed toward each state means operationally, and covers where your MPM changes the interpretation.

What the scoreboard is

Apache’s parent process allocates the scoreboard at startup as a fixed-size shared-memory segment. The total number of slots is bounded by ServerLimit × ThreadsPerChild and cannot grow dynamically. Each slot holds one worker’s current state, updated by the worker itself as it moves through the request lifecycle.

You read it through mod_status. The machine-readable output is at /server-status?auto, which includes a Scoreboard: line with one character per slot:

# Count scoreboard states
curl -s 'http://localhost/server-status?auto' | grep "Scoreboard:" | \
  awk '{print $2}' | fold -w1 | sort | uniq -c | sort -nr

A healthy server under moderate load shows mostly _ (idle workers waiting for connections), a working set of W and R, some K depending on MPM, and . for slots with no process behind them yet. The absolute counts matter less than the shape of the distribution and how it changes over time. mod_status only gives point-in-time snapshots, so sample periodically to see trends.

Two caveats before decoding. First, W has no sub-state detail: a worker writing bytes to a client, waiting on a slow backend, or doing internal processing all show as W. Second, HTTP/2 multiplexing is invisible: one connection with ten concurrent streams occupies one scoreboard slot.

The states, one by one

The lifecycle below shows how a slot moves between states. D sits off the main path because it only appears when something forces per-request DNS resolution.

stateDiagram-v2
    op : . open slot
    st : S starting
    wt : _ waiting
    rd : R reading
    dn : D DNS lookup
    wr : W sending reply
    ka : K keepalive
    cl : C closing
    lg : L logging
    gr : G graceful finish
    ic : I idle cleanup
    op --> st : child spawns
    st --> wt
    wt --> rd : request arrives
    rd --> dn : HostnameLookups on
    dn --> rd
    rd --> wr : request parsed
    wr --> ka : keepalive on
    ka --> rd : next request
    ka --> cl : keepalive timeout
    wr --> cl : keepalive off
    cl --> lg
    lg --> wt
    wt --> gr : graceful restart
    gr --> ic : drained
    ic --> op

What a skewed distribution tells you

Many W: slow clients or a slow backend

W is the normal working state, so some population is expected. It becomes a signal when W climbs toward your worker limit while throughput stays flat or drops. Because W lumps together “writing to the client”, “waiting on a proxied backend”, and “processing”, you must disambiguate:

• If you proxy to a backend, check the backend directly, bypassing Apache: curl -s -o /dev/null -w "%{time_total}\n" http://backend-host:port/health. If the backend is slow, workers pile up in W waiting for it. Apache’s CPU and memory look normal because workers are waiting, not working. This is the most common cause of “Apache outage” in reverse-proxy deployments.
• If you serve content directly, many W with low CPU points to slow clients reading responses, or to disk I/O stalls.

Sustained W above roughly half of all workers, with IdleWorkers trending to zero, is a near-saturation alarm. At the limit, the error log shows AH00484: server reached MaxRequestWorkers setting. See Apache AH00484: server reached MaxRequestWorkers setting.

Many R: slow request bodies or Slowloris

Normal traffic rarely holds more than a few percent of workers in R, because well-behaved clients send requests quickly. A large R population means connections are dribbling data in slowly. Three candidates:

• Legitimate slow uploads or clients on very poor networks.
• A slow-read (Slowloris-style) attack: many connections each holding a worker by sending the request byte by byte.
• A load balancer misconfiguration sending incomplete requests.

Above about 20% of workers in R, investigate. Check source-IP concentration:

# Connection count per source IP
ss -tn 'sport = :80' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head -20

If a few IPs dominate, block them at the firewall, not in Apache: by the time you see this, workers are already exhausted. The standing defense is mod_reqtimeout (loaded by default in 2.4), for example RequestReadTimeout header=20-40,MinRate=500 body=20,MinRate=500, which kills connections that send data too slowly.

Many K: meaning depends entirely on your MPM

This is the state operators misread most often, because the same character means opposite things on different MPMs.

On prefork and worker, a keepalive connection holds a worker hostage for the whole KeepAliveTimeout even though no request is being processed. A scoreboard with 30% or more K on these MPMs means you are wasting a third of your capacity on idle connections. Lower KeepAliveTimeout or accept the connection churn.

On event MPM, keepalive connections are offloaded to a dedicated listener thread and tracked in ConnsAsyncKeepAlive, not in worker slots. A high ConnsAsyncKeepAlive is normal and efficient. Significant K in the scoreboard on event MPM is abnormal: it suggests connections are not being offloaded to the async path, and it deserves investigation.

# Event MPM async connection counters
curl -s 'http://localhost/server-status?auto' | grep -E "^Conns"

Any D: DNS-based access control, stop it

D means a worker is blocked on a DNS lookup. Workers should essentially never do per-request DNS. If you see D at all, something is forcing it: HostnameLookups On, or access control or mod_rewrite rules that resolve client hostnames. Each lookup holds a worker for the duration of the resolver round trip, and a slow or unreachable resolver turns this into worker exhaustion. Turn it off, use IP-based rules, and resolve hostnames offline in log analysis if you need them. Even 5% of workers in D is a problem.

Many L: log stall, disk full, or dead log pipe

Workers finish a request, then block writing the log line. Logging is synchronous by default, so if the log write blocks, the worker blocks. Causes, in order of likelihood:

• The log filesystem is full: df -h /var/log/apache2/ (or /var/log/httpd/).
• A piped logger (rotatelogs or similar) died or stalled, and writes to the pipe block.
• The log filesystem is on saturated or failing storage.

A scoreboard dominated by L with throughput near zero is the log stall deadlock: the process is alive, the port is open, and nothing is served. Check disk space first. If a piped logger’s child process dies, workers can also crash on SIGPIPE, so the error log may show child deaths alongside the stall.

Many G: graceful restart draining, or stuck

G is expected immediately after a graceful restart (apachectl graceful, SIGUSR1): old-generation children finish in-flight requests, then exit. It is a problem when G persists:

• Slow requests (large downloads, slow backends) keep old workers alive for minutes. Frequent restarts then stack generations, multiplying memory. Check restart frequency: grep "resuming normal operations" /var/log/apache2/error.log | tail -20. Set GracefulShutdownTimeout to bound graceful-stop drains — note it does not bound a graceful-restart (SIGUSR1) drain on current 2.4.x, so limiting restart frequency is the effective lever here.
• On event MPM before Apache 2.4.25, a known bug (PR 53555) could leave workers stuck in G indefinitely after a graceful restart until the scoreboard filled and the server stopped accepting connections; the fix shipped in 2.4.25 (the unreleased 2.4.24 line rolled into it). If you are on event MPM and an old 2.4 release, upgrade before doing anything else. A non-zero MaxConnectionsPerChild on event MPM has also been reported to trigger G buildup even on patched versions.

Many . with high load: room in the scoreboard, but no processes

. means the slot exists but nothing is running in it. On a quiet server this is unused capacity. Under load, many . slots alongside few _ and rising queueing means Apache wants to spawn workers but cannot. The limits to check are outside Apache: available memory (per-child RSS times current children against RAM), the per-user process limit, and, on systemd units, TasksMax, which caps total processes plus threads and can silently pin Apache below its configured MaxRequestWorkers.

Few _: headroom is gone

Idle workers are your burst absorber. If _ falls below roughly 10-15% of MaxRequestWorkers during normal peak, you have no margin: the next burst goes into the kernel listen backlog, and once that fills, clients get connection refused while the server still looks “up”. Track this as a capacity signal, not just an incident signal.

Signals to watch in production

How Netdata helps

• Netdata charts the scoreboard state distribution as a time series, so you see W, R, K, L, or G populations building over minutes instead of catching one snapshot mid-incident.
• BusyWorkers versus idle workers is graphed continuously, which turns “headroom shrinking at daily peak” into a visible trend before the cliff edge.
• Correlating a rising W share with upstream response time and flat CPU on the same dashboard is the fastest way to confirm a slow-backend cascade rather than an Apache problem.
• A rising R share next to per-IP connection counts makes slow-read attacks distinguishable from legitimate slow uploads in one view.
• L states correlated with disk space and disk I/O on the log filesystem confirms a log stall without SSH-ing in to run df.
• Event MPM async counters (ConnsAsyncKeepAlive, ConnsAsyncWriting) alongside the scoreboard make the MPM-specific K interpretation explicit.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• How Apache HTTPD actually works in production: a mental model for operators
• Apache AH00484: server reached MaxRequestWorkers setting - worker pool exhausted
• Apache HTTPD monitoring checklist: the signals every production web server needs
• Apache HTTPD monitoring maturity model: from survival to expert