Users report the site is slow. Apache is up, requests eventually complete. Someone says “Apache is slow,” someone else says “the app is slow,” and both are guessing, because from the outside the two are indistinguishable: the client just sees a slow response.

When Apache proxies to a backend, the %D value in your access log bundles three things together: Apache’s own processing, time waiting for the backend, and time transferring the response to the client. A slow backend, a saturated Apache, and a client on a bad connection all produce the same inflated %D. Without separate instrumentation per component, you cannot attribute the latency, and teams routinely burn hours tuning Apache when the database is the problem.

This article is about closing that gap: measuring backend response time separately, reading the scoreboard to see where workers are stuck, and telling “backend down” from “backend slow” from “Apache itself is the bottleneck.”

What this means

A proxied request has three latency contributors, and only one of them is Apache:

  1. Backend time: establishing the backend connection (if no pooled connection exists) and waiting for the response. Usually the dominant term in reverse-proxy deployments.
  2. Apache processing: URI translation, access control, rewrite rules, filter chain (compression, headers). Typically small unless mod_security or complex mod_rewrite is in play.
  3. Client transfer: sending the response body. A 100MB download to a 1Mbps client shows %D of roughly 800 seconds, and none of that is Apache or backend slowness.

The slow-backend cascade is the failure mode to fear. A backend gets slow, each proxied request holds an Apache worker longer, workers accumulate in W state, IdleWorkers drops toward zero, new requests queue in the listen backlog, and you see 504s and then 503s as the pool fully exhausts. Apache is healthy the entire time; it is starved of workers by the backend. This is the most common cause of “Apache outage” in proxy deployments.

The scoreboard cannot resolve the ambiguity on its own: a worker in W state could be writing to the client, waiting on a backend, or doing internal processing. All three show as W. You need backend timing as an independent signal.

Common causes

CauseWhat it looks likeFirst thing to check
Backend application slow (DB locks, GC pause, slow queries)Scoreboard filling with W states, 504s starting, Apache CPU and memory normalcurl the backend directly, bypassing Apache, and time it
Backend down or unreachable502s (connection refused), fast failures rather than slow onesError log for AH01114 and “(111)Connection refused”
Network partition between Apache and backendConnection timeouts to backend, not response timeoutsError log proxy timeout messages; connect vs response time split
Proxy connection pool too small503s under moderate load, workers not exhaustedmax on ProxyPass/BalancerMember vs concurrent proxied requests
Slow clients inflating %DHigh %D only on large responses, backend fast, scoreboard mostly healthyCorrelate %D with response size; check ConnsAsyncWriting on event MPM
Apache-internal processing (mod_security, rewrite, TLS)High %D with fast backend and small responses; elevated Apache CPUCPU per request; TTFB vs total time split

Quick checks

# 1. Scoreboard state distribution: where are workers stuck?
curl -s http://localhost/server-status?auto | grep Scoreboard | \
  sed 's/Scoreboard: //' | fold -w1 | sort | uniq -c | sort -rn

# 2. Busy vs idle workers
curl -s http://localhost/server-status?auto | grep -E "BusyWorkers|IdleWorkers"

# 3. Time the backend directly, bypassing Apache entirely
curl -s -o /dev/null -w "TTFB: %{time_starttransfer}s  Total: %{time_total}s\n" \
  http://backend-host:backend-port/health

# 4. Time the same path through Apache
curl -s -o /dev/null -w "TTFB: %{time_starttransfer}s  Total: %{time_total}s\n" \
  http://localhost/proxied-path

# 5. Proxy errors in the access log (502/503/504 tell different stories)
tail -5000 /var/log/apache2/access.log | awk '$9 ~ /^50[234]$/ {print $9}' | sort | uniq -c

# 6. Proxy error detail in the error log
grep -E "AH01114|AH00898|proxy:|Connection refused" /var/log/apache2/error.log | tail -20

# 7. Current backend connections from this Apache instance
ss -tn state established dport = :8080 | wc -l   # adjust to your backend port

# 8. P95 of %D for proxied requests (assumes %D is last field)
grep "/proxied-path" /var/log/apache2/access.log | tail -1000 | \
  awk '{print $NF}' | sort -n | awk '{a[NR]=$1} END {print "p95 (us):", a[int(NR*0.95)]}'

Checks 3 and 4 are the decisive pair. If the backend is slow when hit directly, the conversation about Apache config is over. If the backend is fast directly but slow through Apache, the problem is on the Apache side: pool exhaustion, queuing, or client transfer.

How to diagnose it

Apache has no built-in per-request backend timing. %D cannot be decomposed after the fact, so diagnosis has two phases: a live comparison you can run immediately, and instrumentation you add so the next incident is answerable from logs alone.

flowchart TD
  A[Slow proxied responses] --> B{Backend slow when curled directly?}
  B -- Yes --> C[Backend is the problem: DB, GC, app, network to backend]
  B -- No --> D{Scoreboard dominated by W states, IdleWorkers near zero?}
  D -- Yes --> E[Workers held by something: check pool exhaustion and 503s]
  D -- No --> F{High %D only on large responses?}
  F -- Yes --> G[Client transfer time, not server slowness]
  F -- No --> H[Apache-internal: CPU, mod_security, rewrite, TLS]
  1. Probe the backend directly. Run check 3 from the Apache host itself, so the network path matches what mod_proxy uses. Probe the actual slow endpoint, not just a health page: a backend can answer /health in 2ms while the real query takes 20 seconds.
  2. Probe through Apache on the same path. Compare TTFB and total time. The delta between direct and proxied measurements is Apache’s overhead plus client transfer. On localhost with a small response, Apache’s own overhead should be milliseconds.
  3. Read the scoreboard. Workers accumulating in W state plus high direct backend timing means workers are waiting on the backend. Healthy scoreboard plus fast backend timing points at response sizes and client behavior instead.
  4. Split connect failure from response slowness. A backend that refuses connections fails fast (502, “Connection refused”). A backend that accepts and never answers fails slow (504 at ProxyTimeout, default 60s). Different incidents, different owners. The error log distinguishes them.
  5. Check the proxy pool. If the backend is fast but you see 503s under load, suspect the pool. The default max for proxy workers equals ThreadsPerChild (1 for prefork), and pools are per-child-process, so total backend capacity is max times the number of children. Pool exhaustion is a cliff: 503 immediately, no queuing.
  6. Instrument for next time. The live comparison works during an incident, but the real fix is making backend timing a permanent log field (see Fixes).

One caveat on ProxyTimeout: a backend that dribbles data slowly but continuously may never trigger it, even if the full transfer takes minutes, so a “no 504s” log does not prove the backend is healthy.

Instrumenting backend timing

Three practical options, in increasing order of effort.

Backend-injected response header. Have the backend application emit its own processing time as a response header, for example X-Backend-Time, and log it in Apache:

LogFormat "%h %l %u %t \"%r\" %>s %b %D %{X-Backend-Time}o" combined_timing
CustomLog /var/log/apache2/access.log combined_timing

%D is total time in microseconds; %{X-Backend-Time}o is the backend’s self-reported time. The difference, minus client transfer, is Apache’s overhead. This gives you a per-request decomposition in the access log, which is exactly what %D alone cannot provide. It requires the backend to cooperate, and the header format is application-defined.

mod_log_debug. With hook=all, mod_log_debug logs a message at each phase of request processing, and the microsecond timestamps in the error log let you reconstruct where time went inside Apache. mod_log_debug is marked Experimental in 2.4 and per-phase logging is verbose; use it for a bounded investigation on a single vhost, not as permanent fleet-wide config.

ProxyStatus On. This makes mod_status display per-backend proxy worker status alongside the scoreboard, so you can see which balancer members are busy, in error, or disabled. It does not give per-request backend timing, but it tells you whether specific backends are being marked down, which distinguishes “one backend sick” from “all backends slow.”

A separate option worth knowing: mod_proxy_hcheck (2.4.21+) runs out-of-band health checks against backends using TCP, OPTIONS, HEAD, or GET, independent of live traffic. This separates “backend alive” from “backend fast under real load,” which are different questions.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Backend direct-probe latency (TTFB and total)The only clean measurement of backend time, independent of ApacheP95 above 2x baseline sustained
Scoreboard W state countWorkers waiting on backends pile up hereW above 50% of workers sustained, with normal CPU
IdleWorkersHeadroom before queuing startsTrending toward zero during a latency event
502 / 503 / 504 split502 = backend invalid/refused, 503 = pool or worker exhaustion, 504 = backend timeoutAny sustained proxy error rate; 504-then-503 is the cascade signature
Proxy pool utilization per backendPool exhaustion is a cliff with no queuingBusy proxy workers approaching max
%D correlated with response sizeSeparates client-induced latency from server-inducedHigh %D concentrated on large responses only
Listen backlog Recv-QConfirms workers cannot keep up once exhaustedSustained Recv-Q > 0 alongside high W count

Fixes

If the backend is slow. The fix belongs to the backend team, but you can limit the blast radius at Apache. Reduce ProxyTimeout temporarily so workers fail fast instead of being held for the full 60-second default; requests fail with 504 quickly, workers recycle, and Apache keeps serving whatever it can. If the backend is fully unresponsive, take the Apache instance out of LB rotation rather than letting it absorb the queue.

If the proxy pool is too small. Raise max on the ProxyPass or BalancerMember directive, remembering the pool is per-child-process: total backend connections are max times the number of children. Size it at roughly 2x expected concurrent proxied requests per child at peak. In prefork, be careful in the other direction: N processes times M backends times pool size can overwhelm the backend with connections. Also set keepalive=On on ProxyPass so backend connections are reused instead of re-established per request. Do not raise MaxRequestWorkers to fix pool exhaustion; that treats the wrong bottleneck and is a common misdiagnosis spiral.

If client transfer is inflating your latency data. Do not “fix” this by changing Apache. Fix your alerting: filter large responses out of latency percentiles, or alert on the backend header time instead of raw %D. On event MPM, slow-client writes are handled asynchronously and tracked in ConnsAsyncWriting, which is the right signal to watch for client-side slowness.

If the latency is genuinely Apache-internal. Check CPU per request. The usual suspects are TLS handshakes without session resumption, mod_security rule cost, complex mod_rewrite, and compression. That is a different article’s worth of diagnosis; you only reach this branch after backend time and client transfer are excluded.

Prevention

  • Log backend time per request. Add the backend timing header to your LogFormat now, not during the next incident. Without it, every latency investigation starts from the same blind spot.
  • Probe backends directly in your monitoring. A health check through Apache tests the whole chain; a health check against the backend tests the backend. You need both, graphed side by side.
  • Alert on the cascade signature, not just the endpoint state. W states rising, IdleWorkers falling, and 504s appearing together is the early phase. Paging on 503s means you find out after the pool is already exhausted.
  • Separate connect failure from response slowness in dashboards. They have different causes, different owners, and different fixes.
  • Size the proxy pool deliberately. The defaults are too small for most production workloads, and pool exhaustion masquerades as an Apache capacity problem.
  • Use health checks that exercise the real service path. A check that only fetches a static file from Apache reports green while every proxied request times out.

How Netdata helps

  • Netdata collects the mod_status scoreboard continuously, so W-state accumulation, BusyWorkers growth, and IdleWorkers drain are visible as time series, not point-in-time snapshots you happened to catch.
  • Per-second collection of request rate, worker utilization, and connection states lets you watch the cascade sequence (workers filling, throughput dropping, backlog growing) as it develops rather than reconstructing it afterward.
  • Correlating Apache worker states with system CPU and memory on the same dashboard makes the “workers waiting, not working” pattern obvious: that combination is the signature of a backend problem, not an Apache problem.
  • 5xx responses broken down by status code let you watch the 504-to-503 progression that marks a slow backend turning into full pool exhaustion.
  • Because Netdata also monitors the backend host and its services, you can put backend latency and resource pressure next to Apache’s worker states in one view, which is the correlation this entire article depends on.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.