$ guides / apache-httpd / apache-httpd-slow-backend-cascade ▌

Operations Guides

Apache slow backend cascade: how one slow upstream starves the whole worker pool

Users report the site down. The load balancer has pulled the node from rotation. You SSH in expecting to find Apache melting, and instead find a perfectly calm process: normal CPU, normal memory, no crash dumps in the error log. The port is listening, but new connections hang or get refused.

This is the slow backend cascade: the classic “Apache outage” in reverse-proxy deployments. Apache is not broken. Every worker is blocked waiting on a backend that has gone slow, and the frontend has run out of execution slots as a consequence.

The failure is deceptive because every conventional host metric looks healthy. Workers are waiting, not working. CPU is idle. Memory is flat. The incident is only visible in the scoreboard, the backend latency, and eventually the listen backlog.

What this means

When Apache proxies a request, the worker that accepted it stays occupied for the entire round trip: accept the client connection, forward the request, wait for the backend response, relay it back. Worker occupancy time per proxied request is therefore roughly the backend response time plus transfer time.

Now apply Little’s law. If you serve 100 proxied requests per second and the backend answers in 200 ms, you need about 20 workers. If the backend degrades to 5 seconds per response, you need 500 workers for the same traffic. If MaxRequestWorkers is below that, the excess does not slow down gracefully. It queues in the kernel listen backlog, and once the backlog fills, new connections are refused. The degradation curve is cliff-edge: normal service right up to saturation, then instant denial.

The cascade:

flowchart TD
  A[Backend goes slow: DB locks, GC pause, dependency down] --> B[Each proxied request holds a worker longer]
  B --> C[Workers pile into W state, IdleWorkers falls to zero]
  C --> D[New connections queue in listen backlog]
  D --> E[Backlog fills, connections refused]
  E --> F[LB health checks time out, node pulled from rotation]
  C --> G[Apache CPU and memory stay normal: workers waiting, not working]

Two properties make this pattern easy to misread:

Direct, non-proxied requests may still work. A static file or a locally served health page can return instantly if a worker is free, while every proxied path hangs. “Apache responds to curl” does not rule this out.
The request rate paradoxically drops. Total Accesses counts completed requests. When workers are stuck, completions fall even though client demand is unchanged or rising.

Common causes

The root cause is always on the backend side. Apache is the victim, not the perpetrator.

Cause	What it looks like	First thing to check
Backend database lock contention or slow queries	Backend response time 5x or more above baseline, gradual onset	Backend health endpoint directly, bypassing Apache
Backend memory exhaustion or GC pause	Latency spikes in bursts, backend recovers then degrades again	Backend’s own memory and GC metrics
Network partition or packet loss between Apache and backend	Connection timeouts rather than slow responses, 502s alongside 504s	Direct curl to backend from the Apache host
Backend’s own external dependency failure	Backend up and accepting connections but slow on specific paths	Which proxied URL patterns are slow versus fast
Proxy connection pool exhaustion on top of slow backend	503s appearing before `MaxRequestWorkers` is reached	Error log for proxy errors, balancer-manager status

Quick checks

Run these from the Apache host. All are read-only.

# 1. Scoreboard state distribution: the signature of this incident
curl -s http://localhost/server-status?auto | grep "Scoreboard:" | \
  awk '{print $2}' | fold -w1 | sort | uniq -c | sort -nr

# 2. Worker utilization
curl -s http://localhost/server-status?auto | grep -E "BusyWorkers|IdleWorkers"

# 3. Listen backlog depth: Recv-Q is current queue, Send-Q is the max
ss -ltn | grep -E ':80\s|:443\s'

# 4. Listen overflow counter: connections already dropped
nstat -a | grep ListenOverflows 2>/dev/null || netstat -s | grep -i "listen\|overflow"

# 5. Proxy and worker-exhaustion errors
grep -E "AH01114|AH00484" /var/log/apache2/error.log | tail -20
# RHEL path: /var/log/httpd/error_log

# 6. 5xx breakdown from recent traffic
tail -1000 /var/log/apache2/access.log | \
  awk '$9 ~ /^5/ {c[$9]++} END {for (k in c) print k, c[k]}'

# 7. Backend health directly, bypassing Apache entirely
curl -s -o /dev/null -w "TTFB: %{time_starttransfer}s Total: %{time_total}s HTTP: %{http_code}\n" \
  --max-time 10 http://backend-host:backend-port/health

# 8. Does Apache itself still work? Request something local and non-proxied
curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" --max-time 5 http://localhost/server-status?auto

What you are looking for: a scoreboard dominated by W with IdleWorkers at or near zero, a growing Recv-Q, 504s (and then 503s) in the access log, and a backend that is slow or unreachable when queried directly.

How to diagnose it

Read the scoreboard first. Count the state distribution. The cascade signature is W states climbing toward MaxRequestWorkers. Note that W is ambiguous by design: a worker in W could be writing to the client, waiting on the backend, or doing internal processing. Disambiguate with the steps below.
Confirm Apache is not resource-bound. Check CPU and memory. In this pattern both are normal because workers are blocked on I/O, not computing. High CPU or climbing RSS points to a different failure pattern (see the related guides).
Check the listen backlog. Non-zero and growing Recv-Q on the listening socket confirms new connections are arriving faster than workers free up. A rising ListenOverflows counter means connections are already being dropped.
Separate 504 from 503. 504 means a backend connected but did not respond within ProxyTimeout: the backend is slow or hung. 503 means the proxy path had nothing usable: proxy pool exhausted, a balancer member in error state, or all balancer workers busy. Worker saturation itself queues excess connections in the listen backlog rather than answering 503. 504s appearing first, then 503s as the pool drains, is the textbook progression.
Test the backend directly from the Apache host. This splits the problem in half. If the backend is slow when hit directly, Apache is exonerated and the incident moves to the backend. If the backend is fast directly but slow through Apache, suspect the network path, the proxy connection pool, or DNS resolution.
Test a non-proxied path on Apache. If /server-status or a static file answers quickly while proxied paths hang, the diagnosis is confirmed: the frontend is healthy and starved by the upstream.
Check the error log for corroboration. Proxy connection failures (AH01114) and AH00484: server reached MaxRequestWorkers setting tell you how far the cascade has progressed. AH00484 is the definitive confirmation that the worker pool is fully saturated.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Scoreboard `W` state count	Direct view of workers blocked mid-request	`W` above 50% of workers sustained
`IdleWorkers`	The last buffer before queuing starts	Falling toward zero at stable traffic
Backend response time	The root cause, and the earliest signal	P95 above 2x baseline; 5x is active cascade
504 rate	Backend responses exceeding `ProxyTimeout`	Any sustained occurrence
503 rate	Proxy pool exhaustion or balancer member error	Any occurrence in production
Listen backlog `Recv-Q`	Connections queuing before reaching a worker	Sustained non-zero; approaching `ListenBacklog` (default 511) is critical
`ListenOverflows` counter	Connections already refused	Any increase
Request completion rate (`Total Accesses` delta)	Falls paradoxically during the cascade	Drop while client demand is unchanged
`AH00484` in error log	Apache explicitly reporting pool saturation	Any occurrence
Apache CPU and RSS	The negative signal that distinguishes this pattern	Normal resources plus all of the above

The correlation that closes the diagnosis in one glance: high W count, elevated backend response time, zero idle workers, and normal Apache CPU and memory. No other failure pattern produces that combination.

Fixes

Immediate mitigation

Check the backend and act on it, not on Apache. Take the node out of LB rotation if health checks are flapping, then work the backend problem. Restarting Apache buys minutes at best: the new workers immediately re-block on the same slow backend. Do not make Apache restarts your first move.

Fail fast with a lower ProxyTimeout. If the backend is slow but not dead, temporarily reducing ProxyTimeout makes workers give up sooner and return an error instead of holding the slot for the full timeout. This converts “hang until the pool drains” into “fast 504s that clients can retry.” It is a pressure valve, not a cure: some legitimate slow requests will now fail, and the change requires a graceful reload to take effect.

If the backend is dead, say so quickly. A fast 503 is better for clients behind a retrying load balancer than a 60-second hang.

Backend-side resolution

The actual fix lives wherever the backend is: kill the locking query, resolve the GC pressure, restore the failed dependency, fix the network path. Until backend response time returns to baseline, every Apache-side change is symptom management.

Proxy pool sizing

The proxy connection pool is per child process, and its default max equals ThreadsPerChild (1 for prefork). That default is too small for most production workloads, and it produces 503s under moderate load that look like worker exhaustion but are not. Size the pool for expected concurrency, and remember the headroom rule from the playbook: pool_utilization = request_rate x backend_avg_response_time / pool_size. When backend latency doubles, pool utilization doubles. Aim for 2x expected concurrent proxied requests per child at peak.

Prevention

Monitor the backend as a first-class signal. Backend response time is the leading indicator for this entire failure class. Alert on backend P95 exceeding 2x baseline before workers start piling up. The playbook’s capacity model applies directly: if peak BusyWorkers trends track backend latency, fixing the backend buys more headroom than raising MaxRequestWorkers.
Set a deliberate ProxyTimeout. The default inherits from Timeout. Pick a value tied to your backend’s actual SLA so a hung upstream fails in seconds, not a minute.
Health-check the critical path, not a static file. A health check that only exercises local content will keep reporting green while every proxied path is down. Probe through the proxy to the backend.
Watch the scoreboard state distribution continuously. A rising W fraction at flat request rate is the earliest frontend-side warning, minutes before the backlog fills.
Know your MPM. This cascade hits all MPMs, but interpretation differs: on event MPM, keepalive connections are handled by the listener thread and tracked via ConnsAsyncKeepAlive, so proxied-request blocking shows up cleanly as W states rather than being mixed with K noise.
Keep idle headroom. At least 25% of MaxRequestWorkers idle at peak. The degradation curve is cliff-edge; there is no graceful degradation zone to catch you.

How Netdata helps

Netdata’s Apache collector polls server-status and charts BusyWorkers, IdleWorkers, and the full scoreboard state breakdown per second, so the W-state pile-up is visible as it builds rather than after the backlog overflows.
Correlating worker utilization against request completion rate exposes the signature paradox of this incident: workers maxed while completed requests fall.
Backend response time and 502/503/504 rates from log or endpoint monitoring sit on the same dashboard as Apache’s own CPU and memory, making the “healthy Apache, dying upstream” contrast obvious in one view.
Listen backlog depth and TCP listen overflow counters from the host are collected alongside Apache metrics, closing the loop from backend latency to worker saturation to refused connections.
Anomaly detection on backend latency and scoreboard states catches the slow drift phase (backend degrading over minutes) before it becomes the cliff-edge phase.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

The Netdata solution

Apache HTTP Server monitoring with Netdata

Netdata monitors Apache HTTP Server with per-second metrics from mod_status, pre-built dashboards, and ML-powered anomaly detection. Watch busy versus idle workers and the scoreboard state mix, requests per second, bytes served per second, and request processing duration alongside the rest of your stack, so you catch the worker-exhaustion, slow-backend, and memory incidents in these runbooks before they page anyone.

See web server monitoring → Start monitoring free