Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion

Your access log is filling with 503s and users are reporting the site is down. The status code tells you almost nothing: Apache returns 503 for two root causes that look identical in the access log but need completely different fixes.

The first is frontend worker exhaustion. Every worker slot in the scoreboard is busy, Apache has hit MaxRequestWorkers, and new connections queue in the kernel backlog until they time out. The second is a mod_proxy failure: a backend is down, marked errored, or the per-child proxy connection pool is too small, so Apache refuses to forward requests even though its own workers are mostly idle.

The access log cannot distinguish these. The error log can. Triage order: error log first, then the scoreboard, then balancer state. Fix the pool that is actually exhausted.

What this means

A 503 from Apache means “I cannot service this request right now,” but the reason lives in one of two layers:

1. Frontend layer (MPM workers). All worker slots are occupied. The exact worker-limit code varies by MPM: AH00484 (event), AH00286 (worker), or AH00161 (prefork), followed by server reached MaxRequestWorkers setting; Apache logs the condition once per generation. New connections pile into the listen backlog (default ListenBacklog is 511, capped by net.core.somaxconn) and eventually get refused or time out.

2. Proxy layer (mod_proxy / mod_proxy_balancer). Apache has free workers but cannot get a usable backend connection. Causes: the backend is down and the proxy worker is in error state (default retry=60 means Apache will not retry that backend for 60 seconds after a failure), all balancer members are errored, or the proxy connection pool (max parameter) is exhausted. The error log shows AH00959: ap_proxy_connect_backend disabling worker for (hostname) for 60s and AH01114: HTTP: failed to make connection to backend.

The dangerous misdiagnosis is conflating them: seeing 503s, assuming MaxRequestWorkers is too low, raising it, and making things worse. If the real bottleneck is the proxy pool or a dead backend, more frontend workers just means more workers blocked waiting on the same dead backend, plus more memory consumed.

flowchart TD
  A[503s in access log] --> B{Error log: MPM worker-limit message present?}
  B -- Yes --> C[Frontend worker exhaustion]
  B -- No --> D{Error log: AH00959 / AH01114 / proxy errors?}
  D -- Yes --> E[Backend down or in error state]
  D -- No --> F{Scoreboard idle workers?}
  C --> G[Check scoreboard state mix: W vs R vs K]
  E --> H[Check balancer-manager and retry state]
  F -- "IdleWorkers = 0" --> C
  F -- "IdleWorkers > 0" --> I[Proxy pool too small: tune max on ProxyPass/BalancerMember]

Common causes

Quick checks

All read-only and safe to run during an incident. Paths shown for Debian/Ubuntu; on RHEL use /var/log/httpd/error_log and httpd in place of apache2.

# 1. The single most decisive check: which 503 is this?
grep -E "AH00484|AH00286|AH00161|AH00959|AH01114" /var/log/apache2/error.log | tail -20

# 2. Worker utilization right now
curl -s http://localhost/server-status?auto | grep -E "BusyWorkers|IdleWorkers"

# 3. Scoreboard state mix: where are workers stuck?
curl -s http://localhost/server-status?auto | grep "Scoreboard:" | \
  awk '{print $2}' | fold -w1 | sort | uniq -c | sort -nr

# 4. Is the listen backlog filling? (Recv-Q on LISTEN sockets)
ss -ltn | grep -E ':80\s|:443\s'

# 5. Backend health, bypassing Apache entirely
curl -s -o /dev/null -w "backend: %{http_code} in %{time_total}s\n" \
  --max-time 5 http://backend-host:port/health

# 6. Established connections from Apache to the backend (adjust port)
ss -tn state established dport = :8080 | wc -l

# 7. Balancer member state, if balancer-manager is enabled
curl -s http://localhost/balancer-manager 2>/dev/null | grep -E 'Worker|Status'

# 8. 5xx breakdown from the access log (status is field 9 in combined format)
tail -1000 /var/log/apache2/access.log | \
  awk '$9 ~ /^5/ {c[$9]++} END {for (s in c) print s, c[s]}'

Note on checks 2 and 3: /server-status requires mod_status and should be IP-restricted. ExtendedStatus On has been the default since 2.3.6 when mod_status is loaded, so the scoreboard line is normally present.

How to diagnose it

1. Read the error log first. AH00484, AH00286, or AH00161 with server reached MaxRequestWorkers setting means the frontend pool is the problem. AH00959 ... disabling worker for (hostname:port) for 60s or AH01114: HTTP: failed to make connection to backend means proxy/backend. If you see neither, check whether the 503s come from an ErrorDocument or application handler instead of Apache itself.

2. Snapshot the scoreboard. IdleWorkers = 0 with BusyWorkers at or near configured MaxRequestWorkers confirms frontend saturation. IdleWorkers > 0 while 503s continue means Apache has capacity and the failure is downstream.

3. If frontend exhaustion, look at the state mix. The distribution tells you what is holding workers:

   • Mostly W: workers sending replies or, in proxy mode, blocked waiting for backends. Correlate with backend latency.
   • Mostly R: slow request bodies, slow clients, or Slowloris. Normal traffic rarely exceeds 5% in R.
   • Mostly K on prefork or worker MPM: keepalive connections holding workers hostage; KeepAliveTimeout too long. On event MPM, significant K in the scoreboard is abnormal because keepalive is handled by the listener thread (ConnsAsyncKeepAlive).
   • Many G: graceful restart pile-up; old generations lingering.
   • Many L: log disk full or log pipe stall. Check df -h.

4. If proxy failure, check the backend directly. Curl the backend from the Apache host. If it is down or slow, that is your incident; Apache is a victim, not the cause. With the default retry=60, the proxy worker stays in error state for 60 seconds after a failure, so 503s persist for up to a minute after the backend recovers. Do not restart Apache over this; it fixes itself when the retry window expires.

5. If the backend is healthy and IdleWorkers > 0, check pool sizing. The default max for a proxy worker equals ThreadsPerChild for the active MPM, and is 1 for prefork. Pools are per-child-process: total backend connections = max x number of children. Under moderate concurrency the default exhausts quickly and Apache returns 503 with completely idle frontend workers. This is the most misdiagnosed cause of Apache 503s.

6. Check the listen backlog. Sustained Recv-Q > 0 with high worker utilization means connections are queuing at the kernel level. Brief spikes during bursts are normal; sustained growth is the cliff edge before connection refused.

Metrics and signals to monitor

Fixes

Frontend worker exhaustion

Raise MaxRequestWorkers, but only with the memory math done. MaxRequestWorkers x per-child RSS must stay under roughly 70% of RAM. For prefork with mod_php, measure real per-child RSS first (ps -C apache2 -o rss --no-headers) because 50MB+ per child is common. For threaded MPMs, ServerLimit must be >= MaxRequestWorkers / ThreadsPerChild, and ServerLimit defaults to 16 for worker and event (256 for prefork) if unset, which silently caps how high MaxRequestWorkers can go. Raising ServerLimit requires a full restart, not a graceful reload. MaxRequestWorkers was renamed from MaxClients in 2.3.13/2.4; the old name still works as a deprecated alias.

Fix what is holding workers instead. If the scoreboard is full of W and the backend is slow, raising the limit only buys a bigger queue of blocked workers. Reduce ProxyTimeout temporarily to fail fast, and fix the backend. If it is full of R, tighten mod_reqtimeout: RequestReadTimeout header=20-40,MinRate=500 body=20,MinRate=500. If it is full of K on prefork/worker, lower KeepAliveTimeout or migrate to event MPM. If it is full of G, reduce graceful-restart frequency; GracefulShutdownTimeout applies to graceful-stop, not a graceful reload. Bound request and keepalive time with Timeout, KeepAliveTimeout, and ProxyTimeout, and investigate any stuck request.

Backend down or in error state

Fix the backend, then wait out the retry window. With the default retry=60, Apache will not retry an errored backend for 60 seconds. You can lower retry on the ProxyPass or BalancerMember (for example retry=5) so recovered backends come back into rotation faster, at the cost of probing a flapping backend more often. If all members of a balancer are in error state, forcerecovery=On (the default since Apache 2.4.2) instructs the balancer to recover all workers immediately, without considering each worker’s retry timeout. Apache’s documentation warns that this can deepen an already overloaded backend’s trouble; in that case set it Off.

Stock mod_proxy only learns a backend is dead by failing a real user request. Without active health checks, the first requests after a backend dies always eat the failure. Apache 2.4.21 and later include mod_proxy_hcheck; if it is not already enabled, verify with apachectl -M | grep proxy_hcheck before configuring active checks.

Proxy pool too small

Size max to your real concurrency. Set max on ProxyPass or BalancerMember to roughly 2x the expected concurrent proxied requests per child at peak. Pools are per-child: on prefork, N children x max connections hit the backend, which can overwhelm it; on worker/event, fewer children means max must be larger to reach the same total. Estimate with pool_utilization = request_rate x backend_avg_response_time / pool_size and keep utilization well under 1.

Enable backend keepalive. keepalive=On on ProxyPass reuses backend connections instead of opening a new TCP connection per request, reducing latency and connection churn on both sides.

Prevention

• Derive MaxRequestWorkers from memory, never from a guess: available_RAM x 0.7 / measured_per_child_RSS. Re-measure after module or application changes.
• Set MaxConnectionsPerChild to a finite value (5000-10000) so leaky modules cannot grow children without bound. The default of 0 is wrong for mod_php and mod_perl deployments.
• Size proxy pools explicitly. Never run production reverse proxies on the default max. Document the per-child math next to the directive.
• Alert on MPM worker-limit messages (AH00484/AH00286/AH00161) and proxy error patterns, not just on 5xx rate. The error log is the only place the two 503 causes are distinguishable automatically.
• Sample the scoreboard continuously. Point-in-time snapshots during an incident are too late; the state distribution trend tells you whether workers are filling with W, R, or K before the cliff.
• Keep logs on their own filesystem so a log explosion cannot starve the OS or trigger the log-stall variant of worker exhaustion.
• Test health checks on the critical path. A health check that only fetches a static file will pass while every proxied request 503s.

How Netdata helps

• Scoreboard state distribution over time. Netdata collects the Apache scoreboard continuously, so you can see the W/R/K mix building toward exhaustion minutes before IdleWorkers hits zero, instead of discovering it from 503s.
• BusyWorkers and IdleWorkers trends. Worker utilization graphed against configured MaxRequestWorkers makes the cliff edge visible and gives you the runway estimate for capacity planning.
• 5xx correlation with request rate. A paradoxical drop in completed requests alongside rising 503s and full workers is the signature of the slow-backend cascade; seeing all three on one dashboard shortens the “is it Apache or the backend” question to seconds.
• Backend latency alongside worker states. Correlating direct backend response time with W-state accumulation confirms or rules out the proxy cause without log spelunking during an incident.
• Error log pattern alerting. Alerting on MPM worker-limit and proxy error messages as distinct conditions routes the page to the right fix: frontend capacity versus backend health.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• How Apache HTTPD actually works in production: a mental model for operators
• Apache AH00484: server reached MaxRequestWorkers setting - worker pool exhausted
• Apache HTTPD monitoring checklist: the signals every production web server needs
• Apache HTTPD monitoring maturity model: from survival to expert