Apache reverse proxy worker busy: proxy connection pool exhaustion (AH00941)

You are seeing intermittent 503s on proxied endpoints under what looks like moderate load. The error log shows AH00941: failed to acquire connection for (backend-host:port). BusyWorkers is nowhere near MaxRequestWorkers, CPU and memory are fine, and requests keep failing. Raising MaxRequestWorkers changed nothing.

That is the signature of proxy connection pool exhaustion. The bottleneck is not Apache’s worker pool. It is the smaller, less visible pool of backend connections that mod_proxy maintains per child process, and its default size is too small for production load.

What this means

When Apache reverse-proxies a request, it needs a connection to the backend. mod_proxy maintains a pool of reusable backend connections per backend worker, and that pool is per child process. It is not shared or coordinated across children.

The pool size is controlled by the max= parameter on ProxyPass (or BalancerMember when using mod_proxy_balancer). The default is the number of threads per process in the active MPM:

• prefork: ThreadsPerChild is effectively 1, so the default pool size is 1 connection per child. Worse, with prefork MPM connections to backends are not pooled across requests at all, and the pooling parameters (acquire, ttl, min, smax, hmax) have no effect. If you are proxying on prefork, that alone is a reason to move to event or worker MPM.
• worker/event: default max = ThreadsPerChild (commonly 25 or 64 depending on your config and distribution).

When every connection in a child’s pool is in use and another proxied request arrives, that request cannot get a backend connection. Apache logs AH00941: failed to acquire connection for (backend-host:port) and the client gets a 503; when every worker of a balancer is busy at once, it logs AH00934: all workers are busy. Unable to serve instead. There is no queuing by default (acquire defaults to zero): pool full means immediate failure. Total backend connection capacity is max multiplied by the number of running children, so the ceiling moves as Apache spawns and kills children.

The causes below assume a threaded MPM (worker or event), where pool exhaustion is a tuning problem rather than an architectural one.

Common causes

Two gotchas to know before tuning. First, if two ProxyPass directives have overlapping URL prefixes, the first worker created is reused and the second worker’s parameters are silently ignored (a warning is logged). Second, if you proxy via RewriteRule ... [P] instead of ProxyPass, you get the default reverse proxy worker, which does not use HTTP keep-alive or connection reuse at all: a fresh TCP connection to the backend per request.

Quick checks

These are all read-only.

# 1. Confirm the error and its frequency
grep -c "AH00941" /var/log/httpd/error_log        # RHEL/CentOS
grep -c "AH00941" /var/log/apache2/error.log      # Debian/Ubuntu
grep "AH00941" /var/log/httpd/error_log | tail -20

# 2. Check which MPM you are running (this changes everything)
httpd -V 2>/dev/null | grep MPM || apache2ctl -V | grep MPM

# 3. Find your ProxyPass / BalancerMember config and look for max=
grep -rn "ProxyPass\|BalancerMember" /etc/httpd/ /etc/apache2/ 2>/dev/null | grep -v "^#"

# 4. Count current established connections to the backend (adjust port)
ss -tn state established dport = :8080 | wc -l

# 5. Check worker utilization at the time of failure
curl -s http://localhost/server-status?auto | grep -E "BusyWorkers|IdleWorkers"

# 6. Look at the scoreboard state distribution (many W states = waiting on backend)
curl -s http://localhost/server-status?auto | grep "Scoreboard:" | \
  awk '{print $2}' | fold -w1 | sort | uniq -c | sort -nr

# 7. Time the backend directly, bypassing Apache
curl -s -o /dev/null -w "TTFB: %{time_starttransfer}s Total: %{time_total}s\n" \
  http://backend-host:8080/health

# 8. If using mod_proxy_balancer, check member status
curl -s http://localhost/balancer-manager 2>/dev/null | grep -E "Worker|Status"

The key diagnostic contrast: if AH00941 is firing while check 5 shows plenty of IdleWorkers, you have isolated the bottleneck to the proxy pool, not the Apache worker pool. See Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion for the full triage between these two 503 causes.

How to diagnose it

1. Confirm the error is AH00941 and not AH00484. AH00484: server reached MaxRequestWorkers setting is frontend worker exhaustion. AH00941 is backend pool exhaustion. Both produce 503s and demand different fixes. Grep the error log for both.

2. Identify the MPM. If the answer is prefork, pool parameters mostly do not apply. Your real fix is migrating to event (or worker) MPM, which usually means moving PHP from mod_php to PHP-FPM if that is what pinned you to prefork.

3. Find the effective pool size. Read the ProxyPass/BalancerMember lines. If there is no max=, the default is ThreadsPerChild. Check for overlapping ProxyPass prefixes that silently discard your settings, and for RewriteRule [P] usage that bypasses pooling entirely.

4. Estimate the concurrency you actually need. Concurrent proxied requests per child at peak is roughly peak proxied request rate per child multiplied by average backend response time in seconds (Little’s law). If your backend answers in 200 ms and each child serves 30 proxied req/s at peak, you need about 6 concurrent connections plus headroom. If backend p95 is 2 seconds, that number is 60 plus headroom.

5. Determine whether the pool filled from demand or from backend slowness. Compare the timeline of AH00941 bursts against backend response time (check 7) and the scoreboard W state count (check 6). If backend latency rose before the 503s started, the pool size may be fine and the backend is the real problem. A slow backend holds pool connections longer, so any pool eventually fills; see the slow backend cascade pattern in How Apache HTTPD actually works in production.

6. Check backend-side connection counts. From the backend, count established connections from the Apache hosts. If the total is far below max x children while AH00941 fires, connections may be dying (firewall idle timeouts, backend-side keepalive limits) rather than saturating.

flowchart TD
  A[503s on proxied paths] --> B{Error log shows?}
  B -->|AH00484| C[MaxRequestWorkers exhausted - frontend pool]
  B -->|AH00941| D[Proxy pool exhausted - backend connections]
  D --> E{Backend latency normal?}
  E -->|Yes| F[Pool too small - raise max=, enable keepalive=]
  E -->|No| G[Slow backend holding connections - fix backend or fail fast]
  D --> H{MPM is prefork?}
  H -->|Yes| I[No pooling on prefork - migrate to event/worker MPM]

Metrics and signals to monitor

Fixes

Set max= explicitly on ProxyPass or BalancerMember

Size the pool at roughly 2x the expected concurrent proxied requests per child at peak, using the estimate from step 4 above. Example:

ProxyPass /api/ http://backend:8080/api/ max=40 ttl=120

Tradeoffs: total backend connections = max x number of children, and child count fluctuates with load. Your backend (and anything between, like a database connection pool or a firewall session table) must tolerate the worst case: max x maximum children. Verify the backend’s own connection limits before raising max aggressively. Changes to ProxyPass parameters require a restart to take full effect reliably; a graceful restart spawns new children with the new pool but old children drain with the old one.

Enable connection reuse with keepalive=

Without reuse, Apache churns TCP connections to the backend, adding latency and TIME_WAIT pressure:

ProxyPass /api/ http://backend:8080/api/ max=40 keepalive=On

Make sure the backend’s own keepalive timeout is longer than the idle gaps Apache creates, or Apache will hand out connections the backend has already closed, producing intermittent 502s.

Mind smax and ttl

smax defaults to max and sets the soft cap on idle pooled connections; connections above smax that sit unused longer than ttl are freed. If you see constant connect/disconnect churn against the backend during traffic valleys, your idle retention is too aggressive. If you see stale connection failures after idle periods (a stateful firewall or NAT between Apache and the backend dropping idle flows), lower ttl so Apache discards connections before the middlebox does.

Fix worker sharing and rewrite-based proxying

Order ProxyPass directives from most specific to least specific, and check the error log at startup for warnings about workers being reused. Replace RewriteRule ... [P] proxying with explicit ProxyPass directives so the traffic gets a configurable pooled worker instead of the unpooled default worker.

If the backend is the problem

Raising max= when the backend is slow only lets you hold more connections to a slow backend. The pool fills again at the new ceiling. Fix backend latency first, or temporarily reduce ProxyTimeout so requests fail fast instead of holding pool connections and Apache workers. Set acquire (milliseconds to wait for a free pool connection) only if you prefer short queuing over instant 503s; remember it has no effect on prefork.

If you are on prefork MPM

There is no pool-size fix. Connections are not pooled, the pooling parameters are ignored, and the default worker behavior dominates. Plan the migration to event MPM; that is the durable fix.

Prevention

• Set max= explicitly on every ProxyPass and BalancerMember. Never accept the ThreadsPerChild default in production. Document the sizing math next to the directive.
• Monitor the proxy pool as a first-class resource. Track established backend connections against the computed ceiling (max x children) and alert at 80% sustained. Pool exhaustion is cliff-edge: full pool, immediate 503, no queue.
• Alert on AH00941 as its own pattern, separate from AH00484. They have different owners and different fixes.
• Baseline backend response time per backend. A rising backend p95 is the leading indicator that your pool will fill at current traffic levels.
• Audit after every config change for overlapping ProxyPass prefixes and new RewriteRule [P] usage.
• Load test through Apache, not just the backend. Backend-only load tests never exercise the proxy pool, which is why this limit survives until production traffic finds it.

How Netdata helps

• Netdata’s Apache collector scrapes server-status?auto per second, so BusyWorkers, IdleWorkers, and the scoreboard state breakdown are captured at the granularity where AH00941 bursts actually happen, not smoothed into 1-minute averages.
• The scoreboard state distribution over time shows W states climbing before the 503s start, which distinguishes “pool filled because backend slowed” from “pool filled because traffic grew”.
• Correlating Apache worker utilization against 5xx rates on the same dashboard makes the AH00941-versus-AH00484 distinction immediate: 503s with idle workers means proxy pool, 503s with zero idle workers means MaxRequestWorkers.
• Netdata’s TCP connection and netstat charts on the Apache host surface established backend connections and TIME_WAIT churn, so you can watch pool utilization and reuse effectiveness without hand-rolling instrumentation.
• Error log monitoring catches AH00941 events as they occur, and anomaly detection flags deviation from the normal error baseline rather than relying on a static threshold you have to guess.

See Apache HTTP Server monitoring with Netdata for how these signals come together.

Related guides

• Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion
• Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status
• Apache CLOSE_WAIT and TIME_WAIT: connection leaks versus normal churn
• How Apache HTTPD actually works in production: a mental model for operators
• Apache keepalive consuming workers: KeepAliveTimeout, the K state, and MPM choice
• Apache listen queue overflow: Recv-Q growth, ListenBacklog, and refused connections
• Apache AH00484: server reached MaxRequestWorkers setting - worker pool exhausted
• Apache MaxRequestWorkers tuning: sizing the worker pool against memory
• Apache HTTPD monitoring checklist: the signals every production web server needs
• Apache HTTPD monitoring maturity model: from survival to expert
• Apache scoreboard states explained: what _ S R W K D C L G tell you
• Apache ListenBacklog vs net.core.somaxconn: the silently truncated accept queue