You are seeing intermittent 503s on proxied endpoints under what looks like moderate load. The error log shows AH01136: Reverse proxy worker busy. BusyWorkers is nowhere near MaxRequestWorkers, CPU and memory are fine, and requests keep failing. Raising MaxRequestWorkers changed nothing.

That is the signature of proxy connection pool exhaustion. The bottleneck is not Apache’s worker pool. It is the smaller, less visible pool of backend connections that mod_proxy maintains per child process, and its default size is too small for production load.

What this means

When Apache reverse-proxies a request, it needs a connection to the backend. mod_proxy maintains a pool of reusable backend connections per backend worker, and that pool is per child process. It is not shared or coordinated across children.

The pool size is controlled by the max= parameter on ProxyPass (or BalancerMember when using mod_proxy_balancer). The default is the number of threads per process in the active MPM:

  • prefork: ThreadsPerChild is effectively 1, so the default pool size is 1 connection per child. Worse, with prefork MPM connections to backends are not pooled across requests at all, and the pooling parameters (acquire, ttl, min, smax, hmax) have no effect. If you are proxying on prefork, that alone is a reason to move to event or worker MPM.
  • worker/event: default max = ThreadsPerChild (commonly 25 or 64 depending on your config and distribution).

When every connection in a child’s pool is in use and another proxied request arrives, that request cannot get a backend connection. Apache logs AH01136: Reverse proxy worker busy and the client gets a 503. There is no queuing by default (acquire defaults to zero): pool full means immediate failure. Total backend connection capacity is max multiplied by the number of running children, so the ceiling moves as Apache spawns and kills children.

The causes below assume a threaded MPM (worker or event), where pool exhaustion is a tuning problem rather than an architectural one.

Common causes

CauseWhat it looks likeFirst thing to check
max= never set, default too small503s start at a consistent, low concurrency level; AH01136 bursts at peakgrep ProxyPass your config for an explicit max=
Slow backend holding connectionsPool fills because connections are checked out longer, not because traffic grewTime the backend directly, bypassing Apache
No connection reuse to backendHigh backend connection churn, TIME_WAIT buildup, latency on every proxied requestCheck for keepalive= on ProxyPass; check if you use RewriteRule [P]
smax churnConnections repeatedly created and destroyed, elevated backend connection rateReview smax= and ttl= settings
Worker sharing silently overriding settingsYour tuned max= on a second ProxyPass is ignoredError log warnings about worker reuse; overlapping ProxyPass URL prefixes

Two gotchas to know before tuning. First, if two ProxyPass directives have overlapping URL prefixes, the first worker created is reused and the second worker’s parameters are silently ignored (a warning is logged). Second, if you proxy via RewriteRule ... [P] instead of ProxyPass, you get the default reverse proxy worker, which does not use HTTP keep-alive or connection reuse at all: a fresh TCP connection to the backend per request.

Quick checks

These are all read-only.

# 1. Confirm the error and its frequency
grep -c "AH01136" /var/log/httpd/error_log        # RHEL/CentOS
grep -c "AH01136" /var/log/apache2/error.log      # Debian/Ubuntu
grep "AH01136" /var/log/httpd/error_log | tail -20

# 2. Check which MPM you are running (this changes everything)
httpd -V 2>/dev/null | grep MPM || apache2ctl -V | grep MPM

# 3. Find your ProxyPass / BalancerMember config and look for max=
grep -rn "ProxyPass\|BalancerMember" /etc/httpd/ /etc/apache2/ 2>/dev/null | grep -v "^#"

# 4. Count current established connections to the backend (adjust port)
ss -tn state established dport = :8080 | wc -l

# 5. Check worker utilization at the time of failure
curl -s http://localhost/server-status?auto | grep -E "BusyWorkers|IdleWorkers"

# 6. Look at the scoreboard state distribution (many W states = waiting on backend)
curl -s http://localhost/server-status?auto | grep "Scoreboard:" | \
  awk '{print $2}' | fold -w1 | sort | uniq -c | sort -nr

# 7. Time the backend directly, bypassing Apache
curl -s -o /dev/null -w "TTFB: %{time_starttransfer}s Total: %{time_total}s\n" \
  http://backend-host:8080/health

# 8. If using mod_proxy_balancer, check member status
curl -s http://localhost/balancer-manager 2>/dev/null | grep -E "Worker|Status"

The key diagnostic contrast: if AH01136 is firing while check 5 shows plenty of IdleWorkers, you have isolated the bottleneck to the proxy pool, not the Apache worker pool. See Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion for the full triage between these two 503 causes.

How to diagnose it

  1. Confirm the error is AH01136 and not AH00484. AH00484: server reached MaxRequestWorkers setting is frontend worker exhaustion. AH01136 is backend pool exhaustion. Both produce 503s and demand different fixes. Grep the error log for both.

  2. Identify the MPM. If the answer is prefork, pool parameters mostly do not apply. Your real fix is migrating to event (or worker) MPM, which usually means moving PHP from mod_php to PHP-FPM if that is what pinned you to prefork.

  3. Find the effective pool size. Read the ProxyPass/BalancerMember lines. If there is no max=, the default is ThreadsPerChild. Check for overlapping ProxyPass prefixes that silently discard your settings, and for RewriteRule [P] usage that bypasses pooling entirely.

  4. Estimate the concurrency you actually need. Concurrent proxied requests per child at peak is roughly peak proxied request rate per child multiplied by average backend response time in seconds (Little’s law). If your backend answers in 200 ms and each child serves 30 proxied req/s at peak, you need about 6 concurrent connections plus headroom. If backend p95 is 2 seconds, that number is 60 plus headroom.

  5. Determine whether the pool filled from demand or from backend slowness. Compare the timeline of AH01136 bursts against backend response time (check 7) and the scoreboard W state count (check 6). If backend latency rose before the 503s started, the pool size may be fine and the backend is the real problem. A slow backend holds pool connections longer, so any pool eventually fills; see the slow backend cascade pattern in How Apache HTTPD actually works in production.

  6. Check backend-side connection counts. From the backend, count established connections from the Apache hosts. If the total is far below max x children while AH01136 fires, connections may be dying (firewall idle timeouts, backend-side keepalive limits) rather than saturating.

flowchart TD
  A[503s on proxied paths] --> B{Error log shows?}
  B -->|AH00484| C[MaxRequestWorkers exhausted - frontend pool]
  B -->|AH01136| D[Proxy pool exhausted - backend connections]
  D --> E{Backend latency normal?}
  E -->|Yes| F[Pool too small - raise max=, enable keepalive=]
  E -->|No| G[Slow backend holding connections - fix backend or fail fast]
  D --> H{MPM is prefork?}
  H -->|Yes| I[No pooling on prefork - migrate to event/worker MPM]

Metrics and signals to monitor

SignalWhy it mattersWarning sign
AH01136 count in error logDefinitive proof of pool exhaustionAny occurrence under production load
Established connections to backend portActual pool utilization vs max x childrenSustained approach to the computed ceiling
Backend response time (direct probe)Slow backends fill pools faster than traffic doesp95 > 2x baseline
Scoreboard W state shareWorkers waiting on backends show as WW > 50% of workers sustained
502/503/504 rate on proxied pathsDistinguishes pool exhaustion (503) from backend death (502) and timeout (504)Any sustained proxy error rate
BusyWorkers / MaxRequestWorkersConfirms the frontend pool is NOT the bottleneckHigh alongside AH01136 means you have both problems

Fixes

Set max= explicitly on ProxyPass or BalancerMember

Size the pool at roughly 2x the expected concurrent proxied requests per child at peak, using the estimate from step 4 above. Example:

ProxyPass /api/ http://backend:8080/api/ max=40 ttl=120

Tradeoffs: total backend connections = max x number of children, and child count fluctuates with load. Your backend (and anything between, like a database connection pool or a firewall session table) must tolerate the worst case: max x maximum children. Verify the backend’s own connection limits before raising max aggressively. Changes to ProxyPass parameters require a restart to take full effect reliably; a graceful restart spawns new children with the new pool but old children drain with the old one.

Enable connection reuse with keepalive=

Without reuse, Apache churns TCP connections to the backend, adding latency and TIME_WAIT pressure:

ProxyPass /api/ http://backend:8080/api/ max=40 keepalive=On

Make sure the backend’s own keepalive timeout is longer than the idle gaps Apache creates, or Apache will hand out connections the backend has already closed, producing intermittent 502s.

Mind smax and ttl

smax defaults to max and sets the soft cap on idle pooled connections; connections above smax that sit unused longer than ttl are freed. If you see constant connect/disconnect churn against the backend during traffic valleys, your idle retention is too aggressive. If you see stale connection failures after idle periods (a stateful firewall or NAT between Apache and the backend dropping idle flows), lower ttl so Apache discards connections before the middlebox does.

Fix worker sharing and rewrite-based proxying

Order ProxyPass directives from most specific to least specific, and check the error log at startup for warnings about workers being reused. Replace RewriteRule ... [P] proxying with explicit ProxyPass directives so the traffic gets a configurable pooled worker instead of the unpooled default worker.

If the backend is the problem

Raising max= when the backend is slow only lets you hold more connections to a slow backend. The pool fills again at the new ceiling. Fix backend latency first, or temporarily reduce ProxyTimeout so requests fail fast instead of holding pool connections and Apache workers. Set acquire (milliseconds to wait for a free pool connection) only if you prefer short queuing over instant 503s; remember it has no effect on prefork.

If you are on prefork MPM

There is no pool-size fix. Connections are not pooled, the pooling parameters are ignored, and the default worker behavior dominates. Plan the migration to event MPM; that is the durable fix.

Prevention

  • Set max= explicitly on every ProxyPass and BalancerMember. Never accept the ThreadsPerChild default in production. Document the sizing math next to the directive.
  • Monitor the proxy pool as a first-class resource. Track established backend connections against the computed ceiling (max x children) and alert at 80% sustained. Pool exhaustion is cliff-edge: full pool, immediate 503, no queue.
  • Alert on AH01136 as its own pattern, separate from AH00484. They have different owners and different fixes.
  • Baseline backend response time per backend. A rising backend p95 is the leading indicator that your pool will fill at current traffic levels.
  • Audit after every config change for overlapping ProxyPass prefixes and new RewriteRule [P] usage.
  • Load test through Apache, not just the backend. Backend-only load tests never exercise the proxy pool, which is why this limit survives until production traffic finds it.

How Netdata helps

  • Netdata’s Apache collector scrapes server-status?auto per second, so BusyWorkers, IdleWorkers, and the scoreboard state breakdown are captured at the granularity where AH01136 bursts actually happen, not smoothed into 1-minute averages.
  • The scoreboard state distribution over time shows W states climbing before the 503s start, which distinguishes “pool filled because backend slowed” from “pool filled because traffic grew”.
  • Correlating Apache worker utilization against 5xx rates on the same dashboard makes the AH01136-versus-AH00484 distinction immediate: 503s with idle workers means proxy pool, 503s with zero idle workers means MaxRequestWorkers.
  • Netdata’s TCP connection and netstat charts on the Apache host surface established backend connections and TIME_WAIT churn, so you can watch pool utilization and reuse effectiveness without hand-rolling instrumentation.
  • Error log monitoring catches AH01136 events as they occur, and anomaly detection flags deviation from the normal error baseline rather than relying on a static threshold you have to guess.

See Apache HTTP Server monitoring with Netdata for how these signals come together.