Apache balancer member in error state: reading balancer-manager and failover

You opened balancer-manager, or someone pasted a screenshot into the incident channel, and one of your BalancerMembers shows Err in the status column. Traffic is still flowing, but the pool is quietly running on fewer backends than you think. If enough members flip to error state, Apache stops proxying entirely and returns 503s even though httpd itself is healthy.

This state is mod_proxy_balancer doing its job: it detected failures against a backend and pulled that member out of rotation so requests stop dying on it. The problem is that the balancer tells you almost nothing about why, and it keeps the member sidelined on its own retry schedule regardless of whether the backend has recovered.

This article covers how to read the error state in balancer-manager, the directives that govern when a member is marked bad and when it comes back, how failover to spares and standbys behaves, and how to find the backend-side root cause.

What this means

When mod_proxy cannot establish a connection to a backend, or when a configured failure condition is met, the balancer marks that worker as being in error state and stops routing requests to it. In the error log you will see a line like:

AH00959: ap_proxy_connect_backend disabling worker for (backend-host) for 60s

That 60s is the member’s retry interval. The default retry is 60 seconds: after a worker enters error state, the balancer will not attempt to use it again until the interval elapses, at which point it tries the worker on a new request. If the backend is still broken, the worker goes straight back to error state for another interval.

In balancer-manager, worker status is shown as a set of single-letter flags. The ones that matter for this incident:

The state machine for a normal member:

stateDiagram-v2
  Ok --> Error: connection failure or failonstatus
  Error --> Ok: retry interval elapsed, backend accepts request
  Error --> Error: retry attempt fails, interval restarts
  Ok --> Drain: set to +N via balancer-manager
  Error --> HotSpareTakesOver: spare (R) in same lbset

Two properties of this design matter during an incident. First, a member in error state with active traffic is a silent capacity reduction: nothing pages, but your pool just lost a fraction of its throughput. Second, if every member of a balancer ends up in error state, the balancer has nowhere to send requests and Apache returns 503 for every proxied request. In proxy troubleshooting, 503 from an otherwise healthy Apache almost always means the pool is exhausted or all members are errored.

Common causes

Quick checks

All read-only. Run them from the Apache host.

# 1. See current member status in balancer-manager (if enabled and reachable)
curl -s http://localhost/balancer-manager | grep -E 'Worker|Status|Err'

# 2. Find the moments members were disabled, and the retry interval in effect
grep "AH00959" /var/log/apache2/error.log | tail -20
# RHEL path: /var/log/httpd/error_log

# 3. Test each backend directly, bypassing Apache
curl -sv --max-time 5 http://backend-host:port/health -o /dev/null

# 4. Check proxy error rates in the access log (502 = bad response/refused,
#    503 = pool exhausted or all members errored, 504 = backend timeout)
tail -5000 /var/log/apache2/access.log | awk '$9 ~ /^50[234]$/ {print $9}' | sort | uniq -c

# 5. Check whether Apache workers are piling up waiting on backends
curl -s http://localhost/server-status?auto | grep -E "BusyWorkers|IdleWorkers|Scoreboard"

# 6. Look for proxy connect and read failures
grep -E "AH01114|AH00898" /var/log/apache2/error.log | tail -10

# 7. Rule out SELinux on RHEL-family systems
grep -i "denied.*httpd" /var/log/audit/audit.log 2>/dev/null | tail -5

Note on check 1: balancer-manager only shows balancers defined outside <Location> containers. If your balancer is defined inside a <Location> block, it will not appear or cannot be controlled dynamically, and you will have to read state from the error log instead. If your balancer-manager page looks empty or incomplete, verify that first.

How to diagnose it

1. Confirm which members are affected and since when. Use balancer-manager or grep AH00959 from the error log. The AH00959 line names the worker and prints the retry interval, so you know exactly when the balancer will try it again.

2. Determine the failure type: refused, timeout, or status-based. Connection refused means the backend is not listening (process down, wrong port, firewall). Timeout means the path or the backend is slow. Status-based ejection only happens if you configured failonstatus, in which case the backend is answering but returning codes you told Apache to treat as fatal. The remediation is completely different for each.

3. Test the backend directly from the Apache host. A fast direct curl with a normal response plus a member stuck in Err points at the balancer configuration or a transient that has not yet hit its retry window. A slow or failing direct curl means the backend is the incident; Apache is just the messenger.

4. Check whether the pool is still serving. Count how many members remain in Ok state versus the pool total, and compare with current request rate. Two members down out of eight during off-peak is a ticket. Two down out of three during peak means the remaining member is about to be the story: watch BusyWorkers, the listen queue (ss -ltn), and 503 counts.

5. If members flap, correlate with backend events. Pull AH00959 timestamps and line them up against backend restarts, deploys, and GC pauses. A member that re-enters error state within one retry interval of recovering is telling you the backend recovers for a few seconds and then falls over again.

6. Check for the bybusyness trap. If the balancer uses lbmethod=bybusyness, a worker that recovers may stay in error state, or return to Ok but receive no requests, until Apache is restarted. This does not happen with the default byrequests method. If you are on bybusyness and a recovered member is not taking traffic, that is a known behavior, not a new failure.

Metrics and signals to monitor

The key correlation for this incident class: member status changes lead, 5xx rates lag. If you alert only on 503s, you find out after the pool is fully drained. Watching member state directly gives you the interval between “first member errored” and “last member errored”, which is where you can still act.

Fixes

Backend is down or refusing connections

Fix the backend. Apache is behaving correctly. While the backend is down, decide whether the balancer should fail over (spare or standby member) or fail fast (short retry, short timeouts) so clients get a quick error instead of a held worker and a slow 503.

Member flapping because the backend is intermittently slow

A slow backend that occasionally crosses a timeout will ping-pong in and out of error state, and every re-entry costs another retry interval of reduced capacity. Two knobs govern this:

• retry on the BalancerMember: seconds to wait before retrying an errored worker. Default 60. retry=0 means always retry the worker, which effectively disables the sidelining behavior. That is appropriate when you would rather spread load onto a degraded backend than concentrate it on fewer members, but every failed attempt then costs a connection timeout before failover.
• failontimeout: when set, an IO read timeout against the backend forces the worker into error state just like a connect failure.

If the real problem is backend latency, neither knob fixes it; they only decide how gracefully Apache absorbs it. Reducing ProxyTimeout temporarily makes Apache fail fast instead of holding workers in W state, which protects the rest of the server at the cost of more visible errors.

Backend returns 5xx and you want (or do not want) that to eject the member

failonstatus=500,503 on the BalancerMember tells the balancer to treat those response codes from the backend as failures and put the member into error state. This is useful when a backend is “up” but broken. It is dangerous when the 5xx is request-specific (one bad request, not a sick backend), because one poisoned request type can sideline a healthy member for a full retry interval. If members enter error state while direct health checks pass, look here first.

Pool fully errored: 503s on everything

When all members of a balancer are in error state, forcerecovery (default On) makes Apache try all workers again immediately rather than waiting out the retry intervals, on the theory that trying something beats a guaranteed 503. If you set forcerecovery=Off, a fully errored balancer hard-fails with 503 until retry intervals expire.

For planned resilience, configure a hot spare (status=+R, drop-in replacement for an unusable worker in the same lbset) or a hot standby (status=+H, activates only when all workers and spares in the set are unavailable). Expect one visible artifact: the first request during the failover transition can return 503 before the standby takes over from the next request onward. That single 503 is normal behavior, not a second failure.

Recovered member not taking traffic on bybusyness

Known issue with lbmethod=bybusyness: a recovered worker can stay in error state, or return to Ok without receiving requests, until httpd is restarted. If you are affected, either restart Apache during a low-traffic window or switch the balancer to the default byrequests method, which does not have this behavior. A restart is disruptive: it drops connections unless you use apachectl graceful, and even graceful restarts briefly overlap old and new children, so plan for the memory overlap.

Prevention

• Watch member status continuously, not during incidents. A pool that silently loses one of three members has lost a third of its capacity with zero alerts. Alert on any member in Err or C state with active traffic.
• Size retry to your backend’s real recovery time. If backends typically recover in 5 seconds, a 60-second retry leaves capacity on the floor after every blip. If backends take minutes, a short retry just adds failed connection attempts.
• Restrict balancer-manager. It must sit behind Require directives (localhost only is the sane default). The interface has an XSS history in older versions, it lets anyone who can reach it disable your backends, and only balancers defined outside <Location> containers are controllable through it anyway.
• Set ProxyPassInherit Off if you use balancer-manager for dynamic changes. Inheritance of ProxyPass directives into vhosts can cause inconsistent behavior with manager-driven changes.
• Use mod_proxy_hcheck for active health checks if you want members ejected based on probing rather than live request failures. Failed dynamic health checks show as the C flag and are re-enabled by the checker when the backend recovers.
• Monitor the backend independently of Apache. “Apache 503” and “backend down” look identical from outside. Direct backend probes separate the two before the incident call starts.
• Test failover before you need it. Deliberately stop one backend in staging and watch: the AH00959 line, the flag change, the spare takeover, and the single 503 on transition. Any surprise here belongs in staging, not production.

How Netdata helps

• Per-code error rates on proxied paths: Netdata’s Apache collector and web log parsing split 502, 503, and 504, so you can tell “backend refused” from “pool drained” without grepping access logs mid-incident.
• Scoreboard state distribution over time: a rising W count alongside a dropping BusyWorkers-to-capacity ratio is the signature of members being sidelined and remaining backends holding workers longer. Point-in-time balancer-manager checks miss the trend.
• Error log event correlation: AH00959 timestamps aligned against the 5xx timeline show whether member ejections precede or follow user-visible errors, which tells you whether the balancer is protecting you or fighting you.
• Backend-side metrics on the same dashboard: putting the backend’s own latency and process health next to Apache’s proxy errors collapses the “is it Apache or the backend” question to one glance.
• Alerts on proxy error rate with per-member context: alerting on any sustained 502/503/504 rate catches the drained pool before it becomes a full 503 outage.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• Apache 502 Bad Gateway: a backend that returned an invalid response
• Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion
• Apache 504 Gateway Timeout: slow backends, ProxyTimeout, and worker pile-up
• Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status
• Apache CLOSE_WAIT and TIME_WAIT: connection leaks versus normal churn
• How Apache HTTPD actually works in production: a mental model for operators
• Apache keepalive consuming workers: KeepAliveTimeout, the K state, and MPM choice
• Apache listen queue overflow: Recv-Q growth, ListenBacklog, and refused connections
• Apache AH00484: server reached MaxRequestWorkers setting - worker pool exhausted
• Apache MaxRequestWorkers tuning: sizing the worker pool against memory
• Apache HTTPD monitoring checklist: the signals every production web server needs
• Apache HTTPD monitoring maturity model: from survival to expert