Apache proxy connection refused: AH01114 / (111) Connection refused to backend

Your error log is filling with lines like these:

AH00957: HTTP: attempt to connect to 127.0.0.1:8080 (localhost) failed
AH01114: HTTP: failed to make connection to backend: localhost

Clients see 502 or 503 responses. Apache itself is running fine. The failure is on the outbound leg: mod_proxy tried to open a TCP connection to the backend and the kernel told it no.

The important thing about (111)Connection refused is what it rules out. ECONNREFUSED means the backend host’s network stack answered immediately with a TCP RST. The host is reachable; nothing is listening on that port, or something actively rejected the connection. That is a different failure from a backend that is slow, hung, or firewalled, which produces a connect timeout after a long wait and a different error path (AH01075: Error dispatching request to, or a 504 after ProxyTimeout). Learning to separate “fast refusal” from “slow timeout” is the core diagnostic move on this page, and it is the difference between “restart the backend” and “go look at the network.”

What this means

When mod_proxy cannot establish a backend connection, you typically see this sequence in the error log:

AH00957: HTTP: attempt to connect to <host>:<port> (<name>) failed
AH00959: ap_proxy_connect_backend disabling worker for (<name>) for 60s
AH01114: HTTP: failed to make connection to backend: <name>

The AH00959 line matters operationally: after a connect failure, Apache marks that proxy worker as being in an error state and will not use it again for retry seconds (default 60). If the backend comes back in 5 seconds, Apache will still refuse to try it for the rest of the cooldown window. With a single backend behind ProxyPass, that means the site stays down for up to a minute after the backend recovers. With a balancer, requests fail over to healthy members, and the errored member rejoins after its retry expires.

If retry=0 is set, there is no cooldown: every incoming request immediately retries the dead backend, and every failure logs the full AH00957/AH01114 sequence. A busy site with retry=0 and a dead backend can produce a serious error-log flood on top of the outage.

Common causes

One more distinction: if the connect attempt hangs for the full connection timeout rather than failing instantly, that is not “connection refused.” That is an unreachable or packet-dropping path (firewall DROP, wrong subnet, dead route), and it belongs to the timeout playbook, not this one.

Quick checks

All read-only. Run from the Apache host unless noted.

# 1. See the actual error sequence and which target is failing
grep -E "AH00957|AH00959|AH01114|AH01075|AH02454" \
  /var/log/httpd/error_log /var/log/apache2/error.log 2>/dev/null | tail -30

# 2. Test the backend connect directly, with timing
#    time_connect near zero + "Connection refused" = fast RST (this page)
#    time_connect == timeout = slow/unreachable path (different problem)
curl -sv --max-time 5 -o /dev/null \
  -w "connect: %{time_connect}s total: %{time_total}s\n" \
  http://<backend-host>:<backend-port>/

# 3. Confirm what Apache is actually resolving and dialing
getent hosts <backend-hostname>   # does localhost return ::1 first?

# 4. Check SELinux state and recent denials (RHEL/Fedora)
getenforce
ausearch -m avc -ts recent | grep -i httpd | tail -10

# 5. Check the FD situation on Apache children
for pid in $(pgrep 'httpd|apache2'); do
  echo -n "PID $pid: "; ls /proc/$pid/fd 2>/dev/null | wc -l
done
grep "Max open files" /proc/$(pgrep -o 'httpd|apache2')/limits

# 6. Count live backend connections from Apache's side
ss -tn state established dport = :<backend-port> | wc -l

# 7. If using a balancer, check member status
curl -s http://localhost/balancer-manager 2>/dev/null | grep -iE 'worker|status'

On the backend host, the two questions that close the case most of the time:

# Is anything listening where Apache expects it?
ss -ltn | grep ':<backend-port>'

# Is the backend process even alive?
pgrep -a -f '<backend-process-name>'

How to diagnose it

Work the path from Apache outward. Each step eliminates one layer.

flowchart TD
  A[AH01114 in error log] --> B{curl to backend from Apache host}
  B -->|instant refused| C{something listening on port?}
  B -->|permission denied| D[SELinux or AppArmor denial]
  B -->|connects fine| E[Apache-side problem: cooldown, FDs, config]
  C -->|no| F[backend down or wrong port]
  C -->|yes, but different IP family| G[IPv4/IPv6 mismatch]
  E --> H{AH00959 within retry window?}
  H -->|yes| I[wait or reduce retry]
  H -->|no| J[check FD limits and proxy pool]

1. Confirm fast refusal vs slow timeout. Run check 2 above. time_connect returning near-zero with “Connection refused” confirms this page. A time_connect equal to your --max-time means the SYN went nowhere; investigate routing and firewalls instead.

2. Read the target in the log line. AH00957 prints the exact host and port Apache dialed. Verify it matches what you intended in ProxyPass or the BalancerMember line. A surprising port or hostname here means a config or DNS problem, not a backend problem.

3. Check for IPv6. If the log shows [::1]:<port> and the backend binds only 127.0.0.1, that is the whole incident. localhost resolving to ::1 first is common on default installs.

4. Check for permission denied, not refused. If the proxy connect log says (13)Permission denied, the backend may be perfectly healthy. AH02454 is a Unix-domain-socket connect signal in some backported proxy handlers; current upstream commonly logs AH00957 for the failed connect, so check the exact package before relying on either ID. On enforcing SELinux systems, httpd is not allowed to make arbitrary outbound TCP connections by default. Confirm with ausearch before changing anything.

5. Rule out the error-state cooldown. If the backend is now healthy but requests still fail, look at the timestamp of the last AH00959 and compare it to retry. During the cooldown Apache will not even attempt the connection. The balancer-manager page, if enabled, shows the member in error state.

6. Check Apache’s own resources. Per-child FD exhaustion produces AH01114 because the child literally cannot open a new socket. Look for “Too many open files” in the error log; the playbook’s FD exhaustion pattern lists proxy connection failures (AH01114) as a primary symptom.

Metrics and signals to monitor

The down-versus-slow distinction deserves emphasis because it changes the response. A dead backend fails every connect in microseconds; workers are freed immediately and the damage is “only” the 502s. A slow backend holds workers in W state for seconds each, which is the slow-backend-cascade pattern that takes the whole server down. Connect time plus scoreboard state tells you which one you have within a minute.

Fixes

Backend process down

Restore the listener, then find out why it died. Check the backend’s own logs and dmesg for OOM kills. If this is a recurring crash, restarting it without root cause work just schedules the next AH01114 flood. Note the cooldown: even after the backend is up, Apache may refuse to use it until retry expires, or with a balancer you can reset the member from balancer-manager.

Wrong target in ProxyPass

Fix the host or port in ProxyPass / BalancerMember, run apachectl configtest, then apachectl graceful. If the error started right after a deploy or config-management run, diff the current config against the previous revision before touching anything else.

IPv4/IPv6 mismatch

Use 127.0.0.1 explicitly in the ProxyPass target instead of localhost, or make the backend listen on both families. The config fix is one character of intent and removes the dependency on resolver order entirely.

SELinux denial (RHEL/CentOS/Fedora)

The canonical fix for httpd making outbound proxy connections:

# Allow httpd to make network connections (persistent)
setsebool -P httpd_can_network_connect 1

Verify with getsebool httpd_can_network_connect. Do not disable SELinux or set it permissive to fix this; the boolean exists precisely for the reverse-proxy use case.

AppArmor (Debian/Ubuntu)

Less commonly hit, but AppArmor profiles can deny outbound connects the same way. Check the audit log for denials against the apache2 profile and adjust the profile rather than removing it.

Error-state and retry tuning

retry on ProxyPass or BalancerMember controls the cooldown after a connect failure (default 60 seconds). Tradeoffs:

• Lower retry (e.g. 5-10s): faster recovery after a brief backend bounce, at the cost of more failed attempts against a genuinely dead backend.
• retry=0: always retry immediately. Reasonable behind a balancer with multiple members; dangerous with a single backend because every request retries and logs the full failure sequence.
• Balancers: failonstatus lets you push a member into error state when it returns specific HTTP codes, and forcerecovery (2.4.2+) forces immediate recovery of all members if every member is errored, ignoring retry. Both are useful for backends that fail “up” (accepting connections but returning garbage).

Do not confuse the balancer-level timeout (maximum wait for a free member) with the per-member connection timeout; they are different knobs on different lines. connectiontimeout on the member controls how long Apache waits for the TCP connect to complete, and ProxyTimeout (default: the global Timeout, usually 60s) bounds waits on established backend connections. None of these cause ECONNREFUSED, but mis-set values change how the failure presents.

FD exhaustion

If AH01114 arrives alongside “Too many open files,” raise the per-process limit in the systemd unit (LimitNOFILE=65536) and reload. Sizing rule from the playbook: each client connection, each backend proxy connection, and each log file costs an FD, so the limit should cover roughly MaxRequestWorkers x 2 plus log and static overhead. This requires a restart, not a graceful reload.

Prevention

• Health-probe the backend path, not just Apache. A localhost check against a static file will pass while every proxied request 502s. Probe through the proxy to the backend’s health endpoint.
• Alert on the error-log sequence, not just 5xx rate. AH00957/AH01114 rate is a cleaner, earlier signal than access-log 502 percentages, and AH00959 tells you cooldowns are happening.
• Pin the backend address family. Use explicit IPs in proxy targets so resolver behavior cannot change underneath you.
• Set the SELinux boolean at provisioning time. httpd_can_network_connect should be part of the base image or config management for any host running httpd as a reverse proxy.
• Size FD limits for the proxy case. Doubling connections per worker (client plus backend) is the common way FD limits that “were fine” suddenly are not.
• Choose retry deliberately. Default 60s is safe; know that it extends every backend blip into a minute-long outage window for single-backend setups, and that retry=0 trades that for log volume and hammering a dead service.

How Netdata helps

• Netdata’s Apache collector polls server-status every second, so BusyWorkers, idle workers, and request rate are visible at the granularity where a retry storm or client-refresh pile-up actually unfolds.
• The web log collector parses access logs into per-status-code rates, so the 502/503/504 split is a first-class chart rather than an awk pipeline you run during the incident.
• Correlating 5xx-by-code against worker utilization separates “backend dead” (errors up, workers flat) from “backend slow” (errors up, workers saturated in W) at a glance.
• System-level charts (per-process FD usage, TCP connection states, network errors) catch the FD-exhaustion and refused-connection variants without per-host log diving.
• Anomaly detection on error-log rates surfaces the first AH01114 burst rather than the thousandth, which is usually the difference between a ticket and a page.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• Apache 502 Bad Gateway: a backend that returned an invalid response
• Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion
• Apache 504 Gateway Timeout: slow backends, ProxyTimeout, and worker pile-up
• Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status
• Apache CLOSE_WAIT and TIME_WAIT: connection leaks versus normal churn
• How Apache HTTPD actually works in production: a mental model for operators
• Apache keepalive consuming workers: KeepAliveTimeout, the K state, and MPM choice
• Apache listen queue overflow: Recv-Q growth, ListenBacklog, and refused connections
• Apache AH00484: server reached MaxRequestWorkers setting - worker pool exhausted
• Apache MaxRequestWorkers tuning: sizing the worker pool against memory
• Apache HTTPD monitoring checklist: the signals every production web server needs
• Apache HTTPD monitoring maturity model: from survival to expert