Apache 502 Bad Gateway: a backend that returned an invalid response

You are seeing 502 Bad Gateway responses from Apache, usually in bursts, often on some requests and not others. Users report intermittent failures. Apache is running, the port is open, static content may even work. The proxied application path is the thing failing.

A 502 from Apache is almost never an Apache bug. In a mod_proxy deployment, 502 means Apache acted as a reverse proxy, forwarded the request to a backend, and got back something it could not use: a garbled response, an empty response, a connection closed mid-reply, or a connection that refused to carry the request at all. Apache is surfacing a backend signal.

Your job is to find out which backend, which failure mode, and whether it is crashing, overloaded, or misconfigured. The first move is always the same: bypass Apache and talk to the backend directly.

What this means

When Apache proxies a request, it opens (or reuses) a connection to the backend, sends the request, and waits for a valid HTTP response. A 502 is returned when that exchange fails in a way that is not a clean timeout and not a pool exhaustion:

• The backend accepted the connection but sent an invalid or unparsable status line, or closed the connection before sending one. Error log: AH01102: error reading status line from remote server.
• The backend started responding and then the read failed partway through (crash mid-response, connection reset). Error log: AH00898: Error reading from remote server.
• The proxy could not connect or dispatch the request at all. Error log: AH01114: HTTP: failed to make connection to backend (HTTP backends), or AH01075: Error dispatching request to (FastCGI/AJP) paired with a connect error such as (111)Connection refused. These produce 503, not 502; grep for them when triaging mixed 5xx bursts.

Three error codes worth memorizing:

Do not confuse 502 with its neighbors. The distinction determines where you look:

If you are seeing 503s instead of 502s, this is the wrong article. See Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion.

Common causes

One pattern that does not belong here: all workers stuck waiting on a slow-but-alive backend. That produces 504s first, then 503s as the worker pool drains, not 502s. That is the slow backend cascade; the signals and response differ.

Quick checks

These are all read-only and safe to run during an incident.

# 1. Confirm the 502s and their rate (assumes combined log format, status in field 9)
tail -5000 /var/log/apache2/access.log | awk '$9 == 502 {c++} END {print c+0, "502s in last 5000 requests"}'

# 2. See which proxied paths are failing
tail -5000 /var/log/apache2/access.log | awk '$9 == 502 {print $7}' | sort | uniq -c | sort -rn | head

# 3. Pull the actual proxy error lines
grep -E "AH01102|AH00898|AH01075|AH01114" /var/log/apache2/error.log | tail -20

# 4. Bypass Apache: hit the backend directly (adjust host/port/path)
curl -sv --max-time 10 http://backend-host:8080/the/failing/path -o /dev/null

# 5. From the Apache host: count live backend connections and probe backend health
ss -tn state established dport = :8080 | wc -l
curl -s -o /dev/null -w "connect: %{time_connect}s ttfb: %{time_starttransfer}s code: %{http_code}\n" \
  --max-time 10 http://backend-host:8080/health

# 6. Check Apache's own saturation state (rules out a worker-exhaustion disguise)
curl -s http://localhost/server-status?auto | grep -E "BusyWorkers|IdleWorkers"

# 7. Confirm Apache uptime so you can correlate 502 bursts with restarts or deploys
curl -s http://localhost/server-status?auto | grep ServerUptimeSeconds

Adjust log paths for your distribution (/var/log/httpd/ on RHEL-family systems). Check 4 is the important one. Everything before it tells you that 502s are happening; check 4 starts telling you why.

How to diagnose it

Work through these in order. Each step narrows the fault domain.

1. Scope the failures. From check 2 above: is it every proxied path, one path, one vhost, one backend in a balancer? A single failing path points at the application behind that path. All paths failing points at the backend process, the network hop, or the proxy configuration.

2. Read the error log at the failure timestamps. Match access-log 502 lines to error-log entries by timestamp. The AH code tells you the failure class: AH01075/AH01114 means the request never got dispatched (connectivity; those requests return 503, not 502), AH01102 means it was dispatched but the reply was unreadable (backend behavior), AH00898 means the reply started and then broke.

3. Bypass Apache. From the Apache host, curl the backend directly on the failing path. Three outcomes:

   • Backend fails the same way: Apache is innocent. Debug the backend.
   • Backend responds but with something odd (empty body, non-HTTP bytes, giant headers): you have found the invalid response. Capture it with curl -sv and inspect the raw exchange.
   • Backend responds cleanly and fast: the failure is in the Apache-to-backend path. Suspect keepalive races, connection reuse, proxy pool state, or an intermediary (SELinux, firewall, conntrack).

4. Correlate with backend lifecycle events. Overlay 502 bursts with deploy times, backend restarts, and backend crash logs. 502s that cluster tightly around restarts are usually the backend closing connections mid-request during shutdown, not a capacity problem.

5. Check Apache’s own saturation. Confirm BusyWorkers/IdleWorkers from check 6. If IdleWorkers is zero and you also see AH00484: server reached MaxRequestWorkers setting in the error log, worker exhaustion is compounding the backend problem: each failed or retried request still holds a worker. The 502 is the symptom; the backend failure plus worker pile-up is the incident.

6. If the backend is in a balancer, check member state. If balancer-manager is enabled, look at whether the member is marked errored or disabled. A member flapping between ok and error produces intermittent 502s that look random from the outside.

flowchart TD
  A[502/503 in access log] --> B{Error log code?}
  B -->|AH01075 / AH01114| C[Connectivity: backend down, refused, or blocked]
  B -->|AH01102 / AH00898| D[curl backend directly]
  D -->|Fails the same| E[Backend crash or protocol error]
  D -->|Responds cleanly| F[Proxy path: keepalive race, pool state, SELinux]
  C --> G[Check backend listener and firewall]
  E --> H[Check backend logs and restarts]
  F --> I[Fix reuse and pool config]

Metrics and signals to monitor

Fixes

Backend crashing or restarting mid-request

Fix the crash, not Apache. If 502s coincide with deploys, the deploy process is dropping in-flight requests: add connection draining to the backend’s shutdown so it finishes active requests before closing listeners. During an active incident, taking the failing backend out of the balancer (or taking the Apache node out of LB rotation) stops user-facing errors while you fix the root cause.

Backend reachable but response invalid

If the direct curl shows a non-HTTP or malformed response, the usual culprits are a protocol mismatch (Apache proxying plain HTTP to a port that speaks HTTPS, or vice versa) or the backend emitting oversized or malformed headers. Fix the protocol scheme in the ProxyPass target, or fix the backend’s header generation. If a backend legitimately sends headers Apache rejects, that is a backend bug to fix, not something to paper over in Apache. Apache’s ProxyBadHeader directive controls how invalid backend header lines (lines with no colon) are handled: IsError (the default) aborts the request with 502, Ignore drops the bad lines, and StartBody treats the first bad line as the start of the body. The default IsError behavior is what produces your 502s.

Keepalive races on reused backend connections

If the backend’s connection idle timeout is shorter than the time Apache holds pooled connections, Apache can send a request on a connection the backend has just closed, producing an empty-reply AH01102. Align the timeouts: the backend should hold idle connections longer than Apache’s proxy pool does. As a diagnostic, disabling keepalive to the backend (for example with SetEnv proxy-nokeepalive 1) should make the sporadic 502s stop, at the cost of a new TCP connection per request. Treat that as confirmation of the race, then fix the timeout alignment rather than leaving keepalive off permanently.

Connection refused (AH01075)

The backend is not listening, is listening on a different port, is bound to a different interface than Apache is reaching, or something in between (SELinux, firewall, security group) is blocking the connect. ss -ltn on the backend host and a direct curl from the Apache host as an unprivileged user resolve which of these it is. On SELinux-enforcing systems, check the audit log for denials against httpd making outbound connections.

Worker pool draining as a side effect

If the 502 burst has also exhausted workers, fix the backend first. Raising MaxRequestWorkers only gives the failing backend more workers to consume. If you must stop the bleeding before the backend is fixed, temporarily lowering ProxyTimeout makes Apache fail faster and release workers sooner, trading longer hangs for quicker 502/504s. That is a mitigation, not a fix.

Prevention

• Monitor the 5xx classes separately. 502, 503, and 504 have different root causes and different owners. One aggregated 5xx alert hides the distinction that drives diagnosis.
• Watch the error log for the AH proxy codes. AH01102, AH00898, AH01075, and AH01114 are early, specific signals that appear before user reports do.
• Probe backends directly, not only through Apache. A synthetic check that bypasses Apache tells you whether the backend is healthy before Apache’s proxy errors confirm it is not. Teams that do not monitor backend health separately from Apache health waste hours debugging the wrong process.
• Drain connections on backend shutdown. Most 502 bursts in healthy environments trace to deploys and restarts that kill in-flight requests.
• Alert on worker saturation as a compounding signal. A 502 burst plus IdleWorkers at zero plus listen backlog growth is a different incident than a 502 burst alone. Correlate them in one alert path.
• Size the proxy pool deliberately. Default proxy pool sizes are small and per-child-process. Pool exhaustion is a 503 problem, but a starved pool under a flapping backend produces mixed 502/503 pictures that are harder to read.

How Netdata helps

• Netdata’s Apache collector scrapes mod_status every second, so BusyWorkers, IdleWorkers, requests per second, and scoreboard state distribution are captured at the granularity where 502 bursts actually happen, not at 60-second polling intervals that average them away.
• Per-second worker utilization next to your 5xx rate lets you answer the key question immediately: are 502s coming with worker exhaustion (compounding cascade) or without it (pure backend failure)?
• Scoreboard state charts show workers accumulating in W state waiting on backends, which distinguishes “backend slow” from “backend broken” before you open a single log file.
• Netdata’s ML anomaly detection flags deviations in request rate and worker utilization per server, catching the intermittent, low-rate 502 patterns that static thresholds miss.
• Correlating Apache metrics with the backend service’s own metrics on one dashboard shortens the bypass-Apache diagnostic step from a manual curl exercise to a glance.

Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion
• Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status
• How Apache HTTPD actually works in production: a mental model for operators
• Apache keepalive consuming workers: KeepAliveTimeout, the K state, and MPM choice
• Apache listen queue overflow: Recv-Q growth, ListenBacklog, and refused connections
• Apache AH00484: server reached MaxRequestWorkers setting - worker pool exhausted
• Apache MaxRequestWorkers tuning: sizing the worker pool against memory
• Apache HTTPD monitoring checklist: the signals every production web server needs
• Apache HTTPD monitoring maturity model: from survival to expert
• Apache scoreboard states explained: what _ S R W K D C L G tell you
• Apache ListenBacklog vs net.core.somaxconn: the silently truncated accept queue