HAProxy 502 Bad Gateway: backend connection and response failures

Clients are getting 502 Bad Gateway from HAProxy. The backend may look healthy, health checks may be green, and yet some fraction of requests comes back with a 502 the application never logged. That is the signature: a 502 from HAProxy is usually generated by HAProxy itself, not passed through from the backend.

HAProxy emits a 502 in two situations. First, it could not establish or use a connection to the backend server at all. Second, it connected fine, but the server sent an invalid, truncated, or incomplete response, or closed the connection mid-response. These are different failure modes with different fixes, and the fastest path to resolution is separating them before touching anything.

This is also what separates a 502 from a backend-returned 500. A 500 means the application processed the request and failed. A 502 from HAProxy means the request never completed at the protocol level: the connection failed, or the bytes that came back were not a valid HTTP response. The backend may have no log of the request at all.

What this means

In the HAProxy stats model, the two failure modes map to two different counters:

• econ (connection errors): HAProxy tried to connect to a backend server and failed. Connection refused, timed out, or reset. The server is unreachable or overwhelmed.
• eresp (response errors): HAProxy connected, sent the request, and the server sent something that was not a valid HTTP response, or closed the connection mid-response. Server aborts (srv_abrt) track the related case where the server dropped the connection before completing the response.

Both surface to the client as 502 and both increment frontend hrsp_5xx. The diagnostic trick is that hrsp_5xx on backend/server rows counts errors the backend actually returned, while the frontend counter includes HAProxy-generated errors. A frontend 5xx rate higher than the backend 5xx rate means HAProxy is generating errors itself: 502s from connection or response failures, 503s from no available servers, 504s from timeouts.

flowchart TD
  A[502 on frontend] --> B{Frontend 5xx > backend 5xx?}
  B -- "No: equal" --> C[Backend returned the error - check app logs]
  B -- "Yes: HAProxy generated" --> D{econ or eresp rising?}
  D -- econ --> E[Cannot connect: refused, timeout, reset]
  D -- eresp --> F[Connected but response invalid, truncated, or aborted]
  D -- neither --> G[Check logs for timeout flags and http-response deny rules]

Common causes

Quick checks

All of these are read-only and safe to run during an incident. They assume the admin socket at /var/run/haproxy.sock; adjust the path to your deployment. Field numbers below match the standard show stat CSV column order; if your version differs, verify against the header line (show stat row starting with # pxname).

# Compare frontend vs backend 5xx to see who generated the errors
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '{print $1"/"$2": 5xx="$44}'

# Connection errors per backend/server (cannot connect)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": econ="$14}'

# Response errors per backend/server (connected but broken response)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": eresp="$15}'

# Server aborts (server dropped mid-response)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": srv_abrt="$51}'

# Connect time: zero ctime + econ = refusal; high ctime + econ = timeout
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": ctime="$60"ms rtime="$61"ms"}'

# Denied responses: 502s caused by http-response deny rules
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": dresp="$12}'

# Captured malformed request/response samples
echo "show errors" | socat unix-connect:/var/run/haproxy.sock stdio

Two notes on interpretation. All the error counters are cumulative and reset on reload, so compute deltas between two samples to get a rate. And ctime/rtime are rolling averages over the last 1024 requests, so brief spikes get smoothed; the counters tell you that failures happened, the averages tell you the recent shape.

How to diagnose it

1. Confirm HAProxy is the source. Compare frontend hrsp_5xx against the sum of backend hrsp_5xx for the affected backend. If the frontend count is higher, HAProxy is generating the 502s. If they are roughly equal, the backend is returning errors and this is an application problem, not a proxy problem.

2. Split econ from eresp. Take two samples a minute apart and compute the per-server delta for econ and eresp. A rising econ means “cannot connect”. A rising eresp means “connected but the response broke”. Most incidents are dominated by one or the other; chasing both at once wastes time.

3. If econ: distinguish refusal from timeout. Check ctime. Near-zero ctime with econ means immediate refusal: the backend process is down, the port is wrong, or a firewall is rejecting. High ctime before the error means the connect attempt timed out: network congestion, a full accept queue on the backend, or an overloaded server too slow to answer the SYN. Also check server status: if the server is flapping UP/DOWN, some 502s are traffic hitting it during the detection window before health checks mark it DOWN.

4. If eresp: check timing first. Compare rtime against timeout server. If rtime sits near or above the timeout, HAProxy is cutting slow responses off and returning 502 for what would eventually have been a valid response. In the logs, these show up with the sH termination flag: timeout server fired before the response headers arrived.

5. Read the termination state flags in the logs. The session state at disconnection in each log line tells you exactly where the session died. The ones that matter for 502:

   • SH: the server aborted before sending complete response headers, or crashed while processing the request.
   • PH: HAProxy blocked the server’s response because it was invalid, incomplete, or matched a security filter. A 502 is sent to the client.
   • SD: the connection to the server died mid-transfer, typically an RST from the server or an ICMP error from a device in between.
   • sH: timeout server fired before the response headers arrived.
   • SC: the connection to the server was explicitly refused (RST or ICMP in return).

6. Capture the offending response. For PH-style failures, show errors on the admin socket dumps the last captured malformed requests and responses, including where parsing failed. This is the definitive check for “the backend sent something HAProxy could not parse”. If show errors returns nothing but the flag is SH, the failure is on the server side before a full response existed, so there was nothing to capture.

7. Check for masking. Look at wretr and wredis. If retries and redispatches are elevated, HAProxy has been compensating for backend instability for a while and the visible 502s are only the requests retries could not save. The real failure rate is higher than the client-visible one.

8. Rule out resource-side causes on the HAProxy host. If econ spikes with no corresponding backend failure, check for ephemeral port exhaustion: high backend connection churn with poor reuse can exhaust the local port range and make new backend connections fail. A drop in the connection reuse ratio alongside rising ctime is the tell.

Metrics and signals to monitor

Fixes

Backend down or refusing connections

Restore the backend process and verify the port and bind address match the server line. If 502s happen during every deploy, the gap is health check detection latency: while the server is restarting, traffic still hits it until the fall threshold trips. Reduce that window with tighter check intervals or take servers out gracefully before stopping them: disable server via the admin socket removes the server from load balancing (this changes live proxy state, so confirm the pxname/svname first), or use your orchestrator’s drain step. Tradeoff: faster checks mean more probe traffic and more CPU on both sides.

Connect timeouts

If ctime is climbing toward timeout connect, the problem is the network path or a backend too overloaded to accept. Fix the saturation on the backend side; raising timeout connect only converts fast 502s into slow ones and holds queue slots longer. If connection reuse collapsed (reuse ratio dropped, ctime jumped from zero), find what changed: backends adding Connection: close, an HTTP version change, or http-reuse misconfiguration.

Server aborting mid-response

srv_abrt and eresp concentrated on one server usually means the application process is crashing or being OOM-killed during response generation. Check the backend host for OOM killer activity and crash logs. HAProxy-side tuning will not fix a dying backend.

Slow responses truncated by timeout server

If sH flags dominate and rtime is near timeout server, decide which side is wrong. If the backend is legitimately slow for some endpoints (exports, reports), raise timeout server for those routes or move them to a dedicated backend with a larger budget. If the slowness is new, the fix is backend performance, not the timeout. Raising the timeout globally trades 502s for longer connection hold times, which pushes you toward queuing and saturation; see HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade for how that cascade develops.

Invalid responses that HAProxy rejects

When show errors shows a structurally broken response (duplicate or malformed headers, bad chunked framing), the durable fix is on the backend: make it emit valid HTTP. HAProxy offers option accept-invalid-http-response to relax some parsing rules, but it only covers specific violations and does not fix structural problems like conflicting transfer encoding headers; treat it as a temporary bridge, not the resolution. If 502s began immediately after an HAProxy upgrade with an unchanged backend, stricter response parsing in the new version is a prime suspect: capture the failing response with show errors and fix the backend output.

Very large response headers can also fail against buffer limits. HAProxy allocates buffers per connection sized by tune.bufsize (16KB default); if the failing responses carry unusually large headers, that is worth checking before changing anything else.

502s from response ACLs

If dresp tracks the 502 rate, an http-response deny rule is blocking the response and HAProxy returns 502 by design. Review the response ACLs against the blocked traffic; this is configuration intent, not a backend failure.

Prevention

• Alert on the econ/eresp split, not just 5xx. Any sustained nonzero rate on either counter warrants a ticket. They catch the two 502 modes independently of log parsing.
• Alert on the frontend minus backend 5xx delta. A rising delta is the earliest “HAProxy is generating errors” signal and shortens every future 502/503/504 investigation.
• Watch retries and redispatches. wretr/wredis above 1% of requests means instability is being masked. Fix it while it is still invisible to users.
• Keep headroom between rtime and timeout server. If typical response time creeps past half the timeout budget, slow-response truncation 502s are one bad day away.
• Load-test backends for response validity, especially after framework or proxy-layer changes in the application stack, and after HAProxy upgrades.
• Protect connection reuse. Monitor the reuse / (connect + reuse) ratio so churn-driven econ and ephemeral port pressure get caught early.
• Drain servers on deploys. Graceful removal before shutdown eliminates the refused-connection 502 window.

How Netdata helps

• Per-backend and per-server econ and eresp rates collected every second, so the “cannot connect” versus “broken response” split is visible immediately, without sampling the socket by hand.
• Frontend and backend hrsp_5xx side by side, making the HAProxy-generated error delta a direct visual comparison instead of a computation.
• srv_abrt, wretr, and wredis on the same dashboard as error rates, exposing masked instability and mid-response aborts alongside the 502s they eventually produce.
• ctime and rtime trends per server, so you can see refusals versus timeouts and response times approaching timeout server before truncation starts.
• Health check status changes correlated with error spikes, catching 502s caused by traffic hitting servers inside the detection window.
• Counter-reset handling across reloads, so delta-based 502 rates stay accurate through config reloads that zero the stats.

Related guides

• HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade
• HAProxy backend losing servers: active server count and cascade risk
• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy health checks green but the application is broken: when UP does not mean healthy
• HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable
• HAProxy health check L7STS: server DOWN on the wrong HTTP status
• How HAProxy actually works in production: a mental model for operators
• HAProxy maxconn hierarchy: global, frontend, backend, and per-server limits
• HAProxy maxconn reached: new connections queued and rejected
• HAProxy monitoring checklist: the signals every production proxy needs
• HAProxy monitoring maturity model: from survival to expert