HAProxy response errors (eresp): truncated and invalid backend responses

The eresp counter is climbing on a backend, and clients are seeing broken pages, truncated downloads, or intermittent 502s. The backend servers report healthy. Health checks are green. The 5xx rate, if it moved at all, does not explain the volume of user complaints.

This is the failure mode eresp exists to capture: HAProxy connected to the backend successfully, sent the request, and then received a response it could not treat as valid. Either the bytes were not parseable HTTP, or the server closed the connection before the response finished. That is a protocol-level failure, not an application returning an error status code, and it needs a different investigation path than a 5xx spike.

What this means

eresp is field 15 of the CSV stats (1-indexed, as awk counts columns). It is a cumulative counter on BACKEND and SERVER rows and increments when:

• The server sent bytes HAProxy could not parse as valid HTTP (junk headers, malformed status line, protocol garbage).
• The server closed the connection mid-response: a Content-Length body that ends early, a chunked response that never terminates, or a connection drop partway through transfer.

Two semantic traps to know up front:

• eresp is not a 5xx. A backend that returns a well-formed HTTP 500 increments hrsp_5xx, not eresp. eresp fires when the response never made it through the HTTP layer intact. Depending on when the failure happens, the client may see a 502 from HAProxy, a truncated response, or a dropped connection.
• eresp is backend/server-scoped. FRONTEND rows do not carry it; request-phase errors on the frontend side are counted under ereq instead. When you are chasing truncated backend responses, look only at BACKEND and SERVER rows.

In TCP mode (mode tcp), there is no HTTP to parse, but eresp can still increment on abnormal server-side closes after a successful connect.

The most common monitoring mistake with this signal is treating it as interchangeable with econ. They are different failure modes: econ means HAProxy never got a connection to the server (refused, timed out, reset during connect), while eresp means the connection worked and the response broke. Lumping them together, or watching only one, hides which side of the failure you are on.

flowchart TD
  A[Request proxied to backend] --> B{Connection established?}
  B -- No --> C[econ increments]
  B -- Yes --> D{Response received intact?}
  D -- Yes, valid HTTP --> E[Normal path - hrsp_ counters]
  D -- Invalid HTTP bytes --> F[eresp increments]
  D -- Closed mid-response --> F
  F --> G{When did it break?}
  G -- Before/early in headers --> H[Client usually sees 502]
  G -- During body transfer --> I[Client sees truncated response]

Common causes

Quick checks

All read-only. Adjust the socket path if yours differs.

# Per-server and per-backend eresp (field 15, 1-indexed)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": eresp="$15}'

# Correlate with econ, wretr, and wredis on the same rows
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": econ="$14" eresp="$15" wretr="$16" wredis="$17}'

# Server aborts: srv_abrt tracks transfers aborted by the server
# and is counted within eresp. cli_abrt is the client-side twin.
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": cli_abrt="$50" srv_abrt="$51}'

# Captured protocol error samples: the single most useful command
# for eresp. Shows recent invalid requests/responses with the
# actual bytes HAProxy choked on.
echo "show errors" | socat unix-connect:/var/run/haproxy.sock stdio

# Is this eresp or a 5xx problem? Compare frontend vs backend 5xx
# (hrsp_5xx is field 44)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '{print $1"/"$2": 5xx="$44}'

# Check whether rtime is approaching timeout server (timeout hypothesis)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "BACKEND" {print $1": rtime="$61"ms qcur="$3}'

Also grep HAProxy logs for the termination state field. Flags like sD (server closed during data transfer), SH/SD (server timeout during headers/data), and -- (clean close) on sessions to the affected backend tell you which side terminated and when. Per-request log timing is more sensitive than the aggregate counters for catching this pattern.

How to diagnose it

1. Confirm scope and rate. Pull per-server eresp twice, a minute apart, and compute deltas. A counter that went up once and stopped is a past event; a counter incrementing steadily is a live failure. Note whether one server is responsible or all servers in the backend.

2. Rule out the easy confusion: is it eresp or 5xx? If the frontend 5xx rate tracks the backend 5xx rate, the backends are returning well-formed error responses and eresp is not your problem. If eresp is rising while backend 5xx stays flat, you have a genuine protocol-level failure.

3. Pull show errors. This is the decisive step. HAProxy keeps a ring buffer of recent protocol violations, including the raw bytes that failed to parse. If the captured response is junk (non-HTTP bytes, binary garbage, an error page from a middlebox), you have a protocol mismatch or a corrupting middlebox. If show errors is empty but eresp climbs, the failures are mid-response closes rather than unparseable bytes, which points at crashes or timeouts.

4. Check srv_abrt alongside. Server aborts rising in step with eresp points at the server terminating transfers: crashes, OOM kills, application restarts. Check backend application logs and host-level evidence (dmesg for the OOM killer, service restart timestamps).

5. Test the timeout hypothesis. If rtime (rolling average, last 1024 requests) sits near your timeout server, slow responses are being cut off. Identify whether specific endpoints (large reports, exports, streaming responses) correlate with the eresp timing. A request that legitimately needs 45s against a timeout server of 30s will produce exactly this signature.

6. Bypass HAProxy to isolate the layer. From the HAProxy host, issue the same request directly to the backend server with curl -v. If the response is complete and valid direct to the server but broken through HAProxy, look at what sits between: middleboxes, and version-specific HAProxy behavior. If the response is truncated even directly, the backend is at fault.

7. Check connection reuse. If eresp events cluster on what should be reused connections, compare connect vs reuse counters on the backend. A backend that silently closes idle keep-alive connections earlier than HAProxy expects can race with request dispatch and produce mid-response-looking failures. A recent drop in the reuse ratio supports this.

8. Mind the reload caveat. All cumulative counters reset on reload. If Uptime_sec is low, short-window deltas may mislead. Confirm the process has been up long enough for the trend to be real.

Metrics and signals to monitor

Fixes

Backend crashing or OOM-killed mid-response

Fix the backend, not HAProxy. Find the crash evidence (application logs, OOM killer in dmesg, restart timestamps) and address memory limits or the crash bug. Tradeoff to be aware of: a crash that closes the TCP connection cleanly (FIN) at a response boundary may not increment eresp at all; it shows up as a truncated response to the client with little HAProxy-side evidence. That is why per-request logs matter.

timeout server too low

Raise timeout server to accommodate legitimately slow responses, or move the slow endpoints (exports, reports) behind a separate backend with a longer timeout. Tradeoff: longer timeouts hold backend connection slots longer, which increases queuing risk under load (qcur). If you raise the timeout, watch queue depth. The better long-term fix is usually making the slow operation asynchronous.

Protocol mismatch

Align the modes. If the backend speaks plain TCP, the HAProxy backend must be mode tcp. If a backend was recently changed (new framework, HTTP/2 enabled, TLS added or removed), that change is your suspect. show errors output makes this diagnosis nearly instant because the captured bytes are visibly not HTTP.

Network truncation or corruption

Identify and fix the middlebox, or remove it from the path. This is the rarest cause but the hardest to see from either endpoint in isolation; the bypass test in step 6 is how you prove it.

Keep-alive race on reused connections

Tune the idle timeouts so the side with the shorter patience is the client side of the pair: HAProxy’s backend keep-alive timeout should be shorter than the server’s, so HAProxy closes idle connections before the server can. Verify the effect by watching the reuse ratio and eresp after the change.

Masking with retries

HAProxy’s retries directive and option redispatch can absorb some transient failures before clients see them, and a rising wretr/wredis alongside eresp means that compensation is already happening. Do not treat retries as a fix for eresp: they add latency, they cannot safely retry non-idempotent requests, and a truncated response partway through the body is not a retryable condition. Retries buy time; they do not repair a broken backend.

Prevention

• Alert on eresp and econ separately. They are different failure modes with different responses. Ratios, not absolutes: eresp as a percentage of requests to that backend, sustained, is the alertable condition.
• Watch srv_abrt as a leading companion. It often moves before or with eresp when backends are unstable.
• Keep show errors in your incident runbook. It is the fastest path from “eresp is rising” to “here are the actual bytes that broke.”
• Size timeout server against measured response times, including the slow legitimate endpoints, and revisit when the application changes.
• Correlate eresp with deploys and config changes. Protocol mismatches and keep-alive races almost always trace back to a recent change on one side of the connection.
• Log termination states and per-request timing. The aggregate counters tell you something broke; the logs tell you which requests, which endpoints, and who closed first.

How Netdata helps

• Netdata collects the HAProxy CSV stats per server and per backend, so eresp, econ, srv_abrt, and wretr/wredis are charted on the same timeline, which makes the “connect failure vs response failure” split visible at a glance.
• Counter resets on reload are handled automatically, so delta-based eresp rates stay meaningful across config reloads that would otherwise produce phantom spikes.
• Per-server breakdowns let you see immediately whether eresp is isolated to one backend server or spread across the pool, which is the first fork in the diagnostic path.
• Correlating eresp with rtime and qcur on the same dashboard surfaces the timeout-cascade pattern (slow responses being cut off, then queuing) without manual log digging.
• Anomaly detection on a normally-zero counter like eresp catches the low, steady drip of mid-response failures that threshold alerts miss until users complain.

Related guides

• HAProxy 502 Bad Gateway: backend connection and response failures
• HAProxy 503 Service Unavailable: no server is available to handle this request
• HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade
• HAProxy backend losing servers: active server count and cascade risk
• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy connect time (ctime) high: network latency and backend accept-queue overflow
• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy health checks green but the application is broken: when UP does not mean healthy
• HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable
• HAProxy health check L7STS: server DOWN on the wrong HTTP status
• How HAProxy actually works in production: a mental model for operators
• HAProxy maxconn hierarchy: global, frontend, backend, and per-server limits