HAProxy 400 and 408 spikes: bad requests, request timeouts, and Slowloris

Your frontend hrsp_4xx counter just doubled, and it is not 404s. The spike is 400s and 408s, which means HAProxy itself is rejecting or timing out requests before they reach a backend. Unlike backend-generated 4xx, these errors say something about the bytes arriving at your frontend: malformed requests, protocol mismatches, oversized headers, or clients that open connections and never finish the request.

Split the two codes before doing anything else. A 400 means HAProxy tried to parse the request and failed. A 408 means timeout http-request fired because the client never sent a complete request in time. One is a parsing problem, the other is a timing problem, and they point at very different fixes.

One rule of thumb: if the 4xx spike lines up with a config change, a certificate change, or a version upgrade, the change is the cause almost every time. Check your deploy timeline first.

What this means

When HAProxy generates a 400, the request never left the frontend. Common triggers: request headers larger than tune.bufsize minus tune.maxrewrite, a frontend expecting PROXY protocol (or TLS) and getting something else, HTTP/2 framing violations, and request smuggling probes. All of these also increment ereq on the frontend, which is your confirmation signal.

When HAProxy generates a 408, the client connected but did not deliver a full HTTP request before timeout http-request expired. At low volume this is internet noise and browser pre-connect behavior. At high volume with many concurrent connections and almost no completed requests, it is the classic Slowloris signature: connections held open to consume your session slots and buffers.

In HAProxy logs, these show up with distinctive termination flags: PR when the proxy blocked the request (invalid syntax returns 400, a matched deny returns 403), and cR when timeout http-request fired before a complete request arrived (typically a 408).

flowchart TD
    A[4xx spike on frontend] --> B{Which code?}
    B -->|400 + ereq rising| C{Config or cert change recently?}
    C -->|Yes| D[PROXY protocol or SSL termination mismatch]
    C -->|No| E[show errors: oversized headers, H2 framing, smuggling probe]
    B -->|408| F{scur high and req_rate low?}
    F -->|Yes| G[Slowloris pattern: show sess, check source IP spread]
    F -->|No| H[Slow clients or browser pre-connect noise]

Common causes

Quick checks

All read-only. Adjust the socket path if yours differs. The CSV column positions below follow the field order in the HAProxy management documentation.

# Frontend ereq: confirm which frontends are rejecting requests at parse level
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "FRONTEND" {print $1": ereq="$13}'

# 4xx responses per proxy (field 43 = hrsp_4xx)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '{print $1"/"$2": 4xx="$43}'

# Denied requests (separates policy denials from parse failures)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '{print $1"/"$2": dreq="$11}'

# The single most useful 400 diagnostic: captured malformed requests
echo "show errors" | socat unix-connect:/var/run/haproxy.sock stdio

# Slowloris check: sessions high, request rate low
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "FRONTEND" {print $1": scur="$5" slim="$7" rate="$34" req_rate="$47}'

# Who is holding connections open
echo "show sess" | socat unix-connect:/var/run/haproxy.sock stdio | head -50

# Client aborts, for correlation (field 50 = cli_abrt)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": cli_abrt="$50}'

show errors is the highest-value command here. It dumps the actual captured buffers of recent malformed requests with the error position, which turns “we are getting 400s” into “clients are sending a bare TLS ClientHello to a plaintext frontend” in one step. If you only run one command from this list, run that one.

How to diagnose it

1. Split 400 from 408. The stats CSV gives you aggregate hrsp_4xx only. Get the per-code breakdown from your HAProxy logs (the status field) or your log pipeline. Everything downstream depends on which code dominates.

2. Line the spike up against change history. Config reloads, cert rotations, upstream load balancer changes, and HAProxy upgrades. A spike that starts at a change timestamp is that change until proven otherwise. Counters reset on reload, so compare rates, not totals, across the reload boundary. See HAProxy counter resets on reload.

3. For 400s, run show errors. Read the captured buffers. Garbage bytes at position 0 on an SSL frontend means plaintext hitting a TLS listener (or the reverse). A valid-looking request truncated at the buffer boundary means oversized headers. Two requests concatenated with conflicting Content-Length and Transfer-Encoding headers means a smuggling probe.

4. For 400s, verify the protocol contract on the wire. If the frontend has accept-proxy on the bind line, confirm the upstream LB is actually sending the PROXY header. If the frontend terminates TLS, confirm clients are not speaking plain HTTP to it. This mismatch is the most common post-change cause of 400 spikes.

5. For 408s, check the session shape. Pull scur, slim, rate, and req_rate per frontend. High scur with low req_rate and near-zero bin/bout is the Slowloris pattern: many connections, almost no completed requests, backends idle (qcur zero). Normal scur with a modest 408 rate is just slow clients or pre-connect noise.

6. For suspected Slowloris, inspect show sess. Look at the source IP distribution and session age. Thousands of connections from a narrow IP set (or a narrow set of /64s for IPv6) stuck in request receipt confirms it. Wide distribution with short lifetimes points at real clients on bad networks instead.

7. Check termination flags in logs. PR confirms proxy-side rejection (400/403). cR confirms timeout http-request firing. If you see mostly cR with 408, your timeout is doing its job; the question is whether the clients behind it are hostile or just slow.

Metrics and signals to monitor

Fixes

Oversized headers (400)

If show errors shows legitimate requests dying at the buffer boundary, your header payload exceeds tune.bufsize minus tune.maxrewrite. With the defaults (16384 and 1024 respectively), the effective request header ceiling is about 15 KB. Large cookies, JWTs, and accumulated X-Forwarded-* chains can cross it.

Raise tune.bufsize in the global section (for example to 32768) and reload. Tradeoffs: buffer memory is allocated per connection, two buffers each, so doubling bufsize roughly doubles per-connection buffer memory. On a high-maxconn frontend that is a real capacity decision, not a free knob. Note that HTTP/2 requires tune.bufsize of 16384 or more.

PROXY protocol or SSL termination mismatch (400)

Fix the contract, not the symptom. Either the upstream sends PROXY protocol and every frontend bind line has accept-proxy, or it does not and none of them do. Mixed states produce exactly this 400 spike. Same for TLS: if clients speak plaintext, route them to a plaintext frontend or redirect them. This class of bug is why ereq spikes deserve an automatic “what changed” check.

HTTP/2 framing errors (400)

If the 400s started when you enabled alpn h2, and show errors shows H2 preface or frame violations, check that the clients triggering it actually speak HTTP/2 and that intermediaries between the client and HAProxy are not mangling frames. Some of these are version-specific parser behaviors; if the spike started at an upgrade, check the HAProxy issue tracker for regressions in your exact version before assuming client fault.

Request smuggling probes (400)

Internet-facing frontends will see a steady drip of these. A patched HAProxy returns 400 and logs the attempt, which is the correct behavior. What matters operationally: keep HAProxy current. CVE-2023-25725 (HTTP request smuggling via HTX-aware versions) was fixed across the 2.8, 2.7, 2.6, 2.5, 2.4, 2.2, and 2.0 maintenance branches, and similar parsing CVEs keep appearing. If you are on an unpatched version and seeing 400 bursts with smuggling-shaped buffers in show errors, treat the upgrade as the fix, not the logging.

Slowloris and slow clients (408)

timeout http-request is your primary defense: it caps how long HAProxy waits for a complete request. If you do not set it, timeout client applies between request chunks, which is usually far more generous than you want against slow-header attacks. Set timeout http-request tight: 5 to 10 seconds is common practice for frontends that serve fast APIs; loosen it if you legitimately serve slow clients on bad networks.

If the attack persists, add per-source-IP rate limiting with a stick table (conn_rate or http_req_rate tracking) so a concentrated source burns itself out instead of your maxconn. Watch scur/slim while you tune: the goal is that attack connections die at the request timeout before they dent your session budget. For the saturation side of this, see HAProxy scur approaching slim.

Pre-connect and probe noise (408 / 400)

Browsers open speculative TCP connections and sometimes never use them; timeout http-request fires and you get a 408 nobody noticed except your graphs. The documented workaround is errorfile 408 /dev/null, which silently closes the connection instead of sending a response. Similarly, option http-ignore-probes suppresses the 400/408 response, the log line, and the error counter when a connection closes without sending anything.

Both have a real cost: you are throwing away signal. The manual warns http-ignore-probes should not be used on internet-facing frontends because scans and malicious activity stop being logged. Prefer these only on internal frontends or health-check-heavy paths where the noise has a known, benign source, and keep the counters visible everywhere else.

Prevention

• Baseline your 4xx composition. Know your normal 400/408 rate per frontend so “2x baseline” means something. Internet-facing frontends always have noise; internal frontends should be near zero, and any ereq there is a bug.
• Alert on ereq rate, not just hrsp_4xx. Parse failures are the early, specific signal; the 4xx aggregate lags and mixes in benign 404s.
• Correlate alerts with deploy events. Pipe config reload and version change timestamps into your dashboards. Most 400 spikes explain themselves in one look.
• Set timeout http-request deliberately on every HTTP frontend, sized to your real client population, before an attacker sizes it for you.
• Track HAProxy security releases. Request-smuggling and framing fixes land regularly; the 400 counter is partly a function of how current your parser is.
• Load-test header limits after any change that grows cookies or tokens, so you find the tune.bufsize ceiling in staging instead of in show errors.

How Netdata helps

• Netdata collects the HAProxy stats CSV continuously, so hrsp_4xx, ereq, and dreq are per-second time series rather than snapshots you have to poll by hand during an incident.
• Plotting ereq against hrsp_4xx on the same dashboard separates HAProxy-generated 400s from backend-returned 4xx without log diving as a first step.
• The Slowloris signature (scur climbing, req_rate flat, bin/bout near zero, qcur zero) is a correlation pattern across four charts, and having them on one screen makes it a 30-second call instead of a guessing game.
• Counter resets on reload are handled, so rate charts stay honest across the config change that probably caused the spike in the first place.
• Anomaly detection on ereq and hrsp_4xx catches the slow-build cases, like a gradual header-size creep toward the buffer limit, before users hit it.

Related guides

• HAProxy 502 Bad Gateway: backend connection and response failures
• HAProxy 503 Service Unavailable: no server is available to handle this request
• HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade
• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy counter resets on reload: phantom drops and spikes in your metrics
• HAProxy backend queue building (qcur): requests waiting for a free server slot