HAProxy tune.bufsize and 400 errors: buffers too small for large headers

Your frontend 400 rate just spiked. The backends insist they never saw the requests, and they are telling the truth: HAProxy generated these 400s itself, during header parsing, before any backend was selected. The usual trigger is a request header block that no longer fits in the per-connection buffer, and the usual suspects are large cookies, JWT bearer tokens, or a newly added header that pushed total header size past the limit.

The mechanism: every connection gets two buffers of tune.bufsize bytes each (default 16384), one for the request, one for the response. HAProxy must hold the complete request line plus all headers in the request buffer before it can evaluate ACLs and route. If the request line plus headers exceed the usable buffer space, HAProxy cannot parse the request, so it rejects it with its own 400 Bad Request. The request never reaches a backend.

The trap is the fix. Raising tune.bufsize multiplies per-connection memory: two buffers per connection, times every connection up to maxconn. Double the buffer and you roughly double the memory HAProxy needs at full connection load. Operators who bump bufsize to silence 400s without doing this arithmetic sometimes trade a client-visible error for an OOM kill weeks later.

What this means

The default tune.bufsize is 16384 bytes. Not all of it is usable for headers: tune.maxrewrite (default 1024) is reserved headroom so HAProxy can rewrite or add headers (think X-Forwarded-For) without overflowing the buffer. The effective ceiling for the request line plus headers is approximately tune.bufsize - tune.maxrewrite, roughly 15 KB at defaults.

When a request exceeds that ceiling:

• HAProxy aborts the request during parsing and returns its own 400. The termination state in the log is PR-- (proxy abort during header parsing), which distinguishes it cleanly from a backend-returned 400.
• The frontend ereq counter increments.
• No backend counter moves. req_tot on the backend, per-server hrsp_*, econ, and rtime are all unaffected because the request was never dispatched.

The symmetric case on the response side is a backend whose response headers exceed the buffer. That produces a 502, not a 400, and shows up as eresp on the backend. This article covers the request side; for the response side see HAProxy 502 Bad Gateway.

Two version behaviors matter. On HAProxy 3.1 and later, the mux returns more specific codes instead of a generic 400: 414 (URI Too Long) when the request line itself is too large, and 431 (Request Header Fields Too Large) when the header block overflows. If you upgraded recently and your 400 alerts went quiet while client complaints continued, check whether the errors reclassified into 414/431, which still land in hrsp_4xx. A bug in the early implementation reported incorrect status codes in some cases; it was fixed in 3.2.8. Also note that a custom errorfile 400 will no longer fire for these rejections on 3.1+, because the status is no longer 400.

flowchart LR
  C[Client request] --> P{Headers fit in
bufsize - maxrewrite?}
  P -- yes --> R[Parse OK, ACLs,
backend selection]
  R --> B[Backend server]
  P -- no --> X[HAProxy generates 400
or 414/431 on 3.1+]
  X --> E[ereq increments,
log shows PR--]
  B --> N[backend counters,
rtime, hrsp_*]

Common causes

Not every PR-- is bufsize. Malformed requests, PROXY protocol misconfiguration, and plaintext-to-TLS mismatches also increment ereq. The distinguishing test is whether the rejected requests are well-formed but large, which is exactly what show errors reveals.

Quick checks

# Check ereq on frontends - this is the counter that moves for buffer overflows
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "FRONTEND" {print $1": ereq="$13}'

# Check frontend 4xx (CSV field 43 is hrsp_4xx)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "FRONTEND" {print $1"/"$2": 4xx="$43}'

# Inspect captured error samples - shows the exact bytes that caused rejection
echo "show errors" | socat unix-connect:/var/run/haproxy.sock stdio

# Confirm current buffer configuration in the running config
grep -E 'tune\.(bufsize|maxrewrite)' /etc/haproxy/haproxy.cfg

# Global connection count for the memory arithmetic later
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep -E "^(CurrConns|Maxconn|PoolFailed):"

All of these are read-only. show errors is the highest-value check: it dumps captured samples of rejected requests, and for a buffer overflow you will see a well-formed request whose header block runs right up to the buffer limit. If instead you see garbled bytes or a TLS ClientHello on a plaintext listener, your problem is a protocol mismatch, not bufsize.

In the logs, grep for the termination state:

# Find proxy-abort terminations during request parsing
grep ' PR-- ' /var/log/haproxy.log | tail -20

The exact log field position depends on your log-format, but PR-- is the signature: the proxy aborted the session while parsing request headers. A backend 400 shows a normal termination state with the 400 status and a server name; a bufsize 400 shows PR-- with no backend server involved.

How to diagnose it

1. Confirm the 400s are HAProxy-generated, not backend-generated. Compare frontend hrsp_4xx and ereq movement against backend hrsp_4xx. If frontend 4xx rises in lockstep with ereq while backend 4xx is flat, HAProxy is the source. The same frontend-minus-backend logic used for 5xx applies here; see HAProxy 5xx delta for the general technique.

2. Correlate with a change. A bufsize overflow almost never appears spontaneously on a static system. Look for an auth rollout (larger JWTs, new cookie attributes), a new upstream hop injecting headers, a client release, or a WAF/CDN change.

3. Capture the offending request with show errors. Confirm the request is well-formed and that the header block is large. Estimate its size: if the headers approach 15 KB, you are at the bufsize - maxrewrite ceiling.

4. Check the termination state and status code in logs. PR-- confirms the abort happened during header parsing. On 3.1+, expect 414 for oversized request lines and 431 for oversized header blocks; on earlier versions, 400 for both.

5. Rule out the lookalikes. PROXY protocol mismatch and plaintext-to-TLS mismatch also produce ereq, but those requests fail immediately at the first bytes and affect every request from the misconfigured source, not just large ones. If only requests with big headers fail while small requests from the same clients succeed, it is bufsize.

6. Quantify the memory cost before fixing. Note CurrConns peak and Maxconn. Each established connection can hold 2 x tune.bufsize of buffer memory, 32 KB at defaults, before connection and stream metadata on top. Compute what your planned bufsize does to that number at full maxconn.

Metrics and signals to monitor

Fixes

Reduce header size at the source

This is the right fix when the headers grew for accidental reasons: duplicated cookie attributes, an oversized JWT crammed with claims, tracing headers stacked by three proxies in a row. Work with the application or auth team to shrink tokens and prune cookies. Deleting stale cookies server-side (Set-Cookie with an expired date) and moving claims out of the token both reduce header size without touching HAProxy. No memory trade-off, no reload risk.

Raise tune.bufsize (with the memory arithmetic done first)

When large headers are legitimate and unavoidable, raise the buffer in the global section:

global
    tune.bufsize 32768

Rules and caveats:

• Do the memory math. Memory for connection buffers is approximately 2 x bufsize x maxconn. At maxconn 50000, the default 16 KB buffer means about 1.6 GB of buffer memory at full load. Doubling to 32 KB means about 3.2 GB. The official documentation warns that increasing bufsize can cause the system to run out of memory and recommends decreasing maxconn by the same factor you increase bufsize. Treat that as a hard requirement, not a suggestion.
• maxrewrite scales with it. tune.maxrewrite is readjusted to half of bufsize if set larger than that. The recommended value is around 1024; the usable header space is bufsize - maxrewrite.
• Bufsize does nothing for request bodies. Bodies are streamed, not buffered. If you raised bufsize to fix large POST bodies and nothing changed, that is why.
• TLS edge case. TLS 1.2/1.3 plaintext records can be up to 16384 bytes, and with framing overhead plus maxrewrite, a default 16384 buffer is marginally too small for TLS clients that send full-size records. If you see intermittent PR-- errors on TLS frontends with modest headers, a bufsize of at least 18304 covers this case.
• Reload to apply. tune.bufsize is a global directive and requires a reload (SIGUSR2 or systemctl reload haproxy). Counters reset on reload. Raise the value in small steps and watch memory, not in one jump from 16 KB to 256 KB.

On HAProxy 3.4: consider tiered buffers

HAProxy 3.4 introduced tune.bufsize.large and tune.bufsize.small, letting you keep the default buffer for most traffic while allocating larger buffers only where big header fetches, retries, or body waits need them. This avoids paying the 2 x bufsize x maxconn penalty uniformly across every connection. If you are on 3.4 and header size is a chronic problem, this is the architecturally cleaner answer than inflating the global buffer.

Update error pages and alerting for 3.1+

If you customized errorfile 400, replicate it for 414 and 431 or buffer-overflow clients will get the default page. Update any alert or dashboard that counts 400 specifically: include 414 and 431 so the signal does not silently reclassify away after an upgrade.

Prevention

• Alert on ereq rate per frontend. Any sustained nonzero rate on an internal frontend, or a spike above noise baseline on a public one, deserves a ticket. Correlate with deploy and config-change events before anything else.
• Budget header size like you budget payload size. Document a maximum expected header block (request line plus headers) for your applications, and keep it comfortably under bufsize - maxrewrite. When a change adds headers, check the budget.
• Load-test with real headers. Synthetic tests with minimal headers will never catch this. Replay production-shaped requests, including cookies and auth tokens, in staging.
• Watch memory after any bufsize change. Track process RSS and PoolUsed_MB through a full traffic peak. Check PoolFailed stays at zero.
• Pin alerting to the whole 4xx family. On 3.1+, the same failure presents as 414 or 431. Alerts keyed only on 400 will miss it.

How Netdata helps

• Frontend ereq and hrsp_4xx per second, so a buffer-overflow spike is visible within seconds of the header change that caused it, not after client complaints arrive.
• Frontend versus backend 4xx/5xx split, which makes it immediately obvious the errors are generated by HAProxy itself and never reached a backend server.
• Correlation against CurrConns and memory, so you can see what a tune.bufsize increase actually did to per-connection memory at peak load before the OOM killer does it for you.
• PoolFailed and pool memory tracking, confirming whether buffer allocation is keeping up after a bufsize change.
• Historical baselines, letting you line the 400 spike up against the deploy, auth, or CDN change that introduced the large headers.

Related guides

• HAProxy 502 Bad Gateway: backend connection and response failures
• HAProxy 503 Service Unavailable: no server is available to handle this request
• HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade
• HAProxy backend losing servers: active server count and cascade risk
• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy compression overhead: comp_byp, CPU cost, and when compression backfires
• HAProxy connect time (ctime) high: network latency and backend accept-queue overflow
• HAProxy connection errors (econ): backend connections refused, timed out, or reset
• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy 5xx delta: telling HAProxy-generated errors from backend errors
• HAProxy health checks green but the application is broken: when UP does not mean healthy
• HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable