A spike in MAIN.client_req_400 means Varnish is rejecting an unusual volume of client requests at the HTTP parsing layer, before VCL runs. A low steady rate of 400s is normal. Bots, scanners, and sloppy clients are constant background radiation on any public-facing proxy. What matters is the delta from your baseline.

The threshold for concern is a sustained rate spike greater than 5x your established baseline. At that volume, the causes narrow to: active scanning or fuzzing, a broken client SDK pushing malformed requests, request-smuggling attempts probing parser discrepancies, or a configuration change that made Varnish reject traffic it previously tolerated.

Varnish is deliberately stricter than most proxies. It rejects ambiguous HTTP framing that nginx, Apache, or HAProxy might silently accept. This prevents request smuggling and protocol desync attacks, but it also means a 400 spike at the Varnish layer does not prove the requests are malicious. A client that worked fine through a more permissive proxy can start receiving 400s when routed through Varnish.

What this means

MAIN.client_req_400 counts every client request Varnish rejected with HTTP 400. The counter increments before VCL processing begins. These requests never reach vcl_recv. Varnish parses the request line and headers, and if anything violates its HTTP grammar, it returns 400 and discards the transaction.

The related counter MAIN.sc_rx_bad tracks sessions closed because of bad data from the client. The two counters are related but not identical. client_req_400 counts rejected requests. sc_rx_bad counts sessions terminated at the transport level because the client sent something Varnish could not parse as HTTP at all. Both should sit near zero in steady state.

The built-in VCL adds another source of 400s: HTTP/1.1 requests without a Host header get synth(400) from the default VCL. This is RFC enforcement, not a parse error, and is the most common cause of 400s from legitimate-but-misconfigured clients.

Since Varnish 7.7, the client connection is always closed after a malformed request is rejected. Earlier versions sent the 400 response but left the connection open, creating a client-side desync vector. On pre-7.7 versions, a 400 spike may be the visible symptom of an active smuggling probe exploiting that connection behavior.

Modern Varnish rejects requests carrying both Content-Length and Transfer-Encoding headers. This is a smuggling defense. Older versions may not reject this combination.

The varnishlog tags that reveal the specific parse failure are BogoHeader (header-level violations) and HttpGarbage (request-line garbage such as null bytes).

flowchart TD
    A["client_req_400 spike"] --> B["Pull varnishlog parse error"]
    B --> C{"Error category?"}
    C -->|"Header too long"| D["Check http_req_hdr_len"]
    C -->|"Too many headers"| E["Check http_max_hdr"]
    C -->|"Control chars / null bytes"| F["Scanning or fuzzing"]
    C -->|"Missing Host header"| G["Client or LB misconfiguration"]
    C -->|"Both CL and TE headers"| H["Smuggling attempt"]
    D --> I{"Legitimate client?"}
    E --> I
    G --> I
    I -->|"Yes"| J["Adjust parameter or fix client"]
    I -->|"No"| K["Rate-limit or block source"]
    F --> K
    H --> L["Verify patched, add VCL hardening"]

Common causes

CauseWhat it looks likeFirst thing to check
Missing Host headerHTTP/1.1 requests without Host get synth(400) from built-in VCLvarnishlog for absent ReqHeader:Host
Header too largeBogoHeader “Header too long” in varnishloghttp_req_hdr_len parameter and client cookie/header sizes
Too many headersBogoHeader “Too many headers” in varnishloghttp_max_hdr parameter (default 64)
Control characters in headersBogoHeader “Header has ctrl char”Source IP distribution, likely fuzzing
Scanning or fuzzingMany source IPs, garbage methods, null bytes in request lineHttpGarbage tag in varnishlog, source IP concentration
Request smuggling probeBoth Content-Length and Transfer-Encoding on same requestvarnishlog filter for dual headers
Broken client SDKSingle source IP or user-agent, consistent malformed patternUser-agent and source IP correlation

Quick checks

# Check current 400 rate and session close rate
varnishstat -1 -f MAIN.client_req_400 -f MAIN.sc_rx_bad

# See the specific parse errors causing 400s
varnishlog -q 'BogoHeader or ReqParseError' -g request

# Source IPs of requests getting 400s
varnishlog -q 'RespStatus == 400' -g request -i ReqStart

# Look for smuggling attempts (both CL and TE on same request)
varnishlog -q 'ReqHeader:Content-Length and ReqHeader:Transfer-Encoding' -g request

# Count 400s by client IP over a short window
varnishncsa -F '%h %s' -q 'RespStatus == 400' | sort | uniq -c | sort -rn | head -20

# Compare 400 rate to total request rate for proportion
varnishstat -1 -f MAIN.client_req -f MAIN.client_req_400

# Check for header overflow (related but distinct signal)
varnishstat -1 -f MAIN.losthdr

# Verify your Varnish version for smuggling vulnerability context
varnishd -V 2>&1 | head -1

How to diagnose it

  1. Confirm the spike is real. Take two readings of MAIN.client_req_400 ten seconds apart. Compute the per-second rate. Compare to your rolling baseline. A 5-10x jump is the trigger threshold. A rate proportional to a general traffic spike is less concerning than one that is independent of total traffic volume.

  2. Pull the specific parse error. Run varnishlog -q 'BogoHeader or ReqParseError' -g request and read the error reason. The BogoHeader reason string tells you exactly what Varnish rejected: “Header too long” points to header size limits, “Too many headers” points to header count limits, “Header has ctrl char” points to binary garbage from fuzzing. HttpGarbage with a null byte in the method line is a signature of automated scanning.

  3. Correlate with source IPs. Run the varnishncsa count-by-IP command from the quick checks. If the 400s concentrate on one or a few IPs, you have a specific source. If they spread across many IPs with no concentration, the pattern is distributed scanning or a broadly deployed broken client.

  4. Check for smuggling patterns. Filter for requests carrying both Content-Length and Transfer-Encoding. Modern Varnish rejects these outright, so their presence in the log means Varnish caught the attempt. A high volume means someone is actively probing your edge.

  5. Verify no recent changes. Check varnishadm vcl.list for recent VCL reloads. Check whether anyone changed http_req_hdr_len, http_max_hdr, or related parameters via varnishadm param.show http_req_hdr_len and varnishadm param.show http_max_hdr. A parameter tightening can turn previously accepted traffic into 400s overnight.

  6. Compare with upstream logs if smuggling is suspected. Smuggled requests that exploit parser discrepancies may appear as normal requests in varnishlog because Varnish parsed them successfully, just differently from the backend. If you suspect smuggling, compare Varnish request logs with backend access logs for discrepancies in request count, order, or content.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
MAIN.client_req_400Primary counter for rejected requestsRate spike > 5x baseline
MAIN.sc_rx_badSessions closed for bad data, transport levelRate spike independent of client_req_400
MAIN.losthdrHeaders silently dropped past http_max_hdr limitAny nonzero rate, especially with caching bugs
MAIN.client_reqTotal request volume, for ratio context400s rising faster than total requests
Source IP distributionDistinguishes scanning from broken clientHigh concentration on few IPs or uniform spread
varnishd -V outputVersion context for smuggling vulnerabilitiesPre-7.7 without connection-close on malformed requests
MAIN.sess_droppedWhether 400 volume is causing thread pressureNonzero rate alongside 400 spike

Fixes

Scanning and fuzzing traffic

If varnishlog shows control characters, null bytes, or garbage HTTP methods spread across many source IPs, this is automated scanning. Varnish is already rejecting these requests correctly. The fix is not in Varnish. Block or rate-limit the source IPs at the firewall, WAF, or load balancer layer. Varnish is not the right layer for IP-based access control under high-volume scanning because every rejected request still consumes a worker thread for parsing.

Broken client SDK

If the 400s concentrate on a single source IP, ASN, or user-agent string, a specific client is sending malformed requests. Common patterns: missing Host header (often from a load balancer or health checker misconfigured to use HTTP/1.1 without setting Host), oversized cookies exceeding http_req_hdr_len, or a custom SDK sending headers Varnish rejects. Fix the client. If the client cannot be fixed immediately and the traffic is legitimate, you can raise the relevant parameter, but understand the tradeoff: larger http_req_hdr_len or http_max_hdr increases per-request workspace memory consumption.

Header size or count limits

If BogoHeader reports “Header too long” or “Too many headers” on legitimate traffic, you may need to raise http_req_hdr_len (maximum single header length) or http_max_hdr (maximum header count, default 64). Change via varnishadm param.set http_max_hdr 128 at runtime, or in your startup configuration for persistence. Each additional header slot costs workspace memory per request, so raise only what you need.

Missing Host header

The built-in VCL returns synth(400) for HTTP/1.1 requests without a Host header. This is RFC 7230 compliance. The most common source is a health checker or monitoring probe configured to send HTTP/1.1 requests without setting Host. Fix the probe configuration to include a Host header, or downgrade it to HTTP/1.0, which does not require Host. Do not remove the check from VCL. It exists for correctness.

Request smuggling attempts

If varnishlog shows requests with both Content-Length and Transfer-Encoding, Varnish is catching smuggling probes. Verify your version is patched against known smuggling vulnerabilities. Varnish has addressed multiple request-smuggling issues across versions, including CL+TE desync, chunked-encoding boundary parsing, and client-side desync from connection reuse after malformed requests.

For VCL-level hardening, you can explicitly reject ambiguous framing early in vcl_recv:

if (req.http.Content-Length && req.http.Transfer-Encoding) {
    return (synth(400));
}

This is redundant in modern Varnish, which already rejects this combination, but it provides defense in depth and makes the rejection explicit in your VCL.

A VCL workaround that rejects all Transfer-Encoding: chunked requests will also block legitimate chunked uploads. Prefer upgrading to a patched version over this workaround.

Prevention

  • Baseline and alert on client_req_400 rate. Alert on sustained rate greater than 5x baseline, not on absolute count. A static threshold is useless because baseline varies by traffic volume.
  • Monitor sc_rx_bad alongside client_req_400. The two counters catch different failure modes. A spike in one but not the other narrows the diagnosis immediately.
  • Keep Varnish patched. Smuggling vulnerabilities are discovered regularly. Running a supported release ensures you have the latest parser hardening.
  • Run varnishncsa with persistent logging. The shared memory log is ephemeral. Without persistent logs, you lose the source IP and parse error data you need during a 400 spike investigation.
  • Audit health checkers and monitoring probes. Ensure they send a Host header if using HTTP/1.1. Missing-Host 400s from health checkers are the most common false alarm.

How Netdata helps

During a 400 spike, Netdata provides the signals that separate a real incident from background noise:

  • Per-second client_req_400 and sc_rx_bad rates show the exact onset and duration of the spike, and whether the problem is request-level rejection or session-level garbage.
  • Correlation with client_req total rate reveals whether 400s are rising proportionally with traffic or independently.
  • losthdr tracking catches silent header drops that cause mysterious caching bugs before they become visible.
  • Anomaly detection on counter rates flags a 400 spike as anomalous even when no static threshold would fire.
  • sess_dropped correlation shows whether 400 volume is consuming enough worker threads to cause collateral session drops, turning a security nuisance into an availability incident.