$ guides / varnish / varnish-losthdr ▌

Operations Guides

Varnish losthdr: HTTP headers silently dropped past http_max_hdr

MAIN.losthdr increments whenever Varnish drops an HTTP header during request or response processing because there is no room for it — the header count exceeds the http_max_hdr limit (default 64) or the workspace runs out while adding it. Each increment is accompanied by a LostHeader log record. The request appears to succeed, but the header is gone, and Varnish does not reveal which one without querying the shared memory log.

The consequences depend on which header was lost. A dropped Vary header causes cache poisoning: the wrong content variant gets cached and served to subsequent users for that URL. A dropped Authorization header means the backend never sees credentials, causing authentication failures or, depending on backend behavior, authentication bypass. A dropped Cache-Control header changes how the response is cached. None produce a Varnish error. They produce incorrect behavior that is difficult to trace unless you know this counter exists.

This problem typically appears after a deployment. A new analytics SDK adds tracking headers. A proxy layer adds X-Forwarded-For and X-Forwarded-Proto. A CDN adds forwarding metadata. Each addition brings the header count closer to the limit, and when it crosses, headers vanish with no warning beyond MAIN.losthdr.

What this means

Varnish enforces a maximum number of HTTP headers per request and per response using the http_max_hdr parameter. The default is 64 header lines; the allowed range is 32 to 65535.

Varnish reserves five header slots for the request or status line (the first line “occupies five header lines” per the parameter documentation), leaving 59 usable slots at the default setting.

Operators who count headers naively will hit the limit sooner than expected.

When the header count exceeds http_max_hdr, the excess header is not stored. Varnish logs a LostHeader event in the shared memory log and increments MAIN.losthdr. The behavior depends on where the overflow occurs and the Varnish version:

Client request with too many headers: Varnish rejects the request with HTTP 400, logging a BogoHeader: Too many headers record. MAIN.client_req_400 increments — not MAIN.losthdr.
Backend response with too many headers: the fetch fails (FetchError: http format error) and the client receives a 503. MAIN.fetch_failed increments — not MAIN.losthdr.
Headers added during VCL processing: If VCL or a VMOD adds headers (in vcl_recv, vcl_deliver, or vcl_backend_response) that push the total past the limit, the excess header is dropped, LostHeader is logged, and MAIN.losthdr increments. On the client side this also marks the request workspace as overflowed, so HTTP/1 delivery ends with a 500 (workspace_client overflow) and MAIN.client_resp_500 increments. A header lost on the backend-response side does not fail the request: the response can still be delivered and cached, missing that header.

The third case is the most dangerous: no error is visible to the client or in status code distribution. The request succeeds, the response is delivered, but a header is missing. If that header controls caching, authentication, or content negotiation, the application behaves incorrectly with no obvious error signal.

flowchart TD
    A["Request or response
arrives at Varnish"] --> B{"Header count
exceeds http_max_hdr?"}
    B -->|No| C["Normal processing"]
    B -->|Yes| D["Excess header
silently dropped"]
    D --> E["MAIN.losthdr increments
LostHeader logged"]
    D --> F{"Which header
was lost?"}
    F -->|Vary| G["Cache poisoning:
wrong variant served"]
    F -->|Authorization| H["Auth failure
or auth bypass"]
    F -->|Cache-Control| I["Caching behavior
changes silently"]
    F -->|Set-Cookie| J["Session state
lost or misrouted"]

Common causes

Cause	What it looks like	First thing to check
Deploy added tracking or analytics headers	MAIN.losthdr starts incrementing immediately after a release	Deployment history for new header additions
Proxy chain adding forwarding headers	X-Forwarded-For, X-Forwarded-Proto, Via headers accumulating at each hop	Count headers per request with varnishlog
VCL adding synthetic headers	Headers added in vcl_deliver or vcl_backend_response push past the limit	Audit VCL for `set req.http.` and `set beresp.http.` statements
Large Content-Security-Policy split across multiple header fields	Single logical header occupies multiple http_max_hdr slots	Inspect request headers with varnishlog -i ReqHeader
Backend returning excessive headers	MAIN.losthdr increments on backend responses, not client requests	Check backend response headers with varnishlog -i BerespHeader

Quick checks

# Check if headers are being dropped (should be 0)
varnishstat -1 -f MAIN.losthdr

# Check current http_max_hdr setting
varnishadm param.show http_max_hdr

# Identify which headers are being lost (run during active traffic)
varnishlog -q 'LostHeader' -g request

# Check for related workspace overflows
varnishstat -1 -f 'MAIN.ws_*_overflow'

# Check downstream impact: 4xx and 5xx errors
varnishstat -1 -f 'MAIN.client_req_4*' -f MAIN.client_resp_500

# Sample header count per request to find offending requests
varnishlog -i ReqHeader -g request

The last command prints every ReqHeader line grouped by request transaction. Count lines per request group to find requests approaching the limit.

How to diagnose it

Confirm MAIN.losthdr is incrementing. Run varnishstat -1 -f MAIN.losthdr twice, a few seconds apart. If the value increases, headers are actively being dropped. losthdr is cumulative since Varnish start; a nonzero value with long uptime may be stale. Only an increasing rate indicates an active problem.
Check the current http_max_hdr value. Run varnishadm param.show http_max_hdr. The default is 64, but it may have been changed. Subtract the internal reserved slots to get the actual usable header count.
Identify which header is being dropped. Run varnishlog -q 'LostHeader' -g request during active traffic. This prints the name of each dropped header. The header name tells you the consequence: a lost Vary means cache poisoning risk, a lost Authorization means auth problems, a lost Set-Cookie means session breakage.
Determine whether the overflow is client-side or backend-side. If LostHeader entries appear in request context (ReqHeader tags nearby), the client is sending too many headers. If they appear in backend fetch context (BerespHeader tags nearby), the backend response has too many headers. This determines where you apply the fix.
Count headers on a representative request. Use varnishlog -i ReqHeader -g request and count the lines per request transaction. Compare to your http_max_hdr minus the internal reserved slots. If requests are near the limit, you have almost no headroom and the next deployment will push them over.
Check for workspace overflow as a related symptom. Run varnishstat -1 -f 'MAIN.ws_*_overflow'. Workspace overflow and header overflow are related: both involve per-request memory limits. If both are incrementing, you have requests that are large in both header count and header byte size, and you need to address both constraints.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
MAIN.losthdr	Direct indicator of dropped headers	Any nonzero value
MAIN.ws_client_overflow	Related constraint: client workspace exhausted	Any nonzero value
MAIN.ws_backend_overflow	Related constraint: backend workspace exhausted	Any nonzero value
MAIN.client_req_400	Client requests rejected for too many headers	Spike correlating with losthdr
MAIN.client_resp_500	Delivery failures, some from workspace pressure	Spike correlating with losthdr
MAIN.cache_hit / MAIN.cache_miss	Cache poisoning from lost Vary headers shifts hit/miss pattern	Unexpected hit ratio change alongside losthdr
MAIN.fetch_failed	Backend fetch failures from too many response headers	Correlates with backend-side losthdr

Fixes

Increase http_max_hdr

The immediate fix is to raise the limit.

# Increase at runtime (no restart required, but does not persist across restarts)
varnishadm param.set http_max_hdr 128

Choose a value with comfortable headroom. If your peak request has 62 headers, set it to 128 rather than 70. Each additional header slot consumes workspace memory per connection, so avoid setting it to 65535 without reason.

To make the change persistent, add the parameter to your varnishd startup arguments:

-p http_max_hdr=128

Add this to your varnishd command line or systemd unit, depending on your distribution. Without this step, the runtime change is lost on the next restart.

Reduce the header count

If the header count is inflated by headers Varnish does not need, strip them in VCL before they accumulate. Common candidates include tracking headers that only the application uses internally, redundant X-Forwarded-* headers from multiple proxy layers, and debug headers left from troubleshooting.

sub vcl_recv {
    unset req.http.X-Internal-Debug;
    unset req.http.X-Tracking-Id;
}

Removing headers that the backend expects will break functionality. Only strip headers you have verified are unused downstream.

Address the backend response

If the overflow is on the backend response side, the fix targets the backend headers. Options:

Increase http_max_hdr as described above.
Strip unnecessary response headers in vcl_backend_response.
Fix the backend to send fewer headers.

Note that a response whose header count exceeds the limit at parse time never reaches VCL: that fetch fails before any stripping could apply (see the second case above). Stripping in vcl_backend_response helps only when individual headers are large (workspace pressure) or when your own VCL additions are pushing the count over the limit.

Option 2 example:

sub vcl_backend_response {
    unset beresp.http.X-Internal-Metadata;
}

If the backend is a third-party service you cannot modify, option 1 is your path.

Prevention

Alert on any nonzero MAIN.losthdr. Zero is the only acceptable steady-state value. Any increment means a header was dropped and behavior may be incorrect.
Audit header count after every deployment. Add a step to your deployment pipeline that counts request and response headers through Varnish. A new SDK or middleware can silently add 5 to 10 headers.
Monitor workspace overflow counters alongside losthdr. Both indicate per-request memory pressure. If both are incrementing, the problem is compounded and a simple http_max_hdr increase may not be enough.
Track VCL-added header count. If your VCL adds headers in vcl_deliver or vcl_backend_response, count them and compare to your remaining headroom under http_max_hdr.
Document http_max_hdr in your configuration baseline. Record the current value and the reasoning. When a new deployment pushes header counts higher, the baseline gives you a reference point for deciding whether to increase the limit or strip headers.
Test with realistic header counts. During pre-production validation, send requests with the full set of headers your production stack adds, not a minimal subset.

How Netdata helps

Netdata collects MAIN.losthdr at per-second resolution. A spike is visible immediately, not minutes later when a downstream symptom like cache poisoning or auth failures appears.
Correlate losthdr with workspace overflow counters (ws_client_overflow, ws_backend_overflow) to distinguish header count exhaustion from byte-size workspace exhaustion. Both produce similar symptoms but need different fixes.
Correlate losthdr with client_req_400 and client_resp_500 to measure downstream impact. If losthdr is incrementing but 400/500 rates are flat, you are in the silent-drop path where headers are lost on requests that still return 200.
Correlate losthdr with cache hit and cache miss rates. A lost Vary header causes cache poisoning, which may manifest as a sudden hit ratio shift (more hits because all users receive the same variant) or an increase in user complaints about wrong content being served.