$ guides / varnish / varnish-workspace-client-overflow ▌

Operations Guides

Varnish workspace_client overflow: 500 errors from oversized headers and cookies

HTTP 500 responses from Varnish when the backend is healthy, the object is in cache, the thread pool is not saturated, and the 503 backend-fetch path is not involved. The 500s come from Varnish itself, and they cluster on requests carrying large Cookie headers, long URLs, deep Via or X-Forwarded-For chains, or responses where VCL has added many synthetic headers.

The root cause is per-request workspace exhaustion. Varnish allocates a fixed block of memory (workspace_client) for each client request to hold the parsed HTTP object, headers, and VCL string operations. When request headers plus any data VCL pushes into that workspace exceed the allocation, the request fails with a 500.

The ws_client_overflow counter (Varnish 6.2+) is the definitive signal. On older releases, rely on client_resp_500 and losthdr. The fix is either to raise workspace_client or to reduce the size of the headers and cookies arriving at Varnish. The first option works but multiplies across every worker thread, so the second is usually the better long-term fix.

What this means

workspace_client is a per-request memory budget. The default is 64 kB on Varnish 6.x (64-bit) and 96 kB on Varnish 7.0+ (64-bit).

That budget holds:

the parsed HTTP request, including every header line
the Cookie header, often the largest single contributor
strings VCL builds during vcl_recv, vcl_hash, and vcl_deliver (for example regsuball results, synthetic headers, cookie manipulation)
ESI scratch space in some configurations

When the sum exceeds the allocation, the workspace allocator refuses the request and Varnish records the overflow. On Varnish 6.2+ the counter MAIN.ws_client_overflow increments directly. MAIN.client_resp_500 is a broader counter: it tracks all 500 responses delivered to clients, not just workspace-caused ones. MAIN.losthdr tracks a related but distinct condition: headers dropped because they exceeded http_max_hdr (default 64 headers), which also consumes workspace slots.

flowchart TD
  A[Client request with large Cookie/headers] --> B[Request parsed into workspace_client]
  B --> C{Workspace budget exceeded?}
  C -- no --> D[Normal request processing]
  C -- yes --> E[ws_client_overflow increments]
  E --> F[client_resp_500 increments]
  F --> G[500 returned to client]
  B --> H[VCL adds regsuball/synthetic headers]
  H --> C

The failure is per-request and stateless. A single abusive bot, a broken mobile SDK that appends tracking cookies without bound, or an application deploy that adds a new large cookie can trigger sustained 500s on a narrow slice of traffic while everything else looks fine.

Common causes

Cause	What it looks like	First thing to check
Large `Cookie` header	500s concentrated on authenticated or personalized paths; `ws_client_overflow` increments with request rate	`varnishlog -g request -q 'RespStatus == 500' -i ReqHeader` and inspect `Cookie:` length
VCL adding too much data	500s appear after a VCL reload; `regsuball` or synthetic header logic in `vcl_recv` or `vcl_deliver`	`varnishadm vcl.list` for recent reload; review VCL for `regsuball`, `set req.http.*`, `synthetic`
Deep proxy chain	`X-Forwarded-For` or `Via` grows at each hop; 500s correlate with traffic from upstream proxies	`varnishlog -i ReqHeader:X-Forwarded-For` and measure header length
`http_max_hdr` exceeded	`losthdr` increments alongside or instead of `ws_client_overflow`; headers silently dropped	`varnishstat -1 -f MAIN.losthdr` and `varnishadm param.show http_max_hdr`
Long URLs	500s on specific endpoints with long query strings; `req.url` dominates workspace	`varnishlog -i ReqURL` for affected transactions
Undersized `workspace_client` after tuning	500s after increasing `thread_pool_max` without adjusting workspace	`varnishadm param.show workspace_client` and `thread_pool_max`

Quick checks

These commands are read-only and safe to run during an incident.

# Check workspace overflow counters (Varnish 6.2+)
varnishstat -1 -f 'MAIN.ws_*_overflow' -f MAIN.client_resp_500 -f MAIN.losthdr

# Current workspace_client and thread pool configuration
varnishadm param.show workspace_client
varnishadm param.show workspace_backend
varnishadm param.show thread_pool_max
varnishadm param.show http_max_hdr

# Find the failing transactions
varnishlog -g request -q 'RespStatus == 500' -i ReqURL -i ReqHeader -i RespStatus -i Debug

# Search logs for the workspace overflow message directly
varnishlog -q 'Debug ~ "workspace overflow"' -g request

# Confirm backend health is not the cause (500 here is Varnish-side, not backend)
varnishadm backend.list

# Identify the largest Cookie headers in live traffic
varnishlog -i ReqHeader:Cookie -g request | awk '{print length($0), $0}' | sort -rn | head -20

# Check loaded VCL versions and recent reload timestamps
varnishadm vcl.list

The combination that confirms workspace overflow: ws_client_overflow (or client_resp_500 on older releases) is incrementing, varnishlog shows workspace_client overflow in the Debug tag followed by RespStatus 500, and the affected requests carry visibly large Cookie or other headers.

How to diagnose it

Confirm the counter is moving. Run varnishstat -1 -f 'MAIN.ws_*_overflow' -f MAIN.client_resp_500 twice, a few seconds apart, and compute the delta. A nonzero rate means active client-facing failures.
Capture the failing transactions. Run varnishlog -g request -q 'RespStatus == 500' during the incident. Look for the Debug tag containing workspace_client overflow, then inspect ReqHeader:Cookie, ReqURL, and any ReqHeader lines to find the oversized contributor.
Measure the actual header footprint. For the affected requests, sum the byte length of all request headers plus the URL. Compare against the configured workspace_client value. If the headers alone consume most of the budget, the overflow is data-driven, not VCL-driven.
Rule out VCL as the amplifier. If the failing requests do not have unusually large headers, inspect VCL for regsuball, set req.http.* operations that append data, and any VMOD calls that allocate workspace (cookie manipulation VMODs are common offenders). A VCL reload that correlates with the start of the 500s is a strong signal.
Check whether losthdr is also incrementing. If losthdr is nonzero alongside ws_client_overflow, the request has too many distinct headers (over http_max_hdr, default 64) in addition to or instead of raw size. The fix path differs: raise http_max_hdr rather than workspace_client.
Verify the version-specific defaults. On Varnish 6.x the default workspace_client is 64 kB. On 7.0+ it is 96 kB. If you recently upgraded Varnish but not the explicit parameter, or if you migrated to a host with more threads, the effective memory pressure changed even if the workspace value did not.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`MAIN.ws_client_overflow`	Direct count of client workspace allocation failures (V6.2+)	Any sustained nonzero rate
`MAIN.client_resp_500`	All 500 responses delivered to clients; broader than workspace-only	Nonzero rate with healthy backends and cache hits
`MAIN.losthdr`	Headers dropped because `http_max_hdr` was exceeded	Any nonzero value; indicates too many distinct headers, not just raw size
`MAIN.ws_backend_overflow`	Same failure on the backend response side; response headers too large for `workspace_backend`	Nonzero rate; fix is `workspace_backend`, not `workspace_client`
Process RSS vs configured storage	Confirms workspace increases did not push Varnish toward OOM	RSS growing after raising `workspace_client` or `thread_pool_max`
Cookie header size distribution	Leading indicator before overflow triggers	P99 cookie length approaching `workspace_client` budget

Fixes

Raise workspace_client

The immediate remediation is to increase workspace_client:

# Runtime change (persists until restart; add to startup params for permanence)
varnishadm param.set workspace_client 128k

Note: workspace_client has the DELAYED_EFFECT flag — it can be changed at runtime but the new value takes effect for newly created connections, not existing ones.

This works, but it multiplies. workspace_client is allocated per worker thread. The real memory cost is workspace_client multiplied by thread_pool_max multiplied by thread_pools. With thread_pool_max at its default of 5000 per pool and 2 pools, raising workspace_client from 96 kB to 128 kB adds roughly (128 - 96) * 1024 * 5000 * 2, approximately 320 MB, to steady-state memory consumption. On a host where Varnish storage is already sized close to available RAM, this can push the process toward OOM.

Before applying a larger workspace value in production, check the arithmetic against actual memory headroom:

varnishadm param.show workspace_client
varnishadm param.show thread_pool_max
varnishadm param.show thread_pools

Verify the change resolved the overflow by watching ws_client_overflow and client_resp_500 drop to zero rate.

The more durable fix is to stop the oversized data from reaching Varnish’s workspace:

Strip cookies that Varnish does not need for caching or forwarding. In vcl_recv, unset analytics, A/B testing, and feature-flag cookies before the request enters the hash and deliver phases. This directly shrinks the workspace footprint.
Cap or normalize X-Forwarded-For. In deep proxy chains, each hop appends an IP. If Varnish is several hops in, the header can dominate workspace. Normalize it to the client IP plus one trusted proxy.
Review regsuball and synthetic header logic. Each set req.http.X = regsuball(...) allocates workspace for the result. If VCL builds large strings (for example, reconstructing a cookie jar or assembling a forwarding header), that allocation counts against the budget. Prefer unset over rewrite where possible.
Shorten URLs. The URL is already in workspace by the time vcl_recv runs, so early return(pass) does not reduce its footprint. If a specific endpoint accepts arbitrarily long query strings, work with the application to move long parameters into the request body.

Adjust http_max_hdr separately

If losthdr is the incrementing counter, the problem is header count, not header size. Raise http_max_hdr:

varnishadm param.show http_max_hdr
varnishadm param.set http_max_hdr 96

Each additional header slot consumes a small amount of workspace, so this interacts with workspace_client. If you raise both, account for the combined memory cost.

Consider http_req_overflow_status (Varnish 7.x)

Varnish 7.x exposes http_req_overflow_status, which controls the HTTP status returned when http_req_size is exceeded. The default is 0, meaning Varnish closes the connection silently. Setting it to 400 or 414 makes the failure explicit to the client and to your logs, which helps distinguish oversized-request rejections from genuine workspace exhaustion:

varnishadm param.show http_req_overflow_status

Note: http_req_overflow_status applies only to the http_req_size limit (oversized request bodies), not to workspace_client overflow. These are distinct failure modes.

This does not fix workspace overflow, but it surfaces a related class of oversized-request failures that would otherwise look like silent connection drops.

Prevention

Monitor ws_client_overflow, client_resp_500, and losthdr continuously. These counters are frequently unmonitored. Alert on any sustained nonzero rate.
Track cookie header size as a leading indicator. If P99 cookie length is creeping toward the workspace_client budget, you will hit overflow before the counter fires. Capture this via varnishlog sampling or a log pipeline.
Size workspace_client deliberately, not by accident. The default changed between 6.x and 7.0. If you rely on defaults, know which default you are on. If you set it explicitly, document the memory arithmetic (workspace_client times thread_pool_max times thread_pools) alongside the value.
Review VCL changes for workspace pressure. A VCL reload that adds regsuball or synthetic header logic can push borderline requests over the edge. Treat the first 500 after a reload as a signal to check ws_client_overflow.
Distinguish client-side from backend-side overflow. ws_backend_overflow increments when backend response headers exceed workspace_backend. The fix is workspace_backend, not workspace_client, and the failure mode is different. On some versions, backend workspace overflow silently drops headers instead of failing the request, which is worse for correctness because it produces a 200 with missing headers rather than a visible 500.

How Netdata helps

Per-second collection of ws_client_overflow, client_resp_500, and losthdr lets you see the exact second the overflow rate began, narrowing the correlation window with deploys, traffic spikes, or bot activity.
Correlating workspace overflow counters with cache_hit and cache_miss rates confirms whether the 500s are hitting cacheable traffic (where header size is the issue) or pass traffic (where VCL logic is the amplifier).
Tracking threads and thread_pool_max alongside workspace_client gives the memory arithmetic context needed to decide whether raising the workspace is safe or will push the process toward OOM.
Process RSS monitoring catches the memory consequence of a workspace increase before the OOM killer does.
Anomaly detection on cookie and header volume patterns surfaces a creeping increase in header size before it crosses the workspace threshold.