Varnish workspace overflow types: client, backend, thread, and session

Varnish allocates small, fixed-size memory regions called workspaces to hold HTTP headers, VCL string operations, and intermediate request state during processing. Each stage of the request lifecycle draws from a different workspace type, with its own size parameter and overflow counter. When any workspace runs out of room, Varnish cannot finish processing the request and returns HTTP 500 to the client. This is a 500, not a 503: the backend did not fail. Varnish itself ran out of working memory.

Since Varnish 6.2, four dedicated counters break workspace exhaustion down by type: ws_client_overflow, ws_backend_overflow, ws_session_overflow, and ws_thread_overflow. Each points at a different runtime parameter and a different stage of request processing. Knowing which counter is firing tells you immediately whether the problem is oversized request headers, oversized backend response headers, ESI pressure, or thread delivery workspace.

The four workspace types at a glance

The defaults above are from the Varnish 7.x varnishd documentation. In Varnish 6.x, workspace_client and workspace_backend defaulted to 64k, and workspace_session defaulted to 0.50k (512 bytes).

The following diagram shows where each workspace type is used in the request lifecycle:

flowchart LR
    A["Client connects"] --> B["workspace_session"]
    B --> C["workspace_thread"]
    C --> D["workspace_client"]
    D --> E["vcl_recv, vcl_hash"]
    E --> F{"Cache hit?"}
    F -->|"miss / pass"| G["workspace_backend"]
    G --> H["vcl_backend_response"]
    F -->|"hit"| I["vcl_deliver"]
    H --> I
    I --> J["Response sent"]

ws_client_overflow: request headers, cookies, and VCL

Parameter: workspace_client (default 96k in Varnish 7.x)

What lives here: Everything Varnish needs to process the incoming client request: the request URL, all client request headers (including Cookie), any headers your VCL adds synthetically, and the working memory for VCL string operations in vcl_recv, vcl_hash, and vcl_deliver.

What causes it to overflow:

• Large Cookie headers. The single most common cause. Applications that accumulate tracking cookies, session tokens, and analytics identifiers can push the Cookie header past the workspace allocation. A single request with a multi-kilobyte cookie string is enough.
• VCL adding many synthetic headers. Each header injected for logging, tracing, or backend communication consumes workspace_client.
• Long URLs. Deeply nested query strings or REST paths with many path segments can exhaust the workspace with the request line alone.
• Deep proxy chains appending forwarding headers. Each hop adds X-Forwarded-For entries and Via headers, growing the header block.

Client-visible effect: HTTP 500. The client_resp_500 counter increments in lockstep.

Sizing note for HTTP/2: workspace_client must be at least 20k to receive full-size (16k) frames from the client, typically POST or PUT bodies. If you have reduced workspace_client below 20k and serve HTTP/2 traffic, this is a likely failure source.

ws_backend_overflow: backend response headers and proxy chains

Parameter: workspace_backend (default 96k in Varnish 7.x)

What lives here: Everything Varnish needs to process the backend response: all backend response headers, Set-Cookie chains, Vary headers, Cache-Control directives, and VCL operations in vcl_backend_response.

What causes it to overflow:

• Long Via or X-Forwarded-For chains in deep proxy stacks. Each proxy hop (CDN, load balancer, application proxy) appends forwarding headers. In a multi-tier setup with three or four proxy layers, these chains can grow large enough to exhaust workspace_backend.
• Backend adding excessive response headers. Applications emitting many Set-Cookie headers, custom analytics headers, or verbose Link headers can push the response header block past the allocation.
• Backend returning very large Vary header values. An overly broad Vary header with many tokens is stored in workspace_backend.

Client-visible effect: The fetch fails. Depending on VCL and grace configuration, the client may receive a 503 (if Varnish has no cached or stale object) or a stale object served from grace. The fetch_failed counter may also increment.

This counter is easy to miss because it looks like a backend problem. The backend is healthy and responding, but Varnish cannot fit the response headers into workspace. If you see ws_backend_overflow incrementing alongside fetch_failed with healthy backends, the response headers are too large.

ws_session_overflow: session state and ESI pressure

Parameter: workspace_session (default 0.75k, or 768 bytes, in Varnish 7.x; was 0.50k in 6.x)

What lives here: Session-level state that persists across requests within the same client connection. This is the smallest workspace, allocated per session (per TCP connection), not per request.

What causes it to overflow:

• ESI (Edge Side Includes) processing. ESI delivery increases session workspace usage, especially with nested includes. Each level of nesting adds session-level bookkeeping. In older Varnish versions, this workspace was called sess_workspace, and ESI panics referencing sess_workspace exhaustion are well-documented. If you use ESI and see ws_session_overflow or child panics during ESI processing, increasing workspace_session is the first thing to try.

Client-visible effect: Request failure. With ESI, this can manifest as a child process panic (crash and automatic restart by the management process), which empties the cache. Check MGT.child_panic if you suspect this path. See the related guide on Varnish ESI errors for more detail.

The default of 768 bytes is adequate for non-ESI workloads. If you enable ESI, plan to increase it. The minimum allowed value is 384 bytes.

ws_thread_overflow: per-thread delivery workspace

Parameter: workspace_thread (default 2k, hard maximum 8k)

What lives here: Per-thread working memory used during request delivery, including write buffers for sending responses to clients.

What causes it to overflow:

• The least commonly seen overflow. The workspace is small but so is the data it holds. If it fires, something in the delivery path is allocating more thread-level memory than expected.

Sizing considerations:

• workspace_thread has a hard maximum of 8k. You cannot set it higher.
• Setting it too low may increase writev() syscall count during delivery, because Varnish has less buffer space to batch outgoing writes. This is a performance tradeoff, not an overflow risk.
• Total memory cost is workspace_thread * total_threads. At default 2k with 10,000 threads (2 pools x 5,000 max), that is approximately 20 MB. Even at the 8k maximum, the cost is around 80 MB. This is the cheapest workspace to increase.

Identifying which counter is firing

All four counters in a single varnishstat invocation:

# Check all workspace overflow counters plus downstream effects
varnishstat -1 -f 'MAIN.ws_*_overflow' -f MAIN.client_resp_500 -f MAIN.losthdr

Live curses view (press q to quit):

varnishstat -f 'MAIN.ws_*_overflow' -f MAIN.client_resp_500

To find the specific requests triggering the overflow, use varnishlog:

varnishlog -q 'Debug ~ "workspace overflow"' -g request

To check current parameter values:

# Show current workspace parameter values
varnishadm param.show workspace_client
varnishadm param.show workspace_backend
varnishadm param.show workspace_session
varnishadm param.show workspace_thread

All four overflow counters should be zero in steady state. Any nonzero rate means active client-facing errors or request failures.

Sizing workspace parameters

The general approach

1. Identify which ws_*_overflow counter is incrementing.
2. Increase the corresponding parameter.
3. Verify the counter stops incrementing.
4. Check process RSS to confirm the memory cost is acceptable.

Changing values at runtime

You can change any workspace parameter without restarting Varnish:

# Increase workspace_client to 128k (runtime only, does not persist across restarts)
varnishadm param.set workspace_client 128k

This takes effect immediately for new requests. To make it permanent, add the -p flag to the Varnish startup command in your service definition (systemd unit, Docker CMD, or equivalent).

Sizing rules

• Increase in 64k increments for workspace_client and workspace_backend. Large enough to absorb typical header growth without trial-and-error cycles.
• Use multiples of 4k for all workspace parameters larger than 4k. This aligns allocations with virtual memory page boundaries.
• For ESI workloads, increase workspace_session from the 768-byte default. Start with 4k and increase if session overflow persists.
• workspace_thread is capped at 8k. There is no reason to go lower than the 2k default unless you are severely memory-constrained.

Memory cost calculation

Workspace memory scales with concurrency, not with traffic. The cost formula is:

total_workspace_memory = (workspace_client + workspace_backend) * concurrent_threads
                       + workspace_session * concurrent_sessions
                       + workspace_thread * total_threads

At default values with a fully saturated thread pool (2 pools, 5,000 threads each = 10,000 threads):

• workspace_client: 96k x 10,000 = approximately 937 MB
• workspace_backend: 96k x 10,000 = approximately 937 MB
• workspace_thread: 2k x 10,000 = approximately 19 MB

That is nearly 1.9 GB for client and backend workspace alone at maximum thread count. In practice, not all threads are active simultaneously, so actual usage is lower. But doubling workspace_client to 192k doubles the client workspace cost across all concurrent threads. On a memory-constrained host, this can push process RSS toward OOM.

On 32-bit systems, the defaults are lower: workspace_client is 24k and workspace_backend is 20k. This limits the per-thread cost but also makes overflow more likely with large headers.

Common sizing mistakes

• Setting workspace_client too high across all nodes. Not every Varnish instance handles requests with oversized cookies. Profile first. If only one application behind Varnish generates large cookies, consider stripping non-essential cookies in VCL before they consume workspace.
• Increasing workspace instead of fixing the root cause. If an application is sending 40 kB of cookies per request, the fix is to reduce the cookies, not to keep raising workspace. Workspace increases treat the symptom.
• Forgetting that workspace_backend overflow looks like a backend problem. The backend is fine. The response headers are too large. Check ws_backend_overflow before chasing backend latency or health issues.

Signals to watch

The losthdr counter is related but distinct. It fires when the number of headers exceeds the http_max_hdr limit (default 64), regardless of total header size. A request with 70 small headers triggers losthdr but not ws_client_overflow. A request with 5 very large headers triggers ws_client_overflow but not losthdr. Both cause silent failures or 500 errors.

How Netdata helps

Netdata collects all four ws_*_overflow counters, client_resp_500, losthdr, and process RSS per second, so you can correlate workspace pressure with client-visible errors and memory cost without running varnishstat during an incident. The same dashboard shows overflow spikes alongside request rate, helping you distinguish a traffic pattern shift (a new endpoint generating large cookies) from a single abusive client. Anomaly detection on workspace counters can surface gradual header growth before it causes hard overflows.

Related guides

• Varnish Error 503 Backend fetch failed: what the error page actually means
• Varnish backend_fail, backend_unhealthy, and backend_busy: three different backend problems
• Varnish backend connection reuse low: keepalive not working and slow TTFB
• Varnish backend probe configuration: threshold, window, interval, and initial
• Varnish backend is sick: health probes, all-backends-sick, and grace
• Varnish cache hit ratio dropped: hit rate collapse and backend overload
• Varnish cache stampede: a popular object expires and the herd hits the backend
• Varnish ESI errors: broken pages and workspace pressure from Edge Side Includes
• Varnish fetch_failed: backend connected but the fetch broke
• Varnish grace masking a backend outage: the ticking-clock incident
• Varnish Guru Meditation: reading the XID and tracing the failing request
• Varnish cache_hitpass / cache_hitmiss climbing: uncacheable content bleeding to the backend