Pages assembled with Edge Side Includes (ESI) can fail in ways that look like application bugs but are cache-layer problems. When MAIN.esi_errors increments, users receive partial pages with missing fragments or HTTP 500 responses from workspace exhaustion. MAIN.esi_warnings counts ESI tags that Varnish skipped, silently dropping content from otherwise valid responses.

Each <esi:include> tag in a backend response triggers a full sub-request through Varnish’s VCL pipeline. A page with five includes means six VCL cycles: the parent plus five children. Each sub-request consumes a worker thread for its full lifecycle, allocates workspace memory for headers and VCL processing, and can itself contain ESI tags that spawn further sub-requests. The recursion is bounded by max_esi_depth, but hitting that bound produces an error, not graceful degradation.

What this means

When ESI processing fails, symptoms fall into three categories:

Broken or partial pages (MAIN.esi_errors incrementing): The parent response was fetched successfully, but one or more ESI includes failed. The page is served with the failed fragment missing, or the entire response is synthesized as an error. Users see incomplete layouts, missing widgets, or blank sections.

Silently skipped tags (MAIN.esi_warnings incrementing): Varnish encountered ESI markup it could not parse and skipped the tag. The page renders but is missing the content that the skipped include would have provided. This is quieter than an error but equally broken from the user’s perspective.

Workspace pressure and 500 errors (MAIN.ws_client_overflow, MAIN.client_resp_500, MAIN.losthdr incrementing): ESI processing consumes workspace memory for each sub-request’s headers, VCL string operations, and response assembly. Pages with many includes, large response headers, or complex VCL can exhaust the per-request workspace, causing 500 errors. This is amplified when ESI fragments themselves carry large headers such as analytics cookies or Set-Cookie chains.

flowchart TD
    A[Client requests page] --> B[Backend returns parent with ESI tags]
    B --> C[Varnish parses ESI includes]
    C --> D[Sub-request per include through VCL]
    D --> E{Include fetch succeeds?}
    E -->|No| F[esi_error: partial page served]
    E -->|Yes| G{Depth or workspace limit hit?}
    G -->|Yes| H[esi_error or ws_overflow: 500]
    G -->|No| I[Assemble fragments, deliver page]

Common causes

CauseWhat it looks likeFirst thing to check
Failed included URLesi_errors spikes; fragment endpoints returning non-200varnishlog -g request -q 'FetchError' to see failing backend fetches for fragment URLs
Circular includesesi_errors increments; depth limit reached; pages never complete or timeoutCheck whether fragment A includes fragment B which includes fragment A
Malformed ESI tagsesi_warnings increments; page renders with missing sections but no errorvarnishlog -i ESI_xml -g request to see raw ESI markup and parse issues
Workspace exhaustionws_client_overflow or client_resp_500 increments on ESI pages; non-ESI pages unaffectedvarnishstat -1 -f 'MAIN.ws_*_overflow' -f MAIN.client_resp_500
Depth limit exceededesi_errors on deeply nested include hierarchies; correlates with specific page templatesCount ESI nesting levels in templates; compare to max_esi_depth
Version-specific error handling changeBehavior changed after upgrade; previously working pages now failCheck Varnish version with varnishd -V; review esi_include_onerror feature flag

Quick checks

# Check ESI error and warning rates
varnishstat -1 -f MAIN.esi_errors -f MAIN.esi_warnings

# Check workspace overflow indicators
varnishstat -1 -f 'MAIN.ws_*_overflow' -f MAIN.client_resp_500 -f MAIN.losthdr

# See ESI processing in the log stream
varnishlog -i ESI_xml -g request
<!-- TODO: verify ESI_xml is the correct VSL tag name for ESI processing events -->

# See ESI include failures with fetch context
varnishlog -g request -q 'FetchError'

# Check current max_esi_depth setting
varnishadm param.show max_esi_depth

# Check workspace sizes
varnishadm param.show workspace_client
varnishadm param.show workspace_backend

# Check thread pool pressure (ESI amplifies thread usage)
varnishstat -1 -f MAIN.threads -f MAIN.thread_queue_len -f MAIN.threads_limited

# Check backend health for fragment endpoints
varnishadm backend.list -p

# Inspect response codes for ESI sub-requests
varnishlog -i BereqURL -i BerespStatus -g request -q 'BerespStatus ne 200'

How to diagnose it

  1. Confirm ESI is the problem. Check whether MAIN.esi_errors or MAIN.esi_warnings are incrementing at a rate that correlates with user-reported broken pages. If the counters are flat, the problem is elsewhere: backend, VCL, or storage.

  2. Identify the specific failure mode. Run varnishlog -i ESI_xml -g request during the incident window. Look for malformed tags (correlates with esi_warnings) and recursive include chains where the same fragment URL appears at increasing depth.

  3. Trace failed includes. If esi_errors is incrementing, find which included URLs are failing. Use varnishlog -g request -q 'FetchError' to see backend fetch failures. Common patterns: fragment endpoint returns 500 or 503 under load, fragment URL changed during a deploy, or the fragment endpoint requires headers that the ESI sub-request does not forward.

  4. Check for circular includes. If the same fragment URL appears at multiple nesting levels in varnishlog, you have a circular dependency. The max_esi_depth limit will eventually fire, producing an esi_error. Circular includes are usually introduced by template refactoring or shared component includes that reference each other.

  5. Assess workspace pressure. Run varnishstat -1 -f 'MAIN.ws_*_overflow' -f MAIN.client_resp_500. If these are nonzero and correlate with ESI traffic, workspace is too small for the header volume of your assembled responses. Check workspace_client and workspace_backend parameter values.

  6. Check version-specific behavior. Varnish 7.3 introduced strict ESI error handling where sub-requests returning non-200/204 status codes would fail the include. Varnish 7.5 reverted this: sub-requests are processed regardless of status code. The esi_include_onerror feature flag controls this behavior, and its semantics inverted between 7.3 and 7.5. If you upgraded across these versions, check the flag state with varnishadm param.show feature and verify your version with varnishd -V.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
MAIN.esi_errorsCount of ESI processing failures. Each increment means a broken or partial page was served.Any nonzero rate in production.
MAIN.esi_warningsCount of malformed ESI tags skipped. Content is silently dropped.Sustained nonzero rate.
MAIN.ws_client_overflowClient workspace exhausted. ESI amplifies workspace consumption per request through sub-request headers and VCL operations.Any nonzero rate, especially on ESI-heavy pages.
MAIN.client_resp_500Delivery failures, often from workspace exhaustion during ESI assembly.Spike correlating with ESI traffic.
MAIN.losthdrHeaders dropped due to workspace limits. Can break caching (Vary lost) or authentication (Cookie lost).Any nonzero value.
MAIN.thread_queue_lenESI sub-requests consume worker threads. Many includes means thread amplification proportional to include count.Sustained nonzero, especially on ESI-heavy URLs.
MAIN.backend_reqESI misses generate additional backend requests. Fragment endpoints see amplified load.Backend request rate higher than expected for the page count.
MAIN.fetch_failedIncluded URL fetch failures. Direct cause of many ESI errors.Nonzero rate on fragment endpoints.

Fixes

Fix failed included URLs

The most common cause of esi_errors in production. When an ESI fragment endpoint returns an error (500, 503, or timeout), the include fails and the page is broken.

  1. Identify the failing endpoint from varnishlog -g request -q 'FetchError'.
  2. Fix the backend issue, or ensure the fragment endpoint has its own health probe and error handling.
  3. If the fragment is non-critical, use onerror="continue" in the ESI tag so the page renders without it rather than failing entirely.
  4. Cache fragment responses with appropriate TTLs so transient backend failures do not break page assembly. Use beresp.grace to serve stale fragments during backend hiccups.

Fix circular includes

Circular includes (fragment A includes fragment B which includes fragment A) hit the max_esi_depth limit and fail. This is a template bug, not a tuning problem.

  1. Identify the circular chain from varnishlog -i ESI_xml -g request.
  2. Break the cycle in the template layer. Shared components should not include their callers.
  3. Do not raise max_esi_depth to work around a circular include. This defers the failure, increases resource consumption, and does not fix the broken page.

Increase workspace for ESI-heavy pages

ESI processing consumes workspace for each sub-request’s headers and VCL operations. Pages with many includes, large fragment headers, or complex VCL can exhaust the default workspace.

# Check current values
varnishadm param.show workspace_client
varnishadm param.show workspace_backend

Increasing workspace raises per-thread memory consumption. With thread_pool_max threads across thread_pools pools, the total memory cost scales as workspace * max_threads per pool.

Workspace parameter changes require a child restart to take effect. Verify after applying by checking the parameter value with varnishadm param.show workspace_client.

Tune max_esi_depth

If your page layout legitimately requires deep ESI nesting and you are hitting the depth limit on valid (non-circular) includes, increase the limit:

# Check current value
varnishadm param.show max_esi_depth

# Increase (live change, no restart needed)
varnishadm param.set max_esi_depth 10

Tradeoff: Higher depth allows more recursion, which means more sub-requests, more thread time, more workspace consumption, and more delivery latency. Only increase if you have verified the include hierarchy is not circular.

Handle version-specific error behavior

If you upgraded Varnish and ESI behavior changed:

  • Varnish 7.3 introduced strict error handling: ESI sub-requests returning non-200/204 would fail. To allow other status codes, enable param.set feature +esi_include_onerror and use onerror="continue" in ESI tags.
  • Varnish 7.5 reverted this: sub-requests are processed regardless of status code. The esi_include_onerror feature flag now enforces 200/204-only behavior, which is the opposite of 7.3. If you had this flag enabled on 7.3 and upgraded to 7.5, behavior flips.
  • Varnish 6.x and earlier accept all status codes in ESI includes (pre-7.3 behavior).
# Check version and feature flag state
varnishd -V
varnishadm param.show feature

Prevention

  • Monitor MAIN.esi_errors and esi_warnings independently. They indicate different problems (failed includes vs. malformed tags) but both produce broken pages.
  • Add health probes for fragment endpoints. ESI sub-requests go through the normal backend fetch path. If fragment endpoints are sick, pages break.
  • Cache fragment responses with grace. Fragment endpoints should have TTL and beresp.grace set so that transient backend failures do not break page assembly.
  • Audit include hierarchies for cycles before deploying template changes. Circular includes only surface in production when the specific page template is requested.
  • Size workspace for your worst-case ESI page. Count the maximum number of includes per page, estimate header sizes for each fragment, and set workspace_client with headroom. Re-check after any change that adds headers (analytics, tracking, A/B testing).
  • Be cautious with ESI over HTTPS. Varnish does not support ESI includes over HTTPS and switches to HTTP. If your fragment URLs are HTTPS, this can cause mixed-content issues or failed includes.

How Netdata helps

  • Per-second MAIN.esi_errors and esi_warnings rates detect ESI failures within seconds of onset. Per-second granularity catches short bursts that minute-level polling misses.
  • Workspace overflow correlation: Netdata surfaces ws_client_overflow, client_resp_500, and losthdr alongside ESI error counters. When ESI errors and workspace overflow move together, the diagnosis narrows immediately to workspace sizing rather than backend failures.
  • Backend health alongside ESI errors: Fragment endpoint failures are the leading cause of ESI errors. Netdata’s per-backend health probes and fetch_failed counters sit next to ESI metrics, so you can confirm whether the include failure originated from a sick backend.
  • Anomaly detection on ESI counter rates: Flags unusual esi_errors or esi_warnings spikes even when absolute rates are low, useful for catching gradual increases from template drift or header bloat.