Varnish not caching: Set-Cookie, Vary, and Cache-Control killing your hit rate

Your hit rate is low and backend_req tracks client_req almost one-to-one. Varnish is up, storage has room, there are no error spikes, but cache_hitpass or cache_hitmiss dominates the cache outcome counters. The cache is running but not caching.

The root cause is almost never Varnish itself. It is the interaction between backend response headers and the built-in VCL rules that determine cacheability. Three header problems account for the vast majority of cases: Set-Cookie on every response, Cache-Control: private or no-cache on public content, and an overly broad Vary header that fragments the cache into non-shareable variants.

The fix is in VCL: strip cookies for cacheable paths, normalize or override Vary, and set explicit caching headers when the backend sends the wrong ones. But first, identify which header is killing your hit rate.

How Varnish decides cacheability

Varnish applies built-in VCL rules that determine cacheability. These rules run in two phases.

In vcl_recv, the built-in VCL returns pass for any request that carries a Cookie or Authorization header. The request never reaches cache lookup. If your application sets a session cookie on every response, which is the default behavior in Rails, Django, and many other frameworks, every subsequent request from that client carries that cookie and the built-in VCL bypasses the cache.

In vcl_backend_response, the built-in VCL checks the backend response for conditions that make it uncacheable. If any of the following is true, Varnish marks the object as uncacheable (beresp.uncacheable = true) and stores it as a hit-for-miss with the value of uncacheable_ttl (120 seconds by default):

• The response contains a Set-Cookie header
• The response contains Vary: *
• The response is marked not cacheable due to Cache-Control: no-store or private
• The calculated TTL is zero or negative

When an object is marked uncacheable, Varnish remembers that decision. For the duration of the hit-for-miss TTL, subsequent requests for the same object short-circuit directly to the backend without attempting a cache lookup. This shows up as cache_hitmiss or cache_hitpass in your counters.

flowchart TD
    A["Client request"] --> B{"vcl_recv: Cookie present?"}
    B -->|yes| C["return pass"]
    B -->|no| D["Cache lookup"]
    D -->|hit| E["Serve from cache"]
    D -->|miss| F["Fetch from backend"]
    F --> G{"Set-Cookie in response?"}
    G -->|yes| H["Uncacheable: hit-for-miss"]
    G -->|no| I{"Cache-Control: private/no-cache?"}
    I -->|yes| H
    I -->|no| J{"Vary: * ?"}
    J -->|yes| H
    J -->|no| K["Cache the object"]
    H --> L["TTL 120s, pass to backend"]

A broad Vary header (such as Vary: Cookie or Vary: User-Agent) does not make an object uncacheable, but it fragments the cache into separate variants. Each unique combination of varied header values creates a separate cache entry. With Vary: User-Agent, every distinct browser string gets its own object. The result is functionally no caching, because the number of variants approaches the number of requests.

Common causes

Quick checks

All commands below are read-only.

# Check cache outcome distribution
varnishstat -1 -f MAIN.cache_hit -f MAIN.cache_miss -f MAIN.cache_hitpass -f MAIN.cache_hitmiss

# Check backend request ratio (should be much lower than client_req)
varnishstat -1 -f MAIN.client_req -f MAIN.backend_req

# Check pass rate and uncacheable counter
varnishstat -1 -f MAIN.s_pass -f MAIN.beresp_uncacheable

# See which URLs are being passed (not cached)
varnishlog -q 'VCL_call eq "PASS"' -g request -i ReqMethod,ReqURL

# Inspect Set-Cookie headers in backend responses
varnishlog -b -i BereqURL,BerespHeader:Set-Cookie

# Inspect Cache-Control headers in backend responses
varnishlog -b -i BereqURL,BerespHeader:Cache-Control

# Inspect Vary headers in backend responses
varnishlog -b -i BereqURL,BerespHeader:Vary

# Check what cookies arrive on client requests
varnishlog -c -i ReqURL,ReqHeader:Cookie

# List recently loaded VCLs (check for recent changes)
varnishadm vcl.list

How to diagnose it

1. Confirm the symptom. Check whether cache_hitpass and cache_hitmiss are high relative to cache_hit. The hit rate formula is cache_hit / (cache_hit + cache_miss + cache_hitpass + cache_hitmiss) on Varnish 5.1 and later, where cache_hitmiss was added. If hitpass and hitmiss dominate, content is being marked uncacheable, not simply missing.

2. Distinguish pass from miss. s_pass counts requests where VCL deliberately bypassed the cache, typically via return(pass) in vcl_recv because a Cookie header was present. cache_miss counts requests where the cache lookup found nothing. If s_pass is high, the problem is in vcl_recv. If cache_miss is high but s_pass is low, objects are expiring or being evicted, not blocked by headers.

3. Identify which header is responsible. Use the varnishlog commands in the quick checks to inspect backend response headers for Set-Cookie, Cache-Control, and Vary. Look for URLs that should be cached but carry blocking headers.

4. Check the request side. Even if your backend responses are clean, incoming requests with Cookie headers trigger the built-in pass in vcl_recv. Use varnishlog -c -i ReqURL,ReqHeader:Cookie to see what cookies arrive on requests for cacheable paths.

5. Check for recent VCL or application changes. Run varnishadm vcl.list and look at timestamps. A recent VCL reload or application deploy that changed response headers is the most common trigger for a sudden hit rate collapse.

6. Verify cache object count. If Vary fragmentation is the issue, n_object may be unusually high relative to the number of distinct cacheable URLs. Each variant is a separate object.

Metrics and signals to monitor

Fixes

Strip cookies in vcl_recv for cacheable paths

The built-in VCL returns pass for any request with a Cookie header. If your application sets cookies that are not needed for rendering cacheable content (analytics cookies, session cookies on public pages), strip them before the cache lookup.

# VCL pattern: strip cookies for GET/HEAD requests
sub vcl_recv {
    if (req.method == "GET" || req.method == "HEAD") {
        unset req.http.Cookie;
    }
}

Warning: stripping all cookies breaks authenticated pages. Apply this only to paths you know are public and cacheable, or scope it with URL matching:

sub vcl_recv {
    if ((req.method == "GET" || req.method == "HEAD")
        && (req.url ~ "^/static/" || req.url ~ "^/assets/")) {
        unset req.http.Cookie;
    }
}

For more selective stripping, remove specific cookies while preserving ones the application needs:

# VCL pattern: remove specific tracking cookies, keep session cookie
sub vcl_recv {
    if (req.http.Cookie) {
        set req.http.Cookie = regsuball(req.http.Cookie, "_ga=[^;]+(; )?", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "_gid=[^;]+(; )?", "");
        if (req.http.Cookie ~ "^\s*$") {
            unset req.http.Cookie;
        }
    }
}

Gotcha: if you strip some cookies but any cookie remains, the built-in VCL still returns pass. The built-in check is for the presence of any Cookie header, not specific cookie values. You must remove all cookies, or add your own return(hash) before the built-in pass logic runs.

Strip Set-Cookie in vcl_backend_response

If the backend emits Set-Cookie on responses that should be cached, remove it in vcl_backend_response before the cacheability decision is made.

# VCL pattern: remove Set-Cookie for cacheable responses
sub vcl_backend_response {
    if (bereq.method == "GET" || bereq.method == "HEAD") {
        unset beresp.http.Set-Cookie;
    }
}

Do not strip Set-Cookie in vcl_deliver. The cacheability decision happens in vcl_backend_response. By the time vcl_deliver runs, the object is already marked uncacheable and stored as a hit-for-miss. Removing the header at delivery does not change the caching decision.

Override Cache-Control for public content

If the backend sends Cache-Control: private, no-cache, or no-store on content that is actually public and cacheable, override it in vcl_backend_response:

# VCL pattern: override restrictive Cache-Control for known cacheable paths
sub vcl_backend_response {
    if (bereq.url ~ "^/static/" || bereq.url ~ "^/assets/") {
        unset beresp.http.Cache-Control;
        set beresp.ttl = 10m;
    }
}

This removes the header that triggers the built-in uncacheability check and sets an explicit TTL. Apply selectively to avoid caching content that genuinely requires privacy headers.

Normalize Vary headers

A broad Vary header fragments the cache. Vary: User-Agent creates a variant per browser string. Vary: Cookie creates a variant per cookie value. Fix this by normalizing or removing the Vary header for cacheable responses.

# VCL pattern: remove Vary for cacheable responses
sub vcl_backend_response {
    if (bereq.url ~ "^/static/" || bereq.url ~ "^/assets/") {
        unset beresp.http.Vary;
    }
}

For User-Agent normalization, classify clients into broad buckets before the hash to reduce the number of variants. This requires modifying vcl_hash to hash a normalized agent string rather than the raw header.

Vary: * is treated differently from Vary: Cookie or Vary: User-Agent. Vary: * explicitly makes the response uncacheable because the built-in VCL checks for it directly. Other Vary values allow caching but fragment it. If you see Vary: *, the response will never be cached unless you remove it in VCL.

Prevention

• Monitor cache_hitpass and cache_hitmiss independently, not just aggregate hit rate. A slow hitpass increase will not trigger hit-rate alerts because hitpass inflates the denominator. Track the rate of these counters separately.
• Add a VCL check on deploy. After loading new VCL, verify hit rate on canary URLs before routing full traffic. Use varnishlog to inspect response headers on key paths.
• Audit application response headers regularly. Application deploys that add Set-Cookie or change Cache-Control are the most common trigger for hit rate collapse.
• Normalize Vary in VCL, not just in the application. Application frameworks may add Vary headers by default. VCL is the last point of control before the cacheability decision.
• Distinguish pass from miss in your monitoring. s_pass is an intentional bypass, usually because Cookie or Authorization is on the request. cache_miss is a lookup that found nothing. Different root causes, different fixes.

How Netdata helps

• Per-second cache outcome counters let you see cache_hit, cache_miss, cache_hitpass, and cache_hitmiss independently and in real time. A gradual hitpass increase becomes visible immediately, not buried in an aggregate hit-rate percentage.
• Correlation between hit rate and backend request rate shortens diagnosis. When hit rate drops and backend_req rises proportionally, the cache is failing to intercept traffic. When backend_req is stable but hit rate drops, the issue is elsewhere.
• VCL reload event tracking helps connect hit rate changes to deploy events. If n_vcl changes or vcl_fail increments coincide with a hit rate drop, the VCL change is the likely trigger.
• Anomaly detection on beresp_uncacheable surfaces header drift before it becomes a hit rate crisis. If backend responses start being marked uncacheable after an application deploy, the anomaly flag fires on the counter rate change.
• Workspace overflow signals (ws_client_overflow, losthdr) catch the secondary effect of large cookie headers. Cookie-heavy requests that cause hit rate issues can also cause workspace exhaustion and 500 errors.

Related guides

• Varnish cache hit ratio dropped: hit rate collapse and backend overload
• How Varnish actually works in production: a mental model for operators
• Varnish monitoring checklist: the signals every production cache needs
• Varnish monitoring maturity model: from survival to expert
• Varnish sess_dropped vs req_dropped: HTTP/1 connection drops and HTTP/2 stream drops
• Varnish thread pool exhaustion: workers all busy, queue full, sessions dropped
• Varnish thread pool tuning: thread_pool_min, thread_pool_max, and thread_pools
• Varnish thread_queue_len above zero: requests waiting for a worker
• Varnish threads_failed: the OS refusing to create worker threads
• Varnish threads_limited climbing: hitting thread_pool_max