$ guides / varnish / varnish-ban-list-growing ▌

Operations Guides

Varnish ban list growing: O(n) lookups and the lurker falling behind

Varnish’s ban list is a linear chain of invalidation rules. Every cache lookup tests the requested object against all active bans before serving it. When the list grows, each lookup pays O(n) cost. Hit latency rises. CPU climbs. But hit ratio stays flat, backend request rate stays flat, and error counts stay flat. This makes a ban list explosion hard to detect with standard monitoring, because most teams alert on hit rate and error rate, not hit latency.

The ban lurker is a background thread that walks the object store, testing cached objects against ban expressions and evicting matches. When bans arrive faster than the lurker can process them, the list grows without bound. At thousands of entries, every cache hit does a linear scan of the ban chain. At tens of thousands, the CPU cost becomes visible across all requests, and the lurker starts losing lock contention to worker threads, compounding the problem.

The root cause is almost always an application or CMS issuing a ban per content update. A publishing system that bans one URL at a time, hundreds of times per minute, will outrun the lurker. The fix is not to make the lurker faster. The fix is to stop generating bans at that rate, convert them to lurker-friendly forms, or switch to a hash-based invalidation model.

This evaluation happens on the request path, in the worker thread, for every cache lookup. The lurker’s job is to shrink the list proactively so that by the time a request arrives, the relevant objects are already gone and the ban can be removed. When the lurker falls behind:

The ban list grows because bans are not cleaned up.
Every cache lookup becomes more expensive because it scans a longer ban chain.

The cost is O(n) in ban list length, compounded by regex evaluation on each ban expression. The lurker also competes with worker threads for internal locks. When MAIN.bans_lurker_contention starts incrementing, the lurker yields to lookups, processes fewer bans per cycle, and accelerates list growth.

The signature is distinct from other Varnish degradation patterns: hit latency and CPU rise, but hit ratio, backend request rate, and error rates stay flat. It is a pure hit-path performance degradation.

flowchart TD
    A["App/CMS issues ban per content update"] --> B["Bans added to ban list"]
    B --> C{"Lurker keeping up?"}
    C -->|No| D["Ban list grows unbounded"]
    C -->|Yes| E["Bans cleaned up"]
    D --> F["Every cache hit: O(n) ban scan"]
    F --> G["Hit latency rises, CPU rises"]
    G --> H["Signature: hit ratio and backend req stay flat"]

Common causes

Cause	What it looks like	First thing to check
App or CMS issuing a ban per content update	`MAIN.bans_added` rate is high and steady, `MAIN.bans_deleted` lags behind	`varnishadm ban.list` to see ban patterns and frequency
Bans referencing `req.*` fields	Lurker counters near zero, ban list grows, `bans_lurker_tested` is low or zero	Inspect ban expressions for `req.url` or `req.http.*` references
Complex regex in ban expressions	Ban list moderate but CPU high, lurker active but slow	Inspect regex complexity in `ban.list` output
`ban_lurker_sleep` set too high	Lurker cycles slowly, ban list grows steadily	`varnishadm param.show ban_lurker_sleep`

Quick checks

# Ban list size and growth rates (value, per-second rate, average)
varnishstat -1 -f 'MAIN.bans*'

# Actual ban entries: req-level vs obj-level
varnishadm ban.list

# Lurker activity and lock contention
varnishstat -1 -f 'MAIN.bans_lurker*' -f MAIN.bans_dups

# Confirm hit ratio and backend rate are stable (the signature)
varnishstat -1 -f MAIN.cache_hit -f MAIN.cache_miss -f MAIN.backend_req

# Request duration in microseconds on the hit path
varnishncsa -F '%D'

# Lurker runtime parameters
varnishadm param.show ban_lurker_sleep
varnishadm param.show ban_lurker_age
varnishadm param.show ban_lurker_batch

If MAIN.bans_added is accumulating faster than MAIN.bans_deleted, the lurker is losing. The gap between MAIN.bans and MAIN.bans_completed tells you how many bans are still active and affecting every lookup.

varnishadm ban.list shows the actual expressions. Look for req.* references. Bans that reference req.url or req.http.* cannot be processed by the lurker at all. They persist until every cached object has been checked against them at lookup time by worker threads. This is the single most common reason the lurker appears stuck while the ban list grows.

How to diagnose

Confirm the ban list is growing. Run varnishstat -1 -f 'MAIN.bans*' twice, 10 seconds apart. If MAIN.bans is increasing, the lurker is falling behind. Compare bans_added delta versus bans_deleted delta over the interval. If added consistently exceeds deleted, the list will grow without bound.
Check whether bans are lurker-friendly. Run varnishadm ban.list and inspect each expression. Bans referencing obj.* fields (such as obj.http.x-url) can be processed asynchronously. Bans referencing req.* fields (such as req.url or req.http.host) cannot. The lurker only has object context, not request context. If the list is full of req.* bans, the lurker is structurally unable to clean them up.
Check lurker contention. Run varnishstat -1 -f MAIN.bans_lurker_contention. If this counter is incrementing, the lurker is yielding to worker threads on lock contention. It processes fewer bans and falls further behind. High traffic volume worsens this because more worker threads compete for the same locks.
Confirm the signature. Verify that hit ratio and backend request rate are stable while latency rises. Run varnishstat -1 -f MAIN.cache_hit -f MAIN.cache_miss -f MAIN.backend_req and compare to baseline. If hit ratio is normal but request latency (from varnishncsa -F '%D') is elevated, you have confirmed pure hit-path degradation. This rules out cache stampede, storage exhaustion, and backend problems.
Identify the ban source. Varnish has no VSL tag for ban issuance: the only ban-related tag is ExpBan, which logs the VXID of objects evicted by a ban. To see which expressions were added and when, use varnishadm ban.list and correlate timestamps with deploy or publishing activity. In most cases, the source is an automated CMS publishing hook, a deployment script, or an invalidation API endpoint called per content update.
Assess severity. Below 100 bans is normal. Between 100 and 1,000, watch the trend. Above 1,000, investigate immediately. Above 10,000, every cache hit is doing significant work and you are in an active performance incident.

Metrics and signals

Signal	Why it matters	Warning sign
`MAIN.bans`	Active ban count. Directly determines per-lookup cost.	Growing trend above 500
`MAIN.bans_added` vs `MAIN.bans_deleted`	Rate comparison reveals whether the lurker is keeping up.	`added` rate consistently exceeds `deleted` rate
`MAIN.bans_completed`	Bans fully processed. `bans - bans_completed` = active bans on the hot path.	`bans - bans_completed` growing
`MAIN.bans_lurker_contention`	Lurker yielded to worker threads for locks.	Any sustained nonzero rate
`MAIN.bans_lurker_obj_killed`	Objects evicted by the lurker. Should track ban injection rate.	Rate near zero while bans grow
`MAIN.bans_dups`	Duplicate bans collapsed by deduplication.	Zero with many similar bans
Request latency via `varnishncsa -F '%D'`	Hit-path latency.	Rising latency with stable hit ratio
CPU on Varnish process	Ban evaluation is CPU-intensive regex work.	Rising CPU with flat traffic

Varnish does not expose request latency, backend fetch time, or response time as varnishstat counters. These are only available through varnishlog or varnishncsa. Monitoring that relies solely on varnishstat will miss the latency increase that is the primary symptom.

Fixes

Reduce ban injection rate

The most effective fix. If a CMS publishes 50 content updates and issues 50 individual bans, consolidate them into a single pattern. Instead of:

ban("req.url == /article/123");
ban("req.url == /article/124");
ban("req.url == /article/125");

Issue one broader ban:

ban("req.url ~ ^/article/");

Fewer entries means a shorter list for the lurker and a shorter chain for every lookup. Coordinate with the application team to batch invalidations.

Convert req-level bans to obj-level bans

The lurker can only process bans referencing obj.* fields. Bans referencing req.url or req.http.* are never cleaned up asynchronously. They persist until every cached object has been tested against them at request time.

The standard workaround: copy request-context information into response headers during vcl_backend_response, then ban on those headers.

sub vcl_backend_response {
    set beresp.http.x-url = bereq.url;
    set beresp.http.x-host = bereq.http.host;
}

Then issue bans using obj.http.*:

ban("obj.http.x-url ~ ^/article/");

The lurker can now process these proactively. It walks the object store, matches objects, evicts matches, and removes the ban once all objects older than the ban have been tested.

These extra headers consume workspace memory per object. Strip them in vcl_deliver:

sub vcl_deliver {
    unset resp.http.x-url;
    unset resp.http.x-host;
}

Tune lurker parameters

If bans are already lurker-friendly but the list is still growing, the lurker may need to run more aggressively:

ban_lurker_sleep (default 0.010s): sleep between processing batches. Reducing this makes the lurker cycle faster.
ban_lurker_age (default 60s): how old an object must be before the lurker tests it. Reducing this processes newer objects sooner at the cost of CPU.
ban_lurker_batch (default 1000): ban expressions processed per lurker cycle. Increasing this lets the lurker process more bans per wake-up but holds locks longer.

varnishadm param.set ban_lurker_sleep 0.002

Tuning is a stop-gap. If the ban injection rate fundamentally exceeds what the lurker can process, parameter tuning only delays the inevitable.

Emergency: set ban_cutoff

ban_cutoff (default 0, disabled) is a blunt emergency brake. When set, the lurker stops inspecting bans beyond that count and treats all objects as if they matched, causing mass eviction.

varnishadm param.set ban_cutoff 10000

This prevents the ban list from growing past the cutoff but nukes the cache. Use only when the ban list has grown so large that Varnish is effectively unusable. Expect a cold-cache thundering herd as all objects are evicted and re-fetched from backends.

Long-term: switch to xkey VMOD

For tag-based invalidation at scale, the xkey VMOD (part of varnish-modules) uses hash-based lookup instead of linear ban list scanning. You attach secondary keys (xkeys) to objects and invalidate by key. Cost is O(1) per invalidation instead of O(n) per lookup.

This requires VCL changes: adding xkey headers during vcl_backend_response and using xkey.softpurge() or xkey.purge() in vcl_recv for invalidation. The shift eliminates the ban list as a scalability bottleneck.

Prevention

Alert on MAIN.bans. Page at 500, investigate immediately at 1,000, and page regardless of time of day at 10,000.

Track ban addition rate. If MAIN.bans_added consistently exceeds MAIN.bans_deleted, the lurker is losing. This ratio is the leading indicator, more useful than absolute count.

Audit ban patterns. Ensure all bans reference obj.* fields, not req.*. Any req.* ban is a permanent lurker blind spot. A one-time VCL audit prevents the most common structural cause.

Rate-limit ban injection. Batch invalidations. A CMS that publishes a section should issue one pattern ban, not one ban per URL.

Verify ban_dup is enabled (default on). This collapses older identical bans when a new one is added. Check with varnishadm param.show ban_dup. If duplicates are accumulating, deduplication may not match due to slightly different expressions.

How Netdata helps

Netdata surfaces ban list problems before they become user-facing incidents:

MAIN.bans at per-second granularity shows growth trends in real time. The rate of change matters more than the absolute count.
Correlated MAIN.bans_added and MAIN.bans_deleted rates immediately reveal whether the lurker is keeping up. Diverging lines mean the list is growing.
MAIN.bans_lurker_contention as an anomaly signal flags when the lurker loses lock battles to worker threads.
Anomaly detection on CPU utilization catches the rising cost of ban evaluation before static thresholds fire. A slow CPU creep with flat traffic is the earliest indicator.
Correlating ban list growth with request latency (via varnishncsa integration) confirms the signature: latency rises while hit ratio and backend request rate stay flat.