$ guides / varnish / varnish-ban-lurker-not-keeping-up ▌

Operations Guides

Varnish ban lurker not keeping up: contention and ban_lurker_sleep

The ban lurker is Varnish’s background invalidation worker. When it falls behind, the ban list grows and every cache lookup must test the object against all uncompleted bans. The signature pattern is MAIN.bans climbing steadily while MAIN.bans_lurker_contention rises and MAIN.bans_lurker_tested stays low or stalls entirely.

This is a slow-motion degradation. Hit rate often looks fine, but per-request latency on cache hits creeps upward because each lookup scans the full ban list. A ban list in the tens of thousands turns each cache hit into an O(n) scan.

This article covers how to identify ban lurker stall, distinguish lock contention from other causes (disabled lurker, req-only bans, age gating), and tune the relevant parameters without making things worse.

What this means

The ban lurker is a single background thread that walks the object store, testing cached objects against ban expressions that reference object attributes (obj.* bans). When it finds a match, the object is removed. Once all objects older than a ban have been tested, the ban is marked completed and can be removed from the list.

Under high traffic, the lurker competes with worker threads for locks on the object store. Each time a cache lookup needs a lock the lurker holds, the lurker yields. This increments bans_lurker_contention. A rising contention rate means the lurker spends more time backing off than working, so bans accumulate faster than they are processed.

The lurker processes bans in batches controlled by ban_lurker_batch (default 1000 objects). After each batch, it sleeps for ban_lurker_sleep seconds. When it yields due to contention, it sleeps for ban_lurker_holdoff seconds before retrying. If contention is constant, the lurker barely progresses between yields.

Key distinction: the lurker can only process obj.* bans. Bans referencing req.* attributes cannot be tested by the lurker. They persist until every object in cache has been checked at lookup time. If all your bans are req.*, the lurker appears inactive (bans_lurker_tested near zero) but the ban list grows anyway.

flowchart TD
    A[High lookup traffic] --> B[Lurker walks object store]
    B --> C{Lock contention with lookup?}
    C -->|Yes: lurker yields| D[bans_lurker_contention increments]
    D --> E[Lurker sleeps, makes little progress]
    E --> F[Ban list grows]
    F --> G[Each lookup scans more bans]
    G --> H[Lookup latency rises]
    H --> A
    C -->|No contention| I[Lurker tests batch of objects]
    I --> J[Pause for ban_lurker_sleep]
    J --> B

Common causes

Cause	What it looks like	First thing to check
Lock contention under high traffic	`bans_lurker_contention` rising, `bans_lurker_tested` nonzero but slow, ban list growing	Rate of `MAIN.bans_lurker_contention`
Ban lurker disabled	`bans_lurker_tested` flat at zero, ban list growing	`varnishadm param.show ban_lurker_sleep` (0.000 means disabled)
All bans are req-level	`bans_lurker_tested` near zero, `bans_obj` at zero, bans persist	`varnishadm ban.list` for `req.*` patterns
`ban_lurker_age` masking activity	Lurker inactive right after a ban, then starts processing roughly 60s later	`varnishadm param.show ban_lurker_age`
Ban injection exceeds processing	`bans_added` rate consistently higher than `bans_deleted` rate	Compare delta rates over 60s

Quick checks

# Check ban list size and lurker counters
varnishstat -1 -f 'MAIN.bans*' -f MAIN.bans_dups

# Inspect the ban list (check for req.* vs obj.* patterns)
varnishadm ban.list

# Check lurker tuning parameters
varnishadm param.show ban_lurker_sleep
varnishadm param.show ban_lurker_age
varnishadm param.show ban_lurker_batch
varnishadm param.show ban_lurker_holdoff

# Check if ban lurker is active (take two readings 10s apart)
varnishstat -1 -f MAIN.bans_lurker_tested -f MAIN.bans_lurker_tests_tested

# Check the mix of obj vs req bans
varnishstat -1 -f MAIN.bans_obj -f MAIN.bans_req

# Check contention rate (take two readings 10s apart, compute delta)
varnishstat -1 -f MAIN.bans_lurker_contention

# Check whether bans are being completed
varnishstat -1 -f MAIN.bans_completed -f MAIN.bans_added -f MAIN.bans_deleted

How to diagnose it

Confirm the ban list is actually growing. Take two readings of MAIN.bans at least 60 seconds apart. If the count is stable, the lurker is keeping up. If it is climbing, proceed.
Check whether the lurker is doing any work at all. Look at MAIN.bans_lurker_tested and MAIN.bans_lurker_tests_tested. If both are flat at zero, the lurker is either disabled or every ban is req-level. Jump to step 4.
Check contention rate. If bans_lurker_tested is nonzero but the ban list is still growing, compute the rate of MAIN.bans_lurker_contention over 60 seconds. A high contention rate means the lurker is constantly yielding to lookup traffic. This is the core contention scenario.
Rule out a disabled lurker. Check ban_lurker_sleep. A value of 0.000 disables the ban lurker entirely. The default is 0.010 seconds. If someone set it to zero, that is your root cause.
Rule out req-only bans. Run varnishadm ban.list and inspect the ban expressions. The lurker can only process bans that match obj.* attributes. If every ban references req.* (for example, req.url), the lurker cannot help. Check MAIN.bans_obj vs MAIN.bans_req for a quick summary.
Check ban_lurker_age. The default is 60 seconds. The lurker will not test objects newer than this against bans. If you check bans_lurker_tested immediately after adding a ban, you will see zero activity. Wait at least 60 seconds before concluding the lurker is stuck.
Check the ban injection rate. Compare bans_added and bans_deleted rates over a 60-second window. If bans_added consistently outpaces bans_deleted, the application is invalidating faster than the lurker can clean up, regardless of contention.
For very large ban lists, raise CLI limits. The CLI truncates output by default after 48KB. If ban.list output is cut off, raise cli_limit temporarily to inspect the full list:
```
# Raise limit for inspection, then reset to default after
varnishadm param.set cli_limit 5000000b
```
Reset afterward: varnishadm param.set cli_limit 48k.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`MAIN.bans`	Active ban count. Each uncompleted ban adds lookup overhead.	Growing trend over minutes
`MAIN.bans_lurker_contention`	Lurker yielding locks to lookups. Rising rate means falling behind.	Rate climbing with traffic
`MAIN.bans_lurker_tested`	Lurker actively testing objects. Flat zero means disabled or req-only bans.	Zero while bans grow
`MAIN.bans_completed`	Bans fully processed and eligible for removal.	Stagnant while bans_added climbs
`MAIN.bans_added` vs `MAIN.bans_deleted`	Injection vs cleanup rate. Mismatch means list growth.	Added rate exceeds deleted rate
`MAIN.bans_obj` vs `MAIN.bans_req`	Whether bans are lurker-processable. req bans bypass the lurker entirely.	All bans are req-level
`MAIN.bans_dups`	Duplicate bans collapsed. Healthy deduplication.	Rising is normal and good
`MAIN.bans_lurker_obj_killed`	Objects evicted by the lurker. Confirms it is doing useful work.	Zero while bans grow and lurker is active

Fixes

Lower `ban_lurker_sleep` for contention-driven stall

The default ban_lurker_sleep is 0.010 seconds (10ms). The lurker sleeps this long between batches. Lowering it makes the lurker wake more frequently and process bans faster:

# Make the lurker more aggressive (default is 0.010)
varnishadm param.set ban_lurker_sleep 0.002

Tradeoff: the lurker competes more aggressively for locks, which can slightly increase lookup latency during processing. Monitor bans_lurker_contention after the change. If contention drops and the ban list starts shrinking, the tuning is working.

Do not set ban_lurker_sleep to 0.000. That disables the lurker entirely.

Increase `ban_lurker_batch` to amortize lock acquisition

The default batch size is 1000 objects. Raising it means the lurker does more work per lock acquisition cycle:

varnishadm param.set ban_lurker_batch 10000

Tradeoff: the lurker holds locks longer per cycle, which can increase contention spikes. Test this alongside ban_lurker_sleep tuning and watch the effect on bans_lurker_contention.

Rewrite req-level bans as obj-level bans

If ban.list shows bans referencing req.* attributes, the lurker cannot process them. Rewrite the VCL or application code to issue obj.* bans instead:

Instead of	Use
`ban("req.url ~ " + pattern)`	`ban("obj.http.x-url ~ " + pattern)`
`ban("req.http.host == " + host)`	`ban("obj.http.x-host == " + host)`

This requires storing the relevant request attributes as custom response headers in vcl_backend_response so they are available as obj.* attributes:

sub vcl_backend_response {
    set beresp.http.x-url = bereq.url;
    set beresp.http.x-host = bereq.http.host;
}

sub vcl_deliver {
    # Strip the internal headers so clients never see them
    unset resp.http.x-url;
    unset resp.http.x-host;
}

Then ban against obj.http.x-url instead of req.url. The lurker can process these proactively.

Switch to xkey or hash-based purging

If the application issues bans on every content update, the fundamental problem is that ban-based invalidation does not scale to high update rates. The xkey VMOD provides surrogate key purging: tag objects with secondary keys during vcl_backend_response, then purge by key without regex evaluation on the ban list.

Hash-based purging (using hash_data() in VCL and purging by hash) is another alternative that avoids ban list accumulation entirely. Both approaches sidestep the lurker bottleneck.

Use `ban_cutoff` as a safety net

ban_cutoff caps the effective ban list length. When the lurker reaches the ban_cutoff-th ban, it treats all objects as if they matched that ban and all older bans, evicting them from cache:

# Set a hard cap to prevent unbounded ban list growth
varnishadm param.set ban_cutoff 10000

This is a blunt instrument. Once the cutoff is reached, cached objects matching the cutoff ban and older are evicted, causing a temporary hit rate drop and backend load spike as objects are re-fetched. Use it as a safety valve, not a primary strategy.

Emergency: clear the ban list

If the ban list has grown so large that lookups are severely degraded, restarting the child process clears the ban list along with the cache. This is destructive: the cache empties and must warm up from scratch, sending full load to backends.

varnishadm stop
varnishadm start

This should be a last resort. Prefer the tuning and architectural fixes above.

Prevention

Monitor MAIN.bans and alert on growth. Set a threshold at a few hundred bans with a growing trend. Do not wait for tens of thousands before investigating.
Prefer obj.* bans or xkey-based purging. Regex bans on req.* attributes are the most common architectural cause of lurker stall.
Batch invalidations. If a CMS publishes multiple updates, coalesce bans into fewer, broader expressions rather than issuing one ban per URL.
Watch ban_lurker_age interactions. The 60-second default means the lurker ignores freshly cached objects. If your objects have very short TTLs, the lurker may never catch up because objects expire before they are old enough to be tested.
Track bans_dups as a health signal. Rising bans_dups is good: Varnish is collapsing duplicate bans before they add to the list.

How Netdata helps

Per-second ban counter collection. Netdata collects all MAIN.bans* counters at one-second resolution, so you can see ban list growth, lurker activity, and contention rate without manual varnishstat sampling.
Correlate ban list growth with cache hit latency. Overlay MAIN.bans against request processing time or backend request rate to confirm ban accumulation as the root cause of latency degradation.
Anomaly detection on lurker counters. Netdata’s ML-based anomaly detection flags unusual spikes in bans_lurker_contention or unexpected stagnation in bans_lurker_tested before the ban list grows to a problematic size.
Alerts on ban list growth. Custom alerts on the rate of change of MAIN.bans or the ratio of bans_added to bans_deleted catch lurker stall early.
Distinguish contention from req-only bans. Charting bans_obj vs bans_req alongside bans_lurker_tested quickly determines whether the lurker is blocked by contention or structurally unable to process the bans.