Varnish purge vs ban vs xkey: choosing an invalidation method that scales

Varnish offers three mechanisms for invalidating cached objects: purge, ban, and the xkey VMOD (surrogate-key invalidation). Each has a fundamentally different cost model. Purge is O(1) and immediate but works only on a single exact hash. Bans are expression-based and flexible but accumulate in a list that every cache lookup must evaluate. xkey operates on secondary key indexes and avoids list growth entirely, but the open-source implementation has known scaling limits.

The choice matters most under load. An application that issues one ban per content update seems harmless until the ban list reaches tens of thousands of entries and every cache hit becomes an O(n) scan. A CMS that purges by individual URL works fine with dozens of pages but breaks when it needs to invalidate every object associated with a template, author, or category. This article covers the cost model of each method at the cache-object level, why per-URL bans are the anti-pattern that drives teams toward xkey, and which Varnish counters reveal when your invalidation strategy is hurting cache performance.

How each invalidation method works

Purge: exact hash, immediate removal

Purge, triggered by return(purge) in vcl_recv when a PURGE request arrives, removes the single object matching the request’s hash. The hash is composed of whatever vcl_hash assembles, typically the Host header plus the URL. Purge discards the object and all its Vary variants immediately. It is O(1): one hash lookup, one object removal, no lingering state.

Purge is the right tool when you know the exact URL or hash components of the object to invalidate. It leaves no trace, adds no overhead to future lookups, and does not depend on any background thread to complete.

The limitation is scope. If you need to invalidate every object derived from a particular database record, and those objects live under different URLs, purge requires you to enumerate every URL and issue a separate PURGE request for each one.

Ban: expression-based, lazy, accumulates

A ban adds an expression to an in-memory ban list. The expression can match object metadata (obj.* variables such as obj.http.x-tag) or request properties (req.* variables such as req.url or req.http.host). Bans are evaluated lazily: when a ban is inserted, Varnish does not scan existing cached objects. Instead, every cache lookup tests the found object against all outstanding bans.

The ban lurker is a background thread that walks the object store testing objects against obj.* bans. When the lurker finds a match, it evicts the object. This is asynchronous and spreads the cost across idle periods. Bans using req.* variables cannot be processed by the lurker because req.* only has meaning in the context of a live client request. Those bans are checked synchronously on every cache hit and persist in the list until every cached object has been tested against them through the lookup path.

The cost model is O(n x m) where n is the number of objects and m is the number of outstanding bans. The ban list grows until the lurker has verified that every object older than the oldest ban has been tested. If bans are added faster than the lurker processes them, the list grows without bound.

xkey: surrogate-key, tag-based

The xkey VMOD (vmod_xkey from varnish-modules) provides surrogate-key invalidation. Objects are associated with one or more secondary keys (tags) set via response headers. When you call xkey.purge("tag"), all objects sharing that key are invalidated. xkey.softpurge() marks matching objects as expired but extends their grace and keep timers, so stale content is still served while the backend refreshes.

xkey avoids the ban list entirely. It maintains an index of secondary keys to objects. Invalidation is a lookup in that index, not a linear scan of the ban list. This means invalidation cost does not grow with the number of outstanding invalidations.

The tradeoff is operational. xkey is a VMOD, not a core Varnish feature. It must be imported and configured in VCL. Keys are registered when objects are inserted, so objects cached before import xkey; was present in VCL cannot be purged via xkey. The open-source implementation also has documented performance limitations under high insert or purge rates.

flowchart TD
    A["Object needs invalidation"] --> B{"Method?"}
    B --> C["Purge: exact hash"]
    B --> D["Ban: expression"]
    B --> E["xkey: surrogate key"]
    C --> C1["Immediate, O(1)
No list, no lurker"]
    D --> D1["obj.* bans: lurker
processes asynchronously"]
    D --> D2["req.* bans: checked on
every cache hit, lurker skips"]
    D1 --> D3["List grows until all
objects tested - O(n x m)"]
    D2 --> D3
    E --> E1["Key index lookup
All tagged objects removed
No list growth"]

The per-URL ban anti-pattern

The most common scaling failure with Varnish invalidation is per-URL bans. An application or CMS issues a ban matching a specific host and URL path, using req.* variables:

ban("req.http.host == ... && req.url == ...")

This combines the worst properties of both other methods. It targets a single exact URL, which is exactly what purge does in O(1). But it uses req.* variables, so the ban lurker cannot process it. The ban sits in the list and is tested on every cache lookup until every object in the cache has been checked against it. If the application issues one such ban per content update and updates happen frequently, the ban list grows continuously and every cache hit pays the price.

The fix is straightforward. For exact-URL invalidation, use purge. For group-based invalidation (all objects associated with a category, template, or dependency), use xkey surrogate keys. Reserve bans for genuine pattern-matching use cases, and prefer obj.* variables so the lurker can process them asynchronously.

When to use which method

A practical decision flow:

• One URL, known exactly. Use purge (return(purge)).
• Multiple related URLs sharing a tag. Use xkey with surrogate keys set on the backend response.
• Genuine pattern match across unknown URLs. Use ban with obj.* variables only, and monitor the ban list.
• Never. Per-URL bans using req.*. This is always the wrong tool.

xkey scaling limits and maintenance mode

As of August 2024, vmod_xkey is officially in maintenance mode. The project notice states that the implementation has known scalability problems that will not be addressed, and recommends Varnish Enterprise’s ykey VMOD as the production-grade replacement for high-scale tag-based invalidation.

The specific scaling problems are worth understanding if you depend on xkey:

• Locking contention on busy sites. xkey piggybacks on the expiry data structure’s mutexes. Under high insert rates or frequent purges, lock contention can degrade performance on sites with heavy cache churn.
• Objects inserted before xkey import cannot be purged. If Varnish started with VCL that did not include import xkey;, objects cached during that period have no secondary key index entries and are invisible to xkey purge. This is a real problem when config management deploys xkey VCL after Varnish has already started with boilerplate configuration.
• Persisted cache reindexing on restart. With MSE (Massive Storage Engine, Varnish Enterprise), restarting Varnish with persisted objects requires xkey to reindex every object one by one. For caches with millions of objects, this means millions of disk operations on restart before xkey is functional.

For open-source Varnish Cache, there is currently no direct replacement for xkey’s tag-based invalidation. Teams that need group invalidation on open-source Varnish have two practical options: use bans with obj.* variables and carefully monitor the ban list, or restructure the application to issue individual purges for each URL in the group.

Signals to watch in production

When invalidation volume spikes unexpectedly, identify the source:

# Check current ban list size and contents
varnishadm ban.list

# Monitor for ban activity in real time
varnishlog -q 'VCL_Log ~ "ban" or CLI ~ "ban"'

# Monitor for PURGE method requests
varnishlog -q 'ReqMethod eq "PURGE"'

If your VCL handles PURGE or BAN methods, verify that it enforces ACL checks on client.ip, not on X-Forwarded-For headers, which can be spoofed. Without ACL enforcement, any network actor can invalidate cache content.

How Netdata helps

• Per-second MAIN.n_purges and MAIN.bans_added rates. A sudden spike in either counter, correlated with a drop in MAIN.cache_hit and a rise in MAIN.backend_req, pinpoints mass invalidation as the cause of a backend load surge rather than a VCL change or TTL expiry.
• MAIN.bans as a gauge with trend tracking. Anomaly detection flags sustained growth in the ban list before cache lookup latency degrades visibly to users.
• MAIN.bans_lurker_contention correlation. When the lurker is losing lock contention, the ban list grows and cache hit latency increases. Seeing both signals on the same dashboard makes the connection immediate.
• Invalidation-to-hit-rate overlay. Overlaying purge and ban counters against cache hit ratio and backend request rate reveals whether your invalidation strategy is the root cause of a hit rate drop, as opposed to a VCL change, storage pressure, or TTL expiration.
• MAIN.n_obj_purged tracking. When purge volume is high, this counter confirms how many objects were actually removed, distinguishing a broad purge from a narrow one and helping estimate the cache warmup cost that follows.

Related guides

• Varnish ban list growing: O(n) lookups and the lurker falling behind
• Varnish cache hit ratio dropped: hit rate collapse and backend overload
• Varnish cache stampede: a popular object expires and the herd hits the backend
• Varnish grace masking a backend outage: the ticking-clock incident
• Varnish backend is sick: health probes, all-backends-sick, and grace