BIND cache eviction storms: DeleteLRU, an undersized max-cache-size, and the pressure spiral

DeleteLRU is rising fast while DeleteTTL barely moves. Cache hit ratio is dropping. Outbound recursive queries are climbing. RecursClients is trending toward its limit. This is a cache eviction storm: a self-reinforcing loop that can end in SERVFAIL for any query requiring recursion.

The cache is too small for the working set. BIND evicts entries by LRU before their TTLs expire. Each eviction forces a cache miss on the next query for that name, triggering an outbound recursive fetch. More fetches mean more concurrent recursive clients, higher latency per resolution, and less cache room as new entries from upstream compete with entries still under eviction pressure.

The diagnostic signal is the ratio of DeleteLRU to DeleteTTL. DeleteTTL counts entries removed by TTL expiry: normal lifecycle. DeleteLRU counts entries removed by memory pressure: the cache cannot hold the working set. DeleteTTL as the dominant eviction path is healthy. DeleteLRU dominating is the warning.

What this means

When DeleteLRU dominates DeleteTTL, the cache is thrashing. Every eviction is a future cache miss. The feedback loop:

flowchart TD
    A["max-cache-size too small
for working set"] --> B["Cache fills to memory ceiling"]
    B --> C["DeleteLRU evictions begin"]
    C --> D["Evicted entries cause
cache misses on requery"]
    D --> E["Outbound recursive queries spike"]
    E --> F["New entries arrive and
compete for cache space"]
    F --> C
    E --> G["RecursClients climbs"]
    G --> H["Latency increases per query"]
    H --> D
    G --> I["recursive-clients limit approached"]
    I --> J["SERVFAIL for new queries"]

The cache stabilizes at a bad equilibrium: depressed hit ratio, elevated outbound query rate, recursive-clients near saturation. On a busy resolver this looks like gradual degradation that operators often misattribute to upstream slowness or traffic growth. The upstream is not the problem. The cache is discarding entries it should still be serving.

Common causes

Quick checks

The commands below query BIND’s statistics-channel HTTP endpoint. BIND does not enable this by default; replace the port with your configured statistics-channels port.

# Dump raw JSON to confirm field names and structure for your BIND version
curl -s http://localhost:8653/json/v1/server | python3 -m json.tool | head -200

# Check DeleteLRU vs DeleteTTL ratio (the core diagnostic)
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  [print(f'{v}: DeleteLRU={cs.get(\"DeleteLRU\",0)} DeleteTTL={cs.get(\"DeleteTTL\",0)}') \
  for v,vd in d.get('views',{}).items() if (cs:=vd.get('resolver',{}).get('cachestats',{}))]"

# Check cache memory utilization and node count
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  [print(f'{v}: TreeMemInUse={cs.get(\"TreeMemInUse\",\"?\")} HeapMemInUse={cs.get(\"HeapMemInUse\",\"?\")} CacheNodes={cs.get(\"CacheNodes\",\"?\")}') \
  for v,vd in d.get('views',{}).items() if (cs:=vd.get('resolver',{}).get('cachestats',{}))]"

# Check configured max-cache-size (path may be /etc/bind/named.conf on Debian/Ubuntu)
named-checkconf -p /etc/named.conf 2>/dev/null | grep -i max-cache-size

# Check current cache hit ratio per view
# NOTE: CacheHits and CacheMisses are cumulative since process start. This computes
# lifetime ratio. To detect recent degradation, sample twice and compute deltas.
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  [print(f'{v}: Hits={cs.get(\"CacheHits\",0)} Misses={cs.get(\"CacheMisses\",0)} Ratio={cs.get(\"CacheHits\",0)/(cs.get(\"CacheHits\",0)+cs.get(\"CacheMisses\",1))*100:.1f}%') \
  for v,vd in d.get('views',{}).items() if (cs:=vd.get('resolver',{}).get('cachestats',{}))]"

# Check recursive client pressure (downstream effect of cache misses)
# TODO: in some BIND versions nsstats is a list of {name, counter} objects, not a flat dict.
# If this returns N/A, check the raw JSON for the actual structure.
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  print('RecursClients:', d.get('nsstats',{}).get('RecursClients', 'N/A'))"

# Check process RSS
awk '/VmRSS/{print $2/1024 " MB"}' /proc/$(pgrep -x named)/status

# Check whether named was recently restarted (cold cache possibility)
# Service name may be 'bind9' on Debian/Ubuntu
systemctl show named --property=NRestarts 2>/dev/null || echo "N/A"

How to diagnose it

1. Confirm DeleteLRU is the dominant eviction path. Pull cachestats at two points separated by a known interval and compute deltas. These are cumulative counters since process start. If the DeleteLRU delta greatly exceeds the DeleteTTL delta in the same window, the cache is evicting by memory pressure, not by TTL expiry.

2. Check cache memory against the configured ceiling. Compare TreeMemInUse and HeapMemInUse to max-cache-size. If the cache is at or near its ceiling, it has no room to absorb the working set. CacheNodes tells you how many entries are currently stored and whether that count is growing.

3. Verify the cache hit ratio has degraded. A hit ratio below 80% on a warm recursive resolver (more than 30 minutes uptime) is suspicious. Below 70% with rising DeleteLRU confirms thrashing. Use deltas over a short window, not lifetime cumulative values, to detect recent degradation.

4. Check recursive client pressure. If RecursClients is climbing alongside rising DeleteLRU and declining hit ratio, the eviction storm is feeding into a recursive resolution cascade. This is the stage where SERVFAIL becomes a real risk as recursive-clients approaches its limit.

5. Identify whether a traffic shift triggered the problem. Compare current query rate, QType distribution, and query name diversity against your baseline. A new set of clients querying different domains can push the working set past what the cache can hold. A random subdomain attack (water torture) also drives DeleteLRU because each unique query name enters the cache, fills it with useless entries, and evicts useful ones.

6. Rule out a cold cache. If uptime is under 30 to 60 minutes, the cache is still warming. DeleteLRU may spike as the cache fills to capacity and then stabilizes. Check restart history before treating this as a capacity problem.

Metrics and signals to monitor

Fixes

Increase max-cache-size

The most direct fix. If the cache cannot hold the working set, give it more room. But max-cache-size only bounds the cache and Address Database (ADB) memory contexts. Zone data, client state, RPZ datasets, and DNSSEC key material are unconstrained. Setting max-cache-size too high on a multi-purpose server risks OOM when other memory contexts grow.

Before increasing, verify the host has the memory headroom. BIND’s allocator does not return freed memory to the OS efficiently: RSS climbs and stays even after the cache subsequently shrinks. Plan for the high-water mark, not the steady state.

# Set max-cache-size in named.conf
# options {
#     max-cache-size 2g;
# };
# Then validate and apply:
named-checkconf /etc/named.conf && rndc reconfig

Reduce the working set

If you cannot increase cache size, reduce demand on it:

• Apply RPZ to block known-useless traffic such as telemetry to dead domains or scanning traffic. This prevents wasted cache entries that evict useful ones.
• Use forward zones for internal domains to a dedicated resolver, keeping the general-purpose cache focused on internet resolution.
• Split resolvers by client community so different working sets do not compete for the same cache space.

Flush the cache (temporary relief only)

rndc flush clears the cache entirely. This does not fix the underlying capacity problem: the cache will refill to the same ceiling and resume evicting. Use it only to break a stuck state or as a stopgap while preparing a max-cache-size change.

# WARNING: flushes entire cache, causing immediate cold-cache conditions
# All subsequent queries will miss cache until it warms again
rndc flush

Lower resolver-query-timeout (mitigation, not fix)

If the eviction storm has pushed RecursClients near saturation, lowering resolver-query-timeout causes stuck queries to fail faster, freeing recursive-client slots. This trades SERVFAIL for faster timeout, which may be preferable during acute pressure. It does not address the cache capacity problem.

Prevention

• Monitor DeleteLRU and DeleteTTL as rates, not raw counters. Both are cumulative since process start. A monitoring system must compute deltas over fixed intervals to detect when the eviction pattern shifts from TTL-dominated to LRU-dominated.
• Set max-cache-size explicitly on every recursive resolver. The default behavior is dangerous on multi-purpose servers where other processes compete for RAM, and may leave too little room for BIND’s own non-cache memory contexts (zones, ADB, client state) on dedicated hosts.
• Track TreeMemInUse and HeapMemInUse as trends against max-cache-size. When these approach the ceiling during daily peaks, you are one traffic shift away from thrashing.
• Watch cache hit ratio as a trend, not a static threshold. A gradual decline over weeks indicates the working set is growing faster than the cache can accommodate. Plan capacity before DeleteLRU becomes the dominant eviction path.
• Account for the ADB. The Address Database shares the max-cache-size memory budget. Resolvers that reach many unique upstream authoritative servers consume ADB memory that reduces effective cache capacity.
• Know your BIND version’s eviction model. BIND 9.18 uses a hybrid model: TTL-based cleaning still runs, but LRU eviction triggers when the cache approaches max-cache-size. The cleaning-interval option was removed and has no effect.

How Netdata helps

• Per-second cachestats collection means DeleteLRU and DeleteTTL rates are computed automatically, not left as raw cumulative counters. The eviction pattern shift is visible within seconds of the cache hitting its ceiling.
• Correlated cachestats, nsstats, and OS-level memory in a single timeline. When DeleteLRU spikes, immediately verify whether CacheHits dropped, RecursClients climbed, and RSS approached system limits.
• TreeMemInUse and HeapMemInUse trended over days and weeks reveal the cache approaching its ceiling before eviction becomes acute.
• Anomaly detection on cache hit ratio and RecursClients catches gradual degradation that fixed thresholds miss.

Related guides

• How BIND actually works in production: a mental model for operators
• BIND monitoring checklist: the signals every production resolver and authoritative server needs
• BIND monitoring maturity model: from survival to expert
• BIND recursive resolution cascade: one slow upstream taking down all resolution
• BIND RecursClients climbing toward the limit: reading the recursive saturation gauge
• BIND ’no more recursive clients: quota reached’: the recursive-clients circuit breaker
• BIND resolver NumFetch per view: per-view recursive pressure in split-horizon setups
• BIND NXDOMAIN spike: DGA malware, water torture, and Windows suffix search lists
• BIND clients-per-query and max-clients-per-query: duplicate recursion for popular names
• BIND forwarding loops: recursion that never terminates and burns recursive slots
• named not responding on port 53: total outage versus UDP-works-TCP-fails
• BIND REFUSED responses: ACL denials, recursion policy, and clients that get locked out