BIND cache hit ratio dropping: the leading edge of recursive pain

A dropping cache hit ratio is rarely the problem itself. It is the leading indicator that something is about to get worse. When fewer queries are answered from cache, each miss consumes a recursive-client slot, adds latency, and increases exposure to upstream slowness. On a busy resolver, a sustained hit-ratio decline from 95% to 80% can roughly triple outbound query volume and push recursive-client utilization into the danger zone.

The counter pair to watch is per-view CacheHits and CacheMisses in the BIND statistics channel cachestats. The ratio CacheHits / (CacheHits + CacheMisses) is what most monitoring systems alert on. Public recursive resolvers should sustain above 90% after warm-up. Alert with an uptime gate above 1800 seconds: cache warming takes 30 to 60 minutes after restart, and a cold cache naturally shows a near-zero hit ratio.

What this means

The cache-pressure spiral is self-reinforcing. Undersized cache produces low hit ratio, which drives more outbound queries, which adds latency, which fills recursive-client slots, which leads toward resource exhaustion. Once recursive-client slots approach the hard limit (default 1000), BIND starts rejecting new recursive queries with SERVFAIL.

BIND defines a formal soft quota at 90% of recursive-clients (900 with the default 1000). Above the soft quota, BIND aborts the oldest pending recursive query to make room for each new one and keeps accepting new queries; at the hard limit, new recursive queries are refused (SERVFAIL).

The real danger is the cascade. More outbound queries means more exposure to upstream timeouts, each of which holds a slot for the duration of the timeout (default 10 seconds). A resolver comfortably handling traffic at 95% hit ratio can hit the recursive-client wall at 75% if upstream latency degrades simultaneously.

flowchart LR
  A["Hit ratio falls"] --> B["More outbound queries"]
  B --> C["More concurrent fetches"]
  C --> D["Recursive slots consumed"]
  D --> E["Upstream slowness amplified"]
  E --> F["SERVFAIL cascade"]

Common causes

Quick checks

Safe, read-only commands. Adjust the statistics channel port (8653 here) to match your deployment.

# Per-view cache hit ratio
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  [print(f'{v}: Hits={cs.get(\"CacheHits\",0)} Misses={cs.get(\"CacheMisses\",0)} Ratio={cs.get(\"CacheHits\",0)/(cs.get(\"CacheHits\",0)+cs.get(\"CacheMisses\",1))*100:.1f}%') \
  for v,vd in d.get('views',{}).items() if (cs:=vd.get('resolver',{}).get('cachestats',{}))]"

# Uptime (suppress hit-ratio alerts below 1800s)
rndc status | grep -i uptime

# Per-view cache eviction and memory counters
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  [print(f'{v}: DeleteLRU={cs.get(\"DeleteLRU\",0)} CacheNodes={cs.get(\"CacheNodes\",0)} TreeMemInUse={cs.get(\"TreeMemInUse\",0)}') \
  for v,vd in d.get('views',{}).items() if (cs:=vd.get('resolver',{}).get('cachestats',{}))]"

# Recursive client pressure
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  print('RecursClients:', d.get('nsstats',{}).get('RecursClients', 'N/A'))"

# NXDOMAIN rate (attack indicator)
curl -s http://localhost:8653/json/v1/server | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
  print('QryNXDOMAIN:', d.get('nsstats',{}).get('QryNXDOMAIN',0))"

# max-cache-size configuration
named-checkconf -p /etc/named.conf 2>/dev/null | grep -i "max-cache-size"

# Process RSS for memory pressure
awk '/VmRSS/{print $2, $3}' /proc/$(pidof named)/status

# See which names are currently being recursed
rndc recursing | head -40

How to diagnose it

1. Gate on uptime. If uptime is below 1800 seconds, the low hit ratio is expected cache warming. Wait 30 to 60 minutes before investigating. Do not alert during this window.

2. Check per-view, not aggregate. Each view has its own cache. A drop in one view may be masked by another view’s healthy ratio if you only look at aggregate stats. Compare each view against its own baseline.

3. Check DeleteLRU. If DeleteLRU is non-zero and climbing, the cache has reached max-cache-size and is evicting entries by LRU. The cache is undersized for the working set. Cross-reference with TreeMemInUse to confirm the cache is at its memory ceiling.

4. Check NXDOMAIN patterns. If QryNXDOMAIN is spiking above 3x baseline, the hit-ratio drop may be caused by a random subdomain attack (water torture). Each unique query for a non-existent name is a cache miss that cannot benefit from caching. Near-zero repetition per unique name indicates an attack, not legitimate traffic.

5. Check recursive client count. If RecursClients is climbing toward 50% of the limit (default 1000), the hit-ratio decline is cascading into recursive pressure. Above 90% (900, the soft quota), BIND aborts the oldest pending recursive query to make room for each new query; at the hard limit (1000), new recursive queries are refused (SERVFAIL).

6. Check upstream RTT distribution. Use the per-view RTT bucket counters (QryRTT10, QryRTT100, QryRTT500, QryRTT800, QryRTT1600, QryRTT1600+). A shift toward higher buckets means upstream nameservers are slow, which amplifies the cost of each cache miss. Bucket names may vary in BIND 9.18+.

7. Check for recent events. Did someone run rndc flush? Did someone change max-cache-size? Did the resolver restart? Correlate the hit-ratio drop with BIND logs and configuration changes.

8. Verify BIND version. If you are running 9.18.x and memory grows faster than expected with aggressive eviction below expected thresholds, a version regression may be the cause.

Metrics and signals to monitor

Fixes

Cold cache after restart or flush

Expected behavior. Do not change cache size or configuration in response to a post-restart hit-ratio drop.

• Suppress hit-ratio alerts for the first 1800 seconds of uptime. Cache warming takes 30 to 60 minutes. Alerting during this window generates noise.
• On high-traffic resolvers, a cold start creates a cache warming storm. Every query triggers recursion, which can overwhelm upstream nameservers if the resolver normally handles tens of thousands of queries per second. Consider staggered restarts in anycast deployments.

Undersized max-cache-size

If DeleteLRU is non-zero and climbing, the cache is too small for the working set.

• Increase max-cache-size. The default is 90% of physical memory for views with recursion yes (2 MB for views with recursion no); this default was introduced in BIND 9.11. If it was explicitly set low, increase it. On multi-purpose servers, balance against other memory consumers.
• Verify the configuration took effect. Use named-checkconf -p to confirm the running configuration matches your intent.
• Check BIND version. If you are on 9.18.x, verify whether you are affected by a known cache-cleaning regression before assuming the cache is genuinely undersized.

Workload shift

A gradual decline with no eviction spike and normal NXDOMAIN rate may indicate a legitimate change in query patterns.

• Check QType distribution. A shift toward short-TTL domains (CDN steering records, DNS-based load balancing) naturally produces lower hit rates because entries expire faster.
• This is not necessarily a problem. If upstream load and latency are acceptable, the lower hit rate may be the new normal. If upstream load is a concern, add resolver capacity rather than fighting cache size.

Water torture attack

If QryNXDOMAIN is spiking and query name cardinality is high (near-zero repetition), the resolver is under a random subdomain attack.

• Identify the targeted domain from query logs or rndc dumpdb -cache analysis (the dump is written to the server’s working directory as named_dump.db unless dump-file is set).
• Apply RPZ to limit or refuse queries for the targeted domain.
• Enable rate-limit to throttle responses (RRL).
• See BIND NXDOMAIN spike: DGA malware, water torture, and Windows suffix search lists for full attack-response procedures.

BIND version regression

A cache-cleaning regression on the 9.18 branch caused memory to grow significantly faster than expected until max-cache-size was reached, triggering LRU eviction that depresses hit ratio. It was introduced by the fix for CVE-2023-6516 (GL #4383) and fixed in BIND 9.18.25 (GL #4596).

• Upgrade to 9.18.25 or later. This is a code fix, not a configuration change.

Prevention

• Alert on sustained hit-ratio drop with an uptime gate. Condition: ratio below rolling baseline by a meaningful margin (e.g., 15 percentage points), sustained for 15+ minutes, uptime > 1800 seconds.
• Monitor DeleteLRU alongside hit ratio. A non-zero DeleteLRU with a declining hit ratio signals an undersized cache.
• Track per-view, not aggregate. In split-horizon setups, one view’s cache failure can hide behind another view’s healthy ratio.
• Track cache memory utilization. Monitor TreeMemInUse and HeapMemInUse as a percentage of max-cache-size. Alert at 85%.
• Track RecursClients as a percentage of limit. This is the downstream effect of a falling hit ratio. If hit ratio drops and RecursClients climbs, the spiral has started.
• High hit ratio can serve bad answers. Stale or poisoned data in cache still produces hits. Correlate hit ratio with SERVFAIL rate and NXDOMAIN patterns to catch cache integrity issues.
• Validate BIND version after upgrades. Cache-related regressions have shipped in stable releases.

How Netdata helps

Netdata collects per-view CacheHits and CacheMisses from the BIND statistics channel and computes hit ratio per view automatically. For this symptom, the operational value is correlation:

• Hit ratio vs DeleteLRU: distinguishes a capacity problem (eviction is happening) from a workload shift (no eviction, just different queries).
• Hit ratio vs RecursClients: shows whether the decline has begun cascading into recursive pressure, before SERVFAIL starts.
• Hit ratio vs NXDOMAIN rate: flags water torture attacks in real time, distinguishing attack-driven miss inflation from organic cache misses.
• Per-view breakdown: prevents a split-horizon deployment from masking one view’s degradation behind another view’s health.
• Uptime tracking: enables the alert gate so hit-ratio alerts are suppressed during cache warming after restart.

Related guides

• BIND clients-per-query and max-clients-per-query: duplicate recursion for popular names
• BIND forwarding loops: recursion that never terminates and burns recursive slots
• How BIND actually works in production: a mental model for operators
• BIND monitoring checklist: the signals every production resolver and authoritative server needs
• BIND monitoring maturity model: from survival to expert
• BIND ’no more recursive clients: quota reached’: the recursive-clients circuit breaker
• named not responding on port 53: total outage versus UDP-works-TCP-fails
• BIND resolver NumFetch per view: per-view recursive pressure in split-horizon setups
• BIND NXDOMAIN spike: DGA malware, water torture, and Windows suffix search lists
• BIND recursive resolution cascade: one slow upstream taking down all resolution
• BIND RecursClients climbing toward the limit: reading the recursive saturation gauge
• BIND REFUSED responses: ACL denials, recursion policy, and clients that get locked out