$ guides / bind-dns / bind-dns-cold-cache-warming ▌

Operations Guides

BIND cold cache after restart: the warming storm and elevated upstream load

After a named restart, the recursive cache is empty. Every incoming query is a cache miss. On a resolver doing 50,000 qps, that means at least 50,000 outbound recursive queries per second to upstream authoritative servers – a 10x increase over steady-state outbound volume. This is the cache-warming storm, and it lasts 30 to 60 minutes.

Three things happen simultaneously: upstream load spikes, the recursive-clients table fills toward its limit, and cache hit ratio starts at zero and climbs. For the first 30 to 60 seconds, some queries may return SERVFAIL while root priming completes and authoritative zones finish loading. All of this is expected behavior.

The operational problem is alerting. Monitoring that does not account for cold starts will page on every restart: cache hit ratio alerts fire at 0%, SERVFAIL alerts fire during root priming, and recursive client alerts fire as the table fills. The fix is uptime gating on every alert that depends on cache state.

What happens during a cold start

BIND has no mechanism to persist its cache to disk and reload it on startup. rndc dumpdb produces a human-readable dump for manual inspection, but there is no corresponding load operation. Every restart produces a full cold cache.

The cold-start sequence proceeds in phases:

Process startup and zone loading (seconds to minutes). named starts, reads configuration, and loads authoritative zone data into memory. For servers with many large zones, validation and loading takes minutes. During this window, the server may not respond to queries at all.
Root priming (first 30 to 60 seconds). Root hint priming happens on demand when the first recursive query arrives that requires contacting a root nameserver. BIND forwards the client query to a root hint server while priming proceeds in parallel. Until priming completes, some queries return SERVFAIL.
RTT probing (seconds to minutes). After startup, BIND probes upstream nameservers to estimate round-trip times for server selection. Server selection is suboptimal during this phase and resolution latency is temporarily higher than steady state.
Cache warming (30 to 60 minutes). The cache fills as queries arrive and get resolved. Hit ratio climbs from zero toward steady-state levels. Outbound recursive query volume is far higher than normal, placing elevated load on upstream authoritative servers.

flowchart TD
    A["named restart"] --> B["Zone load\n(seconds to minutes)"]
    B --> C["Root priming\n(first 30-60s)"]
    C --> D["RTT probing\n(seconds to minutes)"]
    D --> E["Cache warming\n(30-60 min)"]
    C -.->|"SERVFAIL possible"| F["Brief query failures\nuntil priming completes"]
    E -.->|"Elevated upstream load"| G["Outbound QPS\napproaches inbound"]
    E -.->|"Slot pressure"| H["RecursClients fills\ntoward limit"]
    E --> I["Steady state:\nhit ratio normalizes"]

The warming storm: why restarts hit upstream hard

In steady state, a healthy resolver answers 80 to 95% of queries from cache, generating only 5 to 20% of inbound volume as outbound recursive queries. After a restart, that ratio inverts: outbound volume approaches or equals inbound volume because every query is a miss.

On a resolver doing 50,000 qps with a steady-state hit ratio of 90%, normal outbound volume is approximately 5,000 qps upstream. After restart, outbound volume jumps toward 50,000 qps. That is a 10x amplification of upstream load, and it persists until the cache reaches working-set coverage.

The amplification is worse for names with complex resolution chains. Domains with long CNAME chains and out-of-bailiwick nameserver names require multiple sequential queries per resolution. A single user query for a domain with a four-step CNAME chain spread across multiple TLDs can require dozens of upstream queries when the cache is cold. Each upstream query occupies a recursive-client slot for the duration of its resolution.

This is why RecursClients spikes after restart. Every cache miss generates one or more in-flight recursive queries. With recursive-clients defaulting to 1000, a high-traffic resolver can approach the soft quota (900) within seconds of startup. At the hard limit (1000), all new recursive queries receive SERVFAIL.

What is normal after restart

Several behaviors that look alarming are expected during the cold-start window.

SERVFAIL in the first 30 to 60 seconds. Root priming, zone loading, and managed-keys validation all happen at startup. Some queries fail during this window. It resolves on its own as priming completes.

Cache hit ratio at zero. The cache is empty. Hit ratio starts at zero and climbs over 30 to 60 minutes as the working set populates.

Elevated recursive client count. Every cache miss generates recursive traffic. On a high-traffic resolver, RecursClients approaches or temporarily exceeds the soft quota. This is the warming storm in progress, not a sign of upstream failure.

Suboptimal upstream server selection. BIND’s RTT estimates reset on restart. During the probing phase, queries may route to slower upstream nameservers until BIND learns which are fastest.

Memory growth. RSS climbs for 30 to 60 minutes as the cache fills. RSS should stabilize after the working set is cached. BIND’s internal allocator does not return freed memory to the OS efficiently, so RSS may remain elevated even after cache contents cycle.

What is NOT normal after restart

SERVFAIL persisting beyond 60 seconds. If SERVFAIL rate remains elevated after the first minute, something beyond root priming is wrong. Check DNSSEC managed-keys (rndc managed-keys status), zone load status in logs, and upstream reachability.

Cache hit ratio not recovering. If hit ratio stays near zero after 30 minutes, the cache is not warming. This indicates either a workload dominated by random subdomains (water torture attack) or a systematic problem with upstream resolution.

RecursClients pinned at the limit. Brief elevation during cache warming is expected. Sustained saturation beyond 5 minutes, combined with rising SERVFAIL, indicates a recursive resolution cascade that is not self-correcting.

Zones not loading. named starts successfully even if individual zones fail to load. Check logs for zone load errors after every restart. A server can report “running” via rndc status while serving REFUSED or SERVFAIL for specific zones.

Why alerts need uptime gates

Every alert that depends on cache state must gate on uptime. Without an uptime gate, every restart triggers a cascade of false pages.

Alert condition	Uptime gate	Rationale
SERVFAIL rate above 1%	Greater than 300 seconds	Root priming causes SERVFAIL in first 30-60s
RecursClients above 90% of limit	Greater than 300 seconds	Cold cache fills recursive slots immediately after restart
DNSSEC ValFail spike	Greater than 300 seconds	Key initialization produces transient validation noise
Cache hit ratio below baseline	Greater than 1800 seconds	Cache warming takes 30-60 minutes to reach steady state

The 300-second gate covers root priming and RTT probing. The 1800-second gate covers cache warming. For planned restarts, consider suppressing alerts during a maintenance window rather than relying solely on uptime gates. The uptime gate is the safety net for unplanned restarts, OOM kills, and crash recovery.

Handling restarts on high-traffic resolvers

On resolvers handling significant traffic, the warming storm can cause real upstream impact. Several strategies reduce the blast radius.

Stagger restarts in multi-resolver or anycast setups. Restarting all nodes simultaneously multiplies the upstream load. Restart nodes sequentially, allowing each to warm before restarting the next.

Consider pre-warming. There is no built-in cache pre-warming mechanism, but a script that sends a representative sample of production queries to the freshly restarted resolver can accelerate cache population. This does not eliminate the warming storm but shortens it by seeding the cache with the most common names. Rate-limit the pre-warm queries to avoid amplifying the upstream load you are trying to mitigate.

Monitor RecursClients during warming. If recursive clients hit the hard limit during warming, the resolver returns SERVFAIL for all new recursive queries. Temporarily increasing recursive-clients (with sufficient FD and memory headroom) can provide breathing room during the warm-up phase.

Reset statistics after warming. On BIND 9.21 and later, rndc reset-stats clears high-water counters, allowing clean measurement of steady-state performance without the cold-start peak distorting the baseline.

Signals to watch during cache warming

Signal	Why it matters	Expected pattern during warming
Cache hit ratio	Core warming indicator	Starts at 0%, climbs over 30-60 min
RecursClients	Pressure on recursive slots	Spikes immediately, declines as cache fills
Outbound query rate	Upstream load amplifier	Approaches inbound rate, then drops as hit ratio climbs
SERVFAIL rate	Distinguishes normal priming from real failure	Brief spike in first 60s, then near-zero
Process RSS	Memory growth from cache population	Climbs for 30-60 min, then stabilizes
QryRTT distribution	Upstream server selection quality	Suboptimal initially, improves as RTT estimates populate

The key correlation: if cache hit ratio is climbing and RecursClients is declining, the warming storm is progressing normally. If either stalls, investigate. A restart event followed by rising cache misses, climbing RecursClients, and a brief SERVFAIL spike is normal cold-start behavior. The same signals without recovery after 30 minutes point to a different problem.

How Netdata helps

Netdata’s per-second metrics make the warming curve visible in real time. During a cold start, correlate:

Cache hit ratio and cache misses per second: per-second resolution shows the warming curve in detail, making it clear when the cache has reached steady state versus stalled at a low level.
Recursive clients as a percentage of limit: tracking RecursClients against the configured recursive-clients limit shows whether the warming storm is approaching the circuit breaker threshold.
SERVFAIL rate correlated with process uptime: combining SERVFAIL metrics with uptime distinguishes root-priming noise in the first 60 seconds from real resolution failures that persist.
Outbound query rate: the ratio of outbound to inbound queries directly shows cache warming progress and upstream load amplification.
QryRTT distribution per view: RTT buckets show when BIND’s server selection has converged on optimal upstream nameservers after the post-restart probing phase.

These signals are most useful on a single timeline. The warming storm has a recognizable shape: cache misses spike, RecursClients surges, SERVFAIL blips briefly, then all three recover as the cache fills. Deviation from that shape is the diagnostic signal.