CoreDNS cache collapse: the cold-cache thundering herd after a rollout

You just rolled out a CoreDNS config change or image bump. Within seconds, DNS latency across the cluster jumps, upstream DNS traffic spikes to several times baseline, and SERVFAILs appear in application logs. One to five minutes later, it all goes away on its own. The dashboard is green again.

That is the cache collapse pattern: every CoreDNS pod restarted at roughly the same time, every in-memory cache emptied at once, and every client in the cluster re-queried the same names simultaneously. The flood hit your upstreams harder than they could absorb. In the worst version, upstreams return SERVFAIL, CoreDNS caches those SERVFAILs for 5 seconds each, and a brief overload amplifies into a visible outage.

This article covers how to confirm the pattern during and after the event, why the SERVFAIL cache amplifies it, and how to prevent it with rollout hygiene. For the broader signal taxonomy, see the CoreDNS monitoring checklist.

What this means

The CoreDNS cache plugin is an in-memory cache with separate positive (success) and negative (denial) caches. It lives inside each pod’s process. There is no shared cache between replicas: two pods behind the kube-dns Service each hold their own cache, and a restart drops it to zero entries.

When a rolling update replaces pods, each new pod starts cold. That alone is survivable: 100% cache miss after a single restart is expected and resolves in seconds to minutes as the cache warms. The failure mode appears when the rollout is too aggressive:

• maxUnavailable set too high (or a Recreate strategy), so all replicas go down and come up together.
• A manual restart of all pods at once.
• A ConfigMap change plus a reload pattern that flushes caches cluster-wide within the same minute.

The distinguishing feature is the signal ordering: cache-miss spike first, upstream-load spike second, latency and possibly SERVFAIL third. The event is temporally correlated with a deploy or restart, and it is transient unless the upstreams fold under the load.

flowchart TD
  A[Rollout: all CoreDNS pods restart together] --> B[All caches empty at once]
  B --> C[Clients re-query the same names simultaneously]
  C --> D[Upstream request rate spikes]
  D --> E{Upstreams absorb the flood?}
  E -->|Yes| F[Cache repopulates in 1-5 min, self-corrects]
  E -->|No| G[Upstreams return SERVFAIL]
  G --> H[CoreDNS caches SERVFAIL for 5s]
  H --> I[Cached SERVFAIL served to all clients, amplifying failure]
  I --> F

Common causes

Quick checks

All of these are read-only and safe to run during an incident. The curl commands assume you are exec’d into a CoreDNS pod or port-forwarding to its metrics port (kubectl port-forward -n kube-system <coredns-pod> 9153:9153).

# Confirm the temporal correlation: when did the CoreDNS pods last start?
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide

# Check the deployment rollout history
kubectl rollout history deployment/coredns -n kube-system

# Inspect the update strategy (maxUnavailable is the usual suspect)
kubectl get deployment coredns -n kube-system -o jsonpath='{.spec.strategy}'

# Cache state right now: entries should be near zero just after a collapse
curl -s http://localhost:9153/metrics | grep '^coredns_cache_entries'

# Cache hits vs requests: hit ratio craters during the event
curl -s http://localhost:9153/metrics | grep -E 'coredns_cache_(hits|requests)_total'

# Are upstreams failing health checks under the flood?
curl -s http://localhost:9153/metrics | grep 'coredns_proxy_healthcheck_failures_total'

# SERVFAIL responses, broken out by plugin
curl -s http://localhost:9153/metrics | grep 'coredns_dns_responses_total' | grep 'SERVFAIL'

These curl commands sample counters at one instant. The real confirmation comes from graphing them over the incident window: cache hit ratio dropping toward zero at the same moment upstream request rate spikes.

How to diagnose it

1. Establish the timeline. Get the pod start times and the deployment rollout history. The pattern requires a restart or reload event within a minute or two of the latency/SERVFAIL spike. If there is no restart event, this is not cache collapse; look at CoreDNS all upstreams down or CoreDNS slow upstream instead.

2. Confirm the cache emptied. coredns_cache_entries{type="success"} should show a drop to zero (restart) or a sharp decrease at the event time on every affected pod. The cache hit ratio (coredns_cache_hits_total / coredns_cache_requests_total) should crater toward 0% simultaneously.

3. Verify the ordering. The cache-miss spike must lead the upstream-load spike. If upstream latency or SERVFAIL rose before the cache emptied, the direction of causality is reversed: the upstream failed first and this is a different incident.

4. Check whether upstreams folded. Look at coredns_proxy_healthcheck_failures_total{to=...} and coredns_proxy_request_duration_seconds{to=...} per upstream. If upstreams held, the event self-corrects and your job is prevention. If they folded, check whether the forward plugin also started rejecting queries: coredns_forward_max_concurrent_rejects_total incrementing means the flood exceeded the forward plugin’s concurrency cap and clients got REFUSED on top of everything else. See CoreDNS forward max_concurrent rejects.

5. Check for SERVFAIL cache amplification. During the window, coredns_dns_responses_total{rcode="SERVFAIL"} includes responses served from the negative cache, not just fresh upstream failures. The tell: SERVFAILs continuing for several seconds after upstream health metrics recover. That is the 5-second default SERVFAIL cache at work.

6. Rule out a sizing problem. If coredns_cache_evictions_total was already climbing before the rollout, your cache was undersized and the collapse is partly chronic, not purely rollout-induced.

Metrics and signals to monitor

Fixes

If you are in the event right now

If upstreams are holding, the correct action is usually wait. The cache repopulates in 1-5 minutes and the event self-corrects. Restarting pods again just re-empties the caches and restarts the clock.

If upstreams are failing under the flood:

• Check per-upstream health with the to label. If one upstream is collapsing, consider temporarily removing it from the Corefile so traffic concentrates on the healthy ones. This is a disruptive change to a running resolver during an incident: apply it via ConfigMap and verify the reload took effect before declaring victory.
• Do not restart anything. Let the caches refill.
• If clients are being REFUSED due to max_concurrent, that is a capacity cap doing backpressure; the fix is upstream capacity or a higher limit, not more restarts.

Reduce the SERVFAIL amplification

CoreDNS caches SERVFAIL responses for 5 seconds by default. During a thundering herd, a one-second upstream hiccup becomes five seconds of cached SERVFAIL served to every client asking for that name. The cache plugin supports a servfail DURATION directive to change this; setting the duration to 0 disables SERVFAIL caching entirely. Tradeoff: disabling it means every retry during a real upstream outage goes to the already struggling upstream, so you trade client-visible errors for upstream load. A short nonzero value is the conservative middle ground. The servfail directive was introduced in CoreDNS v1.10.0.

Stop all replicas from restarting together

This is the root fix and belongs in Prevention below, but if your update strategy is maxUnavailable: 100% or a Recreate strategy, change it before the next rollout.

Prevention

• maxUnavailable=1. Only one CoreDNS pod should be down at a time during a rollout. Each remaining pod keeps serving its warm cache, and only a fraction of cluster traffic hits the one cold pod.
• PodDisruptionBudget. Set a PDB so voluntary disruptions (drains, upgrades) cannot take out more than one replica. This covers the cases the deployment strategy does not.
• Stagger manual restarts. If you restart CoreDNS manually (config change without the reload plugin, for example), delete pods one at a time and wait for each replacement to pass readiness and warm its cache before proceeding.
• Use lameduck for graceful shutdown. The health plugin’s lameduck DURATION keeps /health returning 200 for the duration while the pod shuts down, giving endpoints time to drain before the pod (and its cache) disappears. Endpoint propagation delays through kube-proxy and the CNI can still cause brief client timeouts even with lameduck configured.
• Consider serve_stale. The cache plugin’s serve_stale option (default 1 hour when enabled) serves expired entries while refreshing in the background. It does not survive a pod restart (the cache is in-memory), so it does not fix the cold-cache flood itself, but it blunts the impact of upstream SERVFAILs during recovery. Watch coredns_cache_served_stale_total if you enable it.
• Size the cache for the working set. If coredns_cache_evictions_total is nonzero at baseline, the cache is too small and every restart is more expensive than it needs to be.
• Prefetch popular entries. The cache plugin’s prefetch directive refreshes popular items before expiry, keeping hot names warm and reducing the miss burst after any disruption.
• NodeLocal DNSCache reduces but does not eliminate the risk. A node-local caching layer absorbs much of the re-query storm, but its caches start cold too. It reduces how many clients hit CoreDNS directly; it does not prevent the initial cache-miss surge.
• Keep CoreDNS current. Older releases had cache bugs that made cold starts worse: expired denial entries obscuring positive entries (fixed around v1.6.8/1.6.9) and a DO-bit cache-miss bug that doubled miss load for DNSSEC-enabled clients (fixed in v1.7.1). If you are on an old release, the thundering herd hits harder than it should.

How Netdata helps

• Netdata charts coredns_cache_hits_total and coredns_cache_requests_total per second, so the hit-ratio collapse and recovery are visible at the resolution this event actually happens at. One-minute aggregation can miss a 3-minute collapse entirely.
• The cache-entries gauge alongside pod restart events makes the temporal correlation with a rollout obvious on one screen.
• Per-upstream breakdowns (to label on proxy latency and health check failures) show whether upstreams absorbed the flood or folded, which decides whether you wait or intervene.
• SERVFAIL rates split by rcode and plugin expose the 5-second cached-SERVFAIL tail persisting after upstream health recovers, confirming the amplification loop rather than a continuing upstream outage.
• Anomaly detection on upstream request rate flags the flood signature (miss spike leading upstream spike) even when no static threshold was crossed.

Related guides

• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed
• How CoreDNS actually works in production: the plugin chain mental model
• CoreDNS monitoring checklist: the signals every production resolver needs
• CoreDNS monitoring maturity model: from survival to expert
• CoreDNS NOERROR with zero answers: the resolution failure that reports success
• CoreDNS NXDOMAIN vs SERVFAIL: why alerting on the wrong one buries real incidents
• CoreDNS per-upstream health check failures: degraded redundancy before total loss
• CoreDNS query rate dropped to zero while the process looks healthy
• CoreDNS returning REFUSED: no matching zone, an ACL, or the forward concurrency limit
• CoreDNS returning SERVFAIL: the resolver is failing queries and what to check first
• CoreDNS slow upstream: per-upstream latency, goroutine pileup, and the to label