CoreDNS SERVFAIL cache amplification: a one-second blip becomes a five-second outage

Your dashboards show an upstream DNS hiccup that lasted one second. Your users report an outage that lasted five. Both are accurate, and the gap between them is the CoreDNS cache plugin doing what it was designed to do.

CoreDNS caches SERVFAIL responses for 5 seconds by default. The cache plugin keeps separate positive (success) and negative (denial) caches, and SERVFAIL goes into the denial cache alongside NXDOMAIN and NODATA. The intent is sound: RFC 2308 permits caching server-failure responses to shield an already-struggling upstream from a retry storm. The side effect is that a single SERVFAIL answer for a hot record is served to every client that asks during the 5-second window, even after the upstream has fully recovered.

This article covers the amplification mechanism, how to confirm it is what you are seeing (rather than a genuinely longer upstream outage), and when tuning the SERVFAIL cache TTL is the right call.

What this means

The sequence:

1. An upstream resolver flaps for one second. A packet drop, a rate-limit kick, a brief overload.
2. A client queries a popular name. The forward plugin gets no usable answer and returns SERVFAIL.
3. The cache plugin stores that SERVFAIL in the denial cache with a 5-second TTL.
4. The upstream recovers one second later.
5. For the remaining four seconds, every client query for that record is answered from cache: SERVFAIL, instantly, without touching the upstream.

The amplification factor is the query rate for the affected name. At 1,000 QPS for a hot record, one upstream failure becomes roughly 5,000 SERVFAIL responses. Client-side retry logic does not rescue you: the system resolver (glibc) treats SERVFAIL as a final answer and does not retry, and applications with their own retry loops receive the cached SERVFAIL again. A one-second flap becomes a five-second outage for every client, and your SERVFAIL rate chart looks five times worse and lasts five times longer than the underlying fault.

flowchart TD
  A[Upstream flaps for 1 second] --> B[Client query for hot name]
  B --> C[Forward fails: SERVFAIL returned]
  C --> D[Cache stores SERVFAIL in denial cache, TTL 5s]
  D --> E[Upstream recovers at t=1s]
  E --> F{Another query, t=2s}
  F -->|denial cache hit| G[SERVFAIL served from cache]
  F -->|t > 5s, entry expired| H[Fresh forward: NOERROR]

There is a second trap during cache collapse events. After a rolling restart empties the cache, clients re-query everything at once. If the upstream stumbles under that flood, CoreDNS caches the resulting SERVFAILs for 5 seconds each, stretching a self-correcting thundering herd into something that looks like a sustained outage. The SERVFAIL cache does not cause the upstream failure, but it determines how long your users keep seeing it after the cause is gone.

Common causes

The amplification mechanism is always the same; what varies is the brief underlying failure that seeds the cached SERVFAIL.

If you are running a CoreDNS version old enough to still use the deprecated proxy plugin, the equivalent metric names are coredns_proxy_* instead of coredns_forward_*.

Quick checks

All read-only. Run these during or immediately after the event; the denial cache entries themselves expire in 5 seconds, so you are mostly working from counters. The commands assume the default metrics endpoint on port 9153.

# SERVFAIL responses, split by the plugin that generated them
curl -s http://localhost:9153/metrics | grep 'coredns_dns_responses_total' | grep 'SERVFAIL'

# Denial cache hits: are SERVFAIL/NXDOMAIN answers coming from cache?
curl -s http://localhost:9153/metrics | grep 'coredns_cache_hits_total'

# Did upstreams actually flap, and which one?
curl -s http://localhost:9153/metrics | grep 'coredns_forward_healthcheck_failures_total'
curl -s http://localhost:9153/metrics | grep 'coredns_forward_healthcheck_broken_total'

# What does the Corefile cache block say right now?
kubectl get cm -n kube-system coredns -o yaml        # Kubernetes
# or: cat /etc/coredns/Corefile                      # standalone

Two things to note in the output. First, the plugin label on SERVFAIL responses tells you whether forward produced them (upstream problem) or another plugin did. Second, coredns_cache_hits_total{type="denial"} covers NXDOMAIN, NODATA, and SERVFAIL together; you cannot separate SERVFAIL cache hits from ordinary negative-cache hits by metric alone. In Kubernetes, most denial hits are search-domain NXDOMAIN noise, so treat a denial-hit spike as corroborating evidence, not proof.

How to diagnose it

1. Establish the true upstream outage window. Look at coredns_forward_healthcheck_failures_total and per-upstream latency over the incident window. If health checks barely moved but SERVFAIL spiked, the upstream fault was shorter than the visible impact.

2. Check the SERVFAIL tail. Plot the SERVFAIL rate second by second (or as finely as your scrape interval allows). The signature of cache amplification is a sharp onset followed by a cliff-edge drop roughly 5 seconds after the upstream signal recovers. A genuine ongoing upstream failure decays messily or not at all.

3. Compare client-facing reports with the upstream timeline. If users report a 5-second failure but upstream health checks show 1 second of trouble, cache amplification is the bridge. If both show 5 seconds, go investigate the upstream instead. See CoreDNS returning SERVFAIL: the resolver is failing queries and what to check first.

4. Confirm the cache is involved. Denial cache hit rate climbing during the SERVFAIL window, while forward traffic to the affected upstream stays flat, means queries are being answered from the denial cache rather than re-forwarded.

5. Check for the cache-collapse variant. If the event followed a restart, reload, or rolling update, look at coredns_cache_entries dropping to zero and upstream request rate spiking. The SERVFAILs are secondary; the root cause is the cold-cache thundering herd.

6. Read the Corefile. Confirm whether the cache block sets a custom denial TTL or a servfail directive. A denial TTL set far above the default makes the amplification window much longer than 5 seconds.

Metrics and signals to monitor

On alerting: SERVFAIL alone is not page-safe, and cache amplification is one reason why. Transient flaps produce SERVFAIL ratios that self-resolve within seconds. Page only on composite rules: SERVFAIL corroborated by upstream failure evidence, sustained for more than 5 minutes, and not during a cold-start window. For the distinction between SERVFAIL and benign negative answers, see CoreDNS NXDOMAIN vs SERVFAIL: why alerting on the wrong one buries real incidents.

Fixes

Tune the SERVFAIL cache TTL

Since CoreDNS v1.9.4, the cache plugin accepts a servfail DURATION directive that overrides the 5-second default. Setting servfail 0 disables SERVFAIL caching entirely. The TTL is capped at 5 minutes per RFC 2308.

. {
    forward . 10.0.0.2 10.0.0.3
    cache 30 {
        servfail 1s
    }
}

Tradeoffs, stated plainly:

• servfail 0 or a very low TTL: clients see upstream recovery almost immediately, and retry loops start working again. The cost is that during a genuine upstream outage, every query for a failing record re-forwards and re-fails, adding load to an upstream that is already down and latency to every client. On a hot record at high QPS, that can be the difference between an upstream that recovers and one that stays down.
• Keep the 5-second default: hot records are shielded, upstreams get breathing room during real outages, but every brief flap is multiplied by five for every client.
• A middle value (1-2s): most of the retry-storm protection, most of the fast recovery. Reasonable for clusters where upstreams flap briefly but often (cloud provider DNS rate limiting is the classic case).

Before v1.9.4 there is no knob: SERVFAIL caching is hardcoded at 5 seconds, and the only way to stop it is to disable the cache plugin entirely, which is almost never worth it.

Fix the flap, not the cache

Tuning the TTL shrinks the blast radius of brief failures, but the failures themselves have a cause. If per-upstream health checks show recurring short failures on one upstream, replace or remove it. If the pattern is rate limiting (AWS VPC DNS resolver limits per ENI are a documented trigger), spread CoreDNS across nodes, add upstreams, or deploy a node-local cache. See CoreDNS per-upstream health check failures: degraded redundancy before total loss.

Prevent the cold-cache amplifier

If your SERVFAIL bursts follow rollouts, the fix is rollout hygiene, not cache tuning: stagger restarts, set maxUnavailable=1, and use PodDisruptionBudgets so the cluster never cold-starts its entire cache at once. A Corefile reload flushes the cache too, so config changes carry the same thundering-herd risk.

Be careful combining serve_stale with SERVFAIL caching

With serve_stale enabled, CoreDNS versions before v1.14.4 could prefer a cached SERVFAIL in the denial cache over a still-valid positive entry, hiding upstream recovery until the stale window expired. v1.14.4 changed the lookup to prefer the positive entry. If you run serve_stale and observe recovery delays longer than 5 seconds, check your CoreDNS version first.

Prevention

• Alert on composites, not raw SERVFAIL. Require upstream failure evidence plus duration. The 5-second amplification guarantees that brief flaps produce scary-looking spikes that resolve on their own.
• Dashboard the denial cache separately. A denial-hit spike overlapping a SERVFAIL spike is your fastest confirmation of amplification during an incident.
• Know your Corefile. Record whether servfail, custom denial TTLs, and serve_stale are set. During an incident, the configured value is the expected tail length; if the observed tail does not match it, the cache is not your problem.
• Remove flaky upstreams aggressively. One upstream that flaps for a second every few minutes generates a permanent low-grade SERVFAIL background at 5x amplification.
• Stagger anything that clears the cache. Rollouts, reloads, and config pushes are the other multiplier.

How Netdata helps

Netdata surfaces the specific correlations that make this failure mode recognizable in seconds rather than after a postmortem:

• SERVFAIL responses split by plugin and zone, so you can tell a forward-plugin failure (upstream origin) from other sources without grepping metrics by hand.
• Per-second SERVFAIL rate against upstream health check failures on the same timeline, which makes the amplification signature (SERVFAIL tail outlasting the upstream blip) directly visible.
• Denial cache hits and cache hit ratio alongside forward traffic, so you can see SERVFAILs being served from cache instead of re-forwarded.
• Per-upstream latency breakdown, catching the slow-then-fail pattern where latency leads the cached SERVFAIL wave.
• Cache entries dropping to zero correlated with pod restarts, identifying the cold-cache thundering-herd variant before you tune the wrong knob.
• Anomaly detection on response codes, flagging SERVFAIL bursts that are short enough to slip under ratio-based alert thresholds but frequent enough to indicate a flapping upstream.

Related guides

• CoreDNS returning SERVFAIL: the resolver is failing queries and what to check first
• CoreDNS NXDOMAIN vs SERVFAIL: why alerting on the wrong one buries real incidents
• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• CoreDNS per-upstream health check failures: degraded redundancy before total loss
• CoreDNS monitoring checklist: the signals every production resolver needs
• How CoreDNS actually works in production: the plugin chain mental model