CoreDNS serve_stale: keeping resolution alive while masking upstream failure

The serve_stale option in the CoreDNS cache plugin is a resilience feature with a monitoring side effect that bites teams during real incidents. When it is enabled, CoreDNS answers queries from expired cache entries instead of failing them when the upstream resolver is unreachable. Clients keep resolving names through an upstream outage. Dashboards stay green. The upstream can be dead for an hour before anyone notices, because the one metric that would tell you is almost never graphed.

This article explains how serve_stale behaves, why it masks upstream failure from end-to-end success metrics, and how to instrument it so the resilience does not turn into blindness. It assumes you understand the CoreDNS plugin chain and the cache plugin’s role in it. If not, start with How CoreDNS actually works in production.

What serve_stale is and why it exists

Without serve_stale, the cache plugin has a hard rule: an entry past its TTL is useless. A query for an expired name goes upstream, and if the upstream is down or slow, the client gets SERVFAIL or a timeout. During a full upstream outage, every cached name expires within minutes to an hour, and resolution then fails completely. The playbook’s “Upstream Black Hole” pattern describes exactly this: cache temporarily masks the failure, then all queries fail once TTLs expire.

With serve_stale enabled, the cache plugin keeps expired entries for a configurable window (default 1 hour past expiry) and serves them to clients when a fresh answer cannot be fetched quickly. The design follows unbound’s serve-expired behavior, not RFC 8767: it deliberately favors fast responses over answer correctness. An expired entry younger than the configured duration is served immediately, and a background refresh attempts to fetch a fresh answer from the upstream.

The operational trade is explicit: you accept that some answers may be outdated in exchange for resolution surviving upstream blips, slow upstreams, and short network partitions. For most external dependencies, a stale A record from 20 minutes ago is far better than SERVFAIL. The problem is not the feature. The problem is that it changes what your success metrics mean.

How it works

The configuration lives inside the cache plugin block in the Corefile:

cache 30 {
    serve_stale 1h
}

The syntax is serve_stale [DURATION] [REFRESH_MODE [VERIFY_TIMEOUT]], with two refresh modes:

immediate (the default): the stale entry is sent to the client right away, and a refresh to the upstream is dispatched in the background. Client latency stays at cache-hit speed even while the upstream is degraded.
verify: CoreDNS tries the upstream first and falls back to the stale entry only if the upstream does not answer. In earlier releases this could block the client for the full upstream timeout; recent releases added an optional verify timeout, for example serve_stale 1h verify 100ms, after which the stale entry is served and verification continues in the background.

Two behavioral details matter operationally. First, stale responses are served with a TTL of 0, so downstream resolvers and clients do not re-cache the stale answer. Second, the served-stale path emits exactly one signal: the counter coredns_cache_served_stale_total (labels server, zones, view). That counter is the entire observability surface of the feature.

flowchart LR
  Q[Client query] --> C{Cache entry fresh?}
  C -- yes --> H[Serve from cache]
  C -- expired --> U{Upstream reachable?}
  U -- yes --> F[Fetch fresh answer and update cache]
  U -- no or slow --> S[Serve stale entry, TTL 0]
  S --> M[coredns_cache_served_stale_total increments]
  S --> B[Background refresh retries upstream]

The success metric the client sees (coredns_dns_responses_total{rcode="NOERROR"}) does not distinguish the fresh path from the stale path. That is the crux of the masking problem.

Where the masking bites in production

The playbook calls out the general version of this trap in its “cache masks upstream failure” anti-pattern: when the cache is warm, an upstream failure is invisible in end-to-end success rates, and teams see “everything is fine” until the cache drains. serve_stale extends the masking window from “until TTL expiry” to “until TTL expiry plus the stale duration.” With the default 1-hour stale window, an upstream can be completely unreachable for over an hour while your NOERROR ratio stays at 99.9%.

Three concrete consequences:

Upstream health alerts tied to SERVFAIL stop firing. Without serve_stale, a dead upstream produces a SERVFAIL spike within minutes as entries expire. With serve_stale, SERVFAIL never comes, so any alert rule built on coredns_dns_responses_total{rcode="SERVFAIL"} stays silent for the duration of the stale window.
Latency dashboards look healthy. In immediate mode, stale serves are cache-speed responses. The “Upstream Black Hole” pattern’s distinguishing feature (SERVFAIL with low latency) never appears, and the “Slow Upstream Drag” pattern’s high P99 never appears either.
The failure surfaces as data staleness, not errors. Applications keep resolving, but to increasingly old answers. For records that change (failover IPs, traffic-shifted endpoints, recently rotated service addresses), this becomes a correctness incident that looks like an application bug.

There is also a deployment-specific caveat: the CoreDNS documentation recommends enabling serve_stale for custom forwarded zones, not for the server block that contains the kubernetes plugin. Headless service pod IPs change frequently, and serving expired pod IPs during an API or upstream hiccup can send traffic to dead pods. For Kubernetes-internal staleness failure modes, see the API disconnect pattern in CoreDNS query rate dropped to zero while the process looks healthy and the monitoring checklist.

Two historical bugs are worth knowing if you run older versions: a race between serve_stale and prefetch caused redundant upstream fetches, and stale negative-cache entries could mask a name that had started resolving. Both were fixed years ago. On current releases these are not concerns, but they explain confusing behavior reports from older clusters.

Tradeoffs and when to use it

Use it for external forwarded zones. SaaS endpoints, package registries, identity providers: for these, a 30-minute-old answer during an upstream blip is almost always better than an error.
Prefer immediate mode when client latency matters and you accept staleness. Prefer verify with an explicit timeout when correctness matters more but you still want a bounded fallback.
Do not use it on the Kubernetes server block for the reasons above. Stale pod IPs are worse than a fast failure.
Size the stale window deliberately. The 1-hour default is generous. Ask how stale an answer can be before it causes harm for the records you actually serve, and set the duration accordingly.
Treat it as a warning system, not a comfort blanket. Every stale serve means an upstream interaction failed. If the counter climbs for hours, you do not have resilience, you have an outage you have not paged on.

Signals to watch in production

Signal	Why it matters	Warning sign
`coredns_cache_served_stale_total` (rate)	The only direct measure of stale serving; each increment is a query that could not get a fresh answer	Any sustained nonzero rate; rising trend
`coredns_proxy_healthcheck_failures_total{to=...}`	Per-upstream health, independent of what clients see	Sustained increments for any upstream while stale serves rise
`coredns_forward_healthcheck_broken_total`	All upstreams failing health checks simultaneously	Any increment, especially alongside rising stale serves
`coredns_proxy_request_duration_seconds{to=...}`	Shows whether the upstream is slow (dragging) rather than dead	P99 climbing while stale serves rise
Cache hit ratio (`coredns_cache_hits_total` / `coredns_cache_requests_total`)	Stale serving inflates apparent cache effectiveness	Hit ratio looks stable or better while upstream health degrades
`coredns_dns_responses_total{rcode="SERVFAIL"}`	With serve_stale, absence of SERVFAIL proves nothing by itself	Flat zero during a known upstream event means the stale path absorbed it

The alert rule that matters most is simple: a rate of coredns_cache_served_stale_total greater than zero sustained over a few minutes is a ticket, and a steeply rising rate combined with coredns_forward_healthcheck_broken_total incrementing is a page. You are then in the “all upstreams down” incident, just with the client-facing symptoms delayed. The triage path in CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken applies; the only difference is that clients are not yet feeling it.

One limitation: the stale counter does not tell you whether the upstream was slow or fully down. There is no label for that. Correlation with the per-upstream health and latency metrics above is how you tell the difference, which is why monitoring upstream health independently of end-to-end success is non-negotiable when this feature is on.

How Netdata helps

Netdata collects the CoreDNS Prometheus endpoint and charts coredns_cache_served_stale_total alongside cache hits, request rates, and response codes, so a rising stale-serve rate is visible in the same view as the success metrics it is masking.
Per-second granularity catches short upstream blips that minute-resolution scraping averages away, which is exactly the timescale at which serve_stale activates.
Upstream health metrics (coredns_proxy_healthcheck_failures_total per upstream, coredns_forward_healthcheck_broken_total) sit next to the stale counter, making the slow-versus-dead distinction a two-chart correlation instead of a log dive.
Latency percentiles for forwarded queries are charted per upstream, so you can see a degrading upstream before the stale rate climbs.
ML-based anomaly detection on the stale counter flags the first deviations from the normal zero baseline rather than waiting for a static threshold.