CoreDNS serving stale cluster DNS: why you need a functional freshness test

Your CoreDNS dashboards are green. Query rate normal, latency sub-millisecond, SERVFAIL near zero, cache hit ratio healthy. Meanwhile, a Service deleted an hour ago still resolves, and the new Service your team just deployed returns NXDOMAIN. Applications are failing, but nothing in your monitoring fired.

This is the silent stale data pattern. The kubernetes plugin builds its DNS records from a watch on the API server. If that watch disconnects and cannot re-establish, CoreDNS keeps answering from its in-memory snapshot. The snapshot drifts from reality. No errors go back to clients. No performance metric moves, because the server is genuinely healthy and fast. The data is simply wrong.

Standard metrics cannot catch this because they measure how well CoreDNS answers, not whether the answers are current. The only reliable detector is a synthetic check that changes something in the Kubernetes API and verifies CoreDNS reflects the change. This article explains the failure mechanism, how to confirm it during an incident, and how to build a functional freshness probe that pages you when DNS programming lags reality.

What this means

The kubernetes plugin does not query the API server per DNS request. It maintains a persistent watch (an HTTP/2 stream) and builds an in-memory record set from the events that stream delivers. Serving DNS and receiving updates are two independent paths. The serving path can be perfect while the update path is dead.

When the watch drops silently:

• New Services and EndpointSlices never appear in DNS.
• Deleted Services keep resolving indefinitely.
• Scaled-down pods keep appearing in headless Service answers.
• coredns_dns_requests_total, latency histograms, cache metrics, and RCODE distributions all look normal.

There is no binary “watch broken” metric in CoreDNS. The coredns_kubernetes_dns_programming_duration_seconds histogram comes closest, but it has two fatal gaps for this failure: it only measures events that actually arrive (a dead watch delivers no events, so the histogram goes flat and quiet), and it currently only works reliably for headless_with_selector services. A silent full disconnect produces no anomaly in it at all.

flowchart LR
  API[Kubernetes API server] -->|watch events| K[kubernetes plugin snapshot]
  K -->|answers from snapshot| CL[cluster clients]
  NET[network partition / RBAC / API overload] -.->|watch drops silently| K
  K -->|stale answers, no errors| CL
  M[standard metrics] -->|all green| OPS[operator]
  PROBE[synthetic freshness probe] -->|create Service, verify resolution| API
  PROBE -->|detects staleness| OPS

Common causes

One version note: CoreDNS before 1.7.0 had a tombstone-handling bug where delete events received after a watch reconnection were silently dropped, so Services deleted during an API outage stayed in DNS indefinitely. This was fixed in 1.7.0. The same release removed the resyncperiod option, which never actually re-listed from the API server anyway. If you are running anything older than 1.7.0, upgrade before doing anything else. On current versions the same mechanism still applies: a watch that never reconnects, or reconnects and misses events, leaves stale data until the pod restarts.

Quick checks

These are all read-only or create a trivial throwaway Service.

# 1. Look for watch/list errors in CoreDNS logs (the only place this failure leaves a trace)
kubectl logs -n kube-system <coredns-pod> --tail=200 | grep -iE "watch|list|error|retry|disconnect"

# 2. Check API request health by status code
kubectl exec -n kube-system <coredns-pod> -- wget -qO- http://localhost:9153/metrics \
  | grep 'coredns_kubernetes_rest_client_requests_total'

# 3. Test API reachability from inside the pod
kubectl exec -n kube-system <coredns-pod> -- \
  wget -qO- https://kubernetes.default.svc.cluster.local/healthz --no-check-certificate

# 4. Functional freshness test: create a Service and see if it resolves
kubectl create service clusterip test-dns-delay --tcp=80:80
nslookup test-dns-delay.default.svc.cluster.local <coredns-ip>
kubectl delete service test-dns-delay

# 5. Negative test: does a Service you deleted earlier still resolve?
nslookup <deleted-service>.<ns>.svc.cluster.local <coredns-ip>

# 6. Check programming duration (helps only for headless_with_selector services)
kubectl exec -n kube-system <coredns-pod> -- wget -qO- http://localhost:9153/metrics \
  | grep 'coredns_kubernetes_dns_programming_duration'

Step 4 is the decisive check. If a freshly created Service does not resolve within a few seconds, or a deleted Service still resolves in step 5, you have stale data regardless of what the dashboards say.

How to diagnose it

1. Confirm the symptom is staleness, not a resolution failure. A new Service returning NXDOMAIN could also be a readiness or zone-matching issue. Staleness is confirmed when step 5 above (deleted Service still resolving) reproduces. Old data persisting is the signature.

2. Check the logs. Watch disconnect and re-list failures are logged by the kubernetes plugin. This is one of the few CoreDNS failures where logs, not metrics, are the primary evidence.

3. Check the rest client metric by code. Sustained 5xx or connection errors on coredns_kubernetes_rest_client_requests_total corroborate API-side trouble. 403 points at RBAC. Absence of recent WATCH activity is itself suspicious.

4. Test connectivity from the pod. If wget to the API healthz endpoint fails, the problem is the network path or a NetworkPolicy, not the API server.

5. Check every replica. With two CoreDNS pods behind kube-dns, only one may have a broken watch. Half your queries get fresh answers and half get stale ones, which makes application symptoms maddeningly intermittent. Run the freshness test against each pod IP individually, not just the ClusterIP.

6. Estimate the drift window. CoreDNS logs and the rest client metric tell you roughly when the watch died. Everything created, deleted, or rescaled since then is suspect. This matters for the post-incident sweep.

Metrics and signals to monitor

Note what is deliberately absent: query rate, latency, cache hit ratio, SERVFAIL rate. They stay green through this entire failure, and including them as detectors would be noise.

Fixes

Restart the affected pod

Restarting CoreDNS forces a full re-list from the API server and rebuilds the snapshot from current state. This is the one case where a restart is the actual fix, not a superstition: there is no runtime command to force reconciliation, and CoreDNS has no periodic full-resync mechanism.

# Restart only the replica confirmed stale (from diagnosis step 5)
kubectl delete pod -n kube-system <stale-coredns-pod>

Tradeoff: the re-list in a large cluster is a memory spike and an API load burst. If the pod was already near its memory limit, the restart can trigger an OOM loop, so check headroom first. Restart one replica at a time and verify freshness on each before moving on.

Fix the underlying cause

A restart restores fresh data, but the watch will die again if the root cause persists:

• NetworkPolicy: allow CoreDNS pods egress to the API server on 443.
• RBAC: restore the ServiceAccount’s watch/list permissions on Services, EndpointSlices, and (if used) Pods.
• API overload: that is a control plane incident of its own; stale DNS is a downstream symptom.

Post-incident sweep

List everything that changed in the cluster during the drift window and verify each one resolves correctly now. Deployments made during the window may have pods that cached the bad answer or failed at startup; some workloads need a nudge to recover.

Building a functional freshness probe

This is the prevention that actually matters. The probe answers one question: does a change I just made in the Kubernetes API show up in CoreDNS answers within N seconds?

Design decisions:

• Create-based, not read-based. Reading a known Service only proves the snapshot contains it, not that it is current. Creating and deleting a throwaway Service exercises both the add path and the delete path. The API overhead of one tiny Service per minute is negligible.
• Probe every replica individually. Query each CoreDNS pod IP directly. Probing through the kube-dns ClusterIP load-balances across replicas and can mask one stale pod for hours.
• Measure programming lag, not just pass/fail. Record the seconds from Service creation to first successful resolution. A rising trend warns you before hard failures start.

Run it as a CronJob (or a small always-running Deployment) every 30 to 60 seconds:

# Probe loop, run per CoreDNS pod IP
kubectl create service clusterip dns-freshness-probe --tcp=80:80 -n kube-system
START=$(date +%s)
# Rely on the nslookup exit code: it is non-zero on NXDOMAIN.
# Do not grep the output for "Address"; the server line matches even on failure.
while ! nslookup dns-freshness-probe.kube-system.svc.cluster.local <coredns-pod-ip> >/dev/null 2>&1; do
  sleep 1
  if [ $(( $(date +%s) - START )) -gt 30 ]; then
    echo "FAIL: probe Service not resolving after 30s"; break
  fi
done
echo "Programming lag: $(( $(date +%s) - START ))s"
kubectl delete service dns-freshness-probe -n kube-system
# Follow-up: verify the deleted probe name stops resolving (delete path)

Alerting policy:

• Page on 2 to 3 consecutive probe failures against any single replica. One failure can be a transient API hiccup; three in a row means the watch is dead.
• Ticket when programming lag exceeds 30 seconds without failing outright, matching the severity threshold for coredns_kubernetes_dns_programming_duration_seconds.
• Pair probe failures with coredns_kubernetes_rest_client_requests_total error codes before paging, to distinguish “CoreDNS watch broken” from “whole API server down” (the latter pages via its own alerts).

If you already run Prometheus Blackbox Exporter you can execute the DNS half of the check with its DNS probe module, but the create/delete step still needs something with a Kubernetes client. Keep both halves in one job so the lag measurement is end-to-end.

Prevention

• Deploy the freshness probe. This is the non-negotiable item. Every other measure on this list reduces frequency; only the probe guarantees detection.
• Alert on rest client errors. Sustained 5xx or any 403 on coredns_kubernetes_rest_client_requests_total is a ticket before staleness sets in.
• Monitor per replica. Divergence between replicas (one pod’s programming duration rising, one flat) is an early staleness indicator.
• Log watch events somewhere searchable. During this incident, logs are the primary evidence. Make sure CoreDNS container logs are retained long enough to reconstruct the drift window.
• Keep CoreDNS current. The post-disconnect tombstone bug was fixed in 1.7.0; running anything older turns every API blip into permanent stale deletes.
• Review NetworkPolicy and RBAC changes as a class. Any change touching kube-system egress or the CoreDNS ServiceAccount should trigger a freshness probe run as a post-deploy check.

How Netdata helps

• Netdata charts coredns_kubernetes_rest_client_requests_total broken down by HTTP status code and method, so a watch going quiet or flipping to 403s is visible as a pattern shift, not buried in an aggregate.
• coredns_kubernetes_dns_programming_duration_seconds percentiles are graphed per service_kind, making programming-lag regressions on headless services visible before the probe starts failing.
• Per-pod metric collection keeps replica divergence visible: one CoreDNS pod with a dying watch shows up as that pod’s API activity diverging from its peers rather than being averaged away.
• Correlating rest client errors with CoreDNS logs and pod restarts in one timeline shortens the “is it the network, RBAC, or the API server?” triage loop.
• Netdata’s anomaly detection on the rest client request rate catches the subtle version of this failure, where watch traffic fades out gradually rather than erroring.

Related guides

• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• CoreDNS cache collapse: the cold-cache thundering herd after a rollout
• CoreDNS cache evictions: the cache is too small for the working set
• CoreDNS cache hit ratio dropping: latency and upstream load climbing together
• CoreDNS CPU throttling: CFS limits making a green dashboard lie about latency
• CoreDNS not resolving external domains: the missing catch-all forward zone
• CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed
• CoreDNS GC pauses adding tail latency: go_gc_duration_seconds and heap pressure
• CoreDNS goroutine count climbing: blocked upstream calls and leaks
• CoreDNS high request latency: reading P99 by zone to find the cause
• How CoreDNS actually works in production: the plugin chain mental model
• CoreDNS memory climbing: heap growth, post-GC minima, and leak detection