CoreDNS per-replica divergence: why averaged metrics hide a failing pod

Your CoreDNS dashboard looks slightly off. Average latency drifted up, SERVFAIL ratio is elevated but under the threshold, and nobody is being paged. Meanwhile, half the cluster’s DNS queries are slow or failing, because one of your two replicas is degraded and kube-proxy is still sending it traffic.

This is the per-replica divergence failure mode. CoreDNS in Kubernetes is almost never a singleton: the default deployment runs two replicas behind the kube-dns ClusterIP Service, and kube-proxy load-balances across them. Any aggregation that averages or sums across pods blends a sick replica’s numbers with healthy ones until the result looks like mild degradation instead of a partial outage.

The fix is not better thresholds on the aggregate. It is keeping per-pod series and alerting on divergence between replicas.

What this means

CoreDNS Prometheus metrics are per-process. Each replica exposes its own counters and histograms on its own :9153/metrics endpoint. Nothing inside CoreDNS knows about its sibling replica.

When you query average request duration or total SERVFAIL rate across all pods, you get the mean of two populations. If replica A answers at 2 ms and replica B at 400 ms, the average is 201 ms: bad, but often under the alert threshold, even though 50 percent of queries are taking 400 ms. The same math applies to error ratios, cache hit rates, and goroutine counts.

Two structural facts make this worse:

• kube-proxy keeps sending traffic to a bad endpoint. Service load balancing removes an endpoint only for failing readiness, never for being slow. And readiness is a weak gate: neither /health nor /ready issues a real DNS query. A pod can be Ready and returning SERVFAIL for every lookup.
• Stale routing after termination. When a CoreDNS pod terminates, kube-proxy updates its rules, but traffic can keep flowing to the dead pod IP through stale iptables/IPVS rules or stale UDP conntrack entries on the node. Those queries vanish. The surviving replica’s metrics stay green, and the dead pod emits nothing at all. The aggregate only shows a small throughput dip.

flowchart LR
  client[Client pods] --> svc[kube-dns ClusterIP]
  svc --> kp[kube-proxy rules]
  kp -->|healthy share| a[Replica A - healthy]
  kp -->|degraded share| b[Replica B - slow or failing]
  a --> agg[Averaged dashboard]
  b --> agg
  agg -->|looks slightly off| alert[No alert fires]

Common causes

Quick checks

All commands are read-only. Run them against each replica individually, not through the Service.

# List CoreDNS pods and their nodes
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide

The CoreDNS container image is scratch-based: there is no shell, wget, or curl inside the pod, so kubectl exec will not work. Use kubectl port-forward to scrape one specific pod:

# Forward the metrics port of ONE pod (repeat per replica; Ctrl-C when done)
kubectl port-forward -n kube-system pod/<coredns-pod-a> 9153:9153 &

# Latency sum/count for this pod only
curl -s localhost:9153/metrics | grep -E '^coredns_dns_request_duration_seconds_(sum|count)'

# SERVFAIL responses per pod
curl -s localhost:9153/metrics | grep 'coredns_dns_responses_total' | grep SERVFAIL

# Per-pod cache state
curl -s localhost:9153/metrics | grep -E 'coredns_cache_(hits|requests)_total'

# Per-pod goroutines and heap
curl -s localhost:9153/metrics | grep -E '^go_goroutines|^go_memstats_heap_inuse_bytes'

# Readiness of each pod independently
kubectl port-forward -n kube-system pod/<coredns-pod-a> 8181:8181 &
curl -s localhost:8181/ready

# Recent restarts (a restarted pod explains a cold cache; restart loops explain QPS imbalance)
kubectl get pods -n kube-system -l k8s-app=kube-dns

Also check the nodes hosting CoreDNS pods for the infra-level causes that never appear in CoreDNS metrics:

# UDP receive buffer errors on the node (packets dropped before CoreDNS sees them)
netstat -su | grep -iE 'rcvbuferrors|receive errors'

# Conntrack pressure
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
dmesg | grep "nf_conntrack: table full"

How to diagnose it

1. Confirm divergence exists. Take any latency, error, or QPS query you normally run and re-run it grouped by pod. If you do not have a pod or instance label on your CoreDNS series, that is the first thing to fix: the server label on CoreDNS metrics is the listen address (both pods typically report the same value, for example dns://:53), so it cannot distinguish replicas. Your scrape configuration must attach pod identity, for example via Prometheus kubernetes_sd_configs with role: pod and relabeling the pod name onto a label.

2. Check QPS symmetry first. rate(coredns_dns_requests_total[5m]) per pod. Replicas behind a ClusterIP should carry roughly equal share over time. A pod carrying far less traffic may be out of the endpoints list, or traffic may be routed somewhere dead (stale kube-proxy rules or stale conntrack entries pointing at a terminated pod IP). A pod carrying far more may be saturating alone.

3. Compare error ratios per pod, not absolute counts. A pod serving half the traffic will naturally have half the SERVFAILs. Compare SERVFAIL / total responses within each pod. Also check the plugin label on coredns_dns_responses_total per pod: SERVFAILs from the forward plugin point at upstreams, from the kubernetes plugin at API state.

4. Compare latency distributions per pod. Use histogram_quantile over coredns_dns_request_duration_seconds grouped by pod. One pod with elevated P99 and normal P50 suggests upstream slowness or GC pressure on that pod; elevated P50 suggests systemic saturation (CPU, throttling) on its node.

5. If one pod is sick, classify the cause. Work through the causes table: node-level infra (UDP buffer errors, conntrack), upstream path (per-pod coredns_forward_healthcheck_failures_total and coredns_forward_request_duration_seconds{to=...}), API watch (per-pod coredns_kubernetes_rest_client_requests_total by code, plus pod logs), cold cache (pod age vs coredns_cache_hits_total / coredns_cache_requests_total). Note: older CoreDNS versions exported the forwarding metrics as coredns_proxy_* instead of coredns_forward_*; check which names your version exposes.

6. If both pods look clean but clients still time out, suspect the routing layer. Check whether a CoreDNS pod was recently terminated. Stale kube-proxy rules or stale UDP conntrack entries can keep sending queries to the dead pod IP; the surviving replica reports green while a fraction of queries disappear. Verify with client-side evidence: application DNS latency histograms, or a synthetic lookup loop from a pod on the affected node.

Metrics and signals to monitor

The rule for every signal below: keep the per-pod series and alert on divergence between replicas, not on the cluster-wide mean.

For the alert itself: compare each pod against the replica set, not against a static threshold. A practical pattern is alerting when a pod’s error ratio or P99 exceeds the median of its peers by a meaningful multiple for a sustained window. Exact PromQL depends on your label schema, so validate it against your own series before relying on it.

Fixes

One replica degraded by its node

Move the pod: delete it and let the Deployment reschedule, ideally onto a different node. Use pod anti-affinity so replicas never share a node; a node-level problem (conntrack exhaustion, CPU contention, UDP buffer pressure) should not be able to take both replicas at once. Then fix the node cause itself, for example raising UDP receive buffers or conntrack limits.

Stale routing to a terminated pod

Short term, flushing the stale conntrack entries on affected nodes stops the bleeding. This is disruptive: deleting conntrack entries kills the flows they track, so scope it to the dead pod IP and UDP only, and expect brief disruption to matching traffic:

# Disruptive. Match only the dead pod IP as reply source, UDP only.
conntrack -D -p udp --reply-src <dead-pod-ip>

Longer term, keep kube-proxy and the kernel current (conntrack cleanup behavior has had known regressions), and consider NodeLocal DNSCache, which removes the NAT/conntrack hop for DNS entirely.

One replica with a broken API watch

Restart the pod to force a fresh re-list, then find out why the watch broke: API server health, RBAC, or a NetworkPolicy blocking CoreDNS to the API server. If you restart as the first move without capturing logs, you lose the evidence.

Cold cache after a rolling update

Not a bug, but a self-inflicted divergence: stagger rollouts with maxUnavailable=1 and a PodDisruptionBudget so you never flush both caches at once. See the related guide on cache collapse.

Scrape configuration

If you cannot see per-pod series at all, fix the scrape: per-pod scraping with pod identity labels, never scraping through the kube-dns Service (that hits one random backend and gives you a load-balanced sample of the truth). This is the highest-leverage fix in this article because everything else depends on it.

Prevention

• Alert on replica divergence for error ratio, P99 latency, and QPS share, in addition to any aggregate alerts.
• Per-pod dashboards by default. Any CoreDNS panel that shows an average should have a per-pod breakdown next to it.
• Anti-affinity across nodes for CoreDNS replicas, so node failures and node-local resource pressure cannot degrade both.
• Staggered rollouts with maxUnavailable=1 and a PDB.
• Synthetic freshness and resolution checks per replica where possible, since readiness does not test DNS answers.
• Client-side DNS latency visibility. CoreDNS processing time excludes kernel buffer waits, throttling, conntrack drops, and network transit; the application-observed number is the one that catches routing-layer divergence.

How Netdata helps

• Per-instance series preserved: Netdata keeps CoreDNS metrics broken out per pod/instance rather than collapsing them into a cluster aggregate, so a single degraded replica stays visible as its own line instead of diluting into the mean.
• Replica-to-replica divergence at a glance: plotting coredns_dns_requests_total, SERVFAIL responses, and request duration per instance on the same chart makes QPS imbalance and latency splits obvious within seconds.
• Correlation across layers on one node view: when one replica is slow because of its node, Netdata’s node-level metrics (UDP buffer errors, conntrack utilization, CPU throttling) sit next to the CoreDNS charts, so the “pod or node?” question gets answered without switching tools.
• ML anomaly detection per series: anomaly scoring runs on each per-pod series independently, so a replica drifting away from its own baseline gets flagged even when the aggregate stays inside static thresholds.
• Per-second granularity: brief divergence windows during rollouts or upstream flaps are not averaged away by coarse scrape intervals.

Related guides

• CoreDNS 5-second DNS timeout: the Kubernetes glibc A+AAAA conntrack race
• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• CoreDNS cache collapse: the cold-cache thundering herd after a rollout
• CoreDNS cache evictions: the cache is too small for the working set
• CoreDNS cache hit ratio dropping: latency and upstream load climbing together
• CoreDNS conntrack table full: silent UDP packet drops with a node-wide blast radius
• CoreDNS Corefile parse error: startup failures and invalid plugin config
• CoreDNS CPU throttling: CFS limits making a green dashboard lie about latency
• CoreDNS CrashLoopBackOff: triaging loop, OOM, config, and port-bind causes
• CoreDNS DNS programming latency: how long a Service takes to become resolvable
• CoreDNS not resolving external domains: the missing catch-all forward zone
• CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed