CoreDNS query rate dropped to zero while the process looks healthy

The dashboard shows coredns_dns_requests_total flatlined at zero. You check the pod: Running, zero restarts. /health on port 8080 returns 200. The process is fine. So either nobody in the cluster is resolving names anymore, or the traffic is dying somewhere between your clients and the CoreDNS process, and CoreDNS has no idea.

The second possibility is the trap. CoreDNS only counts queries it actually receives. A UDP packet dropped by the kernel before it reaches the CoreDNS socket is never counted, never logged, and never reflected in any CoreDNS metric. A full conntrack table, an overflowing UDP receive buffer, a network policy isolating the pod, or a readiness failure that pulled the pod out of the Service endpoints all look identical in CoreDNS metrics: nothing.

There is a third reading of “zero QPS” that wastes on-call time: a genuinely idle server. A test cluster at 3 a.m., a newly deployed replica the load balancer has not picked up, or CoreDNS behind NodeLocal DNSCache serving almost no cache misses all produce the same flat line. The job is to tell those three situations apart quickly, then find which layer is eating the traffic.

What this means

coredns_dns_requests_total increments when a query enters the plugin chain. Everything before that point is invisible to CoreDNS: the client stub resolver, conntrack and DNAT through kube-proxy, the node’s UDP receive path, the Service endpoint list. If a packet dies in any of those hops, CoreDNS’s own metrics show a perfectly healthy, low-latency, zero-error server that happens to have no traffic. That is the “too good to be true” profile: latency low, errors zero, throughput suspiciously below expected demand.

Meanwhile /health only checks process liveness. It does not test DNS resolution, and it says nothing about whether packets are arriving. A pod can pass every probe while receiving zero queries.

flowchart LR
  C[Client pod] --> CT[conntrack / DNAT]
  CT -->|table full: silent drop| X1((dropped, not counted))
  CT --> SVC[kube-dns Service endpoints]
  SVC -->|pod not ready: removed| X2((no endpoints, not counted))
  SVC --> BUF[UDP receive buffer]
  BUF -->|overflow: RcvbufErrors| X3((dropped, not counted))
  BUF --> CD[CoreDNS plugin chain]
  CD --> M[coredns_dns_requests_total counted]

The diagnostic path is a walk from the client toward CoreDNS: find the first hop where traffic is still present, then the hop where it disappears.

Common causes

Quick checks

All read-only. Run these before touching anything.

# 1. Confirm what CoreDNS itself reports
curl -s http://localhost:9153/metrics | grep '^coredns_dns_requests_total'

# 2. Confirm the process health signals (both, not just /health)
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8080/health
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8181/ready

# 3. Check pod and readiness state, and whether the pod is in the Service endpoints
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide
kubectl get endpoints kube-dns -n kube-system

# 4. Check UDP receive buffer drops on the node (kernel-level, invisible to CoreDNS)
netstat -su | grep -i "buffer errors"
grep Udp /proc/net/snmp

# 5. Check conntrack pressure on the node
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
dmesg -T | grep "nf_conntrack: table full"

# 6. Send a real DNS query directly at the CoreDNS pod IP, bypassing the Service
dig @<coredns-pod-ip> kubernetes.default.svc.cluster.local +time=2 +tries=1

Two notes on interpretation. First, check 6 bypasses the Service VIP and kube-proxy DNAT entirely: if the direct query succeeds while clients fail, the problem is in the Service path (endpoints, conntrack, kube-proxy), not in CoreDNS or its socket. Second, run check 4 twice, a minute apart. A static nonzero counter is history; an incrementing counter is the current incident.

How to diagnose it

1. Establish the baseline before assuming an outage. Pull the rolling 24-hour or 7-day query rate for the same hour. If “zero” matches the baseline (idle test cluster, quiet period, NodeLocal DNSCache absorbing nearly everything), stop here. A true idle server reads exactly the same as an unreachable one and should not page you.

2. Check whether clients still experience DNS. If applications are resolving fine, traffic is going somewhere: another replica, the node-local cache, or a different resolver. Compare per-pod metrics. Aggregated averages hide a single starved pod; per-replica divergence is itself a finding.

3. Check readiness and endpoints. If /ready returns non-200 (for example plugin/ready: Still waiting on: "kubernetes" in the logs when the API server is unreachable), Kubernetes removes the pod from the kube-dns endpoints. The pod stays Running, /health keeps returning 200, and traffic drops to zero through the Service. kubectl get endpoints kube-dns -n kube-system should list every ready replica’s IP.

4. Rule the kernel in or out on the node. Incrementing RcvbufErrors in /proc/net/snmp means packets arrived at the node and were dropped before CoreDNS read them. nf_conntrack: table full, dropping packet in dmesg means conntrack exhaustion, which drops DNS and every other new flow on the node. Either finding explains “CoreDNS looks perfect but sees nothing.”

5. Test the data path directly. From a client pod, dig at the CoreDNS pod IP, then at the Service ClusterIP. Direct works, ClusterIP fails: the break is in the Service layer. Both fail: check network policies and node-level drops again. Both succeed but clients still time out: look at the client’s own resolver configuration and dnsPolicy, and at the conntrack race condition that produces intermittent 5-second stalls rather than a flat zero.

6. If CoreDNS is genuinely receiving traffic but the metric reads zero, suspect monitoring, not DNS: scrape target down, metrics port blocked, or an aggregator averaging across a scaled-to-zero replica set. Confirm by curling :9153/metrics on the pod directly.

Metrics and signals to monitor

Fixes

Conntrack table full

Emergency relief: raise the ceiling.

# Emergency: raise conntrack capacity (runtime only; persist via sysctl config or daemonset)
sysctl -w net.netfilter.nf_conntrack_max=262144

This is a node-level change affecting all traffic on the host; coordinate before applying it broadly. The durable fix is NodeLocal DNSCache, which bypasses the iptables DNAT path for DNS and eliminates conntrack pressure from DNS entirely. Also check for stale entries piling up from pod churn.

UDP receive buffer overflow

Raise the kernel socket buffers on the node:

# Increase UDP receive buffers (node-level, not namespaced)
sysctl -w net.core.rmem_max=26214400
sysctl -w net.core.rmem_default=26214400

Buffer increases buy headroom but do not fix a CoreDNS process that cannot drain the socket fast enough. If drops persist after raising buffers, check CPU saturation, CFS throttling, and GC pauses (go_gc_duration_seconds) on the pod.

Pod pulled from Service endpoints

Find out why readiness is failing before forcing anything. If the kubernetes plugin cannot reach the API server, restarting the pod will not help. Fix API connectivity, RBAC, or the network policy blocking it. Verify the readiness probe targets /ready on 8181, not /health on 8080. Using /health as the readiness probe is a known misconfiguration that routes traffic to pods before the kubernetes plugin has synced, producing the inverse symptom: traffic flowing to a pod that returns SERVFAIL for cluster names.

Network partition or policy

Identify the policy or rule blocking UDP/TCP 53 to the pod and correct the selector or rule. Kubelet probes often still pass in this state because the node-to-pod path is not blocked, which is exactly why the pod looks healthy.

Prevention

• Monitor what CoreDNS cannot see. Scrape node-level UDP RcvbufErrors, conntrack utilization, and per-replica query rates. These are the only signals that catch kernel-level drops.
• Alert on baseline deviation, not on zero. “Rate below 10% of the rolling baseline during active hours” catches real outages; “rate equals zero” pages you for idle test clusters.
• Compare per-replica. Two healthy-looking averages can hide one pod receiving no traffic. Alert on divergence between replicas.
• Probe /ready for readiness, /health for liveness. They answer different questions; conflating them creates both false traffic and false silence.
• Deploy NodeLocal DNSCache where conntrack exhaustion is recurring, and remember that it changes what CoreDNS QPS means afterward: low QPS at CoreDNS then reflects cache misses, not total demand.
• Keep a functional check. A periodic synthetic resolution from a client pod through the Service VIP exercises every hop that CoreDNS metrics do not.

How Netdata helps

• Per-second query rate per replica: coredns_dns_requests_total broken down per pod and per zone, so a single starved replica is visible instead of hidden inside a cluster average.
• Node-level correlation: UDP receive buffer errors and conntrack utilization from the same nodes, charted next to CoreDNS QPS, which is the correlation this incident hinges on.
• Baseline-aware anomaly detection: distinguishes “zero traffic that is normal for 3 a.m.” from “zero traffic that deviates from this hour’s history,” which is the difference between a page and a non-event.
• Composite context in one view: readiness state, SERVFAIL rate, and request latency alongside query rate, so “silent” versus “failing” versus “idle” is a glance, not a scavenger hunt.

Related guides

• How CoreDNS actually works in production: the plugin chain mental model
• CoreDNS monitoring checklist: the signals every production resolver needs
• CoreDNS monitoring maturity model: from survival to expert
• CoreDNS returning SERVFAIL: the resolver is failing queries and what to check first