CoreDNS monitoring checklist: the signals every production resolver needs

CoreDNS has one failure mode that defeats most monitoring setups: a pod that passes every health probe while returning SERVFAIL for every query. The /health endpoint on port 8080 checks process liveness. It does not test DNS resolution. The /ready endpoint on port 8181 is plugin-aware, but it does not test resolution either. Teams that alert on pod status and probe results are monitoring the wrong thing.

The actual availability signal for a DNS server is the RCODE distribution of its responses, primarily coredns_dns_responses_total{rcode="SERVFAIL"}. Everything else exists to explain and predict changes in that signal.

This checklist organizes the signals into four maturity levels, from “know there is a fire” to “catch the failure modes that only show up after your third incident.” Use it to audit an existing monitoring setup or to build one. Each level assumes the levels below it are in place.

flowchart TD
  L1["Level 1: Survival
process alive, port 53, metrics endpoint, SERVFAIL"]
  L2["Level 2: Operational
QPS by zone, latency P99, upstream health, cache hit ratio"]
  L3["Level 3: Mature
per-upstream latency, memory and GC, evictions, reloads, panics, FDs"]
  L4["Level 4: Expert
API errors, programming duration, conn cache, security types, node-level"]
  L1 --> L2 --> L3 --> L4

How to use this checklist

Two disciplines apply at every level:

Alert on RCODE metrics, not health probes. A “healthy” pod can fail every query. Probes are a liveness input, never an availability verdict.
Alert on SERVFAIL and REFUSED, never on NXDOMAIN. NXDOMAIN is a normal response, especially in Kubernetes where search domain expansion generates it constantly. Alerting on total errors or absolute NXDOMAIN counts creates alert fatigue and buries real SERVFAIL spikes in noise.

Also note what CoreDNS metrics cannot see. UDP packets dropped by the kernel (full receive buffers, full conntrack table) never reach the process, so CoreDNS can report low latency, zero errors, and suspiciously low throughput while clients time out. Levels 1-3 are CoreDNS’s view of the world. Level 4 includes the node-level view.

Level 1: survival

The minimum to know CoreDNS is alive and answering. If you have only these, you will know there is a fire but will have no diagnostic capability.

Process and pod liveness. Why: a dead resolver is an immediate outage for every client that depends on it. In Kubernetes, watch pod status and restart count; CrashLoopBackOff with “Loop detected” in logs means a forwarding loop in the Corefile, and OOMKilled means the memory limit is too small for the working set.
Port 53 listening, UDP and TCP. Why: a running process that failed to bind serves nothing. Check with ss -ulnp | grep ':53' and ss -tlnp | grep ':53', or a functional query: dig @<coredns_ip> . NS +time=1 +tries=1.
Metrics endpoint reachable on port 9153. Why: every other signal in this checklist comes from here. curl -s http://localhost:9153/metrics should return Prometheus output.
Any SERVFAIL responses. Why: coredns_dns_responses_total{rcode="SERVFAIL"} is the real user pain signal. Any nonzero rate in production warrants investigation. A forwarding loop never shows up here: the loop plugin calls log.Fatalf before metrics exist, so loops appear as CrashLoopBackOff, not as SERVFAIL.

# Level 1 spot check
kubectl get pods -n kube-system -l k8s-app=kube-dns
dig @<coredns_ip> . NS +time=1 +tries=1
curl -s http://localhost:9153/metrics | grep 'coredns_dns_responses_total' | grep 'SERVFAIL'

One caution on paging: process death or CrashLoopBackOff is PAGE. A raw SERVFAIL blip is not. SERVFAIL responses are cached for 5 seconds by default, so a one-second upstream hiccup extends into a short SERVFAIL burst that self-resolves. Page only on composite conditions: sustained SERVFAIL ratio corroborated by upstream failure evidence.

Level 2: operational

The working set for a competent production team. This level covers the primary failure modes: upstream loss, latency degradation, and cache problems.

Query rate by zone. Why: coredns_dns_requests_total with its zone label is your baseline and your blast-radius indicator. A drop toward zero with a healthy process means packets are dying before they reach CoreDNS (conntrack, UDP buffers, network partition). A spike means a retry storm, search domain amplification, or an attack. In Kubernetes, the zone label separates cluster.local problems from forwarded-zone problems.
SERVFAIL ratio, not just count. Why: SERVFAIL / total responses sustained above roughly 1% over 5 minutes (with meaningful traffic) is degraded. The plugin label on coredns_dns_responses_total tells you which plugin generated the failure: forward points at upstreams, kubernetes points at the API watch or cluster state.
Request latency P99. Why: coredns_dns_request_duration_seconds is user-perceived responsiveness. Cache hits should be sub-millisecond to single-digit milliseconds. P99 above 100ms sustained in a mostly-cached workload is a ticket; above 500ms is strong degradation. High P99 with normal P50 means slow upstreams or GC pauses; elevated P50 means systemic trouble. Split by zone: high latency in cluster.local implicates the Kubernetes plugin path, high latency in forwarded zones implicates upstreams.
All upstreams down. Why: coredns_forward_healthcheck_broken_total increments when every configured upstream fails health checks simultaneously. Any increment is a ticket. Two gotchas: with a single upstream it is flap-prone, and by default CoreDNS still tries a random unhealthy upstream, so this counter alone does not prove total failure unless failfast_all_unhealthy_upstreams is set.
Per-upstream health check failures. Why: coredns_proxy_healthcheck_failures_total{proxy_name="forward", to=...} identifies which upstream is failing so you can pull it before it drags everything. The name is proxy-subsystem since CoreDNS 1.11.0; older releases emit coredns_forward_healthcheck_failures_total.
Cache hit ratio. Why: coredns_cache_hits_total / coredns_cache_requests_total determines upstream load and most of your latency profile. Kubernetes clusters typically see 80%+ for repeated service names. A sudden drop means cold cache (restart), eviction pressure, TTLs too short, or a shift to unique-name traffic. Do not alert during the first minutes after a restart; a cold cache is expected. (coredns_cache_misses_total is deprecated; derive misses as requests minus hits.)

Level 3: mature

Signals that catch degradation before users do, and that explain the Level 2 alarms.

Per-upstream latency breakdown. Why: coredns_proxy_request_duration_seconds{proxy_name="forward", to=...} separates “CoreDNS is slow” from “one upstream is slow.” A single slow upstream degrades only the queries routed to it; aggregates hide this. P99 above 250ms for an upstream is a ticket.
Memory: RSS against the container limit. Why: OOM kill is a cliff edge with no graceful degradation, and the post-restart re-list spike can immediately OOM the pod again. Use process_resident_memory_bytes (what the OOM killer sees) against the container limit: 80% is warning, 90% is critical. Use go_memstats_heap_inuse_bytes for leak detection: watch the post-GC minimum trend, not instantaneous peaks. Rising post-GC minimums plus lengthening go_gc_duration_seconds pauses is the classic pre-OOM pattern.
Goroutine count trend. Why: go_goroutines tracks in-flight work. Sustained growth past 2x baseline with flat QPS means blocked upstream calls or a leak. After a load spike, the count should return to baseline; if it does not, suspect a leak.
Cache eviction rate. Why: coredns_cache_evictions_total rising means the cache is too small for the working set. Evictions force upstream queries that cache should have absorbed, raising latency and upstream load gradually rather than all at once. Correlate with coredns_cache_entries sitting at the configured maximum.
Reload failures. Why: coredns_reload_failed_total nonzero means a Corefile change failed to apply and the old config is still running. DNS keeps working, but the running configuration no longer matches what the operator thinks is deployed. That drift is the risk.
Panic count. Why: coredns_panics_total (no dns in the name) must be zero. Any increment is a recovered crash in a query handler: the process survived, but that query got no response. Check versions for known bugs.
Forward max concurrent rejects. Why: coredns_forward_max_concurrent_rejects_total means the forward plugin hit its max_concurrent cap and is rejecting queries with REFUSED. max_concurrent defaults to unlimited, so a nonzero counter means either someone set the cap or you are looking at serious upstream-induced pileup. Correlate with the REFUSED rate.
File descriptor utilization. Why: process_open_fds against process_max_fds. Each upstream connection, watch stream, and listener consumes one FD. Past 80% is warning; exhaustion is a cliff edge of “too many open files” errors.

Level 4: expert

The signals teams add after the incidents that Level 3 did not catch.

Kubernetes API request errors by code. Why: coredns_kubernetes_rest_client_requests_total with the code label is the only metric-level view into watch health. Sustained 5xx means CoreDNS is serving stale cluster data: existing services still resolve, new ones are invisible, and nothing in the standard dashboards moves. Any 403 is an RBAC problem. There is no binary “watch broken” metric; this is how you infer it.
DNS programming duration. Why: coredns_kubernetes_dns_programming_duration_seconds measures how long Service/Endpoint changes take to become resolvable. P99 above 30s means service discovery is materially delayed. Known limitation: it currently works reliably only for headless_with_selector services.
Upstream connection cache performance. Why: coredns_proxy_conn_cache_hits_total versus coredns_proxy_conn_cache_misses_total (with the proxy_name label on current CoreDNS). A miss ratio above 50% sustained means connections are being re-established per query, adding setup latency and FD churn. Usually an upstream closing connections aggressively or a keepalive misconfiguration.
SERVFAIL by plugin. Why: the plugin label on responses isolates forward versus kubernetes versus other plugins as the error source. Rarely dashboarded, decisive during triage.
Request type distribution for security. Why: coredns_dns_requests_total{type="AXFR"} in a Kubernetes deployment is reconnaissance (CoreDNS should not be serving zone transfers to arbitrary clients). A spike in type="ANY" beyond a low baseline suggests amplification abuse. Both are tickets on any nonzero or anomalous rate.
Response size distribution. Why: coredns_dns_response_size_bytes shifting to larger responses suggests amplification reflection or oversized record sets. Sustained average above 2x baseline is worth investigating. There is no truncation metric, so UDP-to-TCP fallback is not directly observable.
Node-level conntrack utilization. Why: every UDP DNS query through Kubernetes iptables DNAT creates a conntrack entry. When the table fills, the kernel drops packets silently and CoreDNS metrics stay green. Track node_nf_conntrack_entries against node_nf_conntrack_entries_limit (node_exporter): 80% warning, 90% critical, and any “nf_conntrack: table full” in kernel logs is a page. The long-term fix is NodeLocal DNSCache, which takes queries off the per-query DNAT path. After deploying it, remember that CoreDNS metrics reflect only cache misses from the node-local layer, not total cluster DNS demand.
UDP buffer errors. Why: /proc/net/snmp Udp RcvbufErrors incrementing means queries were dropped before CoreDNS saw them. The CoreDNS-side signature is metrics that look too good: low latency, no errors, low throughput. This is a node-level check (netstat -su), not a CoreDNS metric.
Cache stale serves. Why: if serve_stale is configured, coredns_cache_served_stale_total shows stale entries being served in place of fresh upstream data. It keeps clients working during upstream failures, but it also masks them.
Health self-check latency. Why: coredns_health_request_duration_seconds rising above roughly 100ms means the process itself is overloaded (CPU starvation, GC thrashing) even if query metrics have not moved yet.

Quick reference: signals by failure mode

Failure mode	First signals to move	Confirm with
All upstreams unreachable	SERVFAIL spikes, low latency (fast failure), `coredns_forward_healthcheck_broken_total`	`dig @<upstream_ip> . NS +time=1 +tries=1`
One slow upstream	P99 latency high, goroutines rising	`coredns_proxy_request_duration_seconds{proxy_name="forward", to=...}`
Cache collapse after rollout	Hit ratio near zero, upstream rate spike	Correlate with deploy time; self-resolves as cache warms
Kubernetes API disconnect	`coredns_kubernetes_rest_client_requests_total` 5xx, cluster.local only affected	`kubectl logs` for watch errors; test resolving a new Service
Forwarding loop	CrashLoopBackOff, no metrics	Pod logs: “Loop detected”; fix the Corefile forward target
Memory blowout	RSS climbing toward limit, GC pauses lengthening	`process_resident_memory_bytes` vs limit; post-GC heap minimum trend
Conntrack exhaustion	Client timeouts, CoreDNS metrics clean	`node_nf_conntrack_entries` vs limit; kernel logs
UDP buffer cliff	Throughput flat or low, latency “too good”	`netstat -su` RcvbufErrors

How Netdata helps

Netdata collects the CoreDNS Prometheus endpoint on port 9153 per pod, so per-replica divergence (one degraded pod hiding inside a healthy average) stays visible instead of being smoothed away.
RCODE breakdowns from coredns_dns_responses_total are charted by rcode, zone, and plugin, which makes the SERVFAIL-versus-NXDOMAIN and forward-versus-kubernetes distinctions immediate rather than a PromQL exercise at 3 a.m.
Cache hit ratio, eviction rate, and entry counts sit on the same dashboard as latency, so a cache collapse shows up as the expected linked movement across all of them.
Go runtime metrics (goroutines, heap, GC pauses, RSS) are collected alongside CoreDNS metrics, letting you overlay memory pressure against latency and restart events without switching tools.
Node-level collection covers conntrack utilization and UDP buffer errors on the same hosts, closing the gap where CoreDNS metrics look clean but the kernel is dropping queries.