CoreDNS monitoring maturity model: from survival to expert

Most CoreDNS monitoring setups are either a single “is the pod up” check or a wall of dashboards nobody reads during an incident. Neither works. The right question is not “how many metrics do we collect” but “which failure modes can we actually detect right now.” That is what a maturity model answers: it orders signals by the failures they catch, so you can see exactly which class of outage would currently reach your users before it reaches your alerts.

This article defines four levels: survival, operational, mature, and expert. It is grounded in the failure archetypes CoreDNS actually exhibits in production: upstream black holes, cache collapses, Kubernetes API disconnects, OOM kills, UDP buffer drops, and conntrack exhaustion. Use it to audit your current setup, find the gaps, and decide what to add next.

How to use this model

• Levels are cumulative. Level 3 assumes everything in Levels 1 and 2. Skipping ahead leaves holes: per-upstream latency breakdowns are useless if you do not already alert on SERVFAIL.
• Levels are per-deployment, not per-organization. Your production cluster DNS should run at a higher level than a throwaway dev cluster.
• Incidents are the upgrade trigger. If an outage reached users without firing an alert, find the level that contains the missing signal and move to it. Do not level up speculatively.
• Higher levels mean more correlation work, not just more metrics. Level 4 signals only pay off when someone or something correlates them during triage.

flowchart TD
  L1["Level 1: Survival - is it alive?"]
  L2["Level 2: Operational - is it serving correctly?"]
  L3["Level 3: Mature - will it fail soon?"]
  L4["Level 4: Expert - what is it hiding?"]
  L1 --> L2 --> L3 --> L4

Level 1: survival

The absolute minimum. You know CoreDNS is down or returning errors. You have almost no diagnostic capability, but you know there is a fire.

A minimal manual check looks like this:

# Survival checks: process, listeners, metrics, SERVFAIL
pgrep -f coredns
ss -ulnp | grep ':53'
ss -tlnp | grep ':53'
curl -s --max-time 2 http://localhost:9153/metrics | grep 'coredns_dns_responses_total' | grep 'SERVFAIL'
dig @<coredns_ip> . NS +time=1 +tries=1

Two things to internalize at this level. First, the /health endpoint on port 8080 only checks process liveness. It does not test DNS resolution. A pod can return 200 OK to health probes while returning SERVFAIL for every query, which is why the SERVFAIL counter is the real availability signal, not the probe. Second, a pod in CrashLoopBackOff with “Loop detected” in the logs is a forwarding loop in the Corefile. The loop plugin calls log.Fatalf at startup, so no metrics exist. Detect loops via pod status and logs, not metrics.

What Level 1 misses: slow upstreams, stale Kubernetes data, cache problems, silent packet drops, and every resource cliff. Anything that degrades instead of dying is invisible.

Level 2: operational

This is where a competent production team should sit. Level 2 covers the failure modes that account for most CoreDNS incidents: upstream failures, latency degradation, and cache problems.

Key interpretation rules at this level:

• SERVFAIL is fast failure. An upstream black hole produces high SERVFAIL with low latency, because the forward plugin fails immediately rather than timing out. High latency with moderate SERVFAIL is the opposite pattern: a slow upstream dragging queries into timeouts. These need different responses.
• The cache masks upstream failure. While the cache is warm, a dead upstream is invisible in end-to-end success rates until TTLs expire. That is why upstream health check failures are monitored independently instead of inferred from SERVFAIL alone.
• SERVFAIL amplifies through the cache. SERVFAIL responses are cached for 5 seconds by default, so a 1-second upstream blip becomes a 5-second outage for that record. Short SERVFAIL bursts that self-resolve are often this effect, not a real incident.
• Do not alert on NXDOMAIN. NXDOMAIN is a normal response, and in Kubernetes it is a large fraction of traffic due to search domain expansion. Alert on SERVFAIL and REFUSED; watch NXDOMAIN only as a ratio against its baseline.
• Derive cache misses. coredns_cache_misses_total is deprecated. Compute misses as requests minus hits.

For paging, composite rules beat raw thresholds. SERVFAIL alone is not page-safe: cold starts, batch processing, and transient upstream flaps all produce self-resolving blips. Page only when SERVFAIL is corroborated by upstream failure evidence and sustained for more than 5 minutes.

What Level 2 misses: everything inside the process and the node. Resource exhaustion, goroutine leaks, reload failures, and kernel-level drops are still invisible.

Level 3: mature

Level 3 adds the signals that predict failure instead of reporting it. Most CoreDNS outages at scale are resource-driven cliff edges: the pod is fine, the pod is fine, the pod is OOMKilled. These metrics give you the runway before the cliff.

Interpretation notes:

• Watch post-GC heap minima, not peaks. Go memory is naturally spiky. A leak shows as post-GC minimums that keep rising and never return to baseline. For OOM-kill risk specifically, use process_resident_memory_bytes against the container limit, since that is what the OOM killer sees. Warning at 80% of limit, critical at 90%.
• Goroutines are the earliest upstream-slowdown signal. Baseline is typically 20-50 goroutines. A count that grows while QPS stays flat means queries are blocked on upstreams and accumulating. This precedes the SERVFAIL and memory signals, sometimes by minutes.
• Reload failures are config drift, not an outage. On a failed reload, CoreDNS keeps the old configuration and DNS continues working. The danger is the gap between what you think is running and what is running. There is also an edge case where a reload that opens a listener on a new port and fails can leave health or metrics ports broken while DNS continues, which is another reason to keep the Level 1 metrics-port check.
• Panics are always a ticket. Any nonzero coredns_panics_total indicates a bug. Note that the metric name has no dns subsystem; it is coredns_panics_total. Panics in background goroutines crash the process without incrementing this counter, so correlate with pod restarts.
• Version note on proxy metrics. The coredns_proxy_* metrics were renamed from coredns_forward_* in CoreDNS 1.11.0. On older versions, per-upstream latency and health check failures appear under the forward prefix. coredns_forward_healthcheck_broken_total and coredns_forward_max_concurrent_rejects_total keep the forward prefix regardless.

What Level 3 misses: failures that never reach the process. Stale Kubernetes data, kernel packet drops, conntrack exhaustion, and security-relevant traffic patterns.

Level 4: expert

Level 4 exists because the nastiest CoreDNS failures produce clean dashboards. These signals cover what the process cannot see about itself.

Why these matter:

• Stale data is the most dangerous failure mode. When the Kubernetes API watch disconnects, CoreDNS keeps serving its last known state. Existing services resolve. New services are invisible. Every performance metric stays green while the answers drift from reality. There is no “watch broken” metric, so API error codes by code label are the proxy, and 403s specifically mean RBAC problems.
• Kernel drops make CoreDNS metrics lie. When the node UDP receive buffer or the conntrack table fills, packets are dropped before CoreDNS counts them. The result is dashboards that look too good to be true: low latency, no errors, suspiciously low throughput, while clients time out. Conntrack exhaustion in Kubernetes drops packets for the entire node, not just DNS, and the kernel log line “nf_conntrack: table full, dropping packet” is definitive. These are node-level signals, so they must come from node monitoring, not from CoreDNS.
• SERVFAIL by plugin cuts triage time. The plugin label tells you which plugin wrote the error response. SERVFAIL from forward points at upstreams; SERVFAIL from kubernetes points at cluster DNS state. One caveat: if the cache plugin serves the response, it appears as the plugin value, which can hide the original source.
• The self-check latency is a canary. The health plugin checks itself once per second and should complete in under 10ms. Self-check latency rising above 100ms means the process is overloaded even if query metrics have not moved yet.
• DNS programming duration has partial coverage. It currently works reliably for headless_with_selector services, so treat it as a lower bound on propagation delay, not a complete SLI.

Knowing when to level up

Map your last few incidents against the levels:

• An upstream outage that users reported before your alerts did: you were at Level 1, and the SERVFAIL ratio plus upstream health signals of Level 2 would have caught it.
• An OOM kill with no warning: Level 3 heap, GC, and goroutine trends give days to hours of runway on the slow-leak variant.
• “DNS was broken but all dashboards were green”: classic Level 4 territory, either stale Kubernetes data or kernel-level drops.
• A config change that “applied” but did not: Level 3 reload failure tracking.

If you operate NodeLocal DNSCache, remember that CoreDNS then only sees cache misses from the node-local layer. Low CoreDNS QPS is expected, not a sign of light load, and the upstream-facing levels matter more, not less.

How Netdata helps

• Netdata collects the CoreDNS Prometheus endpoint on :9153 per instance, which keeps per-replica visibility intact instead of averaging away a single degraded pod.
• RCODE responses, query rate by zone, and request latency are charted together, so the SERVFAIL-versus-latency distinction (fast failure vs slow drag) is visible without building queries by hand.
• Go runtime metrics (heap, GC duration, goroutines) are collected alongside CoreDNS-specific metrics, which is what makes Level 3 correlation between blocked goroutines, memory growth, and upstream latency practical during triage.
• Node-level collection covers conntrack utilization and UDP buffer errors on the same host, closing the Level 4 gap where CoreDNS metrics look clean while the kernel drops packets.
• Per-upstream breakdowns by the to label and per-plugin SERVFAIL breakdowns are available as dimensions, so incident-time drill-down does not require pre-built dashboards.

Related guides

• CoreDNS monitoring checklist: the signals every production resolver needs
• How CoreDNS actually works in production: the plugin chain mental model
• CoreDNS returning SERVFAIL: the resolver is failing queries and what to check first
• CoreDNS returning REFUSED: no matching zone, an ACL, or the forward concurrency limit
• CoreDNS NXDOMAIN vs SERVFAIL: why alerting on the wrong one buries real incidents
• CoreDNS query rate dropped to zero while the process looks healthy
• CoreDNS NOERROR with zero answers: the resolution failure that reports success
• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• CoreDNS slow upstream: per-upstream latency, goroutine pileup, and the to label
• CoreDNS per-upstream health check failures: degraded redundancy before total loss
• CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed
• CoreDNS upstream connection cache misses: new connections adding latency per query