CoreDNS NXDOMAIN vs SERVFAIL: why alerting on the wrong one buries real incidents

Your CoreDNS alert fired again. The “DNS error rate” panel shows a wall of red, so you open it, see a pile of NXDOMAIN responses, and silence the alert. Meanwhile, a genuine upstream failure is producing SERVFAIL responses somewhere in that same wall of red, and nobody will notice until an application team opens a ticket.

This is the most common CoreDNS alerting mistake: treating all non-NOERROR responses as one bucket. NXDOMAIN and SERVFAIL look similar in a naive error counter, but they mean opposite things. NXDOMAIN is the server correctly reporting that a name does not exist. SERVFAIL is the server admitting it failed to answer at all. Alerting on the sum of both guarantees that the noise (NXDOMAIN, constant and expected in Kubernetes) drowns the signal (SERVFAIL, rare and always actionable).

What this means

DNS response codes (RCODEs) tell the client what happened to its query. Four of them matter here:

• NOERROR: the query was answered successfully. This includes NODATA (the name exists but has no record of the requested type), which is also a successful answer.
• NXDOMAIN: the name does not exist. The resolver looked, confirmed the name is not there, and said so. This is a correct, authoritative answer, not an operational error.
• SERVFAIL: the server failed to process the query. Upstream unreachable, plugin failure, API connectivity loss, configuration error. This is an actual failure of the resolution path.
• REFUSED: the server explicitly rejected the query (no matching server block, ACL rejection, or forward max_concurrent limit hit). Also a real failure, usually misconfiguration or capacity.

CoreDNS exposes all of these on one counter family, coredns_dns_responses_total, with an rcode label. The mistake is writing an alert like “rate of responses where rcode != NOERROR.” In a Kubernetes cluster, that numerator is dominated by NXDOMAIN, so the alert either pages constantly (alert fatigue) or gets its threshold raised until real SERVFAIL spikes fall under it.

flowchart TD
  Q[coredns_dns_responses_total by rcode] --> N[NOERROR - answered fine]
  Q --> X[NXDOMAIN - name does not exist]
  Q --> S[SERVFAIL - resolution failed]
  Q --> R[REFUSED - query rejected]
  X --> NX[Expected background in K8s: ndots search-domain expansion]
  NX --> NAL[Do not alert on absolute count]
  S --> SF[Upstream down, API loss, config error]
  SF --> AL1[Alert on SERVFAIL/total ratio vs baseline]
  R --> RF[Missing zone, ACL, max_concurrent]
  RF --> AL2[Alert on any sustained nonzero rate]

Why NXDOMAIN volume is huge in Kubernetes by design

If you run CoreDNS as cluster DNS, your NXDOMAIN count is not a symptom. It is an architectural constant.

Kubernetes pods default to ndots:5 in /etc/resolv.conf. Any name with fewer than 5 dots triggers search-domain expansion before the resolver tries the name as given. A lookup for api.stripe.com (2 dots) from a pod produces this sequence:

1. api.stripe.com.<namespace>.svc.cluster.local - NXDOMAIN
2. api.stripe.com.svc.cluster.local - NXDOMAIN
3. api.stripe.com.cluster.local - NXDOMAIN
4. api.stripe.com. - the actual intended lookup

Every external lookup generates several intermediate NXDOMAINs. In a typical cluster, NXDOMAIN runs at roughly 20-60% of total responses, shifting with traffic mix: deploy a batch job that resolves many external names and NXDOMAIN volume climbs proportionally, with nothing wrong anywhere.

Two consequences follow:

• Absolute NXDOMAIN counts mean nothing. They scale with query volume and workload shape. An alert on “NXDOMAIN > N” is an alert on “the cluster is doing DNS.”
• NXDOMAIN belongs in the denominator, not the numerator. Include it in the total when computing a failure ratio. Excluding it from the denominator inflates the SERVFAIL ratio and makes thresholds meaningless.

Negative caching of these NXDOMAINs is correct behavior, not a leak. coredns_cache_entries{type="denial"} growing steadily is the cache doing its job: absorbing repeated lookups for names that do not exist so they never hit the kubernetes plugin or an upstream. Teams periodically “discover” a big denial cache and treat it as a bug. It is not one.

What SERVFAIL actually tells you

SERVFAIL means CoreDNS could not produce an answer. The usual causes:

The plugin label on coredns_dns_responses_total{rcode="SERVFAIL"} is your fastest triage shortcut. plugin="forward" points at upstreams. plugin="kubernetes" points at the API watch path. Use it.

The SERVFAIL caching gotcha

CoreDNS caches SERVFAIL responses for 5 seconds by default (the cache plugin’s servfail TTL). A 1-second upstream blip gets amplified: for 5 seconds after the upstream recovers, clients querying the affected name get SERVFAIL straight from cache. Transient failures look bigger and longer than they were. This is one reason SERVFAIL alerts should be ratio-based over a multi-minute window rather than “any SERVFAIL ever.” Brief, self-resolving blips happen; a sustained ratio above baseline does not.

REFUSED is a failure too

REFUSED means CoreDNS rejected the query outright: no server block matched the zone, an ACL dropped it, or the forward plugin’s max_concurrent limit rejected it (check coredns_forward_max_concurrent_rejects_total). A missing catch-all forward zone makes external lookups come back REFUSED. Any sustained nonzero REFUSED rate warrants a ticket-level alert alongside SERVFAIL.

Quick checks

Read-only commands to see your actual rcode mix right now:

# All responses broken down by rcode and plugin
curl -s http://localhost:9153/metrics | grep '^coredns_dns_responses_total'

# SERVFAIL only, with the plugin label for triage
curl -s http://localhost:9153/metrics | grep 'coredns_dns_responses_total' | grep 'rcode="SERVFAIL"'

# REFUSED only
curl -s http://localhost:9153/metrics | grep 'coredns_dns_responses_total' | grep 'rcode="REFUSED"'

# Total request rate for context (the ratio denominator)
curl -s http://localhost:9153/metrics | grep '^coredns_dns_requests_total'

# Denial cache size - confirm negative caching, don't panic about it
curl -s http://localhost:9153/metrics | grep '^coredns_cache_entries'

In Kubernetes, hit the metrics endpoint on each CoreDNS pod individually (port 9153), or via your existing Prometheus scrape. Replicas behind the kube-dns Service each report their own counters: one degraded replica is invisible in a fleet average.

How to build the alert correctly

1. Compute the SERVFAIL ratio, not the count. Numerator: rate of coredns_dns_responses_total{rcode="SERVFAIL"} summed across pods. Denominator: rate of all coredns_dns_responses_total. The denominator deliberately includes NXDOMAIN.
2. Gate on minimum traffic. A 5% SERVFAIL ratio on 0.2 queries per second is one failed query. Require a minimum total query rate (10 qps is a reasonable starting point; tune to your fleet) before the ratio means anything.
3. Set thresholds against your baseline. A working starting point: above 1% sustained over 5 minutes is degraded and warrants a ticket; above 5% sustained is critical. In a healthy cluster the SERVFAIL ratio sits near zero, so even these thresholds leave room. The important thing is that they trigger on SERVFAIL alone.
4. Suppress known noise windows. Cold starts after restarts and mass rollouts can produce short SERVFAIL bursts (cache warming, pods serving before ready). Only page when the ratio is corroborated (upstream healthcheck failures, API errors) and the pod has been up long enough that warmup is over.
5. Alert separately on REFUSED. Any sustained nonzero REFUSED rate is a ticket: it means misconfiguration or forward capacity limits, and it does not fluctuate with traffic the way NXDOMAIN does.
6. Never alert on absolute NXDOMAIN. If you want NXDOMAIN visibility at all, track the NXDOMAIN/total ratio against its rolling baseline. A ratio change is meaningful (new misconfigured app, reconnaissance, search-domain storm); the absolute count is not.

Metrics and signals to monitor

Prevention

• Delete any alert on combined error counts. If a rule matches rcode!="NOERROR" or sums rcodes into “errors,” rewrite it now. It is either paging on noise or thresholded so high it cannot catch SERVFAIL.
• Dashboard rcodes as a stacked ratio. NOERROR, NXDOMAIN, SERVFAIL, REFUSED as percentages of total. A healthy cluster shows a stable NXDOMAIN band with SERVFAIL and REFUSED at zero. Deviations are visible at a glance, which makes “why did this alert fire” trivial to answer.
• Fix NXDOMAIN at the source, not in the alert. If a workload generates extreme search-domain amplification, the right fixes are ndots tuning in the pod spec or fully qualified names with a trailing dot in application config. That reduces noise and upstream load without touching failure detection.
• Keep per-replica visibility. Aggregate the ratio across pods for the alert, but keep per-pod breakdowns available. One pod SERVFAILing while the other is healthy is a real incident that an average hides.

How Netdata helps

• Per-rcode response rates out of the box: Netdata charts coredns_dns_responses_total split by rcode and plugin, so NXDOMAIN and SERVFAIL are visually separate streams instead of one blended error counter.
• Ratio context: plotting SERVFAIL rate next to total request rate makes the SERVFAIL/total ratio legible during an incident without ad-hoc PromQL, and shows when a spike is real versus a low-traffic artifact.
• Corroboration on one screen: SERVFAIL spikes land next to forward healthcheck failures, per-upstream latency, and Kubernetes API request errors, so the “forward plugin vs kubernetes plugin” triage step takes seconds.
• Replica-level views: per-pod CoreDNS charts expose single-replica degradation that fleet aggregates mask, which is where quiet SERVFAIL incidents usually live.
• Cache visibility: success vs denial cache entries sit alongside response codes, so you can confirm a high NXDOMAIN rate is being absorbed by the negative cache rather than hammering upstreams.

Related guides

• How CoreDNS actually works in production: the plugin chain mental model
• CoreDNS monitoring checklist: the signals every production resolver needs
• CoreDNS monitoring maturity model: from survival to expert
• CoreDNS returning SERVFAIL: the resolver is failing queries and what to check first