CoreDNS returning SERVFAIL: the resolver is failing queries and what to check first

Applications are failing to resolve names and CoreDNS is answering with SERVFAIL. In Kubernetes this surfaces as connection errors everywhere at once, because nearly all in-cluster communication depends on DNS. SERVFAIL is not one failure. It is CoreDNS saying “a plugin in my chain could not answer this query,” and the cause is usually upstream DNS, the Kubernetes API, or the Corefile itself.

Your health checks are probably lying to you. The CoreDNS /health endpoint on port 8080 only tests process liveness. It does not resolve anything. A pod can return 200 OK to every probe while returning SERVFAIL for every query. The real availability signal is coredns_dns_responses_total{rcode="SERVFAIL"} on the metrics endpoint, split by the zone and plugin labels.

This guide walks through isolating which plugin is failing, which zone is affected, and which of the three usual causes you are dealing with.

What this means

SERVFAIL (RCODE 2) means the server failed to produce an answer for a query it accepted. In CoreDNS’s plugin chain, some plugin was expected to answer and could not. The three usual causes, in order of frequency:

1. Upstream resolvers are unreachable. The forward plugin has no working upstream, so forwarded queries fail immediately.
2. The Kubernetes API is unreachable or unsynced. The kubernetes plugin cannot answer cluster.local queries for records it has not synced.
3. A configuration error. The Corefile forwards to a dead or wrong address, a reload failed, or a zone is misconfigured.

One useful characteristic: SERVFAIL from a hard failure is usually fast. When all upstreams are down, the forward plugin rejects immediately, so request latency stays low even as the error rate spikes toward 100%. That distinguishes a dead upstream from a slow one, where latency climbs and goroutines accumulate instead. SERVFAIL plus high P99 latency plus a rising goroutine count is the “slow upstream drag” pattern, not the “upstream black hole” pattern.

A second characteristic that bites operators: the cache plugin caches SERVFAIL responses for up to 5 seconds by default. A 1-second upstream blip becomes 5 seconds of SERVFAIL served from cache to every client querying that name, even after the upstream recovers. Brief failures get amplified.

Common causes

Note what is not on this list: forwarding loops. A forwarding loop does not produce SERVFAIL metrics. The loop plugin detects the loop at startup and calls log.Fatalf, so the process exits before metrics exist. Loops show up as CrashLoopBackOff with “Loop detected” in the logs, which is a different symptom with a different playbook.

Quick checks

These are all safe, read-only, and fast. Run them in roughly this order. The localhost checks assume you are exec’d into a CoreDNS pod or on a node where the ports are reachable.

# 1. SERVFAIL rate split by zone and plugin (the single most useful command)
curl -s http://localhost:9153/metrics | grep 'coredns_dns_responses_total' | grep 'SERVFAIL'

# 2. Total request rate by zone, for the SERVFAIL ratio denominator
curl -s http://localhost:9153/metrics | grep '^coredns_dns_requests_total'

# 3. Are all upstreams failing health checks?
curl -s http://localhost:9153/metrics | grep 'coredns_forward_healthcheck_broken_total'

# 4. Which specific upstream is failing or slow?
curl -s http://localhost:9153/metrics | grep 'coredns_forward_healthcheck_failures_total'
curl -s http://localhost:9153/metrics | grep 'coredns_forward_request_duration_seconds'

# 5. Is the Kubernetes API answering CoreDNS?
curl -s http://localhost:9153/metrics | grep 'coredns_kubernetes_rest_client_requests_total'

# 6. Did a Corefile reload fail recently?
curl -s http://localhost:9153/metrics | grep 'coredns_reload_failed_total'

# 7. Is the pod actually ready (plugins synced), not just alive?
curl -sf http://localhost:8181/ready && echo "Ready" || echo "Not ready"

# 8. Functional test of internal resolution from a throwaway pod
kubectl run -it --rm dns-debug --image=busybox:1.28 --restart=Never -- nslookup kubernetes.default

# 9. Functional test of upstream reachability (nc -u is not reliable for DNS)
dig @<upstream_ip> . NS +time=1 +tries=1

# 10. Look at the Corefile that is actually running
kubectl get cm -n kube-system coredns -o yaml

Two notes on interpretation. First, on older CoreDNS versions the forward plugin latency and health metrics may appear under the deprecated coredns_proxy_* names (with proxy_name="forward" on some); the coredns_forward_* names are the current ones. If check 4 returns nothing, grep for coredns_proxy_ instead. Second, check logs: kubectl logs -n kube-system deploy/coredns --tail=100 shows watch errors, reload parse errors, and upstream failures that metrics only hint at.

How to diagnose it

The labels on the SERVFAIL metric do most of the triage for you. Work the decision tree.

flowchart TD
  A[SERVFAIL rate elevated] --> B{Which zone label?}
  B -->|cluster.local| C[Kubernetes plugin path]
  B -->|forwarded zones| D[Forward plugin path]
  B -->|both| E[Process or config problem]
  C --> C1{API client errors 5xx or 403?}
  C1 -->|yes| C2[API server, network, or RBAC issue]
  C1 -->|no| C3{Pod ready on :8181?}
  C3 -->|no| C4[Startup sync incomplete or watch broken]
  D --> D1{healthcheck_broken incrementing?}
  D1 -->|yes| D2[All upstreams down - check network path]
  D1 -->|no| D3[Check per-upstream failures and latency by to label]
  E --> E1[Check reload_failed_total and Corefile]

1. Split SERVFAIL by zone. If the elevated series all carry zone="cluster.local." (or your cluster domain), external resolution is fine and the problem is the kubernetes plugin. If the elevated zones are . or your forwarded zones, the problem is the forward path. If both are failing, suspect the process itself (CPU starvation, reload breakage, panic) or a shared dependency like the node’s network.

2. Split SERVFAIL by plugin. The plugin label tells you which plugin generated the response. plugin="forward" means the upstream path failed. plugin="kubernetes" means cluster DNS state is broken. This one label cuts the search space in half before you run anything else.

3. If forward: check upstream health, then the network path. coredns_forward_healthcheck_broken_total incrementing means every configured upstream is failing health checks. Confirm with a real query from the node (dig @<upstream_ip> . NS +time=1 +tries=1). If the upstream answers from the node but not through CoreDNS, look at egress firewalls, NetworkPolicy, or security group changes. Health checks passing does not guarantee queries succeed: an upstream that answers the health probe but drops or rate-limits real query volume looks healthy while SERVFAILing.

4. If kubernetes: check API client errors by code. coredns_kubernetes_rest_client_requests_total with code=5xx or connection errors means the API server is unreachable or overloaded. code=403 means RBAC: the CoreDNS ServiceAccount lost permissions to list or watch Services and Endpoints. During an API disconnect, existing services keep resolving from stale in-memory state, so SERVFAIL often appears only for recently created or unsynced records.

5. If it started right after a config change: check reload state. coredns_reload_failed_total nonzero means the new Corefile failed to parse and CoreDNS is still running the old one. Diff the ConfigMap against the last known-good version and check kubectl logs for parse errors.

6. Check the timeline against latency. SERVFAIL with low latency is fast rejection (dead upstream, unsynced plugin, config error). SERVFAIL with rising P99 and rising go_goroutines is slow upstream drag: queries are piling up waiting on a degraded-but-not-dead upstream, and some eventually fail. The fix paths differ, so establish which pattern you are in before changing anything.

Metrics and signals to monitor

The alert that matters: SERVFAIL / total responses > 1% sustained over 5 minutes, with total query rate above 10 qps. The qps floor keeps idle clusters and low-traffic namespaces from paging you on a handful of failed queries. Page only when corroborated (upstream health checks also failing, or API errors also present) and sustained.

Do not alert on NXDOMAIN. In Kubernetes, NXDOMAIN is a normal product of search-domain expansion and routinely runs at 20-60% of responses. Alerting on “total DNS errors” trains the team to ignore the channel and buries real SERVFAIL spikes in noise. Alert on SERVFAIL and REFUSED only.

Fixes

Upstream unreachable

Restore the path before touching CoreDNS. Verify with dig @<upstream_ip> . NS +time=1 +tries=1 from a CoreDNS pod or node. Common root causes are egress firewall or security group changes, NetworkPolicy blocking UDP/TCP 53 egress, and upstream provider outages. If one upstream of several is bad, remove it from the Corefile forward line temporarily and let the reload plugin pick up the change; queries rebalance to the healthy upstreams.

Two forward plugin behaviors shape what you see. By default, when all upstreams are marked unhealthy, CoreDNS still attempts a random one rather than failing fast, unless failfast_all_unhealthy_upstreams is set, in which case SERVFAIL comes back immediately. And on recent CoreDNS versions, receiving SERVFAIL or REFUSED from an upstream no longer triggers failover to the next upstream; the failover SERVFAIL REFUSED option restores that behavior. If you upgraded recently and SERVFAILs from an upstream suddenly pass straight through to clients instead of failing over, this is why.

Kubernetes API unreachable or RBAC broken

Fix the API path, not CoreDNS. For code=403, restore the ServiceAccount’s list/watch permissions on services, endpoints, and namespaces. For connectivity failures, check NetworkPolicy between the CoreDNS pods and the API server and any API server overload. If a pod is stuck not-ready after startup because its initial list/watch sync is not completing, investigate API list latency; restarting the pod forces a fresh re-list but does not fix an overloaded API server.

One readiness misconfiguration causes recurring SERVFAIL at every rollout: using /health instead of /ready as the readiness probe. /health returns OK before the kubernetes plugin has synced, so the pod receives traffic and SERVFAILs cluster.local queries for its first seconds of life. Point the readiness probe at port 8181.

Config error

For a failed reload, the old config is still live, so fix the ConfigMap and let reload pick it up; coredns_reload_failed_total stops incrementing when a valid Corefile parses. Verify forward targets actually point at real resolvers. If the forward target points back at the cluster DNS Service IP, or at a node resolv.conf that itself points at the cluster, you have a forwarding loop, which will present as CrashLoopBackOff rather than SERVFAIL once the loop plugin fires.

SERVFAIL cache amplification

If brief upstream blips keep turning into multi-second outages for specific names, the 5-second SERVFAIL cache is the amplifier. The cache plugin’s servfail directive controls this; setting it to 0 disables SERVFAIL caching at the cost of re-querying the upstream for every retry during a real outage. For most clusters a short nonzero value is a reasonable middle ground. Do this deliberately: disabling failure caching entirely can hammer a struggling upstream with retries.

Prevention

• Alert on the SERVFAIL ratio, not pod health. /health green means nothing about resolution. The composite page is: ratio > 1% for 5 minutes at > 10 qps, corroborated by upstream health or API error signals.
• Monitor per-upstream, not aggregate. Dashboard coredns_forward_request_duration_seconds and health check failures by the to label so one bad upstream is visible before it drags the aggregate.
• Use /ready for readiness probes. Prevents unsynced pods from taking traffic at every rollout.
• Stagger rollouts. maxUnavailable=1 and a PodDisruptionBudget prevent a full-cache cold start, which floods upstreams and can itself produce a SERVFAIL wave amplified by the 5-second cache.
• Test config changes in a canary pod before applying the ConfigMap cluster-wide, and watch coredns_reload_failed_total after every Corefile edit.
• Read upgrade notes for behavior changes. Forward plugin failover behavior has changed across releases; check release notes for behavioral changes, not just features, before upgrading.

How Netdata helps

• RCODE breakdown out of the box. Netdata collects coredns_dns_responses_total split by rcode, zone, and plugin, so the first triage question (which plugin, which zone) is answered from the dashboard, not from ad-hoc curl commands.
• SERVFAIL ratio alerting. Alert definitions on the SERVFAIL-to-total ratio with duration and traffic floors match the “>1% over 5 min with >10 qps” pattern, avoiding both idle-cluster false pages and NXDOMAIN noise.
• Upstream correlation in one view. Forward health check failures, per-upstream latency, and SERVFAIL rate render side by side, which makes the fast-failure versus slow-drag distinction visible without manual PromQL.
• Kubernetes API watch signals. API client request rates by status code sit on the same dashboard as cluster.local SERVFAIL, so an API-side cause is confirmable in seconds.
• Per-second granularity. SERVFAIL cache amplification events last only seconds; per-second collection catches transients that 30- or 60-second scrape intervals smooth into invisibility.

Related guides

• CoreDNS Operations Guide hub