CoreDNS per-upstream health check failures: degraded redundancy before total loss

One of your upstream DNS resolvers is failing CoreDNS’s health checks. Queries are still being answered because the remaining upstreams are carrying the forwarding load, so nothing is paging yet. This is exactly the window you want: degraded redundancy, not an outage. The next failure in this sequence is coredns_forward_healthcheck_broken_total incrementing, at which point every upstream is marked unhealthy and you are in a real incident.

The signal is coredns_proxy_healthcheck_failures_total, a counter in the proxy subsystem. Filter it by proxy_name="forward" and read the to label: it names the specific upstream address that is failing. That label is the whole point of this metric. It turns “forwarding is degraded” into “10.0.0.2:53 is the problem.”

This article covers how to confirm the failure, work out why that one upstream is failing, decide whether to pull it from rotation, and catch this earlier next time.

What this means

The forward plugin health-checks each configured upstream independently. When one upstream fails health checks, CoreDNS stops routing queries to it and the survivors absorb its share of traffic. From the client’s perspective, DNS still works. From a redundancy perspective, you are one upstream failure away from the all-upstreams-down state.

Two things matter about the failure progression:

1. The cache masks severity. While the cache is warm, cached responses continue to be served and the failure is partially invisible end-to-end. Upstream health must be monitored independently of end-to-end success rates, or you find out only when TTLs expire.
2. The remaining upstreams degrade under added load. Higher per-upstream query volume raises their latency and, in constrained environments, can push them toward their own failure. This is how one dead upstream becomes three.

flowchart TD
  A["One upstream fails health checks
coredns_proxy_healthcheck_failures_total{to=X} increments"] --> B["Forward plugin stops routing to X"]
  B --> C["Remaining upstreams carry full load"]
  C --> D["Per-upstream latency rises
coredns_proxy_request_duration_seconds"]
  D --> E{Another upstream fails?}
  E -->|"yes"| F["coredns_forward_healthcheck_broken_total increments
all upstreams unhealthy"]
  E -->|"no, fixed in time"| G["Redundancy restored"]
  F --> H["SERVFAIL cascade as cache drains"]

The to label on the health check failure counter is the pivot for the entire investigation.

Common causes

A note on rate limiting: an upstream can respond to health checks while dropping real queries, or vice versa, so treat the health check signal as one input, not the full truth. Confirm with a direct query test.

Quick checks

All of these are read-only and safe to run during an incident.

# 1. See which upstream is failing health checks
curl -s http://localhost:9153/metrics | grep 'coredns_proxy_healthcheck_failures_total'
# Filter: proxy_name="forward". The to label names the failing upstream.

# 2. Check whether ALL upstreams are now unhealthy (the escalation signal)
curl -s http://localhost:9153/metrics | grep 'coredns_forward_healthcheck_broken_total'

# 3. Per-upstream latency: is the failing upstream slow, and are survivors degrading?
curl -s http://localhost:9153/metrics | grep 'coredns_proxy_request_duration_seconds'

# 4. SERVFAIL impact: is the failure visible to clients yet?
curl -s http://localhost:9153/metrics | grep 'coredns_dns_responses_total' | grep 'SERVFAIL'

# 5. Test the failing upstream directly with a real DNS query
#    (nc -zv -u is NOT reliable for UDP DNS testing)
dig @<failing_upstream_ip> . NS +time=1 +tries=1

# 6. In Kubernetes, test from the pod network. The CoreDNS image is minimal
#    and does not ship DNS tools, so use a throwaway debug pod:
kubectl run -it --rm debug --image=busybox:1.28 --restart=Never -- nslookup . <failing_upstream_ip>
# busybox:1.28 is a standard Docker Hub image; mirror it if your cluster cannot pull from Docker Hub

# 7. Connection cache behavior toward that upstream
curl -s http://localhost:9153/metrics | grep 'coredns_proxy_conn_cache'

In Kubernetes, replace localhost:9153 with the CoreDNS pod IP, or use kubectl port-forward.

How to diagnose it

1. Read the to label. Pull coredns_proxy_healthcheck_failures_total{proxy_name="forward"} and identify which upstream address has a rising counter. If multiple to values are rising, you are closer to the all-upstreams-down pattern than to a single-upstream issue; treat it as the more severe case.

2. Check whether the failure is complete or partial. Look at coredns_proxy_request_duration_seconds{to="<failing>"}. A slow-but-alive upstream shows high latency before full health check failure. A dead upstream shows no responses at all. A slow upstream that still answers (high latency, moderate SERVFAIL, rising goroutines) needs different handling than a black hole.

3. Test the upstream directly. dig @<upstream> . NS +time=1 +tries=1 from the CoreDNS node or a debug pod. If this fails, the problem is the upstream or the network path, not CoreDNS. If it succeeds while health checks still fail, look at rate limiting or protocol differences between the health check and real queries.

4. Isolate the layer. If the direct query fails from the CoreDNS pod network but works from elsewhere, the issue is the egress path from CoreDNS: firewall, security group, network policy, routing. If it fails from everywhere, the upstream itself is down.

5. Check impact on clients. SERVFAIL ratio by zone (coredns_dns_responses_total{rcode="SERVFAIL"}) tells you whether queries routed to the failing upstream produced errors before health checks pulled it from rotation. A warm cache may be masking this. Do not assume zero SERVFAIL means zero risk.

6. Check the survivors. Per-upstream latency and goroutine count on the remaining upstreams tell you whether the extra load is degrading them. Rising goroutines with stable QPS means queries are piling up on slow survivors.

Metrics and signals to monitor

One versioning caveat: on older CoreDNS versions you may see coredns_forward_healthcheck_failures_total instead of the coredns_proxy_* name; the forward-specific metric name was deprecated in favor of the proxy subsystem metrics with a proxy_name label. If your dashboards or alert rules still reference the old name, verify which one your version actually emits: the proxy-subsystem names ship from CoreDNS 1.11.0 (PR #6178).

Fixes

The upstream is genuinely down or decommissioned

Remove it from the forward line in the Corefile and replace it with a healthy resolver. CoreDNS reloads the Corefile automatically if the reload plugin is enabled; check coredns_reload_failed_total afterward to confirm the new config actually applied. Be aware that a reload clears the cache, which briefly raises upstream load on the survivors.

The network path to one upstream is broken

This is the more common case in practice: the resolver is fine, the route to it is not. Check recent firewall, security group, and network policy changes first, since this failure class almost always has a change event attached to it. Until the path is fixed, removing the unreachable upstream from the Corefile stops the wasted health check traffic and the partial-routing inefficiency.

The upstream is rate-limiting CoreDNS

If failures correlate with load peaks and direct queries succeed intermittently, the upstream is shedding your traffic. Options: distribute forwarding across more upstreams, raise cache TTLs or cache size so fewer queries reach the upstream, or move to a resolver without the limit. In cloud environments, verify whether the default provider resolver has a documented packet rate ceiling.

The upstream is slow, not dead

A slow upstream accumulates blocked goroutines and can cascade into memory pressure. Temporarily removing the slow upstream from the Corefile is a legitimate mitigation. Restarting CoreDNS is not a first-line fix here; it clears the cache and makes everything worse.

Prevention

• Alert on the per-upstream counter, not just the broken total. coredns_forward_healthcheck_broken_total only fires when everything is already down. The per-upstream delta is your early warning; a sustained positive delta over 5 minutes on any single to value should open a ticket.
• Run at least three upstreams so one failure leaves real redundancy. With one upstream, every health check failure is effectively the all-upstreams-down signal.
• Watch per-upstream latency as a leading indicator. Health check failures are often preceded by a latency climb on that upstream. Catching the drift at the latency stage gives you even more runway.
• Account for SERVFAIL cache amplification. A brief upstream failure is cached as SERVFAIL for 5 seconds by default, so the client-visible effect outlasts the actual failure. Factor this into how you interpret recovery.

How Netdata helps

• Netdata charts coredns_proxy_healthcheck_failures_total per to label out of the box, so the failing upstream is visible as its own line rather than buried in an aggregate.
• Correlating the per-upstream failure deltas with per-upstream latency (coredns_proxy_request_duration_seconds) on one dashboard shows whether survivors are degrading under shifted load, which is the escalation risk.
• Plotting SERVFAIL ratio alongside health check failures tells you immediately whether the redundancy loss is client-visible or still masked by the cache.
• Anomaly detection on per-upstream latency surfaces the slow-upstream drift that typically precedes hard health check failures, moving detection one step earlier in the cascade.

Related guides

• How CoreDNS actually works in production: the plugin chain mental model
• CoreDNS monitoring checklist: the signals every production resolver needs
• CoreDNS monitoring maturity model: from survival to expert
• CoreDNS NXDOMAIN vs SERVFAIL: why alerting on the wrong one buries real incidents
• CoreDNS query rate dropped to zero while the process looks healthy
• CoreDNS returning REFUSED: no matching zone, an ACL, or the forward concurrency limit
• CoreDNS returning SERVFAIL: the resolver is failing queries and what to check first