CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken

Every forwarded query through CoreDNS is suddenly returning SERVFAIL, request latency has dropped instead of rising, and coredns_forward_healthcheck_broken_total is climbing. This is the upstream black hole: every upstream resolver configured in the forward plugin is failing health checks at the same time, and CoreDNS has nowhere to send external queries.

The counter-intuitive part is the latency profile. When upstreams die hard, failures are fast. Queries do not sit in timeouts the way they do with a slow upstream. If you only alert on high latency, this failure mode sails under your dashboards until clients start reporting resolution errors.

The second trap: a warm cache masks the outage. Cached answers keep flowing until their TTLs expire, so the incident can look like nothing for minutes, then flip to near-total failure all at once. By the time SERVFAIL rates spike, the upstreams have often been down for a while.

What this means

The forward plugin health-checks each configured upstream and marks it unhealthy after consecutive failures. When every upstream in a server block is unhealthy at once, CoreDNS increments coredns_forward_healthcheck_broken_total. That counter means “the health checking layer believes there is nothing healthy to forward to.”

What happens next depends on your Corefile:

• Default behavior: CoreDNS assumes the health checking mechanism itself may be wrong and sprays each query to a random (unhealthy) upstream anyway. If the upstreams are truly dead, those attempts fail fast and clients see SERVFAIL. If health checks are flapping but the upstreams actually answer, some queries still succeed, which makes the incident look intermittent.
• With failfast_all_unhealthy_upstreams: CoreDNS returns SERVFAIL immediately for all forwarded queries without attempting any upstream. This option was added in v1.12.1 (PR #6999). It converts the black hole into a clean, fast, uniform failure, which is easier to alert on but guarantees zero forwarded resolution while all upstreams are marked down.

Meanwhile the cache plugin keeps serving anything it holds. Positive and negative cache entries are returned as normal until TTL expiry. Once the working set drains, effectively 100% of forwarded queries fail. SERVFAIL responses themselves are also cached (5 seconds by default), so even a brief upstream recovery can be masked by a few seconds of cached SERVFAILs.

flowchart TD
  A[All upstreams unreachable] --> B[Health checks fail for every upstream]
  B --> C[healthcheck_broken_total increments]
  C --> D{failfast_all_unhealthy_upstreams set?}
  D -- Yes --> E[Immediate SERVFAIL for all forwarded queries]
  D -- No --> F[Queries sprayed to random unhealthy upstream]
  F --> G[Fast failure: SERVFAIL with low latency]
  H[Warm cache serves stale-but-valid answers] -.masks until TTL expiry.-> G
  G --> I[Cache drains: SERVFAIL approaches 100% of forwarded queries]
  E --> I

Common causes

One cause to rule out early: in Kubernetes, node-level packet drops (conntrack exhaustion, UDP buffer overruns) can also cut CoreDNS off from its upstreams while CoreDNS itself looks healthy. The distinguishing sign is that the node, not the resolver, is dropping packets.

Quick checks

All of these are read-only and safe to run during an incident.

# 1. Confirm the black hole signal
curl -s http://localhost:9153/metrics | grep 'coredns_forward_healthcheck_broken_total'

# 2. See which upstreams are failing health checks
curl -s http://localhost:9153/metrics | grep 'coredns_forward_healthcheck_failures_total'

# 3. Measure the SERVFAIL impact
curl -s http://localhost:9153/metrics | grep 'coredns_dns_responses_total' | grep 'SERVFAIL'

# 4. Check whether latency is LOW (black hole) or HIGH (slow upstream drag)
curl -s http://localhost:9153/metrics | grep 'coredns_dns_request_duration_seconds'

# 5. Check whether the cache is still masking the outage
curl -s http://localhost:9153/metrics | grep -E 'coredns_cache_(hits|requests)_total'

Then test upstream reachability directly. Use a real DNS query, not a port probe:

# Test each upstream with an actual DNS query (do NOT use nc; UDP port probes
# are unreliable for DNS because a filtered port and an open one look identical)
dig @<upstream_ip> . NS +time=1 +tries=1

Run this from the same network namespace CoreDNS runs in (exec into the pod in Kubernetes). A response, even a REFUSED or SERVFAIL from the upstream itself, means the network path is alive and the problem is the resolver. A timeout means the path is dead: routing, firewall, or upstream outage.

How to diagnose it

1. Confirm the pattern, not just the counter. The black hole is defined by three signals moving together: healthcheck_broken_total incrementing, SERVFAIL rising as a share of forwarded responses, and request latency staying low or dropping. If latency is high instead, you are looking at slow upstream drag (goroutines piling up, max_concurrent pressure), which is a different incident with different fixes.

2. Check the zone and plugin labels on SERVFAIL responses. coredns_dns_responses_total{rcode="SERVFAIL"} carries a plugin label. If SERVFAILs come from forward, this is an upstream problem. If they come from kubernetes, the cluster DNS state is broken and this article is the wrong one. The zone label tells you whether the failure is global or specific to one split-horizon zone.

3. Quantify the cache masking window. Compute the cache hit ratio (coredns_cache_hits_total / coredns_cache_requests_total). A hit ratio that is still near baseline while upstreams are dead means the worst is coming: as TTLs expire, forwarded queries will fail. A hit ratio already collapsing means you are in the full failure phase.

4. Test each upstream individually with dig. List the upstreams from the Corefile and probe each one. All timing out together points at shared infrastructure (routing, firewall, tunnel, provider). One answering while others time out points at per-upstream issues and suggests the health checker may have flapped on transient stress.

5. Check for a local egress break. If the upstreams answer from other nodes but not from the CoreDNS pod or host, the break is local: NetworkPolicy, security group, node firewall, or node-level packet drops. Check for conntrack exhaustion (nf_conntrack: table full in the kernel log) and UDP buffer errors on the node before blaming the resolver.

6. Decide whether the failure is real or a health check artifact. If dig gets answers from the “unhealthy” upstreams but the health checker still marks them down, you have a health check configuration problem, not an upstream outage. The short-term mitigation is a Corefile change, not a network fix.

Metrics and signals to monitor

Alerting note: the counter alone is not page-safe. It flaps with single-upstream configs, cold start instability, and transient stress, and by default it does not guarantee client impact because CoreDNS still tries a random unhealthy upstream. Page on the composite: healthcheck_broken_total incrementing continuously AND sustained SERVFAIL impact AND failed resolution of a critical name. Ticket on any increment.

Fixes

Restore the network path

If dig shows timeouts to all upstreams, the fix is in the path, not in CoreDNS. Revert the firewall or NetworkPolicy change, repair the tunnel or route, or fail over to a secondary egress path. CoreDNS recovers on its own once the health checks start passing again; no restart is needed.

Replace or add upstreams

If the upstreams themselves are down (provider outage, decommissioned resolvers), update the forward directive in the Corefile with working resolvers and let the reload plugin pick up the change. In Kubernetes, edit the coredns ConfigMap. Always configure at least two independent upstreams so a single resolver failure never trips the all-broken state.

Decide on failfast behavior explicitly

Consider setting failfast_all_unhealthy_upstreams (v1.12.1+) if you want deterministic behavior during total upstream loss: every forwarded query gets an immediate SERVFAIL instead of being sprayed at a random dead upstream. The tradeoff: during a health check flap where upstreams actually still answer, default behavior keeps some queries succeeding while failfast fails all of them. If you enable it, pair it with serve_stale in the cache plugin so clients get last-known-good answers during the outage instead of pure failure.

Fix health check configuration, if that is the real problem

If upstreams answer real queries but fail health checks, review the health check settings in the forward block. Setting max_fails 0 disables health checking entirely, which makes upstreams always considered healthy; that eliminates the black hole signal but also eliminates your early warning. It is usually better to fix whatever is dropping the health check probes.

Ride out the SERVFAIL cache

After upstreams recover, SERVFAIL responses may persist for up to the negative cache duration (5 seconds by default). Do not mistake this for continued failure. Verify recovery with a direct dig against CoreDNS for an uncached name.

Prevention

• Multiple independent upstreams. Single-upstream configurations make the broken counter flap-prone and turn one resolver’s bad day into a full outage. Two or more upstreams on independent infrastructure.
• Monitor per-upstream health, not just the aggregate. Dashboard coredns_forward_healthcheck_failures_total by to label so you see one upstream dying before it becomes all upstreams dying.
• Alert on the composite, not the counter. Combine healthcheck_broken, SERVFAIL ratio, and low-latency fast failure into one page-level rule.
• Test egress changes against DNS. Firewall, NetworkPolicy, and security group changes should be validated with a dig from a CoreDNS pod before rollout.
• Consider serve_stale and explicit failfast. Together they make total upstream loss a degraded-but-answering state rather than a cliff.
• Stagger CoreDNS restarts. A rolling restart during a masked upstream outage cold-starts the cache and removes the masking layer at the worst possible time.

How Netdata helps

• Netdata charts coredns_forward_healthcheck_broken_total as a rate, so the onset of the all-broken state is visible the moment it starts rather than after cache TTLs expire.
• Per-upstream health check failures (coredns_forward_healthcheck_failures_total) are broken out by to, making it obvious whether upstreams failed together (shared path) or in sequence (per-resolver issues).
• SERVFAIL rate with the plugin and zone labels is charted alongside query rate, so you can confirm the failures come from forward and compute the failing share of forwarded traffic.
• Request latency percentiles next to SERVFAIL rate expose the defining signature directly: fast failure with low latency, not slow-upstream drag.
• Cache hit ratio on the same dashboard shows the masking window in real time, letting you predict when the full failure will surface.
• Correlating these on one screen shortens the loop from “DNS is broken” to “the egress firewall change 20 minutes ago cut off all upstreams.”

Related guides

• How CoreDNS actually works in production: the plugin chain mental model
• CoreDNS monitoring checklist: the signals every production resolver needs
• CoreDNS monitoring maturity model: from survival to expert
• CoreDNS returning REFUSED: no matching zone, an ACL, or the forward concurrency limit
• CoreDNS returning SERVFAIL: the resolver is failing queries and what to check first