CoreDNS not resolving external domains: the missing catch-all forward zone

The symptom is specific: names inside the cluster resolve, but anything outside the cluster fails. kubernetes.default.svc.cluster.local works. example.com does not. Applications report DNS timeouts, getaddrinfo failures, or intermittent external dependency errors while service discovery inside Kubernetes looks fine.

That split is the clue. CoreDNS is not one global resolver. It matches each query to the most specific server block in the Corefile, then runs that block’s plugin chain. If cluster.local is handled by the kubernetes plugin but there is no catch-all block for ., external names match nothing useful and are refused instead of forwarded.

The usual root cause is a missing or wrong forward . <upstream> statement. In Kubernetes the common pattern is forward . /etc/resolv.conf, but that is only safe when the resolv.conf CoreDNS sees points to a real upstream resolver. If it points back at the cluster DNS Service or another path that returns to CoreDNS, the loop plugin can kill the process at startup and turn a bad Corefile into CrashLoopBackOff.

What this means

A query for api.default.svc.cluster.local should match the Kubernetes zone and be answered from the in-memory Service and Endpoint state built by the kubernetes plugin. A query for api.partner.com must fall through to a server block that covers . and reach the forward plugin.

When the catch-all is absent, CoreDNS has no forwarding path for the name. The failure is immediate and cheap: REFUSED, not a long upstream timeout. When the catch-all exists but points somewhere invalid, the failure changes shape: loops crash at startup, dead upstreams produce SERVFAIL or latency, and a concurrency limit produces REFUSED under load.

flowchart LR
  A[Client DNS query] --> B{Name matches cluster.local?}
  B -- yes --> C[kubernetes plugin answers]
  B -- no --> D{Server block for . exists?}
  D -- no --> E[REFUSED: no matching zone]
  D -- yes --> F{forward target valid?}
  F -- loops back --> G[loop plugin exits at startup]
  F -- dead upstream --> H[SERVFAIL or timeout]
  F -- healthy upstream --> I[External NOERROR]

Common causes

Quick checks

Run these read-only checks before changing the Corefile.

# 1. Confirm internal and external behavior from a client pod
nslookup kubernetes.default.svc.cluster.local
nslookup example.com

# 2. Query CoreDNS directly to bypass local resolver weirdness
dig @<coredns-pod-or-service-ip> kubernetes.default.svc.cluster.local +time=2 +tries=1
dig @<coredns-pod-or-service-ip> example.com +time=2 +tries=1

# 3. Inspect the active Corefile source in Kubernetes
kubectl get cm -n kube-system coredns -o yaml

# 4. Check whether the process is crashing instead of merely refusing
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system <coredns-pod> --tail=200

The metrics endpoint listens on the CoreDNS pod itself, so scrape it with kubectl exec -n kube-system <coredns-pod> -- curl -s http://localhost:9153/metrics, or port-forward first. curl http://localhost:9153/metrics from a random node will hit nothing.

# 5. Look for REFUSED and SERVFAIL by rcode
curl -s http://localhost:9153/metrics | grep 'coredns_dns_responses_total' | grep -E 'REFUSED|SERVFAIL'

# 6. Check forward concurrency rejects
curl -s http://localhost:9153/metrics | grep 'coredns_forward_max_concurrent_rejects_total'

# 7. Check upstream health signals
curl -s http://localhost:9153/metrics | grep -E 'healthcheck_failures_total|healthcheck_broken_total'

A fast REFUSED for example.com while cluster.local answers normally points strongly at zone matching, not upstream slowness. A slow failure points more toward forwarding, upstream health, or packet loss.

How to diagnose it

1. Prove the split. From the same client, resolve one internal name and one external name directly against CoreDNS. If internal is NOERROR and external is REFUSED, stay on the Corefile path. If both fail, check process liveness, Service endpoints, conntrack, and node UDP drops first.

2. Read the Corefile as CoreDNS loaded it. Inspect the coredns ConfigMap, but also check coredns_reload_failed_total and logs. A ConfigMap can contain the intended fix while the running pods still use the old configuration because reload failed.

3. Confirm there is a catch-all path. Look for a server block covering . or an explicit forward . <target> in the block that external queries will match. If the Corefile only defines cluster.local and more specific internal zones, external names have nowhere to go.

4. Check ordering in split-horizon setups. Put the most specific forwarding zones before broader ones. A private zone such as corp.example.internal must be matched before example.com, and both before the final forward . <upstream> catch-all. If a broad zone is placed first, it can capture names you intended for a different upstream.

5. Validate the forward target independently. From the CoreDNS pod or node network namespace, run a real DNS query against the upstream: dig @<upstream-ip> example.com +time=2 +tries=1. Do not rely on a UDP port probe alone. A reachable IP that does not answer DNS is still a dead resolver.

6. Rule out the loop case before applying forward . /etc/resolv.conf. The standard Kubernetes pattern is valid only when the resolv.conf seen by CoreDNS points to a real upstream, not back at the cluster DNS Service or CoreDNS itself. If pods are in CrashLoopBackOff and logs contain loop detection, fix the target first. The loop plugin is doing its job.

7. Separate capacity REFUSED from configuration REFUSED. If REFUSED rises only during bursts, check coredns_forward_max_concurrent_rejects_total, upstream latency, and goroutine growth. That is backpressure, not a missing zone.

8. Use labels to isolate scope. On coredns_dns_requests_total and coredns_dns_responses_total, compare the zone, server, and rcode labels. Exact label sets depend on your CoreDNS version, but the goal is constant: prove cluster.local is healthy while the external path is REFUSED, SERVFAIL, or absent.

Metrics and signals to monitor

Fixes

Add the missing catch-all

If external queries match no server block, add a catch-all forwarding path. In a typical Kubernetes Corefile this means keeping cluster.local on the kubernetes plugin and forwarding everything else:

. {
    errors
    health
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
    }
    prometheus :9153
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
}

Do not copy this blindly. Keep your existing plugins, ports, cache sizing, and zone layout. The important part is that queries outside the Kubernetes zones reach a valid forward . <upstream> target.

Point the catch-all at a real upstream

forward . /etc/resolv.conf is correct only when that resolv.conf contains usable upstream resolvers from CoreDNS’s point of view. If it contains the cluster DNS Service, a loopback stub that routes back into the cluster, or a resolver reachable only from the host and not from pods, replace it with explicit upstream IPs or fix the resolver source.

If you change to explicit upstreams, test each one with dig @<upstream-ip> example.com before rollout. Prefer at least two independent upstreams so one failure degrades redundancy instead of removing external resolution.

Fix split-horizon order

For split-horizon DNS, order matters. Put specific private zones first, broader parent zones second, and the catch-all last. Verify with queries that should hit each branch: one private name, one public name under the same parent, and one unrelated external name.

Treat max_concurrent REFUSED as capacity

If REFUSED correlates with coredns_forward_max_concurrent_rejects_total, do not fix it by adding more zones. Check upstream latency, goroutine accumulation, and whether the configured concurrency limit is too low for peak QPS times upstream response time. Removing backpressure entirely can trade REFUSED for memory growth and OOM risk.

Recover from a loop crash safely

If logs show loop detection, do not remove the loop plugin to make the crash stop. Change the forwarding target so it no longer returns to CoreDNS. The loop check runs at startup, so a runtime config change can still leave you exposed until restart. After fixing the Corefile, watch the next rollout closely.

Prevention

• Keep one tested external path. Every Corefile should have an explicit, reviewed answer for “where does a name outside our zones go?” If the answer is resolv.conf, document what that file contains in the environment where CoreDNS runs.
• Guard ConfigMap changes with reload and canary checks. After any Corefile change, check coredns_reload_failed_total, pod readiness, one internal lookup, and one external lookup.
• Alert on REFUSED and SERVFAIL separately. REFUSED often means policy, zone matching, or concurrency. SERVFAIL often means upstream or plugin failure. Combining them hides the mechanism.
• Dashboard by zone and plugin. Keep views for cluster.local, forwarded zones, and response codes. This is the fastest way to see “internal fine, external broken” during an incident.
• Test the loop path before rollout. Any change to forward ., node resolver configuration, cluster DNS Service IP, or stub resolvers deserves a preflight check for circular resolution.
• Stagger rollouts. Cache is cold after restart. Restarting all CoreDNS pods at once can amplify external failures into an upstream thundering herd.

How Netdata helps

• Netdata can chart CoreDNS responses by rcode, zone, and server so REFUSED from a missing catch-all is visually separate from SERVFAIL from upstream failure.
• Per-upstream to breakdowns on forward latency and health check failures help identify whether external resolution is failing because of one bad upstream or the whole forwarding path.
• Correlating REFUSED with coredns_forward_max_concurrent_rejects_total separates configuration errors from backpressure during traffic spikes.
• Pairing CoreDNS metrics with pod restarts and logs helps catch the forwarding-loop case, where metrics may be absent because the process exits before scrape.
• Tracking reload failures beside Corefile change events prevents the common “we edited the ConfigMap but the old config is still running” trap.

Related guides

• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• How CoreDNS actually works in production: the plugin chain mental model
• CoreDNS monitoring checklist: the signals every production resolver needs
• CoreDNS monitoring maturity model: from survival to expert
• CoreDNS NOERROR with zero answers: the resolution failure that reports success
• CoreDNS NXDOMAIN vs SERVFAIL: why alerting on the wrong one buries real incidents
• CoreDNS per-upstream health check failures: degraded redundancy before total loss
• CoreDNS query rate dropped to zero while the process looks healthy
• CoreDNS returning REFUSED: no matching zone, an ACL, or the forward concurrency limit
• CoreDNS returning SERVFAIL: the resolver is failing queries and what to check first
• CoreDNS slow upstream: per-upstream latency, goroutine pileup, and the to label