CoreDNS Loop detected and CrashLoopBackOff: the forwarding loop that kills the pod

Your CoreDNS pods are in CrashLoopBackOff. kubectl logs shows a line like [FATAL] plugin/loop: Loop (127.0.0.1:53 -> :53) detected for zone ".", and the process exits a few seconds after every start. There are no CoreDNS metrics to look at, because the process dies before the metrics endpoint is ever scraped.

The crash is intentional. The loop plugin detected that the Corefile’s forward target routes queries back to CoreDNS itself, and it killed the process on purpose to prevent an infinite query amplification storm. Kubernetes restarts the pod, the loop is detected again, and the cycle repeats.

The fix is always in the same place: the forward target in the Corefile. This guide covers how the detection works, how to confirm which misconfiguration you have, and how to fix it without removing the safety mechanism.

What this means

CoreDNS ships with a loop plugin whose only job is to detect forwarding loops. At startup, before serving traffic, it sends a self-referencing probe query (type HINFO) through its own forwarding path. If the probe comes back to CoreDNS, there is a loop: queries forwarded “upstream” are actually returning to CoreDNS, which forwards them again. Left running, this amplifies a single query into an unbounded flood that exhausts goroutines and memory.

Instead of letting that happen, the loop plugin calls log.Fatalf, which calls os.Exit(1). The process dies within seconds of starting. Kubernetes sees the exit, restarts the pod per its restart policy, and the loop is detected again. That is the CrashLoopBackOff you are observing.

Two properties of this mechanism matter operationally:

• Detection happens only at startup. The probe runs once during initialization. A loop introduced later (for example by a change on the node or in an upstream resolver) is not detected at runtime.
• There are no metrics. The process exits before the Prometheus endpoint on :9153 serves anything scrapeable. Do not waste time looking for SERVFAIL or latency metrics from the crashing pod. Diagnosis is pod status plus logs.

The typical Corefile misconfigurations that produce this:

• forward . /etc/resolv.conf where the node’s /etc/resolv.conf points back at cluster DNS (directly, or through a local stub resolver that ultimately resolves via cluster DNS).
• forward . <kube-dns ClusterIP> or any address that is CoreDNS’s own Service IP.
• forward . 127.0.0.1:53 (or another loopback address) where whatever listens there forwards back to CoreDNS.

flowchart TD
  A[CoreDNS pod starts] --> B[loop plugin sends HINFO probe at startup]
  B --> C{Probe returns to CoreDNS?}
  C -- yes --> D[log.Fatalf -> os.Exit 1]
  D --> E[Kubernetes restarts pod]
  E --> A
  C -- no --> F[loop plugin disables itself, pod serves DNS]
  G[forward target: node resolv.conf, ClusterIP, or 127.0.0.1] -. routes back .-> B

Common causes

Quick checks

All read-only.

# 1. Confirm the pod state and restart count
kubectl get pods -n kube-system -l k8s-app=kube-dns

# 2. Read the logs from the crashed container (the key check)
kubectl logs -n kube-system <coredns-pod> --previous
# Look for "Loop detected" from plugin/loop, with the zone and addresses involved

# 3. Read the running Corefile
kubectl get cm -n kube-system coredns -o yaml
# Find every "forward" line and note its targets

# 4. On a node that runs CoreDNS, inspect resolv.conf
cat /etc/resolv.conf
# If forward . /etc/resolv.conf is in the Corefile, every nameserver here must be a real upstream

# 5. Verify the intended upstream actually answers
dig @<upstream-ip> . NS +time=1 +tries=1

Two notes on these checks. First, use --previous in step 2 because the current container instance may have just started and not yet printed the fatal line; the previous terminated instance holds the definitive message. Second, TCP connect checks like nc -z tell you nothing about a UDP DNS resolver, which is why step 5 issues an actual DNS query.

How to diagnose it

1. Confirm the signature. Pod in CrashLoopBackOff plus “Loop detected” in the logs is conclusive. The log line names the zone and the addresses involved in the loop, which tells you which server block and which forward target is at fault.
2. Identify the forward target. In the coredns ConfigMap, find the forward directive for the zone named in the log line (usually .). Note exactly what it points to: a file (/etc/resolv.conf), an IP, or a list of IPs.
3. Trace the target back to CoreDNS. If the target is /etc/resolv.conf, read that file on the node. Every nameserver listed there must resolve outside the cluster. If any nameserver is the kube-dns ClusterIP, a loopback stub that chains to cluster DNS, or CoreDNS’s own address, you have found the loop. If the target is a literal IP, check whether it is the cluster DNS Service ClusterIP or a loopback address.
4. Check for a local stub resolver. Some node images run a local DNS stub on a loopback address, and /etc/resolv.conf points at that stub rather than the real upstream. The stub’s own configuration then decides where queries go. If the stub ultimately forwards to cluster DNS, the loop exists even though resolv.conf looks innocuous. Inspect the stub resolver’s configuration on the node to see its real upstreams.
5. Fix the forward target, apply the ConfigMap change, and watch the pods come up. The pods should pass the startup probe and stay running. If they loop again, the new target still routes back; repeat step 3.

If you remove the loop plugin from the Corefile “to test”, understand what you are doing: you are disabling the only protection against a real forwarding loop. A genuine loop left running causes unbounded query amplification, goroutine exhaustion, and OOM. Treat plugin removal as a short diagnostic step only, and never as the fix.

Metrics and signals to monitor

There are no CoreDNS process metrics for this failure. The signals are Kubernetes-level and client-level.

Fixes

Fix the forward target to a real upstream

This is the definitive fix for every variant of this incident. Edit the coredns ConfigMap:

kubectl edit cm -n kube-system coredns

Change the offending forward line so it points at DNS resolvers that are genuinely upstream of the cluster: your cloud provider’s VPC resolver, your organization’s resolvers, or public resolvers, as appropriate for your environment. Never the cluster DNS ClusterIP, never a loopback address, and never a resolv.conf whose nameservers route back into the cluster.

Tradeoff: if you hardcode upstream IPs instead of /etc/resolv.conf, you own keeping them current when the network changes. If you keep /etc/resolv.conf, you own guaranteeing that node provisioning never writes a nameserver that resolves via cluster DNS.

Fix the node’s resolv.conf when using forward . /etc/resolv.conf

If you want to keep forwarding to /etc/resolv.conf (the Kubernetes default Corefile does), then the node file must contain real upstreams. On nodes whose resolv.conf points at a local stub resolver, configure the kubelet to use the stub’s underlying upstream file instead, or reconfigure the stub so it never chains to cluster DNS. The exact mechanism depends on your distro and cluster installer.

On platforms where you cannot touch the node at all (managed or serverless node offerings), the node-side fix is unavailable and editing the Corefile forward target is the only option. On EKS Fargate, for example, the documented workaround is to replace forward . /etc/resolv.conf with the VPC resolver address (the VPC CIDR base plus 2, such as 10.0.0.2 for a 10.0.0.0/16 VPC).

Roll back a bad Corefile edit

If the loop appeared immediately after a ConfigMap change, the fastest path back is restoring the previous Corefile. Keep a known-good copy in version control so this is a revert, not a reconstruction.

Prevention

• Keep the loop plugin in the Corefile. The crash is the protection working. Teams that delete the plugin to stop the restarts trade a loud, fixable CrashLoopBackOff for a silent amplification storm that ends in OOM.
• Gate Corefile changes. The Corefile is a ConfigMap, so nothing validates it before apply. Review forward targets in change management, and roll CoreDNS deliberately after any edit so a bad config is caught on one pod, not discovered during an unrelated incident.
• Audit node images for DNS stubs. When you adopt a new base image or distro, check what /etc/resolv.conf contains and where any local stub resolver forwards. This is the most common way the loop appears on previously healthy clusters: the Corefile never changed, the node did.
• Remember detection is startup-only. A loop introduced at runtime (a node resolver reconfigured, an upstream changed) will not be caught until the next CoreDNS restart, and an undetected loop manifests as goroutine and memory growth, not a clean crash. If you see CoreDNS goroutine count climbing with rising upstream query rates and no client traffic change, suspect a runtime loop. See CoreDNS goroutine count climbing: blocked upstream calls and leaks.

How Netdata helps

• Netdata’s Kubernetes monitoring surfaces pod state and restart counts, so a CoreDNS pod entering CrashLoopBackOff triggers an alert immediately rather than being discovered via application DNS failures.
• Per-pod visibility shows whether all replicas are looping or only those scheduled onto nodes with the bad resolver configuration, which narrows the root cause to node-level versus Corefile-level.
• Correlating the restart storm with client-side signals (application errors, falling cluster DNS query volume on surviving replicas) quantifies blast radius while you fix the forward target.
• After the fix, Netdata’s per-second CoreDNS metrics (query rate, SERVFAIL ratio, latency by zone) confirm the pods are serving again and the cache is rewarming normally, instead of you inferring recovery from pod status alone.
• Alerting on CoreDNS pod restarts in general also catches the adjacent failure modes that produce restart loops, such as OOMKilled from memory pressure, which present similarly at the Kubernetes level but very differently in the logs.

Related guides

• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• CoreDNS cache collapse: the cold-cache thundering herd after a rollout
• CoreDNS cache evictions: the cache is too small for the working set
• CoreDNS cache hit ratio dropping: latency and upstream load climbing together
• CoreDNS CPU throttling: CFS limits making a green dashboard lie about latency
• CoreDNS DNS programming latency: how long a Service takes to become resolvable
• CoreDNS not resolving external domains: the missing catch-all forward zone
• CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed
• CoreDNS GC pauses adding tail latency: go_gc_duration_seconds and heap pressure
• CoreDNS goroutine count climbing: blocked upstream calls and leaks
• CoreDNS high request latency: reading P99 by zone to find the cause
• How CoreDNS actually works in production: the plugin chain mental model