$ guides / coredns / coredns-crashloopbackoff-triage ▌

Operations Guides

CoreDNS CrashLoopBackOff: triaging loop, OOM, config, and port-bind causes

A CoreDNS pod in CrashLoopBackOff means the process starts, dies, and gets restarted by Kubernetes in a tightening backoff loop. While this is happening, at least one DNS replica is contributing nothing. If all replicas are crash-looping, the kube-dns Service has no backends and cluster-wide name resolution is down: queries hang at the client until a pod recovers.

CrashLoopBackOff is not one failure. It is a symptom with a small set of common root causes, and they need different fixes. The four you will see most often: the loop plugin detecting a forwarding loop, the OOM killer terminating the container, a Corefile that fails to parse, and a port-bind failure on startup. RBAC and Kubernetes API access failures show up in the same pod state too, usually as fatal list errors or persistent un-readiness.

The first move is always the same: read the previous container’s logs and the pod’s last-terminated reason. Those two pieces of information route you to the correct runbook in under a minute. This page is that router.

What this means

CrashLoopBackOff is a Kubernetes container state, not a CoreDNS error. Kubernetes runs the container, the container exits, and the kubelet restarts it with exponential backoff (seconds growing to minutes). The exit reason and the logs from the dying process are the diagnosis.

Two consequences matter for triage:

Metrics are usually missing. If CoreDNS exits within seconds of starting (loop detection, parse error, bind failure), the Prometheus endpoint on :9153 never gets scraped. Do not waste time looking for dashboards. The evidence is in pod status and logs.
The backoff hides the real cadence. Restart count climbing by one every few minutes looks calm, but each restart may be an immediate crash. kubectl describe pod shows the actual last state and exit reason.

flowchart TD
  A[CoreDNS pod in CrashLoopBackOff] --> B[kubectl describe pod: Last State reason]
  B --> C{Reason}
  C -->|OOMKilled| D[Memory: limit too low for snapshot and cache]
  C -->|Error / exit 1| E[kubectl logs --previous]
  E --> F{Log content}
  F -->|Loop detected| G[Forwarding loop: fix forward target]
  F -->|Parse error / unknown directive| H[Corefile error: fix ConfigMap]
  F -->|bind: address already in use| I[Port conflict on 53]
  F -->|Failed to list / 403| J[RBAC or API access]

Common causes

Cause	What it looks like	First thing to check
Forwarding loop	Pod restarts within seconds; logs show “Loop detected” for a zone	`kubectl logs --previous` for the loop message; inspect the `forward` target in the Corefile
OOMKilled	`Last State: Terminated, Reason: OOMKilled`; pod reaches Running briefly then dies again	Memory limit vs. steady-state RSS and the restart re-list spike
Corefile parse or plugin config error	Process exits immediately at load; logs show a parse error or unknown directive	The `coredns` ConfigMap and the exact error line in logs
Port-bind failure	Process exits at startup; logs show a bind failure on the listen port	What else is bound to port 53 on that node (node-local DNS, another resolver)
RBAC / API access	Logs show “Failed to list” errors or 403s from the API server; pod may stay unready	CoreDNS ServiceAccount ClusterRole and API reachability

Quick checks

These are all read-only.

# 1. Pod status and restart count
kubectl get pods -n kube-system -l k8s-app=kube-dns

# 2. Last terminated state: OOMKilled vs Error vs Completed
kubectl describe pod -n kube-system <coredns-pod>

# 3. Current container logs
kubectl logs -n kube-system <coredns-pod>

# 4. Previous container logs (the crash you are actually debugging)
kubectl logs -n kube-system <coredns-pod> --previous

# 5. The Corefile as the pod sees it
kubectl get cm -n kube-system coredns -o yaml

Pod status. Restart count incrementing with short intervals between restarts confirms an active crash loop, not a historical one.
describe pod. The Last State block is the fastest fork in the road: Reason: OOMKilled routes you to memory; Reason: Error with an exit code routes you to logs. Also check Events for probe failures and scheduling context.
Current logs. May be empty or truncated if the container has already exited again.
Previous logs. The one people forget. When the current container just started, --previous holds the output from the run that actually crashed. Loop detection and parse errors show up here.
ConfigMap. Confirm the ConfigMap content matches what you think is deployed. A stale or hand-edited ConfigMap is the source of most parse errors.

How to diagnose it

Step 1: Fork on the last-terminated reason

Run kubectl describe pod -n kube-system <coredns-pod> and read Last State:

Reason: OOMKilled: go to Step 3.
Reason: Error: go to Step 2 and read the previous logs.

Step 2: Read the previous logs and match the message

kubectl logs -n kube-system <coredns-pod> --previous

Match what you see against these signatures:

“Loop detected” with a zone name. The loop plugin sends a self-referencing HINFO probe query at startup. If the probe comes back, CoreDNS concludes it is forwarding to itself and calls log.Fatalf, exiting deliberately. The crash is intentional; do not remove the loop plugin to silence it. The probe runs at startup, so a loop introduced by a mid-operation config change may not be caught until the next restart. See CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken for the forwarding-path context, but the fix here is in the Corefile forward target.
Parse error, “unknown directive”, or invalid plugin option. The Corefile failed to load, and the error line names the directive. Common sources: a plugin name that does not exist in this CoreDNS version (the long-removed proxy directive instead of forward, or ready in a build that predates it), a malformed server block, or a typo in a plugin option. A missing space before { in a server block is a classic source of confusing parse errors.
Bind failure on the listen port. Something else on the node already holds port 53 (or whichever port the server block specifies). On nodes running systemd-resolved, dnsmasq, or a node-local DNS cache, expect this if the pod is scheduled there with hostNetwork or a conflicting port.
“Failed to list” or HTTP 403 from the API server. The kubernetes plugin cannot list or watch Services, Endpoints, or EndpointSlices; the ServiceAccount’s ClusterRole is missing or missing verbs. There is no single “watch broken” metric: these failures surface in logs and as code=403 on coredns_kubernetes_rest_client_requests_total once the process survives long enough to be scraped.

Step 3: For OOMKilled, confirm the growth pattern

OOMKilled means the container’s memory crossed its cgroup limit and the kernel killed it. For CoreDNS this is usually one of two shapes:

Gradual climb. Heap grows over hours and never returns to baseline after GC. Causes: cache sized too large for the limit, a goroutine leak holding memory, or the kubernetes plugin’s in-memory snapshot growing with cluster size.
Restart spike. The pod OOMs again almost immediately after starting. On restart, the kubernetes plugin does a full re-list of Services, Endpoints, and optionally Pods, and that re-list is the peak memory event. If the limit was sized to steady-state RSS with little headroom, the re-list peak kills the pod and you get a crash loop. Memory limits need headroom for the restart spike, not just steady state.

Check process_resident_memory_bytes (RSS) against the container limit if you have historical metrics. RSS, not go_memstats_heap_inuse_bytes, is what the OOM killer sees. Practical thresholds: warn at 80% of limit, page at 90%, and keep at least 50% headroom over steady-state RSS because the re-list peak can be 2-3x steady state in large clusters.

Metrics and signals to monitor

These signals matter once the pod survives long enough to scrape, and for predicting the next crash loop.

Signal	Why it matters	Warning sign
Pod restart count	Direct crash-loop indicator	Any restart in the last hour warrants log review
`process_resident_memory_bytes` vs. container limit	RSS is what the OOM killer measures	> 80% of limit; post-GC minimum trending up
`go_goroutines`	Leaks and blocked upstream calls consume memory	Growth that never returns to the 20-50 baseline
`coredns_reload_failed_total`	A pushed Corefile change failed to apply; running config differs from intended	Any nonzero value
`coredns_panics_total`	Recovered panics are bugs; unrecovered ones crash the process	Any nonzero rate
`coredns_kubernetes_rest_client_requests_total` by `code`	403 means RBAC; 5xx or connection errors mean API trouble	Any 403; sustained 5xx
`coredns_dns_requests_total{type="HINFO"}`	The loop plugin’s probe query; normal at startup, abnormal if sustained	Sustained HINFO queries

Fixes

Forwarding loop

Fix the forward target so it points at a real upstream, never back at CoreDNS. The classic Kubernetes case: the Corefile says forward . /etc/resolv.conf, and the node’s /etc/resolv.conf points at the cluster DNS Service or a local stub that resolves through the cluster, closing the circle. Other variants: forwarding to CoreDNS’s own ClusterIP, or forwarding to another resolver that itself forwards back. Change the target to actual upstream resolver IPs (or a resolv.conf that names them), roll the ConfigMap, and confirm the pods come up and stay up.

OOMKilled

Short term: raise the container memory limit so the restart re-list fits, since a crash-looping pod serves no DNS at all. Medium term: right-size against steady-state RSS with at least 50% headroom, because the re-list peak in a large cluster can be 2-3x steady state. If the growth pattern is a leak (post-GC minimum rising for hours, goroutine count climbing), reducing cache size and chasing the leak matters more than raising the limit. See CoreDNS goroutine count climbing: blocked upstream calls and leaks and CoreDNS cache evictions: the cache is too small for the working set.

Corefile parse or plugin config error

Fix the directive named in the log line, then let the ConfigMap roll out. Two guardrails:

If the Corefile has the reload plugin, a bad change triggers coredns_reload_failed_total and CoreDNS keeps the old config. That is survivable. A bad Corefile present at process start is not: the process exits and you are in CrashLoopBackOff. Validate Corefile changes before they reach the ConfigMap.
Keep plugin names consistent with the CoreDNS version you run. Directives from older Corefiles (like proxy) do not exist in current builds and fail at load.

Port-bind failure

Identify what holds the port and decide which process owns it. If a node-local DNS cache or systemd-resolved legitimately owns 53 on that node, the fix is on the CoreDNS side (listen port, scheduling, or hostNetwork choice), not on killing the local resolver. Confirm the port is free on the target node before expecting the pod to start.

RBAC / API access

Restore the ServiceAccount’s ClusterRole permissions to list and watch the resources the kubernetes plugin needs (Services, Endpoints, EndpointSlices, and namespaces). code=403 in the API client metrics is the tell. If the API server itself is unreachable, the pod may start but stay unready; that is a readiness problem, not a crash loop.

Prevention

Validate Corefiles before rollout. A parse error that reaches a running pod via ConfigMap plus restart is a self-inflicted outage. Render and check the Corefile in CI or with a canary pod.
Size memory for the restart spike, not steady state. Keep at least 50% headroom over steady-state RSS so the kubernetes plugin’s re-list does not OOM the pod on every restart.
Alert on restart count and reload failures. coredns_reload_failed_total nonzero and any pod restart in the last hour should both page a human before the backoff loop becomes a full DNS outage.
Pin upstreams explicitly. Avoid forwarding targets that can resolve back to the cluster (node resolv.conf files that point at cluster DNS, ClusterIPs). Loops are configuration errors: cheap to prevent, expensive at 3 a.m.
Know what else binds port 53 on nodes where CoreDNS can schedule, especially alongside node-local DNS.

How Netdata helps

Restart and OOM context in one view. Netdata correlates container restart events, Last State reasons, and cgroup memory against the container limit, so an OOMKilled pod shows its RSS climb into the limit instead of appearing as an unexplained restart.
Pre-crash memory signals. process_resident_memory_bytes, go_memstats_heap_inuse_bytes, and go_goroutines are collected per pod at per-second granularity, which catches the gradual-leak pattern and the re-list spike that minute-resolution scraping misses.
Reload and panic counters. coredns_reload_failed_total and coredns_panics_total are surfaced directly, so a bad Corefile push shows up as a failed reload before someone restarts the pods into a crash loop.
API client error codes. coredns_kubernetes_rest_client_requests_total broken down by code distinguishes RBAC failures (403) from API server trouble (5xx) without log spelunking.
Per-replica comparison. Each CoreDNS pod is charted independently, so a single replica crash-looping on one node (local port conflict or node-level memory pressure) stands out from a fleet-wide config problem.