CoreDNS Corefile parse error: startup failures and invalid plugin config

You changed the Corefile, and now one of two things has happened. Either the CoreDNS pods are in CrashLoopBackOff and cluster DNS is down, or the pods look fine but your change never took effect and the log is full of parse errors. Both are the same root event: the Corefile parser rejected your configuration. The difference is whether the parser ran at process start (fatal) or during a live reload (rejected, old config kept).

The log line operators search for is some variant of:

/etc/coredns/Corefile:5 - Error during parsing: Unknown directive 'xyz'

The error is more informative than most: it names the file, the line number, and usually the exact token the parser choked on. The hard part is that the two failure modes look completely different operationally, and the quiet one (failed reload) is more dangerous because nothing pages.

What this means

CoreDNS loads its entire configuration from the Corefile at startup. The parser is strict: any syntax error, unknown plugin directive, malformed option, or duplicate zone and port combination is fatal at load. There is no “start with warnings” mode.

There are two paths to the same error:

1. Fresh start. The process reads the Corefile, hits the parse error, and exits immediately. In Kubernetes this becomes CrashLoopBackOff: the pod starts, the parse fails, the process exits, Kubernetes restarts it, the parse fails again. No metrics are ever exported because the process dies before the metrics endpoint comes up.
2. Live reload. If the reload plugin is in the Corefile, CoreDNS polls the file every 30 seconds (plus or minus 15 seconds of jitter) and reloads when the SHA512 checksum changes. If the new Corefile has a parse error, CoreDNS logs the error and keeps running the old configuration. The metric coredns_reload_failed_total increments on each failed attempt.

flowchart TD
  A[Corefile changed] --> B{When is it parsed?}
  B -->|Process start| C[Parse fails: process exits]
  C --> D[CrashLoopBackOff, no metrics, DNS down]
  B -->|reload plugin poll, ~30s| E{Parse OK?}
  E -->|Yes| F[New config live, cache flushed]
  E -->|No| G[Log error, coredns_reload_failed_total +1]
  G --> H[Old config keeps serving]

The operational trap in path 2 is configuration drift: you applied a change, the change was rejected, DNS still works, and the running configuration no longer matches what your ConfigMap or Git repo says. Weeks later someone restarts the pods for an unrelated reason, the broken Corefile is parsed at startup, and now you have path 1 with no idea which old change caused it.

Common causes

The version-skew causes deserve emphasis. Most production parse errors are not random typos; they are a Corefile written against one CoreDNS version running on another. This happens constantly during Kubernetes upgrades, image bumps, and when copying a Corefile from a blog post written for a different release. The corefile-migration project tracks which plugins and options were added, deprecated, or removed in each CoreDNS version, and kubeadm runs a preflight check (CoreDNSUnsupportedPlugins) that blocks upgrades when the Corefile contains directives the target version does not support.

One thing that is not a parse error but looks related: plugin ordering inside a server block does not determine execution order. The plugin chain order is fixed at compile time. Rearranging plugins in the Corefile is syntactically valid but changes nothing at runtime, so do not try to fix behavior by reordering lines.

Quick checks

# 1. Get the actual parse error from the logs (Kubernetes)
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100 | grep -iE "error|parse|reload"

# 2. Check pod state: CrashLoopBackOff means the parse failed at startup
kubectl get pods -n kube-system -l k8s-app=kube-dns

# 3. Look at the Corefile exactly as CoreDNS sees it
kubectl get cm -n kube-system coredns -o yaml

# 4. Port-forward the metrics endpoint (9153 by default)
kubectl port-forward -n kube-system deploy/coredns 9153:9153 &

# 5. Check whether reloads are being rejected (old config still running)
curl -s http://localhost:9153/metrics | grep coredns_reload_failed_total

# 6. Confirm which config hash is actually loaded
curl -s http://localhost:9153/metrics | grep coredns_reload_version_info

# 7. Check which CoreDNS version is running (version skew is the usual culprit)
kubectl get deploy -n kube-system coredns -o jsonpath='{.spec.template.spec.containers[0].image}'

All of these are read-only. Two caveats. First, check 1 only works if the pod started at all; if it is crash-looping, use kubectl logs -n kube-system <pod> --previous to get logs from the last terminated container. Second, do not try to kubectl exec into the CoreDNS container to curl localhost: the container image is minimal and typically has no shell, curl, or wget. Port-forwarding is the reliable path.

How to diagnose it

1. Determine which failure mode you are in. kubectl get pods: CrashLoopBackOff means fatal-at-startup. Pods Running but change not applied means a rejected live reload. The diagnostics differ from here.

2. Read the error message literally. The parse error names the file, the line, and the token. Go to that line in the ConfigMap. The named token is either an unknown directive (first word on the line is not a plugin this binary knows), an unknown property (a bad option inside a plugin block), or a structural problem (braces, zone syntax) near that line.

3. Check version compatibility before anything else. Take the named token and check it against the plugin documentation for the exact CoreDNS version you are running, not the latest docs. Known gates: proxy removed in 1.5.0, the kubernetes plugin’s upstream option removed around the same time , max_concurrent in the forward plugin added later, import change detection in reload added in 1.7.0 . If your Corefile came from documentation for a newer release, this is almost certainly your cause.

4. Check brace and whitespace structure. The Corefile is whitespace-sensitive, and this is poorly documented. example.com:1053{ (no space before the brace) is not a syntax error in the way you would expect; it can be misparsed as a zone named example.com:1053{ plus a default zone, producing deeply confusing errors elsewhere in the file. If the reported line looks innocent, look at the lines immediately above it for a missing space or an inline brace.

5. Check for directives outside server blocks. A bare health or prometheus line at the top of the Corefile is parsed as a zone name. Everything after it shifts meaning, and the actual error often lands on a later line (classically Unknown directive '.'). If the error line number does not match anything obviously wrong, scan upward for a stray directive.

6. For rejected reloads, compare intended vs running config. The running process still has the old configuration. Pull the ConfigMap (intended) and check coredns_reload_version_info for the hash of what is actually loaded. Treat every increment of coredns_reload_failed_total as “the cluster is not running what you think it is running.”

7. Be aware of the reload listener edge case. If the new Corefile changes listener ports and the new port fails to bind, the old listener may already be closed. DNS keeps serving on the old config, but health or metrics endpoints can go dark. If probes start failing right after a Corefile change that touched ports, this is the mechanism to suspect.

Metrics and signals to monitor

The key monitoring gap: a rejected reload produces exactly one metric (coredns_reload_failed_total) and a log line. If you are not alerting on that counter, failed Corefile changes are invisible until the next pod restart turns them into an outage. This belongs in any CoreDNS monitoring setup at the “mature” level; see the signal maturity model on the CoreDNS guides hub.

Fixes

Fix the directive or option named in the error

Correct the token the parser named, in the ConfigMap or Corefile source, and let the change roll out again. If the cause is version skew, either downgrade the directive to what your binary supports or upgrade the binary deliberately. Do not split the difference by leaving a directive in place “because the old config still works”; with reload enabled the bad Corefile sits in the ConfigMap as a landmine for the next restart.

For removed directives, use the replacement: proxy becomes forward, and options that moved between plugins need to be rewritten, not just renamed. The corefile-migration documentation lists these transitions version by version.

Fix structural errors

Add the missing whitespace around braces, move stray top-level directives into a server block, and remove duplicate zone and port combinations. These are one-line fixes, but verify by reading the whole file, not just the reported line, because structural errors shift the meaning of everything below them.

If pods are crash-looping right now

Revert the ConfigMap to the last known-good Corefile. If you do not have the previous content, kubectl rollout history will not help you for ConfigMaps; this is why the Corefile should live in version control (see Prevention). In the worst case, a minimal working Corefile for a Kubernetes cluster is well documented in the CoreDNS and kubeadm docs, and kubeadm can regenerate its default. A reverted, working config beats a broken, ambitious one.

If a reload was rejected but pods are running

You have time; DNS is still working on the old config. Fix the Corefile, watch for coredns_reload_failed_total to stop incrementing, and confirm the new hash appears in coredns_reload_version_info. Do not restart the pods to “force” the change: if the Corefile in the ConfigMap is still broken, the restart converts quiet config drift into a CrashLoopBackOff outage.

Prevention

• Validate before rollout. Because a parse error is fatal at load, a cheap validation is to start the CoreDNS binary against the candidate file in a scratch environment (a CI container with the same image tag you run in production): if the process stays up, the file at least parsed; if it exits immediately, the log tells you why. Run this in CI on every Corefile change.
• Pin validation to the running image tag. Validating against latest catches syntax errors but misses version-gated directives. Use the same image the cluster runs.
• Alert on coredns_reload_failed_total. Any nonzero rate is a ticket. This converts silent config drift into a signal.
• Keep the Corefile in version control. ConfigMaps edited by hand have no history. Git gives you the revert path for the crash-loop scenario.
• Roll Corefile changes deliberately. A successful reload flushes the cache, so a config change causes a cold-cache latency bump even when it parses. Treat Corefile changes like small deploys, not edits. See cache collapse after a rollout for what the cold-cache window looks like under load.
• During Kubernetes upgrades, check plugin compatibility first. Kubeadm’s CoreDNSUnsupportedPlugins preflight check exists for this; do not bypass it without reading which directives it flagged.

How Netdata helps

• Netdata collects coredns_reload_failed_total from the metrics endpoint, so a rejected Corefile change shows up as a visible event instead of a silent log line, and you can alert on any nonzero rate.
• Correlating reload failures with pod restart counts distinguishes the two failure modes at a glance: reload failures with stable pods means rejected reload and config drift; restarts climbing with no metrics means fatal parse at startup.
• After a successful reload, cache hit ratio and request latency charts show the expected cold-cache dip and recovery, so you can tell a normal post-reload warmup from a change that actually broke resolution.
• SERVFAIL rate by rcode and plugin, tracked per second, confirms within moments of a config change whether clients are still getting answers or the change degraded resolution.
• Per-pod visibility matters here: with multiple CoreDNS replicas, one pod can fail to reload or crash-loop while the others mask it in aggregate metrics.

Related guides

• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• CoreDNS cache collapse: the cold-cache thundering herd after a rollout
• CoreDNS cache evictions: the cache is too small for the working set
• CoreDNS cache hit ratio dropping: latency and upstream load climbing together
• CoreDNS CPU throttling: CFS limits making a green dashboard lie about latency
• CoreDNS DNS programming latency: how long a Service takes to become resolvable
• CoreDNS not resolving external domains: the missing catch-all forward zone
• CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed
• CoreDNS GC pauses adding tail latency: go_gc_duration_seconds and heap pressure
• CoreDNS goroutine count climbing: blocked upstream calls and leaks
• CoreDNS /health vs /ready: the readiness-probe mistake that serves SERVFAIL
• CoreDNS high request latency: reading P99 by zone to find the cause