CoreDNS reload failed: config drift when the new Corefile never took effect

You pushed a Corefile change, nothing broke, and everyone moved on. Weeks later you discover the cluster is still running the old configuration. The metric that would have told you is coredns_reload_failed_total, and it was nonzero the whole time.

A failed CoreDNS reload is not an outage. The reload plugin polls the Corefile every 30 seconds (with jitter) and triggers a graceful reload when the SHA512 checksum of the file changes. If the new configuration has a syntax error, an invalid plugin directive, or hits a port conflict, CoreDNS logs the error, increments coredns_reload_failed_total, and keeps serving DNS with the old config. Queries keep flowing. The only thing that changed is that reality and your intent have quietly diverged.

That drift is the real incident. Every subsequent “we already fixed that in the Corefile” assumption is wrong until you reconcile what is loaded with what you think is loaded.

What this means

CoreDNS never partially applies a Corefile. A reload either succeeds atomically or the old configuration keeps running in full. There is no error returned to clients, no SERVFAIL spike, no crash. The failure is visible in exactly three places:

• The counter coredns_reload_failed_total increments (typically once per poll cycle while the bad config sits on disk).
• The CoreDNS log carries the parse or startup error for the rejected config.
• The gauge coredns_reload_version_info{hash, value} records the SHA512 hash of the most recent reload attempt. It is updated to the attempted hash even when the reload fails, so a changed hash does not by itself prove the new config is running.

There is a nastier edge case documented upstream: a reload that changes a listener port closes the old listener before opening the new one. If opening the new port fails (for example, the port is already in use), the reload aborts but the old listener is already gone. DNS on port 53 keeps working while the health, ready, or metrics endpoint is permanently broken until the process restarts.

flowchart TD
  A[Corefile changed on disk] --> B[reload plugin polls every 30s with jitter]
  B --> C{SHA512 changed?}
  C -- no --> B
  C -- yes --> D[attempt graceful reload]
  D -- success --> E[new config live, cache flushed, version_info hash updates]
  D -- failure --> F[log error, reload_failed_total +1, old config keeps serving]
  F --> G[config drift: operator believes new config is live]

Any nonzero value of coredns_reload_failed_total is actionable. It means the running configuration does not match what someone intended to deploy.

Common causes

Note on timing: in Kubernetes, the ConfigMap update has to propagate into the pod before the poll loop even sees it, and the Kubernetes documentation tells operators to allow up to two minutes for Corefile changes to take effect. Distinguish “reload has not happened yet” from “reload failed” before you start fixing things.

Quick checks

All read-only.

# 1. Check the reload failure counter
curl -s http://localhost:9153/metrics | grep '^coredns_reload_failed_total'

# 2. Check the hash of the most recent reload attempt (a failed reload still updates it)
curl -s http://localhost:9153/metrics | grep '^coredns_reload_version_info'

# 3. Find the reload error in the logs
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=200 | grep -iE 'reload|corefile'

# 4. Read the intended configuration
kubectl get cm -n kube-system coredns -o yaml

# 5. Verify the HTTP endpoints survived (listener edge case)
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8080/health
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8181/ready
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:9153/metrics

In Kubernetes, run the curl checks with kubectl exec against each CoreDNS pod. Check every replica: drift is frequently per-pod, because each pod polls and reloads independently.

How to diagnose it

1. Confirm the failure is active, not historic. coredns_reload_failed_total is a counter and never decreases. Sample it twice, 60 to 90 seconds apart. If it is still incrementing, the on-disk config is being rejected on every poll cycle. If it is static, someone pushed a bad config in the past and the current file may be fine but was never loaded either.

2. Read the error. The logs tell you exactly what CoreDNS rejected: a parse error with a line number, an unknown plugin option, or a startup failure such as a port bind. This is almost always enough to identify cause 1, 2, or 3 from the table above.

3. Establish drift direction. Compare the intended Corefile (the ConfigMap) against coredns_reload_version_info on each pod. Record the hash, make a trivial known-good edit, wait up to two minutes, and record it again. If the hash never moves on a pod, that pod has not attempted a reload since it started. A moved hash is not proof of success: on a failed reload the gauge records the attempted new hash while the old config keeps running, so confirm with the failure counter and the logs.

4. Rule out the timing false alarm. If the counter is zero and the change is recent, give it the full ConfigMap propagation plus poll interval (up to about two minutes) before concluding anything. If the hash then updates, there was never a failure.

5. Check for the listener edge case. If the change touched any port (a new server block on a nonstandard port, or a moved health, ready, or prometheus directive), test every HTTP endpoint on every pod. A pod that answers DNS but returns nothing on :8080, :8181, or :9153 is in the wedged state: the old listener was closed and the new one never opened. The process cannot recover from this on its own.

6. Check the version-specific failure modes. On v1.6.0 or earlier, edits to files pulled in with import do not trigger a reload at all; the counter stays zero and this looks like “nothing happened.” On v1.8.0, a failed reload triggered by an imported file can wedge the reload path: subsequent attempts fail with “use of closed network connection” and a later change can panic the process (“sync: negative WaitGroup counter”). Correlate coredns_panics_total and pod restarts with your config pushes. If you run lameduck for graceful shutdown, watch readiness closely during config changes.

Metrics and signals to monitor

Fixes

Fix the Corefile and push again

For syntax and plugin errors, correct the file and push it. The next poll cycle computes a new checksum, attempts the reload again, and on success the error loop stops. Confirm recovery by watching the version hash change and the cache transient occur. The counter does not reset; recovery is “stops incrementing,” not “returns to zero.” If the error complained about a duplicated directive, remember reload may only appear once per server block.

Recover from the listener port edge case

If a reload died midway through a port change, first free the conflicting port or pick a different one in the Corefile. The running process may still be missing its health or metrics listeners even after you fix the config, because the old listener was closed during the failed attempt. The reliable recovery is a pod restart. Treat that restart as a cache-flushing event (see below) and do one pod at a time.

When the change was never picked up

On v1.6.0 or earlier with import, make a trivial edit to the Corefile itself so the checksum changes, or plan an upgrade to v1.7.0 or later where imported files are tracked. On v1.8.0 with the imported-file wedge, upgrade; the reporter of upstream issue 5203 resolved it by moving to v1.9.0. If the process is already panic-looping, the pod restart is happening anyway.

Last resort: restart to force the config

If reloads keep failing and you cannot afford to wait, restarting the pod forces the new Corefile to load at startup. This is disruptive in one specific way: the cache comes back empty. Restarting all replicas at once converts your config fix into a cold-cache thundering herd against your upstreams. Delete pods one at a time, wait for readiness and cache warmup between them, and keep the change away from peak traffic if you can. See the cache collapse guide for why this matters.

Prevention

• Alert on any reload failure. Any increase of coredns_reload_failed_total means running config and intended config differ. That is a ticket-worthy condition even though nothing is down.
• Close the loop on every config push. After applying a Corefile change, automation should wait two minutes, then assert that coredns_reload_version_info changed on every replica and that no pod’s failure counter moved. A config push without this check is how drift survives for weeks.
• Validate Corefiles before they ship. Render the final Corefile in CI and load it with a throwaway CoreDNS instance or staging pod before merging. Every rejected reload in production was a config that never got parsed anywhere first.
• Do listener changes with a rollout, not a reload. Any edit that adds, removes, or moves a port should go through a controlled pod restart, because the reload path closes listeners before it knows the new ones will bind.
• Track the version-specific bugs. Know whether you run anything at or below v1.6.0 (import blind spot), v1.8.0 (imported-file wedge), or with lameduck enabled (readiness interaction). Upgrade paths fix the first two.
• Keep readiness on /ready. If a reload wedges a pod or a restart goes badly, correct readiness probes pull it out of the Service instead of letting it serve degraded answers. See the probe guide below.

How Netdata helps

• Netdata scrapes each CoreDNS pod’s :9153 endpoint independently, so per-replica drift (one pod loaded the new config, the other rejected it) is visible instead of averaged away.
• Alerting on any rate of change of coredns_reload_failed_total catches the failure on the first poll cycle, not when someone notices stale behavior weeks later.
• Plotting coredns_reload_version_info alongside config push events closes the drift loop: you can see the hash move, or see it not move, per pod.
• Correlating reload events with the cache hit ratio distinguishes “reload happened” (hit-ratio dip, forward-QPS spike) from “reload silently never happened” (flat lines after a push).
• coredns_panics_total and pod restarts on the same timeline as config changes surface the wedged-reload bug class before it takes pods down.
• The health self-check metrics (coredns_health_request_failures_total, coredns_health_request_duration_seconds) reveal the listener edge case where DNS still answers but the health path is broken.

Related guides

• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• CoreDNS cache collapse: the cold-cache thundering herd after a rollout
• CoreDNS cache evictions: the cache is too small for the working set
• CoreDNS cache hit ratio dropping: latency and upstream load climbing together
• CoreDNS CPU throttling: CFS limits making a green dashboard lie about latency
• CoreDNS DNS programming latency: how long a Service takes to become resolvable
• CoreDNS not resolving external domains: the missing catch-all forward zone
• CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed
• CoreDNS GC pauses adding tail latency: go_gc_duration_seconds and heap pressure
• CoreDNS goroutine count climbing: blocked upstream calls and leaks
• CoreDNS /health vs /ready: the readiness-probe mistake that serves SERVFAIL
• CoreDNS high request latency: reading P99 by zone to find the cause