CoreDNS panics: recovered handler crashes and coredns_panics_total

You found coredns_panics_total above zero, or you saw Recovered from panic in the CoreDNS logs. The pods are not restarting, the health endpoint returns 200, and most DNS queries still resolve. That combination is what makes this failure mode easy to ignore and dangerous to dismiss.

A recovered panic means a query handler crashed inside the plugin chain and CoreDNS’s recovery wrapper caught it, keeping the process alive. The server survived. The query that triggered the panic did not: the client that sent it got no answer and waited out its own timeout. Every increment of the counter is a real query that failed, plus evidence of a genuine bug in CoreDNS or one of its plugins.

Unlike most CoreDNS signals, this one has no acceptable baseline. Any nonzero rate of coredns_panics_total in production is a defect. The question is never “is this serious enough,” it is “which code path is crashing, on which input, and is it fixed in a newer version.”

What this means

CoreDNS handles each DNS query in its own goroutine, running it synchronously through the plugin chain defined in the Corefile. A recovery handler sits around that per-request execution. If a plugin panics while processing a query, for example a nil pointer dereference on unexpected input or a data race, the recovery handler catches it, increments coredns_panics_total, logs the panic, and lets the process continue serving other queries.

Two details follow:

The triggering query is lost. The client that sent it received no response and will time out and retry. If panics are intermittent and tied to a specific query pattern, you see a trickle of client-side DNS timeouts that never rises to a visible outage. Services look flaky; CoreDNS looks fine.

Only per-request panics are recovered. A panic in a background goroutine, such as the kubernetes plugin’s API watch handler or the forward plugin’s health checker, is not caught. It crashes the entire process, does not increment this counter, and shows up as a pod restart instead. So coredns_panics_total at zero does not mean no panics; it means no recovered panics. Check restart counts alongside it.

The metric is a counter with no labels and, unusually, no dns subsystem. It is coredns_panics_total, not coredns_dns_panics_total. Queries or alerts written against the wrong name silently match nothing.

flowchart TD
  Q[Incoming DNS query] --> PC[Plugin chain execution]
  PC -->|panic in query handler| R[Recovery wrapper]
  R --> M[Increment coredns_panics_total]
  R --> L[Log recovered panic]
  R --> F[Triggering query lost: client timeout]
  PC -->|panic in background goroutine| C[Process crash]
  C --> RS[Pod restart, counter NOT incremented]
  PC -->|normal path| OK[Response sent]

Common causes

A useful calibration from the field: operators have reported Recovered from panic log lines every few seconds with no visible DNS resolution failure at all. The affected queries were a tiny fraction of traffic, so nothing paged, but a specific client or query shape was silently failing the whole time. Treat a low, steady panic rate as a bug to fix, not noise to tolerate.

Quick checks

All of these are read-only.

# Current panic count (note: no dns subsystem in the metric name)
curl -s http://localhost:9153/metrics | grep '^coredns_panics_total'

# Find recovered panics in logs (Kubernetes)
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=5000 | grep -i "recovered from panic"

# Check previous container logs for unrecovered panics that killed the process
kubectl logs -n kube-system -l k8s-app=kube-dns --previous | tail -50

# Pod restart counts: background-goroutine panics bypass the recovery handler
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Running CoreDNS version
kubectl get pods -n kube-system -l k8s-app=kube-dns -o jsonpath='{.items[*].spec.containers[*].image}'

# SERVFAIL rate alongside panics (panicked queries may surface here or as client timeouts)
curl -s http://localhost:9153/metrics | grep 'coredns_dns_responses_total' | grep 'SERVFAIL'

# Query type distribution: is an unusual type (AXFR, ANY) present near panics?
curl -s http://localhost:9153/metrics | grep '^coredns_dns_requests_total' | grep -oE 'type="[^"]+"' | sort | uniq -c

How to diagnose it

1. Confirm the rate, not just the value. The counter is cumulative for the process lifetime. A value of 3 that has not moved in a week is a historical artifact; a value climbing steadily is an active bug. Sample it twice a few minutes apart, or check your monitoring system’s rate over the last 24 hours.

2. Get the panic value from the logs. The log line contains the panic reason, typically a nil pointer dereference or an index out of range. That string is your search key. If your Corefile loads the errors plugin with the stacktrace option, you also get a full stack trace naming the exact plugin and code path. If you do not have stack traces and panics are recurring, adding stacktrace to the errors plugin is a low-risk way to get them; it changes logging, not serving behavior.

3. Map the stack trace or panic value to a plugin. With a stack trace this is direct: the frames name the plugin. Without one, work from your Corefile. List every plugin in the affected server block, including any third-party or out-of-tree plugins, which are common panic sources and do not get the same scrutiny as the core set.

4. Check the version against known panic fixes. Panics are frequently fixed in patch and minor releases, and release notes call them out explicitly. Past examples include nil-pointer fixes in rewrite plugin EDNS0 handling, a transfer plugin panic on empty DNS records, a kubernetes plugin AXFR panic triggered by headless services with no declared ports, and kubernetes plugin panics when endpoints are disabled. If your version predates the fix for the code path in your stack trace, you have your answer.

5. Correlate panics with query patterns. Panics are usually triggered by specific input, not random chance. Align panic timestamps with query logs (if the log plugin is enabled), the query type distribution, and deployment events. A panic that fires only when a particular client runs, or only during a specific batch job, points at the triggering input.

6. Check what clients actually experienced. A recovered panic means one lost query. Whether the client saw a SERVFAIL or a silent timeout depends on the CoreDNS version and code path. Either way, correlate the panic rate with your SERVFAIL ratio and with client-side DNS timeout reports to size the real blast radius.

7. Rule out unrecovered panics. If restart counts are climbing but coredns_panics_total is flat, the crash is in a background goroutine. The previous-container logs will show the panic and the goroutine that died.

Metrics and signals to monitor

Fixes

Upgrade CoreDNS

If the stack trace or panic value matches a fixed issue, upgrading is the fix. This is the most common resolution: recovered panics are almost always defects that the project patches, and the release notes call them out. Check plugin fixes as well as core fixes, since the panic usually lives in a plugin. After upgrading, watch coredns_panics_total for 24-48 hours to confirm the specific panic is gone.

Remove or reconfigure the triggering plugin path

If the panic lives in a feature you do not strictly need, disabling it stops the crashes while you wait for a fix. Examples: if a transfer-plugin panic is triggered by AXFR requests and you never intended to serve zone transfers, remove the transfer plugin and AXFR clients get refused instead of crashing a handler. If a rewrite rule is on the crashing path, restructure the rule. The tradeoff is losing that functionality, so confirm nothing depends on it first.

Filter the triggering input as a stopgap

When a specific query pattern triggers the panic, for example AXFR requests from arbitrary clients or malformed packets from a scanner, blocking that traffic upstream (network policy, firewall) stops the panics without touching CoreDNS. This treats the symptom, not the bug. The code path is still broken; the next unexpected input may find it again. Use this to buy time for an upgrade, not as a permanent posture.

Do not use the debug plugin as a diagnostic shortcut in production

The debug plugin disables panic recovery entirely so CoreDNS crashes with a full stack trace instead of recovering. That is useful in a lab to capture a definitive trace, but in production it converts a per-query failure into a full process crash on every occurrence, and the errors plugin (if loaded) recovers panics itself, negating the effect of debug. Prefer the errors plugin’s stacktrace option, which keeps recovery intact.

Restarting is not a fix

Because the process survives recovered panics, restarting pods does nothing except reset the counter and lose your evidence. Extract the logs and stack traces first.

Prevention

• Alert on any nonzero rate of coredns_panics_total. This is one of the few metrics where zero is the only acceptable value. A rate-based alert (rate(coredns_panics_total[5m]) > 0) catches intermittent panics that a threshold would miss.
• Pair it with a restart-count alert. Together they cover both recovered and unrecovered panics.
• Load the errors plugin with stacktrace if your panic volume justifies it, so the first occurrence captures the evidence you need instead of the tenth.
• Watch the counter after every upgrade for 24-48 hours. Version changes are when new panic-inducing code paths get exercised.
• Track your version against release notes for panic and CVE fixes, and treat “fixed a panic in plugin X” as upgrade motivation proportional to whether you run plugin X.

How Netdata helps

• Per-second coredns_panics_total rate, so intermittent panics that a five-minute average would smooth into invisibility show up as discrete events you can align with logs.
• Correlation with SERVFAIL responses on the same timeline, letting you see whether panicked queries are surfacing as SERVFAIL, and how the client-visible error rate tracks the panic rate.
• Pod restart and container-state signals alongside the counter, so unrecovered panics (which never increment the metric) are visible in the same view as recovered ones.
• Query type distribution from coredns_dns_requests_total, helping you spot an unusual query type appearing in the same window as panic increments.
• Go runtime context (goroutines, GC, heap) for distinguishing a per-request handler bug from broader process instability.

Related guides

• How CoreDNS actually works in production: the plugin chain mental model
• CoreDNS goroutine count climbing: blocked upstream calls and leaks
• CoreDNS memory climbing: heap growth, post-GC minima, and leak detection
• CoreDNS GC pauses adding tail latency: go_gc_duration_seconds and heap pressure
• CoreDNS high request latency: reading P99 by zone to find the cause
• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken