How CoreDNS actually works in production: the plugin chain mental model

Most CoreDNS incidents are pipeline incidents, not DNS incidents. An upstream resolver gets slow, goroutines pile up, memory climbs, and the pod gets OOMKilled. Or the Kubernetes API watch drops silently and CoreDNS keeps answering every query correctly except the answers are three hours stale. Each failure mode is a direct consequence of how CoreDNS is built internally, and most CoreDNS runbooks assume you already hold that internal model in your head.

This article is that model: the plugin chain, the goroutine-per-query concurrency model, the cache, forward, and kubernetes plugins, and the Go runtime underneath. It does not cover specific failure patterns or fixes; those get their own guides. The goal is that when a runbook says “the forward plugin is accumulating blocked goroutines” or “SERVFAIL is being served from the denial cache,” you know which piece of the machine that sentence is about.

What the plugin chain is and why it matters

CoreDNS is a DNS server written in Go, built around a plugin chain. There is no monolithic resolver. Every query traverses an ordered chain of plugins defined in the Corefile, and each plugin can inspect the query, modify it, answer it, or pass it to the next plugin.

Three rules define everything:

1. Every query traverses an ordered plugin chain. The chain order is fixed by the build, not by the order plugins appear in your Corefile. Rearranging lines in the Corefile does not change execution order.
2. The most specific server block wins. A query is matched to a server block by zone, using the longest matching suffix. With blocks for . and cluster.local, a query for kubernetes.default.svc.cluster.local runs through the cluster.local chain, and a query for example.com runs through the . chain. Each block has its own chain.
3. A plugin that answers stops the chain. If the cache has the record, the query never reaches the kubernetes or forward plugins. If forward returns SERVFAIL, that is the final answer. The chain is not a failover mechanism: a plugin returning an error response has still handled the query, and later plugins never see it.

Rule 3 is the one that bites operators. It explains why a single dead forward target produces SERVFAIL for every forwarded zone instead of falling through to another plugin, and why a warm cache can completely mask a dead upstream until TTLs expire.

How a query moves through CoreDNS

The full path for a typical Kubernetes deployment:

1. A listener accepts the query on UDP or TCP port 53 (port 53 in the standard Kubernetes container, with the Service targetPort also 53).
2. The Go scheduler hands the query its own goroutine. There is no thread pool and no connection queue.
3. The query is matched to the most specific server block by zone.
4. The block’s plugin chain executes synchronously inside that goroutine. In a standard Kubernetes Corefile, the plugins that actually answer queries are, in chain order: cache, then kubernetes, then forward.
5. The first plugin that produces a response ends the chain. The response goes back to the client and the goroutine exits.

flowchart LR
  Q[Client query] --> L[Listener UDP/TCP 53]
  L --> G[Goroutine per query]
  G --> S{Most specific server block}
  S --> C[cache]
  C -->|hit| R[Response]
  C -->|miss| K[kubernetes]
  K -->|answer| R
  K -->|not in cluster zone| F[forward]
  F -->|answer or SERVFAIL| R
  F --> U[Upstream resolvers]

Alongside the answer-producing plugins, the chain typically carries supporting plugins that change behavior without answering queries: errors and log for output, health and ready for probe endpoints, prometheus for metrics on :9153, loop for startup loop detection, reload for Corefile hot reload, and bufsize (default EDNS0 buffer 1232 bytes). autopath, when enabled, short-circuits Kubernetes search domain expansion server-side at the cost of extra CPU and memory.

The pieces every runbook assumes

Goroutine-per-query

CoreDNS has no thread pool and no inbound connection queue. Each in-flight query occupies one goroutine, and goroutine count is bounded only by memory. This is why “slow upstream” and “memory exhaustion” are the same incident: when forward blocks waiting on an upstream, the query’s goroutine stays alive holding memory, and client retries create more goroutines on top. A healthy CoreDNS idles at roughly 20 to 50 goroutines. A growing count that never returns to baseline means blocked calls or a leak.

The cache plugin

The cache is an in-memory LRU with two stores: a positive (success) cache for NOERROR answers and a negative (denial) cache for NXDOMAIN/NODATA. SERVFAIL responses are also cached, in the denial cache, for 5 seconds by default. Default capacity is 9984 entries per cache, split across 256 shards, with LRU eviction when full.

Two operational consequences matter constantly:

• The cache masks upstream failure. A warm cache keeps serving answers while upstreams are down. The failure only becomes visible as TTLs expire, which is why upstream health must be monitored independently of end-to-end success.
• The cache amplifies upstream failure. A one-second upstream blip writes SERVFAIL into the cache, and every client asking for that record gets SERVFAIL from cache for the next 5 seconds even though the upstream has recovered.

Cache hit ratio (coredns_cache_hits_total / coredns_cache_requests_total) is the single biggest lever on both latency and upstream load. Cluster DNS typically sees 80%+ hit rates on repeated service names. Note that coredns_cache_misses_total is deprecated; derive misses as requests minus hits.

The forward plugin

The forward plugin maintains connection pools to upstream resolvers, with connection reuse, health checking, backoff, and load balancing policies across multiple upstreams (random, round_robin, sequential). Per-upstream state is what makes it debuggable: coredns_forward_request_duration_seconds{to=...} and coredns_forward_healthcheck_failures_total{to=...} break latency and health down by individual upstream. (Older deployments using the deprecated proxy plugin, removed from the default build in 1.5.0, expose the equivalent data under the coredns_proxy_* prefix; the forward plugin has always used the coredns_forward_* prefix.)

Behaviors worth memorizing:

• When all upstreams fail health checks, coredns_forward_healthcheck_broken_total increments. By default CoreDNS still tries a random unhealthy upstream rather than failing instantly, unless failfast_all_unhealthy_upstreams is set. So the counter proves degraded forwarding, not necessarily total failure.
• A fully dead upstream produces fast SERVFAIL with low latency. A slow-but-alive upstream produces high latency and goroutine accumulation. These are different incidents with different fixes.
• If max_concurrent is configured, excess in-flight queries are rejected with REFUSED (SERVFAIL on releases before 1.8.1) and counted in coredns_forward_max_concurrent_rejects_total. That is deliberate backpressure, and preferable to unbounded goroutine growth.
• The loop plugin probes for forwarding loops at startup. A loop introduced by a mid-flight config change is not caught until the next restart, because the plugin does not re-probe while running.

The kubernetes plugin

The kubernetes plugin does not query the API server per DNS query. It runs watch-driven informers against the API server and builds an in-memory record set from the streamed state. DNS answers come from that local snapshot.

This design has three consequences that show up in every CoreDNS outage review:

• Stale is the failure mode, not down. If the API watch disconnects, CoreDNS keeps answering from its last known state. Existing services resolve fine; new services and endpoint changes are invisible. Metrics stay green while the data drifts. There is no binary “watch broken” metric; you infer it from coredns_kubernetes_rest_client_requests_total error codes or from logs.
• Memory scales with the cluster. The snapshot grows with Services and Endpoints, and the post-restart re-list is the peak memory event. Memory limits sized close to steady-state RSS cause OOM crash loops on restart.
• Readiness is plugin-aware. The /ready endpoint on :8181 waits for the initial watch sync before reporting ready. Using /health (process liveness only, :8080) as the Kubernetes readiness probe lets pods receive traffic before they can resolve cluster names.

The Go runtime

CoreDNS inherits the Go runtime wholesale. Stop-the-world GC pauses add directly to P99 latency (go_gc_duration_seconds). Heap that grows without being reclaimed across GC cycles is a leak signal (go_memstats_heap_inuse_bytes), but the OOM killer acts on RSS (process_resident_memory_bytes), which is always larger and is the number to compare against the container limit. Go also returns memory to the OS lazily, so elevated RSS after a load spike is normal and is not, by itself, a leak.

File descriptors are the other runtime resource: one per upstream connection, per listening socket, per API watch stream, per log file. process_open_fds approaching process_max_fds ends in “too many open files,” not graceful degradation.

Where the model pays off in production

Each of CoreDNS’s characteristic failures maps to exactly one piece of the model:

The last row is why the mental model has to include what is not CoreDNS. Packets dropped by the kernel (UDP receive buffer exhaustion, conntrack table full) never reach a listener, never get a goroutine, and never appear in any coredns_dns_* metric. CoreDNS dashboards look too good to be true: low latency, no errors, suspiciously low throughput. The model tells you where visibility ends, which is at the socket.

Common misreadings of the model

• Treating the chain as failover. A plugin that returns SERVFAIL has handled the query. Nothing later in the chain runs. There is no “try the next plugin on error.”
• Reordering the Corefile to change execution. Plugin execution order is fixed at build time. Corefile order does not control it.
• Trusting /health as proof DNS works. /health checks process liveness only. A pod can return 200 while answering SERVFAIL to every query. RCODE metrics are the availability signal.
• Alerting on NXDOMAIN. NXDOMAIN is a normal answer, and high NXDOMAIN rates are expected in Kubernetes because ndots:5 search domain expansion generates them by design. Alert on SERVFAIL and REFUSED.
• Reading QPS as logical lookups. With ndots:5, one application lookup can become 4 to 6 queries. CoreDNS QPS is the amplified number, and with NodeLocal DNSCache deployed it is only the cache-miss remainder.

Signals to watch in production

How Netdata helps

• Netdata charts the response-code distribution per plugin, so SERVFAIL from forward and SERVFAIL from kubernetes are visually separate incidents instead of one blended error rate.
• Per-upstream forward latency and health check failures are graphed individually via the to label, which is the breakdown you need when exactly one of three upstreams is slow.
• Go runtime signals (goroutines, heap, GC pause duration) sit on the same timeline as DNS latency, making the “blocked goroutines, rising memory, GC pauses, then OOM” cascade readable as one story instead of four dashboards.
• Cache hit ratio next to upstream request rate shows the masking effect directly: you can see upstream health degrade while end-to-end success stays green behind a warm cache.
• Per-second collection keeps the 5-second SERVFAIL cache amplification window and post-reload cold-cache spikes visible rather than averaged away.
• In Kubernetes, node-level signals like conntrack utilization and UDP buffer errors can be correlated with CoreDNS metrics from the same node, which is how you catch the failures that never reach the process.

Related guides

• CoreDNS monitoring checklist: the signals every production resolver needs
• CoreDNS monitoring maturity model: from survival to expert
• CoreDNS returning SERVFAIL: the resolver is failing queries and what to check first
• CoreDNS returning REFUSED: no matching zone, an ACL, or the forward concurrency limit
• CoreDNS NXDOMAIN vs SERVFAIL: why alerting on the wrong one buries real incidents
• CoreDNS query rate dropped to zero while the process looks healthy
• CoreDNS NOERROR with zero answers: the resolution failure that reports success
• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• CoreDNS slow upstream: per-upstream latency, goroutine pileup, and the to label
• CoreDNS per-upstream health check failures: degraded redundancy before total loss
• CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed
• CoreDNS upstream connection cache misses: new connections adding latency per query