$ guides / coredns ▌

COREDNS · OPERATIONS PLAYBOOK

CoreDNS: one plugin chain, a cache that hides upstream failure, and a kernel path that drops queries the process never sees

A DNS server built as an ordered chain of plugins running on the Go runtime. Every query is a goroutine, an in-memory cache and a forward connection pool sit on the hot path, and in Kubernetes the conntrack and UDP-buffer layer in front of it can silently drop queries while every CoreDNS metric stays green. We trace how that design behaves under load, where it turns a small fault into a cluster-wide outage, and what to do when it does.

> Start with the monitoring checklist → # Jump to the full guide list

CoreDNS is a few lines of Corefile and it just works — until a config change points it back at itself, an upstream goes dark, or the kernel starts dropping DNS packets the process never sees.

The defaults work. Until a Corefile forwards to the cluster DNS Service or a node's /etc/resolv.conf that routes back, the loop plugin logs Loop detected at startup and the pod drops into CrashLoopBackOff. Until every upstream goes unreachable and forwarded queries return SERVFAIL cluster-wide — fast, not slow, and briefly masked by a warm cache. Until the kubernetes plugin loses its API watch and quietly keeps serving stale records while new Services stay invisible. Until query volume fills the node's conntrack table and the kernel drops packets — nf_conntrack: table full — with every CoreDNS dashboard still green. Until glibc's simultaneous A and AAAA lookups hit a conntrack race and applications stall for exactly five seconds.

These guides are written for engineers who already run CoreDNS, not for people learning what a resolver is. The goal is the mental model of how the plugin chain and the Go runtime beneath it actually behave under load, the failure patterns that keep recurring, the monitoring story that catches them before they page anyone, and the runbooks you wish someone had handed you before your last incident.

How CoreDNS actually runs in production

CoreDNS is not just a DNS server. It is a Go process running an ordered chain of plugins, where every query is a goroutine, two caches and a forward connection pool sit on the hot path, and — in Kubernetes — the kernel network path in front of it can drop packets the process never counts. Most production failures live between these layers, not inside any one of them.

clients + resolv.conf

Pods and hosts resolve through <code>/etc/resolv.conf</code>. Kubernetes' default <code>ndots:5</code> expands short names across the search list, so one logical lookup becomes four to six real queries. Every query first crosses the kernel UDP and conntrack path — before CoreDNS ever sees it.

CLIENT

listeners (:53 UDP/TCP)

CoreDNS accepts queries on port 53 with a goroutine per request and no fixed thread pool. The kernel's UDP receive buffer sits in front of the listener; packets dropped there are never counted by any CoreDNS metric.

LISTEN

server blocks + plugin chain

The Corefile defines one plugin chain per server block. An incoming query matches the most specific zone, then runs the chain in order — the first plugin that answers stops it. Plugin order is behaviour, not decoration.

CHAIN

cache plugin

Separate positive, negative, and SERVFAIL LRU caches, each with its own TTL. A warm cache hides an upstream that has already died; SERVFAIL is cached for five seconds by default, so a one-second blip is served to everyone for five.

CACHE

kubernetes plugin

Watches Services and Endpoints through API informers and builds an in-memory record set. When the API watch disconnects, CoreDNS keeps serving the last known state with no error — existing names resolve, new ones do not.

K8S

forward plugin + upstreams

Maintains connection pools and health checks to upstream resolvers, with a <code>max_concurrent</code> cap on in-flight queries. One slow upstream drags every query routed to it; all upstreams down turns forwarding into an immediate SERVFAIL cascade.

FORWARD

Go runtime (goroutines, heap, GC)

A goroutine per in-flight query, a heap holding the caches and the Kubernetes watch state, and a garbage collector whose pauses land on the query P99. The container memory limit is a hard cliff — cross it and the pod is OOMKilled instantly.

RUNTIME

health / ready / metrics endpoints

<code>:8080/health</code> checks process liveness only, <code>:8181/ready</code> is plugin-aware and waits for the kubernetes sync, and <code>:9153/metrics</code> exposes Prometheus data. None of the three proves that DNS actually resolves.

STATE

Why this matters: 'DNS is broken' can be a forwarding loop crashing the pod, a cluster-wide SERVFAIL from dead upstreams, an OOM kill, conntrack drops the process never sees, a stale Kubernetes record set, a cold-cache thundering herd, or a five-second glibc timeout in the network path. The symptom rhymes but each layer has a different signal — and a different fix.

The failures you'll actually see

Most CoreDNS incidents fall into a small set of recurring patterns. Recognise the shape, and triage gets dramatically faster.

CRITICAL

The upstream black hole

Every configured upstream resolver goes unreachable — a firewall change, a routing failure, or the upstreams genuinely down. The forward plugin fails forwarded queries immediately with SERVFAIL, so latency stays LOW (fast failure, not timeout). A warm cache serves stale hits and masks it briefly; once TTLs expire, everything fails at once. coredns_forward_healthcheck_broken_total incrementing confirms all upstreams are marked unhealthy.

SERVFAIL climbing toward 100% of forwarded queries, latency LOW
coredns_forward_healthcheck_broken_total incrementing
Per-upstream health check failures for every upstream
Cache hit ratio briefly high, then a cliff as TTLs expire

Investigate →

CRITICAL

CrashLoopBackOff from a forwarding loop

The Corefile forwards to an address that routes back to CoreDNS — 127.0.0.1:53, the cluster DNS Service ClusterIP, or a node /etc/resolv.conf that points at cluster DNS. The loop plugin sends a self-referencing probe at startup, detects the cycle, and exits the process. Kubernetes restarts the pod, it loops again, forever. No metrics are scraped because the process dies within seconds.

Pod in CrashLoopBackOff, restarting every few seconds
Loop detected in the CoreDNS logs (plugin/loop)
No Prometheus metrics — process exits before scrape
Started right after a Corefile / forward-target change

Investigate →

CRITICAL

The memory cliff and OOM kill

Memory climbs — an oversized cache, a goroutine or connection leak, a large-cluster watch set, or the startup re-list peak — until RSS crosses the container limit and the runtime OOM-kills the pod instantly. There is no graceful degradation. When the limit is set only just above steady state, the kubernetes plugin's full re-list on restart blows it again, producing a crash loop.

Pod restarts with reason OOMKilled
process_resident_memory_bytes near the container limit
Post-GC heap minima trending up, GC pauses lengthening
Crash loop where each restart is OOM-killed during API re-list

Investigate →

CRITICAL

Silent packet loss beneath the process

Every UDP DNS flow consumes a conntrack entry. Under high QPS the node's conntrack table fills and the kernel drops packets silently — no ICMP, no reset. CoreDNS metrics look perfect because the packets never reach the process; clients simply time out. Because conntrack is a shared node resource, exhaustion drops new TCP and UDP flows for every pod on the node, not just DNS.

nf_conntrack: table full, dropping packet in kernel logs
Client-side DNS timeouts while CoreDNS dashboards stay green
conntrack count near nf_conntrack_max on CoreDNS nodes
Unrelated apps on the same node failing connections too

Investigate →

ACTIVE

The SERVFAIL cascade

Queries start failing with SERVFAIL — the resolver hit an internal error or could not reach what it needed. In Kubernetes the two big sources are dead upstreams (the forward plugin) and a lost API watch (the kubernetes plugin). A five-second SERVFAIL cache turns a brief upstream flap into a longer, wider outage. A pod passing its health probe can still SERVFAIL every single query.

coredns_dns_responses_total{rcode="SERVFAIL"} rising
plugin label pointing at forward vs kubernetes
zone label isolating cluster.local from forwarded zones
Health probe green while resolution is failing

Investigate →

IMMINENT

Stale cluster DNS

The kubernetes plugin loses contact with the API server — overload, network trouble, or a 403 after an RBAC change. CoreDNS keeps serving records from its last known state, so existing Services resolve and every performance metric stays green, while newly created Services and endpoint changes are invisible and the data quietly drifts from reality. There is no binary 'watch broken' metric — you infer it from API error codes.

coredns_kubernetes_rest_client_requests_total showing 5xx / 403
New Services not resolvable while old ones are fine
cluster.local affected; forwarded zones unaffected
A synthetic freshness test failing while metrics look clean

Investigate →

Choosing a tool

Best CoreDNS Monitoring Tools (2026)

A ranked review of the tools teams actually shortlist here, what each one is genuinely good at, and how the pricing behaves as you scale.

Read the buyer's guide →

CoreDNS monitoring maturity levels

CoreDNS observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your resolver matters. Most production clusters should land at the second level.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor. With these signals you can answer one question: is CoreDNS alive and is it returning answers rather than failures? You will not learn what broke, but you will learn that something broke. Survival is enough for dev resolvers and non-critical zones.

Process / pod running Is CoreDNS up and not in CrashLoopBackOff or OOM restart?
Port 53 listening (UDP and TCP) Is the resolver actually accepting queries?
Metrics endpoint on :9153 Is Prometheus data reachable at all?
Any SERVFAIL responses coredns_dns_responses_total{rcode="SERVFAIL"} — the real availability signal.
Restart count Restarts in the last hour point at loops, OOM, or config faults.

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production clusters should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident on its own: failing upstreams, stalled Kubernetes sync, cache disruption, latency, capacity.

Query rate by zone coredns_dns_requests_total; a drop to zero can mean drops upstream of the process.
SERVFAIL rate and ratio Alert on the ratio against baseline, never on NXDOMAIN.
Request latency P99 by zone cluster.local latency points at the API; forwarded zones at upstreams.
All upstreams down counter coredns_forward_healthcheck_broken_total — complete forwarding loss.
Per-upstream health failures coredns_proxy_healthcheck_failures_total{to=...} isolates the bad upstream.
Cache hit ratio hits/requests; a drop drives latency and upstream load up together.
Node conntrack utilisation The single most common unmonitored cause of Kubernetes DNS outages.

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone up. A cache slowly undersized for its working set, goroutines creeping upward, GC pauses lengthening, a config reload that silently failed, one replica quietly degraded. None of these page you on day one. They become incidents on day thirty.

Per-upstream latency coredns_proxy_request_duration_seconds{to=...} finds the slow upstream.
Heap trend and GC pause duration Post-GC minima rising is a leak heading for OOM.
Goroutine count trend A count that never returns to baseline is a leak.
Cache eviction rate Evictions at baseline traffic mean the cache is too small.
Reload failures coredns_reload_failed_total — the running config is not what you think.
Recovered panics coredns_panics_total — a CoreDNS or plugin bug.
Forward max_concurrent rejects Backpressure: the forward plugin is overwhelmed.
File descriptor ratio process_open_fds / process_max_fds; a cliff at the limit.

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. Kubernetes API error breakdowns, DNS programming latency, connection-cache behaviour, per-plugin SERVFAIL, node UDP-buffer errors, response-size and query-type distributions for abuse. Most teams never need every signal here. Add the ones your incident history says you do.

Kubernetes API errors by code coredns_kubernetes_rest_client_requests_total; 403 is RBAC, 5xx is overload.
DNS programming duration How long a Service takes to become resolvable.
Connection cache hit/miss coredns_proxy_conn_cache_*; churn adds latency and FDs.
SERVFAIL by plugin The plugin label separates forward from kubernetes faults.
Node UDP buffer errors netstat -su RcvbufErrors — silent packet loss below the process.
Response size distribution Large-response shifts flag amplification or TCP fallback.
Query type distribution (ANY/AXFR) Amplification and zone-transfer reconnaissance.
Cache stale serves coredns_cache_served_stale_total masks upstream failure.

Operating mistakes worth avoiding

The traps CoreDNS teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

Trusting health probes instead of RCODE metrics

The <code>/health</code> endpoint checks process liveness only — a pod can return <code>200 OK</code> to every probe while returning <code>SERVFAIL</code> to every DNS query. Teams that alert only on pod health miss the single most important failure mode. Monitor <code>coredns_dns_responses_total{rcode="SERVFAIL"}</code> and, in Kubernetes, use <code>/ready</code> — not <code>/health</code> — for the readiness probe so pods don't take traffic before the kubernetes plugin has synced.

Not monitoring conntrack at all

The most common unmonitored root cause of Kubernetes DNS disasters. Teams instrument CoreDNS beautifully — QPS, latency, errors, memory — then get blindsided by node-wide packet drops because nobody watched <code>nf_conntrack_count</code>. Conntrack is shared node infrastructure, invisible to CoreDNS, and its exhaustion is catastrophic with zero warning. Every production cluster running CoreDNS must watch conntrack on its nodes.

Confusing NXDOMAIN with SERVFAIL

NXDOMAIN means the name does not exist — normal, expected, and constant in Kubernetes, where <code>ndots</code> search expansion generates intermediate NXDOMAINs by design. SERVFAIL and REFUSED are failures. Alerting on 'total errors' buries real SERVFAIL spikes in NXDOMAIN noise. Alert on the SERVFAIL/total ratio against a rolling baseline; never on absolute NXDOMAIN count.

Trusting CoreDNS latency for what applications feel

CoreDNS reports processing time. It does not include kernel buffer wait, CFS throttling, network transit, or conntrack drops. Application-observed DNS latency can be 10-100x the CoreDNS number during degradation. Teams that watch only CoreDNS-side latency get blindsided when apps report DNS timeouts while the CoreDNS dashboard is green.

No functional freshness test for Kubernetes DNS

If the kubernetes plugin's API watch disconnects silently, every performance metric stays green — the data is just wrong. Without a synthetic test that creates (or reads a known-recent) Service and confirms CoreDNS resolves it, there is no way to detect stale records. This is CoreDNS's most dangerous failure mode, and almost nobody tests for it.

Ignoring the ndots:5 amplification

Seeing 50,000 QPS at CoreDNS does not mean 50,000 logical lookups — with the default <code>ndots:5</code>, short names expand through the search list into four to six queries each, so it may be 8,000-12,000 real lookups amplified 4-6x. This wrecks capacity planning and hides the real fix: override <code>ndots</code> (or use fully-qualified names) for workloads that mostly resolve external names.

Rolling all CoreDNS pods at once

A rolling update with <code>maxUnavailable</code> too high clears every cache simultaneously. The resulting cold-cache thundering herd hammers upstreams, and if they buckle, the five-second SERVFAIL cache amplifies it. Teams treat the post-rollout latency spike as 'expected' instead of preventing it with <code>maxUnavailable=1</code>, PodDisruptionBudgets, and staggered rollouts.

Treating CoreDNS as a singleton

Two or more replicas sit behind a ClusterIP with kube-proxy balancing across them. If one replica is degraded and the others are healthy, averaged metrics look only 'slightly off' while a real fraction of queries fail or stall. Keep per-pod series and alert on divergence between replicas, not just the aggregate.

CoreDNS runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

Availability: SERVFAIL, REFUSED, and query failures

▸

Upstream forwarding and the forward plugin

▸

Caching, hit ratio, and cold-cache storms

▸

Query latency and slow responses

▸

Go runtime: memory, goroutines, GC, and file descriptors

▸

Kubernetes plugin: API watches, stale data, and probes

▸

Corefile, reloads, and forwarding loops

▸

The kernel path: conntrack, UDP buffers, and the 5-second timeout

▸

DNS protocol: truncation, response size, and TCP fallback

▸

Security: amplification, zone transfers, and enumeration

WHERE TO GO NEXT

Setting up CoreDNS monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.

> Start with the checklist > Back to Operations Guides