CoreDNS Operations Guides on Netdata

CoreDNS /health vs /ready: the readiness-probe mistake that serves SERVFAIL

Mon, 01 Jan 0001 00:00:00 +0000

Your CoreDNS pods look healthy. Liveness passes, readiness passes, the pods are Running and Ready. Yet for the first seconds or minutes after every pod start, clients get SERVFAIL for cluster.local names. New deployments fail to resolve their dependencies, retries pile up, and then the problem vanishes on its own.

This is the classic symptom of pointing the Kubernetes readiness probe at /health instead of /ready. The two endpoints answer different questions, and confusing them makes your readiness gate meaningless.

CoreDNS 403 from the Kubernetes API: RBAC misconfiguration breaking cluster DNS

Mon, 01 Jan 0001 00:00:00 +0000

The symptom usually arrives in one of two forms. Either CoreDNS pods never become ready after an install or upgrade, stuck at 0/1 while kubectl shows no crash and no obvious error, or cluster DNS suddenly returns SERVFAIL for cluster.local names while external resolution keeps working. In both cases, the smoking gun is the same: coredns_kubernetes_rest_client_requests_total{code="403"} is incrementing.

A 403 from the Kubernetes API is not a transient error. It means the API server received the request, authenticated the caller, and refused it because the CoreDNS ServiceAccount lacks the RBAC permissions the kubernetes plugin needs to list and watch Services, Endpoints, and EndpointSlices. Without those watches, the plugin cannot build or maintain its in-memory record set, so it cannot answer queries for the cluster zone.

CoreDNS 5-second DNS timeout: the Kubernetes glibc A+AAAA conntrack race

Mon, 01 Jan 0001 00:00:00 +0000

Your application latency histogram has a cluster of samples at almost exactly 5 seconds. Not 4.8, not 5.3: a tight spike right at the glibc default resolver timeout. Tracing the slow calls shows they stall on name resolution. CoreDNS dashboards are green: low latency, no SERVFAIL, healthy cache hit ratio, both replicas fine.

That combination, application-side DNS stalls at exactly the resolver timeout with clean CoreDNS metrics, is the signature of the glibc A+AAAA conntrack race. The loss is in the kernel network path between the application pod and CoreDNS, not inside CoreDNS. No amount of Corefile tuning will touch it.

CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken

Mon, 01 Jan 0001 00:00:00 +0000

Every forwarded query through CoreDNS is suddenly returning SERVFAIL, request latency has dropped instead of rising, and coredns_forward_healthcheck_broken_total is climbing. This is the upstream black hole: every upstream resolver configured in the forward plugin is failing health checks at the same time, and CoreDNS has nowhere to send external queries.

The counter-intuitive part is the latency profile. When upstreams die hard, failures are fast. Queries do not sit in timeouts the way they do with a slow upstream. If you only alert on high latency, this failure mode sails under your dashboards until clients start reporting resolution errors.

CoreDNS ANY query flood: DNS amplification and reflection abuse

Mon, 01 Jan 0001 00:00:00 +0000

You are looking at coredns_dns_requests_total and the type="ANY" series has jumped from a flat zero to a sustained rate, or ANY queries have crept past a few percent of total traffic. That pattern is the classic fingerprint of DNS amplification and reflection abuse: an attacker sends small ANY queries with a spoofed source address, and your CoreDNS replies with large responses delivered to the victim’s IP. Your server is the reflector; someone else absorbs the blast.

CoreDNS AXFR zone transfer attempts: reconnaissance of internal service topology

Mon, 01 Jan 0001 00:00:00 +0000

An AXFR query asks a DNS server for the entire contents of a zone: every record, not one. Against a Kubernetes cluster running CoreDNS, a successful zone transfer hands the requester a complete map of internal service topology: every Service name, every namespace, every headless Service endpoint, every pod hostname you publish. That is exactly the map an attacker wants before lateral movement.

Stock CoreDNS does not serve AXFR. Unless the transfer plugin is explicitly configured in the Corefile, zone transfer requests are refused, and default kubeadm, EKS, GKE, and AKS Corefiles do not include it. In a typical deployment, an AXFR query is not a data-loss event. It is a tripwire.

CoreDNS cache collapse: the cold-cache thundering herd after a rollout

Mon, 01 Jan 0001 00:00:00 +0000

You just rolled out a CoreDNS config change or image bump. Within seconds, DNS latency across the cluster jumps, upstream DNS traffic spikes to several times baseline, and SERVFAILs appear in application logs. One to five minutes later, it all goes away on its own. The dashboard is green again.

That is the cache collapse pattern: every CoreDNS pod restarted at roughly the same time, every in-memory cache emptied at once, and every client in the cluster re-queried the same names simultaneously. The flood hit your upstreams harder than they could absorb. In the worst version, upstreams return SERVFAIL, CoreDNS caches those SERVFAILs for 5 seconds each, and a brief overload amplifies into a visible outage.

CoreDNS cache evictions: the cache is too small for the working set

Mon, 01 Jan 0001 00:00:00 +0000

coredns_cache_evictions_total climbing steadily, or upstream query load and DNS latency creeping up without any change in traffic volume, both point at the same thing: the cache plugin is evicting entries before their TTL expires because it has run out of room, and queries that should have been sub-millisecond cache hits are being forwarded to upstream resolvers instead.

This is progressive degradation, not a cliff edge. Latency rises, upstream load rises, and the hit ratio slides, slowly enough that it often goes unnoticed until someone asks why DNS got slower this quarter.

CoreDNS cache hit ratio dropping: latency and upstream load climbing together

Mon, 01 Jan 0001 00:00:00 +0000

The cache hit ratio is coredns_cache_hits_total / coredns_cache_requests_total. When it falls, every query that used to be answered from memory in under a millisecond now goes to the kubernetes plugin or out to an upstream resolver. Two things happen at once: coredns_dns_request_duration_seconds climbs, and forwarded query volume climbs with it. That second effect is the dangerous one, because the upstream flood can degrade the upstreams themselves and turn a cache problem into a resolution outage.

CoreDNS conntrack table full: silent UDP packet drops with a node-wide blast radius

Mon, 01 Jan 0001 00:00:00 +0000

Applications across a node start timing out on DNS lookups. Then on database connections. Then on API calls. You check CoreDNS and the dashboards are green: low latency, no SERVFAILs, healthy pods. But the timeouts keep coming, and they are not limited to DNS.

This is conntrack table exhaustion. Every UDP DNS query that traverses Kubernetes iptables DNAT creates a connection tracking entry on the node with a default timeout of 30 seconds. At high DNS QPS, the table fills. When it does, the kernel drops new packets silently: no ICMP error, no reset, no log line the application will ever see. The kernel log says nf_conntrack: table full, dropping packet and nothing else anywhere warns you.

CoreDNS Corefile parse error: startup failures and invalid plugin config

Mon, 01 Jan 0001 00:00:00 +0000

You changed the Corefile, and now one of two things has happened. Either the CoreDNS pods are in CrashLoopBackOff and cluster DNS is down, or the pods look fine but your change never took effect and the log is full of parse errors. Both are the same root event: the Corefile parser rejected your configuration. The difference is whether the parser ran at process start (fatal) or during a live reload (rejected, old config kept).

CoreDNS CPU throttling: CFS limits making a green dashboard lie about latency

Mon, 01 Jan 0001 00:00:00 +0000

Applications are reporting intermittent DNS timeouts and multi-hundred-millisecond resolution spikes. You open the CoreDNS dashboard and everything looks fine: coredns_dns_request_duration_seconds shows sub-millisecond cache hits, SERVFAIL is zero, QPS is normal, pods are healthy. Both things are true at the same time, and that is exactly the problem.

When a CoreDNS pod has a tight CPU limit, the Linux CFS scheduler throttles the container: the process is runnable but the kernel refuses to schedule it until the next quota period. That stall is real latency for the client, but it never appears in any CoreDNS metric, because CoreDNS only measures time spent actively processing a query. Time spent waiting for CPU does not exist as far as coredns_dns_request_duration_seconds is concerned.

CoreDNS CrashLoopBackOff: triaging loop, OOM, config, and port-bind causes

Mon, 01 Jan 0001 00:00:00 +0000

A CoreDNS pod in CrashLoopBackOff means the process starts, dies, and gets restarted by Kubernetes in a tightening backoff loop. While this is happening, at least one DNS replica is contributing nothing. If all replicas are crash-looping, the kube-dns Service has no backends and cluster-wide name resolution is down: queries hang at the client until a pod recovers.

CrashLoopBackOff is not one failure. It is a symptom with a small set of common root causes, and they need different fixes. The four you will see most often: the loop plugin detecting a forwarding loop, the OOM killer terminating the container, a Corefile that fails to parse, and a port-bind failure on startup. RBAC and Kubernetes API access failures show up in the same pod state too, usually as fatal list errors or persistent un-readiness.

CoreDNS DNS programming latency: how long a Service takes to become resolvable

Mon, 01 Jan 0001 00:00:00 +0000

You create a Service, the API server accepts it, kubectl get svc shows it, and yet for some period of time nothing in the cluster can resolve its name. That gap, from “the object exists in the API” to “CoreDNS answers for it”, is DNS programming latency, and CoreDNS exposes it as a first-class metric: coredns_kubernetes_dns_programming_duration_seconds.

This is the Kubernetes DNS Programming SLI. It exists because service discovery delay is a real failure mode: a rollout completes, endpoints are ready, but clients still get NXDOMAIN or stale answers for seconds or tens of seconds. From the application’s perspective the new pods are unreachable by name. From CoreDNS’s request metrics everything looks green.

CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed

Mon, 01 Jan 0001 00:00:00 +0000

coredns_forward_max_concurrent_rejects_total is incrementing and clients are getting REFUSED responses for forwarded queries. Something is piling up between CoreDNS and your upstreams, and the rejects are the valve letting pressure escape.

This counter only increments when the number of in-flight forwarded queries hits the max_concurrent cap you configured. Every rejected query gets a REFUSED response, not SERVFAIL. That distinction matters for triage: REFUSED from this path is a capacity signal, not a resolution failure, and unlike NXDOMAIN it is not subject to negative caching, so it will not linger in client caches after the pressure clears.

CoreDNS GC pauses adding tail latency: go_gc_duration_seconds and heap pressure

Mon, 01 Jan 0001 00:00:00 +0000

Your CoreDNS dashboards look mostly fine: P50 latency is sub-millisecond, SERVFAIL is zero, cache hit ratio is healthy. But P99 is spiking to tens or hundreds of milliseconds at irregular intervals, and a small fraction of clients see DNS timeouts they cannot explain. The process never crashes. Upstream latency is clean. CPU looks acceptable on average.

This is the classic signature of Go garbage collection stop-the-world pauses landing on DNS query latency. CoreDNS is a Go process, and every in-flight query is a goroutine. When the GC stops the world, every one of those goroutines waits. DNS is supposed to be sub-millisecond for cache hits, so a pause that would be invisible in a batch service is user-visible here. The working threshold: GC pauses over 10ms are impactful for DNS and show up as P99 spikes.

CoreDNS goroutine count climbing: blocked upstream calls and leaks

Mon, 01 Jan 0001 00:00:00 +0000

Your go_goroutines graph for CoreDNS is trending up. Maybe it spiked during an incident and never came back down. Maybe it has been climbing slowly for days. Either way, the question is the same: are queries piling up behind a slow upstream, or is something leaking goroutines that will never exit?

The distinction matters because the fixes are completely different. A blocked-upstream spike resolves when the upstream recovers. A leak grows until the OOM killer ends the debate for you.

CoreDNS high request latency: reading P99 by zone to find the cause

Mon, 01 Jan 0001 00:00:00 +0000

P99 on coredns_dns_request_duration_seconds is at 400ms when it used to sit at 15ms. Do not start with the aggregate number: aggregate latency in CoreDNS blends at least two very different workloads. Cache hits should complete in single-digit milliseconds. Forwarded and Kubernetes-backed queries depend on systems outside CoreDNS itself.

The zone label on the duration histogram is the fork in the road. High latency in cluster.local points at the Kubernetes path. High latency in forwarded zones points at upstream resolvers. High latency everywhere, at every percentile, points at the CoreDNS process itself: CPU saturation or GC pressure. Until you split the histogram by zone, you are guessing.

CoreDNS Kubernetes API disconnect: stale records and new services going invisible

Mon, 01 Jan 0001 00:00:00 +0000

The symptom that usually surfaces first is not a DNS error. It is a developer saying “I deployed the service twenty minutes ago and nothing can reach it,” while every existing workload looks fine. nslookup kubernetes.default works. External domains resolve. CoreDNS dashboards are green. But anything created or changed recently in the cluster does not exist as far as DNS is concerned.

This is the Kubernetes API disconnect pattern. The CoreDNS kubernetes plugin maintains its DNS records by watching the API server through informers, not by querying the API per DNS request. When that watch breaks and cannot reconnect, CoreDNS keeps answering from its last known in-memory snapshot. Existing services keep resolving, which is exactly why the failure is so dangerous: everything looks healthy while the answers drift further from reality.

CoreDNS Loop detected and CrashLoopBackOff: the forwarding loop that kills the pod

Mon, 01 Jan 0001 00:00:00 +0000

Your CoreDNS pods are in CrashLoopBackOff. kubectl logs shows a line like [FATAL] plugin/loop: Loop (127.0.0.1:53 -> :53) detected for zone ".", and the process exits a few seconds after every start. There are no CoreDNS metrics to look at, because the process dies before the metrics endpoint is ever scraped.

The crash is intentional. The loop plugin detected that the Corefile’s forward target routes queries back to CoreDNS itself, and it killed the process on purpose to prevent an infinite query amplification storm. Kubernetes restarts the pod, the loop is detected again, and the cycle repeats.

CoreDNS memory climbing: heap growth, post-GC minima, and leak detection

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS memory has been creeping up for hours. The Grafana panel shows a sawtooth that never quite comes back down, and you need to answer one question before the pod gets OOMKilled: is this Go being Go, or is something actually leaking?

The trap is that go_memstats_heap_inuse_bytes is spiky by design. The heap grows, the garbage collector reclaims, the heap grows again. If you react to instantaneous peaks you will chase ghosts all night. The signal that separates normal GC behavior from a leak is the trend of the post-GC minima: the floor each sawtooth returns to. If that floor keeps rising over an hour or more and never returns to baseline, you have a genuine leak heading for the container memory limit.

CoreDNS memory limit and the re-list spike: sizing for the restart peak

Mon, 01 Jan 0001 00:00:00 +0000

A CoreDNS pod that runs fine for weeks gets OOMKilled thirty seconds after a restart. Kubernetes starts it again. It re-syncs, allocates heavily, crosses the limit, and gets killed again. Restart count climbs into the hundreds, and cluster DNS capacity drops by half or more while the crash loop runs. kubectl describe pod shows Last State: Terminated, Reason: OOMKilled, and nothing in steady-state monitoring predicted it.

The mechanism: the memory limit was sized against steady-state RSS, but the kubernetes plugin’s startup re-list is the actual peak memory event in CoreDNS’s lifecycle. In large clusters that peak runs 2-3x steady state. A limit set 10-20% above normal RSS is a limit sized to guarantee a crash loop on the next restart.

CoreDNS monitoring checklist: the signals every production resolver needs

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS has one failure mode that defeats most monitoring setups: a pod that passes every health probe while returning SERVFAIL for every query. The /health endpoint on port 8080 checks process liveness. It does not test DNS resolution. The /ready endpoint on port 8181 is plugin-aware, but it does not test resolution either. Teams that alert on pod status and probe results are monitoring the wrong thing.

The actual availability signal for a DNS server is the RCODE distribution of its responses, primarily coredns_dns_responses_total{rcode="SERVFAIL"}. Everything else exists to explain and predict changes in that signal.

CoreDNS monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Most CoreDNS monitoring setups are either a single “is the pod up” check or a wall of dashboards nobody reads during an incident. Neither works. The right question is not “how many metrics do we collect” but “which failure modes can we actually detect right now.” That is what a maturity model answers: it orders signals by the failures they catch, so you can see exactly which class of outage would currently reach your users before it reaches your alerts.

CoreDNS ndots:5 query amplification: one lookup becoming five

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS dashboards show 50,000 QPS, the denial cache is churning, and NXDOMAIN responses are a third of all traffic, but nothing is broken. What you are looking at is probably not real demand. The default ndots:5 in every pod’s /etc/resolv.conf expands each external lookup through the Kubernetes search domain list, so one logical lookup becomes four to six wire queries before the real name is ever tried.

Teams routinely misread this. They see amplified QPS, assume 50,000 lookups per second of genuine demand, and capacity-plan, scale, and alarm against a number inflated 4-6x. A rule of thumb: 50,000 QPS at CoreDNS often represents only 8,000 to 12,000 logical application lookups.

CoreDNS NOERROR with zero answers: the resolution failure that reports success

Mon, 01 Jan 0001 00:00:00 +0000

Your RCODE dashboards are green. NOERROR rate is healthy, SERVFAIL is near zero, health probes pass. Yet an application team reports their service cannot resolve a dependency. dig returns instantly with status: NOERROR and ANSWER SECTION: 0. No address, no error, no timeout. The resolver library treats this as “the name exists but has no records” and the application fails with a connection error that looks nothing like a DNS problem.

CoreDNS not resolving external domains: the missing catch-all forward zone

Mon, 01 Jan 0001 00:00:00 +0000

The symptom is specific: names inside the cluster resolve, but anything outside the cluster fails. kubernetes.default.svc.cluster.local works. example.com does not. Applications report DNS timeouts, getaddrinfo failures, or intermittent external dependency errors while service discovery inside Kubernetes looks fine.

That split is the clue. CoreDNS is not one global resolver. It matches each query to the most specific server block in the Corefile, then runs that block’s plugin chain. If cluster.local is handled by the kubernetes plugin but there is no catch-all block for ., external names match nothing useful and are refused instead of forwarded.

CoreDNS NXDOMAIN flood from one source: DGA malware and domain enumeration

Mon, 01 Jan 0001 00:00:00 +0000

Your CoreDNS dashboards are green: SERVFAIL is zero, latency is fine, cache hit ratio looks normal. But the NXDOMAIN rate has jumped, and when you pull query logs, one source IP is responsible for thousands of NXDOMAIN responses per hour, querying names that do not exist.

This is a security signal, not a CoreDNS health signal. A single source generating a sustained flood of NXDOMAIN responses is doing one of three things: cycling randomly generated domains looking for a command-and-control server (DGA malware in a compromised pod), probing which service names exist in your cluster (reconnaissance), or misbehaving in a way that merely looks like an attack (a config typo hammering a name that was never created).

CoreDNS NXDOMAIN vs SERVFAIL: why alerting on the wrong one buries real incidents

Mon, 01 Jan 0001 00:00:00 +0000

Your CoreDNS alert fired again. The “DNS error rate” panel shows a wall of red, so you open it, see a pile of NXDOMAIN responses, and silence the alert. Meanwhile, a genuine upstream failure is producing SERVFAIL responses somewhere in that same wall of red, and nobody will notice until an application team opens a ticket.

This is the most common CoreDNS alerting mistake: treating all non-NOERROR responses as one bucket. NXDOMAIN and SERVFAIL look similar in a naive error counter, but they mean opposite things. NXDOMAIN is the server correctly reporting that a name does not exist. SERVFAIL is the server admitting it failed to answer at all. Alerting on the sum of both guarantees that the noise (NXDOMAIN, constant and expected in Kubernetes) drowns the signal (SERVFAIL, rare and always actionable).

CoreDNS OOMKilled: the memory cliff and the restart crash loop

Mon, 01 Jan 0001 00:00:00 +0000

You found the CoreDNS pods in kube-system with OOMKilled as the last terminated state. Maybe one pod restarted once and recovered. Maybe both replicas are flapping and cluster DNS is degraded or down. Either way, the pod status operators paste into a search bar is the same: Last State: Terminated, Reason: OOMKilled.

Memory exhaustion in CoreDNS is a cliff edge, not a slowdown. There is no degradation curve where latency rises and errors climb while you watch. RSS approaches the container memory limit, the kernel OOM killer terminates the process instantly, and every client served by that pod loses DNS until a replacement is ready. The warning phase exists only in your metrics, and only if you are watching the right ones.

CoreDNS panics: recovered handler crashes and coredns_panics_total

Mon, 01 Jan 0001 00:00:00 +0000

You found coredns_panics_total above zero, or you saw Recovered from panic in the CoreDNS logs. The pods are not restarting, the health endpoint returns 200, and most DNS queries still resolve. That combination is what makes this failure mode easy to ignore and dangerous to dismiss.

A recovered panic means a query handler crashed inside the plugin chain and CoreDNS’s recovery wrapper caught it, keeping the process alive. The server survived. The query that triggered the panic did not: the client that sent it got no answer and waited out its own timeout. Every increment of the counter is a real query that failed, plus evidence of a genuine bug in CoreDNS or one of its plugins.

CoreDNS per-replica divergence: why averaged metrics hide a failing pod

Mon, 01 Jan 0001 00:00:00 +0000

Your CoreDNS dashboard looks slightly off. Average latency drifted up, SERVFAIL ratio is elevated but under the threshold, and nobody is being paged. Meanwhile, half the cluster’s DNS queries are slow or failing, because one of your two replicas is degraded and kube-proxy is still sending it traffic.

This is the per-replica divergence failure mode. CoreDNS in Kubernetes is almost never a singleton: the default deployment runs two replicas behind the kube-dns ClusterIP Service, and kube-proxy load-balances across them. Any aggregation that averages or sums across pods blends a sick replica’s numbers with healthy ones until the result looks like mild degradation instead of a partial outage.

CoreDNS per-upstream health check failures: degraded redundancy before total loss

Mon, 01 Jan 0001 00:00:00 +0000

One of your upstream DNS resolvers is failing CoreDNS’s health checks. Queries are still being answered because the remaining upstreams are carrying the forwarding load, so nothing is paging yet. This is exactly the window you want: degraded redundancy, not an outage. The next failure in this sequence is coredns_forward_healthcheck_broken_total incrementing, at which point every upstream is marked unhealthy and you are in a real incident.

The signal is coredns_proxy_healthcheck_failures_total, a counter in the proxy subsystem. Filter it by proxy_name="forward" and read the to label: it names the specific upstream address that is failing. That label is the whole point of this metric. It turns “forwarding is degraded” into “10.0.0.2:53 is the problem.”

CoreDNS pod alive but not ready: the /ready endpoint and API sync

Mon, 01 Jan 0001 00:00:00 +0000

The pod is Running. The process is up, the logs show CoreDNS started, maybe it is even answering some queries. But kubectl get pods -n kube-system shows 0/1 READY, the pod is excluded from the kube-dns Service endpoints, and your cluster is running on one fewer DNS replica than you think. If this is your only replica, or the other one is on the same node, you are one hiccup away from a cluster-wide DNS outage.

CoreDNS query rate dropped to zero while the process looks healthy

Mon, 01 Jan 0001 00:00:00 +0000

The dashboard shows coredns_dns_requests_total flatlined at zero. You check the pod: Running, zero restarts. /health on port 8080 returns 200. The process is fine. So either nobody in the cluster is resolving names anymore, or the traffic is dying somewhere between your clients and the CoreDNS process, and CoreDNS has no idea.

The second possibility is the trap. CoreDNS only counts queries it actually receives. A UDP packet dropped by the kernel before it reaches the CoreDNS socket is never counted, never logged, and never reflected in any CoreDNS metric. A full conntrack table, an overflowing UDP receive buffer, a network policy isolating the pod, or a readiness failure that pulled the pod out of the Service endpoints all look identical in CoreDNS metrics: nothing.

CoreDNS reload failed: config drift when the new Corefile never took effect

Mon, 01 Jan 0001 00:00:00 +0000

You pushed a Corefile change, nothing broke, and everyone moved on. Weeks later you discover the cluster is still running the old configuration. The metric that would have told you is coredns_reload_failed_total, and it was nonzero the whole time.

A failed CoreDNS reload is not an outage. The reload plugin polls the Corefile every 30 seconds (with jitter) and triggers a graceful reload when the SHA512 checksum of the file changes. If the new configuration has a syntax error, an invalid plugin directive, or hits a port conflict, CoreDNS logs the error, increments coredns_reload_failed_total, and keeps serving DNS with the old config. Queries keep flowing. The only thing that changed is that reality and your intent have quietly diverged.

CoreDNS response size distribution: large answers, TCP pressure, and amplification

Mon, 01 Jan 0001 00:00:00 +0000

Most CoreDNS dashboards track QPS, latency, and SERVFAIL. Response size rarely makes the cut until something odd happens: TCP/53 traffic climbs for no obvious reason, a workload starts resolving incomplete answers, or a security review asks whether your resolvers could be used as DDoS reflectors. All three questions are answered by the same signal: the distribution of DNS response sizes CoreDNS is sending.

The metric is coredns_dns_response_size_bytes, a histogram exported by the prometheus plugin with labels server, zone, view, and proto (udp/tcp). This article covers what a healthy distribution looks like, what pushes it toward large answers, why large answers create TCP pressure, and how to separate benign large record sets from amplification abuse using the labels you already have. It is a reference for interpreting the signal; the related guides at the end cover the concrete failure modes.

CoreDNS response truncation: the TC bit, EDNS bufsize, and TCP fallback

Mon, 01 Jan 0001 00:00:00 +0000

Most DNS traffic fits in a single UDP datagram. When it does not, the protocol has an escape hatch: the server sets the TC (truncated) bit in the response, and the client retries the same query over TCP. CoreDNS implements this correctly in the common case, but the edges are where production incidents live: firewalls that silently block TCP/53, legacy clients that do not speak EDNS0, and known CoreDNS bugs where the TC flag never reaches the client at all.

CoreDNS returning REFUSED: no matching zone, an ACL, or the forward concurrency limit

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS is answering queries with REFUSED and clients cannot resolve names. Unlike SERVFAIL, which means “I tried and failed,” REFUSED means “I declined to try.” REFUSED is almost never an upstream or network problem: CoreDNS itself is deciding not to answer, and it does that for exactly three reasons.

Two are configuration problems: no server block matches the query’s zone, or the acl plugin is rejecting the source. One is a capacity problem: the forward plugin’s max_concurrent limit is shedding load. Determine which of the three you are dealing with before touching the Corefile. Editing the Corefile to fix a capacity REFUSED, or scaling replicas to fix a config REFUSED, wastes the incident.

CoreDNS returning SERVFAIL: the resolver is failing queries and what to check first

Mon, 01 Jan 0001 00:00:00 +0000

Applications are failing to resolve names and CoreDNS is answering with SERVFAIL. In Kubernetes this surfaces as connection errors everywhere at once, because nearly all in-cluster communication depends on DNS. SERVFAIL is not one failure. It is CoreDNS saying “a plugin in my chain could not answer this query,” and the cause is usually upstream DNS, the Kubernetes API, or the Corefile itself.

Your health checks are probably lying to you. The CoreDNS /health endpoint on port 8080 only tests process liveness. It does not resolve anything. A pod can return 200 OK to every probe while returning SERVFAIL for every query. The real availability signal is coredns_dns_responses_total{rcode="SERVFAIL"} on the metrics endpoint, split by the zone and plugin labels.

CoreDNS serve_stale: keeping resolution alive while masking upstream failure

Mon, 01 Jan 0001 00:00:00 +0000

The serve_stale option in the CoreDNS cache plugin is a resilience feature with a monitoring side effect that bites teams during real incidents. When it is enabled, CoreDNS answers queries from expired cache entries instead of failing them when the upstream resolver is unreachable. Clients keep resolving names through an upstream outage. Dashboards stay green. The upstream can be dead for an hour before anyone notices, because the one metric that would tell you is almost never graphed.

CoreDNS SERVFAIL cache amplification: a one-second blip becomes a five-second outage

Mon, 01 Jan 0001 00:00:00 +0000

Your dashboards show an upstream DNS hiccup that lasted one second. Your users report an outage that lasted five. Both are accurate, and the gap between them is the CoreDNS cache plugin doing what it was designed to do.

CoreDNS caches SERVFAIL responses for 5 seconds by default. The cache plugin keeps separate positive (success) and negative (denial) caches, and SERVFAIL goes into the denial cache alongside NXDOMAIN and NODATA. The intent is sound: RFC 2308 permits caching server-failure responses to shield an already-struggling upstream from a retry storm. The side effect is that a single SERVFAIL answer for a hot record is served to every client that asks during the 5-second window, even after the upstream has fully recovered.

CoreDNS serving stale cluster DNS: why you need a functional freshness test

Mon, 01 Jan 0001 00:00:00 +0000

Your CoreDNS dashboards are green. Query rate normal, latency sub-millisecond, SERVFAIL near zero, cache hit ratio healthy. Meanwhile, a Service deleted an hour ago still resolves, and the new Service your team just deployed returns NXDOMAIN. Applications are failing, but nothing in your monitoring fired.

This is the silent stale data pattern. The kubernetes plugin builds its DNS records from a watch on the API server. If that watch disconnects and cannot re-establish, CoreDNS keeps answering from its in-memory snapshot. The snapshot drifts from reality. No errors go back to clients. No performance metric moves, because the server is genuinely healthy and fast. The data is simply wrong.

CoreDNS slow upstream: per-upstream latency, goroutine pileup, and the to label

Mon, 01 Jan 0001 00:00:00 +0000

Overall DNS latency is climbing. P99 is above 500ms, but P50 looks only mildly elevated. SERVFAIL is nonzero but nowhere near a full outage. The /health endpoint returns 200, every upstream passes its health checks, and yet clients are complaining that resolution is slow. Meanwhile go_goroutines is trending up and heap is following it.

This is the slow upstream drag pattern: one upstream DNS server is responding slowly but still returning valid answers. Because it never actually fails, the forward plugin’s health checks never mark it down, so queries keep getting routed to it. Each of those queries holds a goroutine while it waits. Latency accumulates, goroutines accumulate, and memory follows. Left alone, this ends in either max_concurrent rejects (if you set one) or OOM (if you did not).

CoreDNS too many open files: file descriptor exhaustion and refused connections

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS pods are logging too many open files and DNS resolution is failing intermittently or completely. Clients see timeouts or refused connections while the process keeps running and /health still returns 200. Restarting the pod fixes it for a while, then it comes back.

This is file descriptor exhaustion, and it is a cliff-edge failure. The moment process_open_fds reaches process_max_fds, CoreDNS cannot open anything new: no upstream connections, no listening sockets, no log files, no Kubernetes API watch streams. There is no queuing and no graceful degradation. New connection attempts fail immediately and DNS breaks.

CoreDNS TTL=0 responses: the anti-pattern that silently bypasses the cache

Mon, 01 Jan 0001 00:00:00 +0000

Your cache hit ratio is sliding. Upstream query volume is climbing. P99 latency is drifting up with it. You open the Corefile and the cache plugin is right there, configured the way it has been for months. Nothing changed on your side, but the cache has effectively stopped working.

A common cause is records arriving with TTL=0. When an upstream resolver, or your own zone data, answers with a zero TTL, those answers are uncacheable by contract: every client query for that name has to be forwarded again. The cache plugin is present and correct, but there is nothing for it to hold on to. It looks like a cache failure but is actually a data problem arriving through the response path.

CoreDNS UDP buffer errors: kernel receive-buffer overflow dropping queries

Mon, 01 Jan 0001 00:00:00 +0000

Applications are reporting intermittent DNS timeouts. You open the CoreDNS dashboard and everything looks fine: latency is low, SERVFAIL rate is zero, the health endpoint returns 200, and QPS is suspiciously flat. Not spiking, not zero, just lower than the client demand you know exists.

This is the UDP buffer cliff. Incoming UDP traffic is exceeding the kernel socket receive buffer on the node, and the kernel is dropping DNS queries before CoreDNS ever reads them. Because CoreDNS only counts the packets it actually receives, every CoreDNS-level metric looks clean. The failure lives one layer below, in the kernel.

CoreDNS upstream connection cache misses: new connections adding latency per query

Mon, 01 Jan 0001 00:00:00 +0000

You are here because forwarded DNS queries got slower, or because coredns_proxy_conn_cache_misses_total is climbing and you need to know whether it matters. It does. Every connection cache miss means CoreDNS opens a fresh connection to an upstream resolver instead of reusing one from its pool. Setup cost lands on that query, and each new connection holds a file descriptor until it is closed or reaped.

This is usually degradation, not outage. CoreDNS keeps answering. The damage shows up as worse tail latency on uncached lookups, rising process_open_fds, and, under sustained churn, resource pressure that can turn into a real incident. Rule of thumb: a miss ratio above 50% sustained for 10 minutes is worth investigating.

CoreDNS with NodeLocal DNSCache: what changes about monitoring

Mon, 01 Jan 0001 00:00:00 +0000

You deployed NodeLocal DNSCache to fix the 5-second timeout race and conntrack pressure, and it worked. DNS latency dropped, the conntrack table stopped filling, and CoreDNS QPS fell off a cliff. Then someone looked at the CoreDNS dashboard, saw near-zero traffic, and concluded CoreDNS was oversized. That conclusion is how the next incident starts.

NodeLocal DNSCache runs a caching DNS proxy as a DaemonSet on every node. It intercepts pod DNS queries before they enter the iptables DNAT and conntrack path, so each query is answered on the local node instead of traversing NAT to a central CoreDNS pod. This eliminates the kernel race condition behind the 5-second glibc timeout and removes DNS as a conntrack consumer.

How CoreDNS actually works in production: the plugin chain mental model

Mon, 01 Jan 0001 00:00:00 +0000

Most CoreDNS incidents are pipeline incidents, not DNS incidents. An upstream resolver gets slow, goroutines pile up, memory climbs, and the pod gets OOMKilled. Or the Kubernetes API watch drops silently and CoreDNS keeps answering every query correctly except the answers are three hours stale. Each failure mode is a direct consequence of how CoreDNS is built internally, and most CoreDNS runbooks assume you already hold that internal model in your head.