CoreDNS dashboards show 50,000 QPS, the denial cache is churning, and NXDOMAIN responses are a third of all traffic, but nothing is broken. What you are looking at is probably not real demand. The default ndots:5 in every pod’s /etc/resolv.conf expands each external lookup through the Kubernetes search domain list, so one logical lookup becomes four to six wire queries before the real name is ever tried.

Teams routinely misread this. They see amplified QPS, assume 50,000 lookups per second of genuine demand, and capacity-plan, scale, and alarm against a number inflated 4-6x. A rule of thumb: 50,000 QPS at CoreDNS often represents only 8,000 to 12,000 logical application lookups.

This article covers the exact expansion mechanism, how to confirm it in your cluster, what it costs CoreDNS and your upstreams, and the two fixes that actually work: per-workload ndots overrides and trailing-dot FQDNs.

What this means

The kubelet writes every pod’s resolver configuration. A typical pod /etc/resolv.conf has a nameserver pointing at the cluster DNS service, a search line with the cluster suffixes, and options ndots:5. The search list typically has three to four entries: <namespace>.svc.cluster.local, svc.cluster.local, cluster.local, and on some cloud providers an additional node-level internal domain.

ndots:5 tells the stub resolver: if the name being resolved contains fewer than 5 dots, treat it as possibly relative and try each search suffix first. Only after every suffix fails does the resolver try the name as-is, as an absolute name.

So when an application calls getaddrinfo("api.partner.com"), the resolver sees 2 dots, which is less than 5, and issues this sequence:

  1. api.partner.com.<namespace>.svc.cluster.local - NXDOMAIN from the kubernetes plugin
  2. api.partner.com.svc.cluster.local - NXDOMAIN
  3. api.partner.com.cluster.local - NXDOMAIN
  4. api.partner.com.<node search domain> - NXDOMAIN (if present)
  5. api.partner.com. - the real query, forwarded upstream, NOERROR

One logical lookup, five wire queries, three to four of them guaranteed NXDOMAINs. If the application also looks up AAAA alongside A (glibc does both by default), double everything.

flowchart TD
  A["App: getaddrinfo(api.partner.com)"] --> B{"dots < ndots (5)?"}
  B -->|"yes: 2 dots"| C["api.partner.com.NS.svc.cluster.local"]
  C -->|"NXDOMAIN"| D["api.partner.com.svc.cluster.local"]
  D -->|"NXDOMAIN"| E["api.partner.com.cluster.local"]
  E -->|"NXDOMAIN"| F["api.partner.com."]
  F -->|"NXDOMAIN"| G["api.partner.com. (absolute)"]
  G -->|"NOERROR via forward"| H["answer returned to app"]
  B -->|"no: trailing dot or 5+ dots"| G

The intermediate NXDOMAINs are served by CoreDNS’s kubernetes plugin or its denial cache, so they are fast and mostly invisible in latency metrics. The cost shows up elsewhere: inflated QPS, denial cache churn, extra upstream load on cache misses, extra conntrack entries per node, and capacity math that is off by a factor of four to six.

Common causes

The amplification is a default, not a bug. What makes it a production problem is usually one of these triggers:

CauseWhat it looks likeFirst thing to check
Default ndots:5 never overriddenSustained high QPS and 30-60% NXDOMAIN ratio since the cluster was builtkubectl exec <pod> -- cat /etc/resolv.conf and look at the options line
New workload that mostly resolves external namesStep change in QPS and NXDOMAIN ratio correlating with a deployment eventQuery logs (if the log plugin is enabled) for the same base name with different cluster.local suffixes
Runtime with no resolver cache (short-lived processes, some language runtimes)Same names re-expanded on every request instead of being cached client-sidePer-pod query volume; a single pod generating disproportionate QPS
Capacity planning against amplified QPSCoreDNS “at capacity” at 50k QPS that is really ~10k logical lookupsRatio of NXDOMAIN to NOERROR responses; query names in logs

Quick checks

All read-only. Run from a pod that exhibits the pattern and against the CoreDNS metrics endpoint (port 9153).

# 1. Confirm the resolver config in an affected pod
kubectl exec <pod> -- cat /etc/resolv.conf
# Look for: options ndots:5 and the search line

# 2. Check whether a workload has an ndots override already
kubectl get pod <pod> -o yaml | grep -A6 dnsConfig

# 3. NXDOMAIN share of all responses (sample the counters)
curl -s http://localhost:9153/metrics | grep 'coredns_dns_responses_total' | grep 'NXDOMAIN'

# 4. Total query rate by zone - cluster.local carrying the expansion load
curl -s http://localhost:9153/metrics | grep '^coredns_dns_requests_total'

# 5. Denial cache pressure from cached NXDOMAINs
curl -s http://localhost:9153/metrics | grep '^coredns_cache_entries'
curl -s http://localhost:9153/metrics | grep 'coredns_cache_hits_total' | grep 'denial'

Two testing gotchas that waste operator time:

  • dig does not reproduce the problem. dig does not use the search list the way the glibc stub resolver does; it effectively treats the name as given. Debug with nslookup or getent hosts from inside the pod, which follow resolv.conf search and ndots semantics. Operators who test with dig, see one clean query, and close the ticket miss the amplification entirely.
  • Test the trailing-dot behavior explicitly. nslookup api.partner.com. (note the trailing dot) from the pod should produce exactly one query. Comparing query counts with and without the trailing dot is the cleanest live demonstration of the amplification factor.

How to diagnose it

  1. Establish the amplification factor. Compare application-side lookup rates (if you have them) with CoreDNS coredns_dns_requests_total. If CoreDNS QPS is 4-6x what applications believe they are issuing, search domain expansion is the difference. Without client-side numbers, the NXDOMAIN ratio is the proxy.
  2. Confirm the NXDOMAINs are expansion artifacts, not failures. If the log plugin is enabled, look for the telltale sequence: the same base name tried with <namespace>.svc.cluster.local, then svc.cluster.local, then cluster.local, each NXDOMAIN, followed by the bare name succeeding. That pattern is diagnostic. NXDOMAIN is a normal response; never alert on it as an error. The signal here is volume and ratio, not the rcode itself.
  3. Identify which pods are driving it. CoreDNS does not expose per-source-IP Prometheus metrics, so this needs the log plugin or dnstap. Aggregate query volume by source IP and map back to pods. One chatty service that resolves external APIs on every request is usually most of the amplification.
  4. Check the secondary costs. Denial cache entries and denial hit ratio (coredns_cache_entries{type="denial"}, coredns_cache_hits_total{type="denial"}), upstream query rate, and node conntrack utilization. Each amplified query is a conntrack entry and, on denial-cache miss, kubernetes plugin work.
  5. Rule out lookalikes. A rising NXDOMAIN ratio can also come from an application querying genuinely nonexistent names (misconfigured service discovery, DGA-like traffic). The distinguishing feature of ndots amplification is that the NXDOMAINs share a base name and differ only in cluster suffix, and each burst ends in a NOERROR for the bare name.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
coredns_dns_requests_total (by zone)The amplified number; this is what capacity math must correct forSustained >2x rolling baseline without a known business event
coredns_dns_responses_total{rcode="NXDOMAIN"} as a ratio of total responsesDirect read on search-domain expansion volumeRatio shift vs rolling baseline; in Kubernetes, 20-60% NXDOMAIN is normal, so alert on change, not the absolute value
coredns_cache_entries{type="denial"}Denial cache filling with expansion NXDOMAINsDenial entries at configured maximum with rising evictions
coredns_cache_hits_total{type="denial"} / coredns_cache_requests_totalHow much expansion load the denial cache absorbs vs re-processingLow denial hit ratio means every expansion re-hits the kubernetes plugin
coredns_dns_request_duration_secondsExpansion adds client-side latency even when server-side latency looks fineElevated P99 combined with a high NXDOMAIN ratio
coredns_proxy_request_duration_seconds (count)Upstream load; cache misses on the final absolute query still forwardUpstream query rate tracking total QPS growth 1:1
Node conntrack entries vs limitEvery amplified UDP query creates a conntrack entrynf_conntrack_count approaching nf_conntrack_max; kernel log “table full”

Alerting posture: do not alert on absolute NXDOMAIN count. Alert on the NXDOMAIN-to-total ratio deviating from its rolling baseline, and keep SERVFAIL alerting strictly separate. Confusing the two is a standard source of alert fatigue.

Fixes

Override ndots per workload

For pods that primarily resolve external names, set ndots lower in the pod spec:

spec:
  dnsConfig:
    options:
      - name: ndots
        value: "1"

With ndots:1, any name containing at least one dot is tried as an absolute name first, so api.partner.com resolves in one query. Single-label names like myservice (0 dots) still expand through the search list, which is what you want for cluster-internal service discovery. The change requires recreating the pod; it is not picked up in place.

Tradeoffs before rolling this out broadly:

  • Dotted internal names pay one extra query. A pod using short forms like myservice.myns (1 dot) with ndots:1 will try myservice.myns. as absolute first, get NXDOMAIN, then fall back to the search list. Workloads that mix heavy external and heavy internal resolution may be better with ndots:2, or with fully qualified internal names.
  • It is per-pod. There is no first-party cluster-wide default for ndots; you set it in every Deployment spec, or enforce it at admission time.
  • musl-based images behave differently. Alpine and other musl libc images do not implement all glibc resolver options and can differ on search-list and option handling. Verify behavior in the actual image, not just in a Debian-based debug container.

Use trailing-dot FQDNs in application config

A name ending in a dot is absolute by definition and skips the search list entirely, regardless of ndots. Setting https://api.partner.com./ in application configuration reduces the lookup to one query with zero pod-spec changes. This is the cheapest fix where you control the configuration, but some TLS and HTTP stacks historically mishandle trailing dots, and it will not survive a config management layer that normalizes hostnames. Prefer the ndots override for anything long-lived.

Consider autopath for cluster-wide relief

The CoreDNS autopath plugin short-circuits search domain expansion server-side: CoreDNS tries the suffixes internally and answers the client with the final result, collapsing the five round trips into one. The tradeoff: it dramatically reduces query amplification but adds CPU and memory cost on the CoreDNS side, and it requires CoreDNS to understand the pod-to-namespace mapping. Treat it as a deliberate architectural choice with its own capacity implications, not a free win.

What NodeLocal DNSCache does and does not fix

NodeLocal DNSCache caches the NXDOMAIN responses locally on each node, so repeated expansions stop hitting CoreDNS and stop adding conntrack pressure beyond the node. That reduces latency and CoreDNS load. It does not change the amplification factor: the first expansion of each name still walks the full search list, and applications still issue five queries per logical lookup. Deploy it for the caching and conntrack benefits, not as an ndots fix.

Prevention

  • Set ndots as part of your workload onboarding standard. Any service template for workloads that call external APIs should carry a dnsConfig ndots override by default, with a documented exception path for internal-heavy services.
  • Capacity-plan against logical lookups, not wire QPS. Record your measured amplification factor (wire QPS divided by application lookup rate) and use it as a divisor in every CoreDNS sizing exercise. An autoscaling trigger on raw coredns_dns_requests_total will scale against phantom demand.
  • Watch the ratio, not the count. Baseline the NXDOMAIN-to-total ratio and alert on deviation. A step change after a deployment is early warning that a new external-heavy workload arrived without an ndots override.
  • Correlate DNS metrics with deployment events. Many CoreDNS “incidents” are application changes. A new deployment that resolves thousands of unique external names, or scales 200 pods that each resolve on startup, shows up as a DNS event. Tie CoreDNS signal changes to rollout timelines before treating them as infrastructure failures.

How Netdata helps

Netdata surfaces the specific signals this problem lives in, at per-second granularity:

  • Query rate by zone and type from coredns_dns_requests_total, so you can see cluster.local carrying expansion traffic separately from genuinely forwarded zones.
  • Response code distribution from coredns_dns_responses_total, letting you track the NXDOMAIN ratio against its own baseline instead of alerting on raw counts.
  • Denial cache behavior: coredns_cache_entries{type="denial"} and denial cache hits, so you can see whether the cache is absorbing the expansion or churning under it.
  • Upstream load correlation: forward request counts and duration alongside total QPS, which separates “upstreams are slow” from “upstreams are being asked 5x too often”.
  • Node-level conntrack and UDP buffer signals next to CoreDNS metrics on the same dashboard, which matters because amplified UDP queries are also amplified conntrack entries.

Correlating these in one place is what turns “CoreDNS looks overloaded” into “one deployment is generating 4x amplification” in minutes rather than hours.