CoreDNS with NodeLocal DNSCache: what changes about monitoring

You deployed NodeLocal DNSCache to fix the 5-second timeout race and conntrack pressure, and it worked. DNS latency dropped, the conntrack table stopped filling, and CoreDNS QPS fell off a cliff. Then someone looked at the CoreDNS dashboard, saw near-zero traffic, and concluded CoreDNS was oversized. That conclusion is how the next incident starts.

NodeLocal DNSCache runs a caching DNS proxy as a DaemonSet on every node. It intercepts pod DNS queries before they enter the iptables DNAT and conntrack path, so each query is answered on the local node instead of traversing NAT to a central CoreDNS pod. This eliminates the kernel race condition behind the 5-second glibc timeout and removes DNS as a conntrack consumer.

The cost is that your monitoring model inverts. Most queries never reach central CoreDNS anymore. CoreDNS QPS now measures node-local cache misses, not cluster DNS demand. Every dashboard, alert threshold, and capacity assumption built on “CoreDNS traffic equals DNS traffic” is now wrong.

What changes when you deploy NodeLocal DNSCache

Without NodeLocal DNSCache, the query path is: pod to kube-dns ClusterIP, through iptables DNAT and conntrack, to a CoreDNS pod. CoreDNS sees every query. Its metrics are a complete picture of cluster DNS demand.

With NodeLocal DNSCache, each node runs a local caching instance (itself CoreDNS, configured as a caching forwarder). Pods are pointed at a link-local address on the node, typically 169.254.20.10. The node-local instance answers from its cache when it can. On a miss, it forwards to the central CoreDNS service, and the node-local Corefile forces TCP for that upstream hop by default, which changes the connection profile between the layers.

Two consequences follow:

• The central CoreDNS becomes a miss handler. Its traffic is whatever the per-node caches could not serve: cold entries, expired TTLs, names with short or zero TTLs, and the long tail of unique names. This traffic is bursty and correlates across nodes during events like rollouts or upstream TTL expiry.
• Failure blast radius moves. A central CoreDNS or upstream failure no longer degrades gracefully per query. Node-local caches absorb it silently until their entries expire, then every node re-queries at once. The failure arrives at central CoreDNS as a synchronized flood.

flowchart LR
  subgraph node["Every node"]
    pod["Application pod"] -->|"query to link-local IP"| nld["node-local-dns cache"]
    nld -->|"cache hit"| pod
  end
  nld -->|"miss, TCP"| cd["Central CoreDNS"]
  cd -->|"cluster.local"| k8s["Kubernetes API state"]
  cd -->|"external zones"| up["Upstream resolvers"]

The old path, pod to ClusterIP through NAT, is gone from the hot path. That is exactly why the signals you used to rely on, conntrack pressure and CoreDNS-side query volume, no longer tell you about total demand.

What changes on the central CoreDNS side

QPS no longer means load

coredns_dns_requests_total on the central CoreDNS pods now counts cache misses from every node-local instance. A drop of 80 to 95 percent after enabling NodeLocal DNSCache is expected and healthy. It is not spare capacity you can reclaim.

The trap is under-provisioning central CoreDNS because “it was not doing much.” When an upstream resolver fails or a wave of TTLs expires, every node’s cache misses simultaneously and the flood lands on a CoreDNS deployment that was scaled down to match its quiet period. Size central CoreDNS for the miss storm, not the steady state.

Latency gets slower on average, and that is fine

Central CoreDNS used to serve a mix of cache hits (sub-millisecond) and forwards. Now node-local caches absorb the hits, so coredns_dns_request_duration_seconds on central CoreDNS skews toward forwarded and kubernetes-plugin queries. Average latency rises. P99 matters more than the mean here; watch coredns_proxy_request_duration_seconds per upstream (to label) to see whether the rise is your upstreams or your CoreDNS.

Cache metrics at the central layer change meaning

The central cache now mostly holds entries that node-local caches also missed on. Its hit ratio (coredns_cache_hits_total / coredns_cache_requests_total) will typically fall, because the easy, repetitive hits are served one layer down. Do not alert on the central cache hit ratio dropping after the rollout; establish a new baseline first. Evictions (coredns_cache_evictions_total) still mean the central cache is undersized for its working set, but the working set itself changed.

SERVFAIL amplification works differently now

CoreDNS caches SERVFAIL for 5 seconds by default, amplifying brief upstream blips. With NodeLocal DNSCache there are two caching layers. A short upstream failure can be cached at the central layer and then re-cached by every node-local instance, so a one-second blip can surface as cluster-wide failures that outlive the actual upstream problem. Watch coredns_dns_responses_total{rcode="SERVFAIL"} at both layers, split by the plugin label, before assuming the upstream is still down.

What appears on the node-local-dns side

The node-local-dns pods run CoreDNS in caching mode and expose the same Prometheus metric families, but on port 9253 rather than 9153. Because the DaemonSet runs one pod per node, every signal is per-node. There is no cluster-wide aggregate unless you build one by summing across pods.

What the layer tells you:

• Real cluster DNS demand. coredns_dns_requests_total summed across all node-local-dns pods is the true cluster QPS. This is the number central CoreDNS QPS used to represent.
• Node-local cache effectiveness. The hit ratio per node shows how much traffic never leaves the node. Low hit ratio on one node with normal ratios elsewhere points at a node-specific workload pattern, not a DNS problem.
• Miss pressure heading for central CoreDNS. Miss rate across all nodes, aggregated, is the best leading indicator of load about to hit the central layer.
• Upstream health from the edge. The node-local instances health-check their upstream, which is central CoreDNS. A simultaneous spike in health-check failures across many nodes is your earliest warning that central CoreDNS is struggling, before SERVFAILs reach applications.
• Connection pressure. The forced-TCP hop to central CoreDNS means each node-local instance holds upstream connections. Watch process_open_fds and the connection cache metrics (coredns_proxy_conn_cache_misses_total) on busy nodes.

One operational caveat: the node-local-dns pod installs iptables rules to intercept DNS traffic. If the pod is OOMKilled, those rules can persist while the cache is down, producing a DNS gap on that node until the container restarts. Monitor per-node pod restarts, not just aggregate DaemonSet health.

The failure mode this architecture hides

The most dangerous property of the two-layer design is how well it masks central-layer failure. A warm cache hides an upstream failure until TTLs expire, then the failure becomes sudden and complete. NodeLocal DNSCache multiplies the effect because the masking cache is distributed across every node and the unmasking is synchronized.

The sequence looks like this:

1. Central CoreDNS or its upstreams degrade. Node-local caches keep answering from warm entries. Application metrics stay green.
2. TTLs expire. Entries age out at roughly the same rate across nodes because they were cached at roughly the same time.
3. Every node-local instance starts forwarding. Central CoreDNS goes from quiet to flooded in seconds. If it was scaled down during the quiet period, it now hits coredns_forward_max_concurrent_rejects_total, REFUSED responses, and goroutine accumulation.
4. Node-local caches receive and re-cache failures, extending the blast radius beyond the original fault.

Your early-warning signals for this sequence all live at the seams: node-local health-check failures toward central CoreDNS, central coredns_forward_healthcheck_broken_total and per-upstream coredns_proxy_healthcheck_failures_total, and the aggregate node-local miss rate. End-to-end success ratio will be the last signal to move, which is exactly why you cannot rely on it.

Signals to watch in production

Two habits make this table usable. First, always split dashboards by layer; a chart that mixes central and node-local CoreDNS metrics with the same metric names will mislead you. Second, baseline everything again after the rollout. Pre-NodeLocal thresholds for QPS, hit ratio, and latency are invalid by construction.

Common misreadings after the rollout

• “CoreDNS QPS dropped 90 percent, something is broken.” It is working. Verify by checking that aggregate node-local QPS matches your pre-rollout cluster demand.
• “CoreDNS is idle, scale it down.” It is a miss handler. Scale it for the correlated miss storm during an upstream failure or mass TTL expiry, and keep enough replicas that a rolling restart does not coincide with one.
• “DNS looks healthy, no alerts fired.” Check whether your alerts only watched the central layer. Node-local failures, per-node OOM kills, and rising miss rates are invisible there.
• “Latency went up after we deployed the cache.” Central-layer latency rising is expected because the hits moved. Compare application-observed DNS latency before and after, not the central CoreDNS histogram.

How Netdata helps

• Per-second scraping of both layers makes the synchronized miss flood visible as it forms, where minute-resolution monitoring shows only the aftermath.
• Netdata discovers CoreDNS Prometheus endpoints and charts the standard metric families, so you can place node-local-dns (port 9253) and central CoreDNS (port 9153) side by side and compare QPS, SERVFAIL by rcode, and cache hit ratios on the same time axis.
• Per-upstream breakdowns of request duration and health-check failure metrics on central CoreDNS let you see a single failing upstream while node-local caches are still masking it from applications.
• Go runtime metrics (go_goroutines, heap, GC pauses) on central CoreDNS expose the accumulation phase of a miss storm before pods start rejecting queries or getting OOMKilled.
• Node-level views alongside DaemonSet pod metrics help you spot the single-node cases: an OOMKilled node-local-dns pod, FD exhaustion on a busy node, or a workload pattern producing an abnormally low local hit ratio.
• ML-based anomaly detection on the aggregate node-local miss rate gives you a leading indicator for central-layer load without hand-tuned thresholds on a traffic pattern that changes with every rollout.

Related guides

• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• CoreDNS cache collapse: the cold-cache thundering herd after a rollout
• CoreDNS cache evictions: the cache is too small for the working set
• CoreDNS cache hit ratio dropping: latency and upstream load climbing together
• CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed
• CoreDNS CPU throttling: CFS limits making a green dashboard lie about latency