CoreDNS cache hit ratio dropping: latency and upstream load climbing together

The cache hit ratio is coredns_cache_hits_total / coredns_cache_requests_total. When it falls, every query that used to be answered from memory in under a millisecond now goes to the kubernetes plugin or out to an upstream resolver. Two things happen at once: coredns_dns_request_duration_seconds climbs, and forwarded query volume climbs with it. That second effect is the dangerous one, because the upstream flood can degrade the upstreams themselves and turn a cache problem into a resolution outage.

A falling ratio is a symptom, not a root cause. Four causes account for almost every case: clients querying unique names (random subdomains, DGA-style traffic), upstream responses arriving with TTL=0, a cache that is too small for the working set, or a cold cache after a restart or reload. The diagnostic work is figuring out which one you have.

One housekeeping note before the math: coredns_cache_misses_total is deprecated. Derive misses as coredns_cache_requests_total - coredns_cache_hits_total. If your dashboards still reference the misses counter, migrate them to the requests/hits pair.

What this means

The cache plugin sits in the plugin chain ahead of kubernetes and forward. A hit short-circuits the chain: no upstream round trip, no API-derived lookup, sub-millisecond response. A miss falls through to the rest of the chain and then populates the cache on the way back.

The type label on coredns_cache_hits_total splits hits into success (positive answers) and denial (NXDOMAIN/NODATA negative answers). Both count as hits, and both are valuable. In Kubernetes, search-domain expansion generates a lot of NXDOMAINs, and a healthy denial cache absorbs them. When you analyze a ratio drop, split by type first: a drop in success hits with stable denial hits points at a different cause than the reverse.

Baseline expectations: cluster DNS with repeated service-name lookups typically sees 80% or higher hit ratios. Workloads that resolve many unique external names will sit lower. The alert that matters is not an absolute threshold, it is deviation: a drop of more than 20% from the rolling 24-hour baseline, sustained, warrants investigation. Ignore the first few minutes after any restart, because a cold cache reads near 0% by definition.

The failure cascade looks like this:

flowchart TD
  A[Cache hit ratio drops] --> B[More queries fall through the cache]
  B --> C[Forwarded query rate climbs]
  B --> D[Request latency climbs]
  C --> E[Upstream load climbs]
  E --> F[Upstreams slow or rate-limit]
  F --> D
  F --> G[SERVFAIL responses]
  G --> H[SERVFAIL cached for 5s, amplifying the failure]

The feedback loop at the bottom is what turns a slow leak into an incident: cache misses overload the upstream, the upstream starts failing, and the failures get cached as SERVFAIL for 5 seconds each, pushing more pain back to clients even after the upstream recovers.

Common causes

Quick checks

All read-only. Run against the metrics endpoint on port 9153 unless noted.

# 1. Current cache hit/requests counters, split by type
curl -s http://localhost:9153/metrics | grep -E 'coredns_cache_(hits|requests)_total'

# 2. Cache occupancy vs capacity, split by success/denial
curl -s http://localhost:9153/metrics | grep '^coredns_cache_entries'

# 3. Eviction pressure: nonzero rate means the cache is too small for the working set
curl -s http://localhost:9153/metrics | grep 'coredns_cache_evictions_total'

# 4. Confirm the latency and upstream-load correlation
curl -s http://localhost:9153/metrics | grep 'coredns_dns_request_duration_seconds'
curl -s http://localhost:9153/metrics | grep 'coredns_proxy_request_duration_seconds'

# 5. Check for stale serves (if serve_stale is configured, this can mask upstream failure)
curl -s http://localhost:9153/metrics | grep 'coredns_cache_served_stale_total'

Note on check 4: coredns_proxy_* metrics come from the legacy proxy plugin; deployments using the forward plugin expose the same data as coredns_forward_request_duration_seconds. Grep for both.

# 6. Check the TTL an upstream actually returns for a hot name
dig @<upstream_ip> example.com +noall +answer +time=2 +tries=1
# Read the TTL column in the answer section. 0 means the response bypasses the cache.

# 7. Kubernetes: correlate with restarts
kubectl get pods -n kube-system -l k8s-app=kube-dns
# Restart count > 0 recently explains a cold cache. Check events for OOMKilled.

If you run Prometheus, the ratio over a window:

sum(rate(coredns_cache_hits_total[5m])) by (type)
  / sum(rate(coredns_cache_requests_total[5m]))

Always compute the ratio from rates, not raw counters, and split by type before drawing conclusions.

How to diagnose it

1. Split the drop by type. Graph hit ratio separately for type="success" and type="denial". A success-hit drop with stable denial hits points to TTL=0 answers, an undersized success cache, or unique-name traffic on real names. A denial-hit drop points to churn in negative answers, often from search-domain expansion on new workloads.

2. Rule out the cold cache. Check pod restart counts and reload events. If the drop is temporally correlated with a restart or Corefile reload and is recovering on its own, this is expected warmup, not an incident. Do not “fix” it by restarting again.

3. Check eviction pressure. If coredns_cache_evictions_total is incrementing at a sustained rate and coredns_cache_entries sits at the configured maximum, the working set exceeds the cache. Default capacity is 9984 entries per cache (success and denial are configured separately), divided across 256 shards. Under capacity pressure, entries are evicted before TTL expiry, which shows up directly as a falling hit ratio.

4. Check for TTL=0 responses. Pick the names that dominate your forwarded traffic and dig them against the upstream. TTL=0 answers are never effectively cached: they occupy cache space until evicted but can never produce a hit. This is a common anti-pattern from upstreams that deliberately defeat caching.

5. Look for unique-name traffic. If QPS is up, evictions are up, but no single name dominates, you likely have randomized query names. CoreDNS exposes no per-client Prometheus metrics, so this requires the log plugin (enable it temporarily; it is expensive at high QPS) or dnstap. Look for high-entropy subdomains or a single source generating many distinct NXDOMAINs. That pattern is either reconnaissance, DGA malware on a compromised pod, or a misbehaving application with per-request unique names (timestamps in hostnames are a classic).

6. Check the upstream side of the correlation. If per-upstream latency (coredns_proxy_request_duration_seconds by to, or the forward-plugin equivalent) rose at the same time as the ratio fell, the upstream may be the initiator, not the victim: a slow upstream inflates miss latency, clients retry, retry amplification increases miss volume, and the ratio drops further. See CoreDNS slow upstream: per-upstream latency, goroutine pileup, and the to label.

7. Watch for cache-poisoning-adjacent noise. coredns_cache_drops_total counts responses excluded from the cache because the response question name did not match the request. A rising drop rate suppresses hits and can indicate mismatched or malformed responses from upstream.

One known wrinkle: there is a long-standing upstream issue where an expired denial (negative) cache entry can shadow a success entry for a name that transitioned from NXDOMAIN to existing, causing repeated backend lookups for a name that should hit the positive cache. If your success hit ratio is inexplicably low for specific names that recently started existing, check whether this affects your version.

Metrics and signals to monitor

Fixes

Unique-name traffic

There is no cache configuration that fixes names that never repeat. The fix is at the source: identify the client from query logs, then either fix the application (remove per-request unique names) or contain it (NetworkPolicy egress rules, or removing a compromised workload). If the traffic is legitimate but inherently unique (per-tenant subdomains, for example), accept the lower ratio and size upstream capacity for it rather than fighting the cache.

TTL=0 responses

The TTL comes from the authoritative server via your upstream. If you control the zone, fix the TTL at the source. If you do not, your options are limited: the cache plugin stores what it is given, and TTL=0 entries never hit. Weigh whether the affected names are hot enough to matter before engineering around an upstream you do not control.

Cache too small

Increase the cache size in the Corefile cache directive. Success and denial caches are configured separately; size them independently based on which type is evicting. The tradeoff is heap memory: cache entries are a major heap consumer, so raise the container memory limit alongside the cache and watch go_memstats_heap_inuse_bytes after the change. If evictions stop and the ratio recovers, the sizing was the cause.

Cold cache after restart

Usually nothing to fix; the ratio recovers as the cache warms. The fix is for the next rollout: stagger restarts so the whole fleet is never cold at once, use maxUnavailable=1 or a PodDisruptionBudget, and do not reload the Corefile during peak traffic if you can avoid it, since a reload flushes the cache.

Upstream-initiated cascade

If the upstream degraded first, fix the upstream path (replace a slow upstream, investigate rate limiting) before touching the cache. The ratio will recover on its own once miss latency normalizes. Do not restart CoreDNS during this: a cold cache on top of a struggling upstream is the worst combination.

Prevention

• Baseline the ratio by type. Alert on a > 20% drop from the rolling 24-hour baseline, not an absolute threshold, and suppress the alert for a few minutes after restarts.
• Alert on evictions at baseline traffic. Sustained coredns_cache_evictions_total at normal load means the cache is undersized before it becomes an incident.
• Stagger rollouts. Never restart all CoreDNS replicas simultaneously. Cold-cache thundering herds are self-inflicted.
• Correlate with deployments. A new workload that queries many unique names is the most common cause of an unexplained ratio drop. Deployment events should be on the same timeline as the ratio graph.
• Watch upstream health independently. A warm cache masks upstream failure until TTLs expire; per-upstream health checks and latency catch it earlier. See CoreDNS per-upstream health check failures.

How Netdata helps

• Netdata charts the cache hit ratio derived from coredns_cache_hits_total and coredns_cache_requests_total per second, split by type, so success and denial drops are visible separately without writing PromQL.
• Cache entries and evictions appear next to the ratio, making the “cache at capacity, evicting, ratio falling” pattern readable on one screen.
• Request latency and per-upstream latency (by to) sit alongside cache metrics, so the latency-and-upstream-load correlation this symptom is named for can be confirmed in seconds.
• Go runtime metrics (goroutines, heap, GC) are collected from the same endpoint, letting you see whether a miss flood is accumulating blocked queries or pushing the pod toward its memory limit.
• Anomaly detection on the ratio flags deviation from the learned baseline, which matches how this alert should work: deviation, not absolute thresholds.

Related guides

• How CoreDNS actually works in production: the plugin chain mental model
• CoreDNS slow upstream: per-upstream latency, goroutine pileup, and the to label
• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• CoreDNS per-upstream health check failures: degraded redundancy before total loss
• CoreDNS returning SERVFAIL: the resolver is failing queries and what to check first
• CoreDNS monitoring checklist: the signals every production resolver needs