CoreDNS cache evictions: the cache is too small for the working set

coredns_cache_evictions_total climbing steadily, or upstream query load and DNS latency creeping up without any change in traffic volume, both point at the same thing: the cache plugin is evicting entries before their TTL expires because it has run out of room, and queries that should have been sub-millisecond cache hits are being forwarded to upstream resolvers instead.

This is progressive degradation, not a cliff edge. Latency rises, upstream load rises, and the hit ratio slides, slowly enough that it often goes unnoticed until someone asks why DNS got slower this quarter.

Two distinct situations produce this metric, and the fix differs. Sustained evictions at normal baseline traffic mean the cache is undersized for the working set: raise the cache size. A sudden spike in evictions with a matching spike in unique query names means something is flooding the cache with one-off names (random subdomains, DGA-style traffic): that is a traffic problem, and enlarging the cache only delays it.

What this means

The cache plugin keeps two independent in-memory caches: a success cache for positive answers and a denial cache for NXDOMAIN/NODATA responses (SERVFAIL is also cached, for 5 seconds by default). Each has its own configured capacity. The default is 9984 entries per cache, divided across 256 shards, roughly 39 entries per shard.

Every entry carries a TTL from the upstream response. Eviction before TTL expiry only happens when capacity is hit: a new entry needs a slot and something has to go. Because capacity is enforced per shard and shards do not fill perfectly evenly, evictions can begin before the total entry count reaches the nominal maximum.

When that happens continuously, entries with useful remaining lifetime are discarded, the next query for that name misses, and CoreDNS forwards it upstream. The cost shows up in three places: higher average and tail latency on coredns_dns_request_duration_seconds, more load on upstream resolvers, and a declining hit ratio. In Kubernetes it also means more conntrack entries and more upstream DNS traffic than the cluster needs.

flowchart TD
  A[coredns_cache_evictions_total rising] --> B{Pattern?}
  B -->|Sustained at baseline traffic| C[Undersized cache]
  B -->|Sudden spike| D{Query names unique per query?}
  D -->|Yes, random subdomains| E[Flood or DGA traffic - fix the source]
  D -->|No, real names| F[Working set grew - cache undersized]
  C --> G[Confirm: entries at max, hit ratio declining]
  F --> G
  G --> H[Raise cache size in Corefile]

Common causes

Quick checks

All of these are read-only.

# Eviction counters, split by success vs denial cache
curl -s http://localhost:9153/metrics | grep 'coredns_cache_evictions_total'

# Current occupancy per cache type: compare against configured capacity
curl -s http://localhost:9153/metrics | grep '^coredns_cache_entries'

# Raw inputs for hit ratio (hits / requests; misses_total is deprecated)
curl -s http://localhost:9153/metrics | grep -E 'coredns_cache_(hits|requests)_total'

# Query rate by zone and type: is traffic actually up, or just misses?
curl -s http://localhost:9153/metrics | grep '^coredns_dns_requests_total'

If you run Prometheus, the useful derived views are:

# Eviction rate per second, by cache type
sum by (type) (rate(coredns_cache_evictions_total[5m]))

# Cache hit ratio
sum(rate(coredns_cache_hits_total[5m])) / sum(rate(coredns_cache_requests_total[5m]))

# Evictions as a fraction of cache requests: sustained above ~10% is a strong undersize signal
sum(rate(coredns_cache_evictions_total[5m])) / sum(rate(coredns_cache_requests_total[5m]))

How to diagnose it

1. Establish the eviction pattern. Is rate(coredns_cache_evictions_total[5m]) a flat sustained line at baseline traffic, or a spike that arrived with a traffic event? Sustained baseline evictions mean undersized. A spike means something changed.

2. Split by cache type. If type="denial" dominates and you are in Kubernetes, the denial cache is churning on search-domain-expansion NXDOMAINs. That cache is sized independently and is commonly left at default while operators tune only the success cache.

3. Check occupancy. Compare coredns_cache_entries{type="success"} and {type="denial"} against the configured capacities. Entries pinned at or near the maximum while evictions run confirms capacity pressure. Entries well below capacity with a low hit ratio points away from sizing and toward TTL=0 responses or one-off query names.

4. Confirm the hit-ratio impact. A falling hit ratio alongside rising evictions is the smoking gun: evicted entries are being re-fetched. If evictions are rising but the hit ratio is flat, the evicted entries were not being reused anyway (flood traffic), and resizing will not help.

5. Rule out a flood. Look at the query-name stream (the log plugin if enabled, or dnstap). Hundreds of unique subdomains under one parent, or high-entropy labels, indicate flooding or DGA-style traffic, not a capacity problem. CoreDNS exposes no per-source-IP Prometheus metrics, so source attribution requires logs.

6. Correlate with deployments. If evictions began right after a rollout, a new workload expanded the working set (new external endpoints, new service naming patterns). The cache did not get smaller; the job got bigger.

Metrics and signals to monitor

Fixes

Raise the cache size

The direct remedy for an undersized cache. The cache directive in the Corefile takes a capacity, and success and denial are sized independently, so you can grow only the cache that is evicting:

.:53 {
    cache 30 {
        success 50000
        denial 25000
    }
    forward . /etc/resolv.conf
}

Notes before you pick a number:

• The capacity is rounded down to the nearest multiple of 256 so all shards are equal, and the minimum allowed cache size is 1024.
• Size from observed data: unique names queried per TTL period, times a safety factor, per cache type. Watch coredns_cache_entries after the change; if occupancy settles well below the new maximum and evictions stop, you sized it right.
• Memory is the tradeoff. Every entry consumes heap, and CoreDNS memory already carries the Kubernetes plugin snapshot and goroutine overhead. Check RSS headroom against the container limit first, and keep at least 50% headroom over steady-state RSS so a restart re-list or a traffic burst does not OOM-kill the pod. A bigger cache that gets you OOMKilled is strictly worse than a small one.

Changing the Corefile triggers a reload (the reload plugin polls the Corefile roughly every 30 seconds), and a reload flushes the cache. Expect a brief cold-cache latency bump and an upstream traffic spike right after the change. Watch coredns_reload_failed_total to confirm the new config actually loaded.

Reduce TTL pressure for low-value entries

If only part of the working set is displacing valuable entries, reduce TTLs for the less-important entries so they age out naturally instead of forcing evictions. This is a narrow fix and only makes sense when you can identify a class of entries (for example, long-TTL records queried once) that is crowding out hot names.

Fix the flood at the source

If diagnosis pointed at random-subdomain traffic, resizing is the wrong tool. Identify the source via query logs, then:

• For a misbehaving internal client, fix or rate-limit that workload.
• For NXDOMAIN-heavy search-domain amplification, reduce ndots for workloads that primarily resolve external names so they stop generating throwaway names that churn the denial cache.
• If it is genuinely hostile traffic, treat it as a security investigation, not a capacity one.

Prevention

• Alert on sustained evictions, not on existence. Occasional evictions during traffic spikes are normal. Alert when the eviction rate stays positive for more than 10 minutes at baseline traffic, or when evictions exceed roughly 10% of cache requests.
• Trend occupancy, not just evictions. coredns_cache_entries approaching capacity is the leading indicator; evictions are the lagging one. Capacity-plan from the occupancy curve the same way you would for disk.
• Size the denial cache deliberately in Kubernetes. Search-domain expansion guarantees heavy negative-cache churn. Do not leave it at default while tuning only the success cache.
• Correlate with deployment events. Most working-set growth arrives with a rollout. Eviction and hit-ratio trends annotated with deploys turn a slow mystery into a one-line diagnosis.
• Recheck after every deliberate resize. Occupancy settling below the new maximum with evictions at zero is the confirmation. If occupancy climbs to the new maximum too, the working set is still growing and you are buying time, not solving it.

How Netdata helps

• Netdata charts coredns_cache_evictions_total split by the type label out of the box, so you can see immediately whether the success or denial cache is the one under pressure.
• Cache entries, hits, and requests are collected together, which puts occupancy-versus-capacity and the hit ratio on the same screen as the eviction rate: the three-signal correlation this diagnosis depends on.
• Per-second granularity catches short eviction bursts from flood traffic that 30- or 60-second scrape intervals average away.
• Correlating cache metrics with request latency and upstream forward latency on one dashboard shows the cost of the evictions directly, which is what justifies the resize.
• Memory metrics from the same agent let you check heap and RSS headroom before raising the cache size, so the fix does not become an OOM kill.

Related guides

• CoreDNS monitoring checklist: the signals every production resolver needs
• CoreDNS monitoring maturity model: from survival to expert
• How CoreDNS actually works in production: the plugin chain mental model
• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• CoreDNS returning SERVFAIL: the resolver is failing queries and what to check first
• CoreDNS NXDOMAIN vs SERVFAIL: why alerting on the wrong one buries real incidents
• CoreDNS NOERROR with zero answers: the resolution failure that reports success
• CoreDNS not resolving external domains: the missing catch-all forward zone
• CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed
• CoreDNS per-upstream health check failures: degraded redundancy before total loss
• CoreDNS query rate dropped to zero while the process looks healthy
• CoreDNS returning REFUSED: no matching zone, an ACL, or the forward concurrency limit