CoreDNS TTL=0 responses: the anti-pattern that silently bypasses the cache

Your cache hit ratio is sliding. Upstream query volume is climbing. P99 latency is drifting up with it. You open the Corefile and the cache plugin is right there, configured the way it has been for months. Nothing changed on your side, but the cache has effectively stopped working.

A common cause is records arriving with TTL=0. When an upstream resolver, or your own zone data, answers with a zero TTL, those answers are uncacheable by contract: every client query for that name has to be forwarded again. The cache plugin is present and correct, but there is nothing for it to hold on to. It looks like a cache failure but is actually a data problem arriving through the response path.

This is easy to miss because nothing errors. Queries succeed and responses are correct. The only symptoms are the slow-motion ones: hit ratio decay, upstream load multiplying, latency climbing in lockstep. This guide covers how to confirm TTL=0 is the cause, how to distinguish it from the other reasons a hit ratio collapses, and how to floor TTLs at the CoreDNS layer when you cannot fix the source.

What this means

The cache plugin keeps an in-memory LRU with separate positive (success) and negative (denial) caches. Whether an answer gets cached, and for how long, is governed by the TTL on the records in the response. A TTL of zero tells any caching layer “do not reuse this.” Every subsequent query for the same name is a cache miss and a fresh upstream round trip.

The blast radius depends on how popular the zero-TTL names are. One hot name with TTL=0 and thousands of clients behind it means the cache does nothing for your highest-volume traffic, while cold names with normal TTLs keep the aggregate hit ratio looking merely mediocre instead of zero. That masking effect is why this survives for weeks in some environments.

flowchart LR
  C[Client query] --> CE{In cache?}
  CE -->|yes| H[Serve from cache, sub-ms]
  CE -->|no| F[Forward to upstream]
  F --> R{Response TTL?}
  R -->|TTL > 0| S[Store in cache, serve]
  R -->|TTL = 0| N[Serve, do not retain]
  N --> C

One nuance before you start: the cache plugin applies a minimum TTL (MINTTL, default 5 seconds) to what it stores, so a zero-TTL answer is not always literally dropped. What those entries do in practice is churn: they expire within seconds, so only names queried faster than that interval get any hits, and everything else misses. The entries still occupy cache slots and add eviction pressure while contributing almost nothing. If an operator has explicitly overridden the minimum TTL down to 0, the bypass is total. Either way, the observable signature is the same: misses on names that should be hot.

Common causes

Quick checks

All of these are read-only. Run them from inside a CoreDNS pod, or wherever the metrics endpoint on port 9153 is reachable.

# Current cache hit/requests counters
curl -s http://localhost:9153/metrics | grep -E 'coredns_cache_(hits|requests)_total'

# Cache population and evictions
curl -s http://localhost:9153/metrics | grep -E 'coredns_cache_(entries|evictions_total)'

# Upstream latency, per upstream
curl -s http://localhost:9153/metrics | grep 'coredns_proxy_request_duration_seconds'

Then check the actual TTLs on the wire. Query through CoreDNS and directly against the upstream, and compare:

# TTL as CoreDNS returns it
dig @<coredns_ip> suspicious.name.example A +noall +answer

# TTL as the upstream returns it
dig @<upstream_ip> suspicious.name.example A +noall +answer

The TTL column is the second field of each answer record. If the upstream returns 0, you have found it. If the upstream returns a real TTL but CoreDNS returns 0, something in your plugin chain (rewrite rules, serve_stale) is rewriting it.

Also read the Corefile with fresh eyes:

# Kubernetes
kubectl get cm -n kube-system coredns -o yaml

# Standalone
cat /etc/coredns/Corefile

Look at the cache block for any minimum TTL override, and at the kubernetes block for a ttl directive.

How to diagnose it

1. Confirm the hit ratio drop is real and sustained. Compute coredns_cache_hits_total / coredns_cache_requests_total over a 30-minute window, not from raw counters. Rule out the cold-cache case first: if a CoreDNS pod restarted or the Corefile reloaded within the window, a low hit ratio is expected and self-correcting.

2. Check whether cache entries are churning. If coredns_cache_entries stays low or flat while request volume is high, entries are not sticking around. If entries are high but coredns_cache_evictions_total is climbing, the cache is full of something useless (short-lived entries being evicted as shards fill) and you have a sizing-plus-TTL problem, not just a sizing problem.

3. Identify which names are missing. CoreDNS does not expose per-name cache metrics, so this step is log-based. Enable the log plugin temporarily on one replica if traffic volume allows, sample the repeated queries, and pick the top recurring names. Do not leave full query logging on in a high-QPS production environment.

4. Read the TTLs for those names using the dig commands above, both through CoreDNS and directly against the upstream. Zero from the upstream means the source is the problem. Non-zero from the upstream but zero through CoreDNS means your plugin chain is the problem.

5. Audit the Corefile for TTL manipulation. Check the cache block for a minimum TTL set to 0, the kubernetes block for ttl 0, and any rewrite ttl rules that might be clamping downward. If serve_stale is configured, remember that stale responses intentionally carry TTL 0; that is expected behavior, not this bug.

6. Quantify the impact. Compare upstream latency and the forwarded query rate before and after the hit ratio drop. The increase in upstream queries attributable to the zero-TTL names is your remediation priority list.

Metrics and signals to monitor

Fixes

Fix the source if you can. Fix CoreDNS if you cannot.

Fix the authoritative source

The correct fix is at whatever is emitting TTL=0: the upstream’s authoritative zone, your own zone file ($TTL 0 or per-record zeros), or the backend driving the etcd/file/auto plugins. A zero TTL is almost never a deliberate, load-aware decision. It is usually a default someone forgot, or a “make changes instant” setting left behind from a migration. Raising it to even 30 to 60 seconds recovers most of the cache benefit for hot names.

Floor the TTL in the cache plugin

When the upstream is not yours to fix, the cache plugin’s minimum TTL forces a floor on what gets cached. With the default minimum of 5 seconds, zero-TTL answers are held briefly rather than never. If an explicit override in your Corefile set the minimum to 0, removing that override restores the floor. You can also raise the minimum for the success cache to something larger (for example 30 seconds) if the offending names change infrequently in practice.

The tradeoff: you are deliberately serving data staler than the authoritative source asked for. For records that genuinely change second-to-second (some load-balancer and failover setups), that staleness can be worse than the extra upstream load. Pick the floor based on the actual change rate of the records, not on how annoyed you are at the upstream.

Clamp TTLs with the rewrite plugin

The rewrite plugin can clamp response TTLs into a range, for example flooring anything below 30 seconds up to 30 (and optionally capping the top end too). This is more surgical than the cache minimum because it applies per-zone and changes what clients see, not just what the cache keeps. It carries the same staleness tradeoff, amplified, because clients and intermediate resolvers will also hold the record longer. Use it when you control the resolution path end to end and understand the change semantics of the names involved.

Check the kubernetes plugin TTL

If only cluster.local names are affected, look for ttl 0 in the kubernetes block. Setting the plugin TTL to 0 prevents its records from being cached and is rarely what you want in a busy cluster. Restore a small positive TTL unless you have a specific reason for per-query freshness on service names.

What not to do

Do not respond to a falling hit ratio by blindly enlarging the cache. If entries are zero-TTL, a bigger cache just holds more immediately-dead entries and the hit ratio does not move. Do not restart CoreDNS to “warm the cache” either; a restart guarantees a cold cache and makes the immediate problem worse.

Prevention

• Alert on hit ratio trend, not just restarts. A ratio drop of more than 20% from the rolling daily baseline, sustained over 5 minutes, catches TTL regressions, eviction storms, and traffic-shape changes with one rule. Suppress it for a few minutes after pod restarts and reloads.
• Watch evictions alongside entries. Rising evictions with stable entries means the cache is full of something. That is your early warning for both undersizing and worthless entries.
• Sample TTLs in change review. When a new upstream, zone, or backend data source is onboarded, dig a few of its names and look at the TTLs before it goes into the forwarding path. Zero-TTL sources should be a conscious decision, not a discovery.
• Keep the Corefile’s TTL knobs visible. Any explicit minimum TTL, rewrite ttl rule, or kubernetes plugin TTL should have a comment saying why it exists. These directives are invisible in metrics until they hurt.
• Correlate hit ratio with upstream load in dashboards. A hit ratio falling while forwarded QPS rises proportionally is the TTL=0 signature. A hit ratio falling with flat forwarded QPS is a traffic-shape change. The pair separates the two in seconds.

How Netdata helps

• Netdata charts coredns_cache_hits_total and coredns_cache_requests_total together, so the hit ratio decay from TTL=0 shows up as a divergence you can see at per-second resolution, not as a weekly report.
• Correlating the cache ratio against coredns_dns_request_duration_seconds on the same dashboard confirms the latency cost is miss-driven, not upstream-driven, which is the first branch of the diagnosis.
• Cache entries and eviction rate side by side expose the churn pattern: entries not accumulating, or evicting early, while requests stay high.
• Per-upstream latency lets you see whether the extra forwarded load is starting to degrade the upstream itself, which changes the urgency.
• Goroutine and Go runtime metrics catch the escalation case where miss-driven forwarding starts accumulating in-flight queries.
• Because Netdata keeps per-second history, you can line the hit ratio drop up against deploys, Corefile reloads, and upstream changes to find the moment the zero-TTL records appeared.

Related guides

• CoreDNS cache hit ratio dropping: latency and upstream load climbing together
• CoreDNS cache collapse: the cold-cache thundering herd after a rollout
• CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed
• CoreDNS NOERROR with zero answers: the resolution failure that reports success
• CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken
• CoreDNS per-upstream health check failures: degraded redundancy before total loss
• CoreDNS not resolving external domains: the missing catch-all forward zone
• CoreDNS NXDOMAIN vs SERVFAIL: why alerting on the wrong one buries real incidents
• CoreDNS query rate dropped to zero while the process looks healthy
• How CoreDNS actually works in production: the plugin chain mental model
• CoreDNS monitoring checklist: the signals every production resolver needs
• CoreDNS monitoring maturity model: from survival to expert