Consul stale DNS queries: the agent is answering from cache

When consul.dns.stale_queries climbs, the Consul agent is doing what it was designed to do: keep answering DNS from local state when it cannot get a fresh answer from a server. That behavior has been the DNS default since Consul 0.7. But “stale” means two very different things depending on how stale, and why.

A stale answer two seconds old during a leader handoff is harmless availability. A stale answer five minutes old, still pointing at instances that crashed four minutes ago, is silently routing traffic to dead services. The counter alone cannot tell you which case you are in. Correlate it with agent-to-server RPC health, server health, and your consistency configuration.

This guide covers what consul.dns.stale_queries measures, the three consistency modes, the configuration knobs that control staleness, and how to tell whether a rising counter is benign or actively dangerous.

What this means

The consul.dns.stale_queries counter increments each time the agent serves a DNS response whose server LastContact is more than 5 seconds old. The threshold is the hard-coded staleCounterThreshold = 5 * time.Second; there is no knob to change it.

Staleness enters the DNS path through two distinct mechanisms with different operational meaning.

Server-side stale reads. With allow_stale = true (the default since Consul 0.7), DNS queries forwarded to a server can be answered from that server’s local FSM state, which may lag the leader. This is normal read scaling. A follower answering slightly behind the leader is the intended behavior that lets Consul distribute reads across all servers instead of pinning every read to the leader. A steady low rate of stale queries under healthy servers is this mechanism working as designed.

Agent-side cache serving. With use_cache = true, the agent caches DNS results locally and can serve them without contacting any server at all. When the agent cannot reach a server (RPC port 8300 blocked, server overloaded, network partition), it falls back to its local cache. This is the connectivity signal that matters: the agent is alive and answering DNS, but it is disconnected from the source of truth. A sudden spike that coincides with consul.client.rpc.failed rising on the same agent means the agent has lost its server connection and is serving whatever it last knew.

The counter does not distinguish between the two. Interpretation requires correlation.

flowchart TD
  Q[Client DNS query] --> A[Agent port 8600]
  A --> C{use_cache
and cache fresh?}
  C -->|yes| CACHE[Serve from local cache]
  C -->|no| R{Server reachable
on port 8300?}
  R -->|no| STALE[Serve stale from cache
stale_queries++]
  R -->|yes| F{allow_stale true
and within max_stale?}
  F -->|yes| ANS[Follower answers
from local FSM]
  F -->|no| LEADER[Forward to leader
default mode]

Consistency modes: stale, default, consistent

Consul exposes three consistency modes. DNS can use stale or default. The consistent mode is HTTP API only.

The trade-off is availability against freshness. Stale mode keeps DNS answering through leader elections, server failures, and brief partitions. Default mode sacrifices that availability for answers closer to current. Consistent mode adds a quorum round-trip on every read and is too expensive for high-volume DNS paths; it exists for HTTP operations that must be linearizable, such as lock acquisition reads.

For DNS, max_stale caps how stale a stale-mode answer can be before the server refuses and forwards to the leader. Its default is 87600h (10 years) since Consul 0.7.1, which effectively means unbounded. Setting max_stale low sounds like a safe way to bound staleness, but it converts cheap follower reads into expensive leader reads. If servers are already overloaded, a low max_stale makes the overload worse. The official guidance is explicit: only set max_stale low if you prefer total unavailability over stale results.

Common causes

Quick checks

Run these read-only checks on the agent showing the elevated counter.

# Stale query counter (run twice, 30s apart, to compute rate)
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -i stale_queries

# Agent-to-server RPC failures (the real connectivity signal)
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -i "client.rpc"

# Verify a leader exists
curl -s http://127.0.0.1:8500/v1/status/leader

# How many servers this agent knows about
curl -s http://127.0.0.1:8500/v1/agent/self | grep -A2 known_servers

# Time a direct DNS query against the local agent
dig @127.0.0.1 -p 8600 myservice.service.consul SRV +stats

# Inspect the dns_config block in the agent config
grep -E "allow_stale|max_stale|use_cache|cache_max_age|node_ttl|service_ttl" /etc/consul/*.hcl /etc/consul.d/*.hcl 2>/dev/null

If consul.client.rpc.failed is zero and the leader endpoint returns a stable address, the stale counter is most likely server-side stale reads under allow_stale = true, which is benign. If RPC failures are climbing, the agent is disconnected and the staleness is the symptom of a connectivity problem.

How to diagnose it

1. Confirm the counter is actually rising. Pull the counter twice, 30 seconds apart, and compute the rate. A monotonically increasing counter with zero rate is just historical accumulation.

2. Check agent-to-server RPC health. consul.client.rpc.failed should be near zero. Any sustained non-zero rate means the agent cannot push state to servers or pull fresh reads. This is the single most important correlation. If RPC failures and stale queries rise together, you have a connectivity problem, not a tuning problem.

3. Verify server health. Check /v1/status/leader from multiple servers. If the response is empty or oscillating, the server cluster has no stable leader and all reads are necessarily stale. Check consul.raft.commitTime for write pipeline saturation and server file descriptor usage for connection exhaustion.

4. Distinguish cache-driven staleness from server-driven staleness. If use_cache = true and cache_max_age is set, the agent serves from its local cache until the cache entry expires, then re-fetches. If the agent cannot re-fetch because the server is unreachable, it keeps serving the cached value and the counter climbs. If use_cache is not enabled, the staleness is server-side: a follower is answering behind the leader under allow_stale = true.

5. Check for the cache_max_age = 0s trap. A value of 0s does not mean “always fresh.” It disables max-age entirely, so the agent never re-fetches based on age. The workaround documented in Consul issue #7073 is to use 1ns instead. If you intended near-immediate refresh and set 0s, you have accidentally configured permanent caching.

6. Inspect downstream resolvers. If clients query through dnsmasq, systemd-resolved, or corporate DNS forwarders, those layers add their own caching and negative-response caching. Windows caches negative DNS responses for 15 minutes by default. A service that appears “down” to clients may be down only in a downstream cache, not in Consul itself.

Metrics and signals to monitor

Fixes

If the agent cannot reach a server

This is the case that matters most. Stale DNS is the symptom; broken agent-to-server connectivity is the cause.

• Verify port 8300 connectivity from the agent to each server. A common pattern is a firewall or security group change that blocks RPC (8300) while leaving gossip (8301) open. Gossip looks healthy, but RPC is dead.
• Check server file descriptor usage. Servers near their FD limit stop accepting new RPC connections. The agent cannot reach the server even though the server process is running.
• Check for TLS mismatch. If certificates were rotated on servers but not on agents, RPC handshakes fail. Look for TLS errors in both agent and server logs.
• Check server overload. If consul.raft.commitTime is elevated, servers may be too slow to handle RPC read requests, causing the agent to time out and fall back to cache.

Tuning max_stale

The default max_stale = 87600h is effectively unbounded. That is intentional: it prioritizes availability. Do not set max_stale low to “bound staleness” unless you have explicitly decided that stale answers are worse than no answers. A low max_stale under server load converts follower reads into leader reads and accelerates overload. If you need bounded staleness, the safer approach is use_cache with an explicit, non-zero cache_max_age, accepting that the agent will serve from cache when it cannot refresh in time.

use_cache and cache_max_age

use_cache enables agent-side DNS caching and implies allow_stale. With it on, the agent can serve DNS without contacting servers at all, which reduces server load and DNS latency but introduces a freshness gap controlled by cache_max_age.

• Never use cache_max_age = 0s. It disables max-age entirely instead of meaning “always fresh.” Use 1ns if you want near-immediate re-fetch.
• There is no stale-if-error for the DNS cache. The HTTP API agent cache supports stale-if-error semantics (return stale when servers are unreachable), but the DNS cache does not. If you need that behavior, put a caching resolver in front of Consul DNS, such as Unbound configured with serve-expired.

Downstream caching and TTLs

Consul’s default TTL for all DNS records is 0s, meaning clients do not cache and every lookup hits the agent. If you want clients to cache, set node_ttl and service_ttl explicitly. Higher TTLs increase the window in which clients hold stale results during service failures.

If clients query through systemd-resolved, on systemd v246+ you must specify the port explicitly (DNS=127.0.0.1:8600). Without the port, queries go to port 53 instead of 8600. dnsmasq and corporate forwarders add their own TTL and negative-caching behavior that can extend apparent staleness beyond what Consul reports.

Prevention

• Monitor consul.client.rpc.failed on every agent. This is the leading indicator that stale DNS is connectivity-driven rather than load-distribution-driven. A sustained non-zero rate means the agent is disconnected and the catalog is going stale.
• Decide your staleness policy explicitly. Document whether your deployment treats stale DNS as acceptable availability or as a correctness risk. That decision drives your allow_stale, max_stale, use_cache, and cache_max_age settings.
• Track cache_max_age configuration across the fleet. The 0s trap is easy to hit during configuration refactors and produces no error, only silently stale answers.
• Watch server FD usage and consul.raft.commitTime. Both are upstream causes of agents falling back to stale cache. Catching them early prevents the stale counter from spiking.
• Validate downstream resolver behavior. If you run dnsmasq, systemd-resolved, or Unbound in front of Consul, understand their caching and negative-response behavior independently of Consul’s own metrics.

How Netdata helps

• Per-second consul.dns.stale_queries rate shows the exact moment the counter starts climbing, not a minute-later aggregate. Correlate that timestamp with other signals to identify the trigger.
• consul.client.rpc.failed alongside the stale counter on the same timeline is the fastest way to distinguish cache-driven staleness (connectivity problem) from server-side stale reads (normal behavior under allow_stale = true).
• Server-side signals (consul.raft.commitTime, leader transitions, file descriptor usage, memory) let you check whether the stale DNS is caused by server overload or quorum loss without switching tools.
• ML anomaly detection on the stale query rate separates the expected low background of stale reads from a genuine spike, reducing alert noise on a signal that is non-zero by design.
• DNS query latency metrics show the user-facing impact. If stale DNS is keeping latency low while servers are unreachable, that is the feature working as intended. If latency is climbing alongside staleness, the agent itself is struggling.

Related guides

• Consul catalog bloat: too many services and checks slowing everything down
• Consul registration storm: catalog churn overwhelming Raft
• Consul anti-entropy not syncing: local agent state and the catalog drifting apart
• Consul client rpc failed: agents alive but the catalog is going stale
• Consul DeregisterCriticalServiceAfter: instances vanishing from the catalog
• Consul DNS SERVFAIL: service discovery is broken for your applications
• Consul on EBS: burst-credit exhaustion and the sudden latency cliff
• Consul gossip encryption key mismatch: a botched keyring rotation splits the pool
• Consul gossip flapping: nodes oscillating between alive, suspect, and failed
• Consul serf queue backlog: an agent falling behind on gossip
• Consul gossip storm after mass recovery: rejoin floods and anti-entropy spikes
• Consul health check flapping: the passing/critical oscillation that churns the catalog