Consul DNS latency high: slow lookups stalling connections and failovers

Slow Consul DNS lookups rarely look like a DNS problem at first. Applications see connection timeouts, retries, and slow startup. Load balancers see health check flapping. Service mesh sidecars see upstream resolution failures. The DNS layer is invisible to most teams until consul.dns.domain_query crosses the page threshold.

The first decision in a Consul DNS latency incident is structural: is the slowness in the server cluster (Raft, disk, catalog size) or in the DNS path itself (TTL, agent CPU, query complexity, downstream resolvers)? The signals are all in agent telemetry, but you have to know which ones to correlate.

What this means

Consul DNS latency is measured by consul.dns.domain_query.node_query and consul.dns.domain_query.service_query timers, which break down query latency by type. The signal that maps directly to user impact is the p99 latency of these timers and the failure rate (SERVFAIL or NXDOMAIN responses). Cached results should resolve in single-digit milliseconds. Fresh lookups that hit the server cluster should resolve in the low tens of milliseconds. Sustained p99 above 500ms or a failure rate above 1% is page-worthy: service discovery is broken for every application that depends on it.

A query travels from the application, through a system resolver (glibc, systemd-resolved, dnsmasq, Unbound), to the Consul agent’s DNS listener on port 8600, and then optionally via RPC to a Consul server that scans the catalog. Latency can appear at any layer. TTL configuration shapes this path: with the default TTL of 0, every lookup requires an agent-to-server RPC. With non-zero TTLs, results are cached at clients and downstream resolvers, reducing load but delaying visibility into service changes.

Common causes

Quick checks

Confirmed: consul.rpc.request and consul.client.rpc.failed are counters, not latency timers. consul.api.http is the only request-level timer metric (ms). HTTP API timing below is a practical alternative for layer isolation.

# Time a direct DNS query against the agent listener
dig @127.0.0.1 -p 8600 myservice.service.consul SRV +stats

# Pull DNS query timers from agent telemetry
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep "consul.dns"

# Check whether the agent is serving stale answers
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep "consul.dns.stale_queries"

# Time an HTTP API call to compare server-side latency against DNS latency
time curl -s http://127.0.0.1:8500/v1/catalog/service/myservice > /dev/null

# Verify leader exists (empty response = no leader = writes blocked)
curl -s http://127.0.0.1:8500/v1/status/leader

# Check Raft commit time on the leader (write pipeline health)
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep "consul.raft.commitTime"

# Check agent-to-server RPC failures (catalog going stale)
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep "consul.client.rpc.failed"

# Disk I/O latency on the Raft data volume (most common server-side cause)
iostat -x 1 5

How to diagnose it

The fastest path to root cause is layer isolation: compare DNS latency against server-side latency. If both are elevated, the server cluster is the bottleneck. If only DNS is elevated, the problem is in the DNS path itself.

flowchart TD
    A[DNS p99 latency high
or failures > 1 percent] --> B{Is the HTTP API
also slow?}
    B -- Yes, all paths slow --> C[Server-side bottleneck]
    C --> C1[Raft commit time]
    C --> C2[Leader disk I/O latency]
    C --> C3[Catalog churn or bloat]
    B -- No, DNS only --> D[DNS path issue]
    D --> D1[TTL=0 forcing every
query to server]
    D --> D2[Agent CPU or GC pauses]
    D --> D3[Large result sets over UDP]
    D --> D4[Downstream resolver caching]

1. Isolate the layer. Pull consul.dns.domain_query.node_query and consul.dns.domain_query.service_query mean and p99 from the agent metrics endpoint. Then time a direct HTTP API call (time curl -s http://127.0.0.1:8500/v1/catalog/service/myservice). If both DNS and HTTP API are slow, jump to the server-side checks. If only DNS is slow, the issue is in the DNS path.

2. Check for stale answers. Elevated consul.dns.stale_queries means the agent is serving DNS from cache because it cannot reach any server within the staleness window. This is a connectivity problem, not a query-performance problem. Cross-reference with consul.client.rpc.failed.

3. Verify leader exists. An empty response from /v1/status/leader means no leader. All writes, including health updates that feed DNS, are blocked. DNS will continue to serve stale results until the catalog stops updating.

4. Check Raft commit time on the leader. consul.raft.commitTime should be well under 50ms. Anything approaching the heartbeat timeout risks leader elections. If commit time is high, check disk I/O next.

5. Check disk I/O on the leader’s Raft volume. Run iostat -x 1 5 on the volume holding data_dir/raft/. Write latency (await) sustained above 10ms is the most common root cause of cascading Raft and DNS latency. EBS gp2 burst credit exhaustion and shared volumes are typical culprits.

6. Check TTL configuration. The default TTL of 0 forces every DNS lookup into an agent-to-server RPC. With high query volume, this directly multiplies into server load and DNS latency. Inspect dns_config in your Consul agent configuration for node_ttl and service_ttl values.

7. Check agent CPU and GC pauses. DNS-only slowness on a single agent often points to CPU starvation or Go runtime GC pauses. Pull consul.runtime.gc_pause_ns and consul.runtime.num_goroutines and compare with baseline.

8. Check downstream resolver caching. If applications query through dnsmasq, systemd-resolved, or Unbound, those layers cache according to their own TTL logic and Consul’s advertised TTL. Negative caching (NXDOMAIN) can cause services to appear down for far longer than the actual outage. Compare direct queries to the agent (dig @127.0.0.1 -p 8600) against queries through the system resolver.

9. Check for large result sets over UDP. Services with many healthy instances can produce UDP responses that exceed the datagram size limit and require TCP retry. This adds a round-trip and visibly increases latency for those specific queries.

Metrics and signals to monitor

Fixes

If the server cluster is the bottleneck

Server-side DNS latency is downstream of Raft. The fixes target the Raft pipeline.

• Disk I/O is the most common cause. Move the Raft data directory to a dedicated SSD. Avoid EBS gp2 in favor of io1/io2 with provisioned IOPS, or gp3 with reserved throughput. Do not colocate Raft data with logs, application data, or other databases. See Consul on EBS: burst-credit exhaustion and the sudden latency cliff.
• Reduce catalog churn. Identify flapping health checks and runaway registration loops. Each transition is a Raft write that consumes commit pipeline capacity. See Consul health check flapping and Consul registration storm.
• Reduce catalog size. Track total service instances and check counts. Snapshots, DNS scans, and HTTP catalog endpoints all scale with catalog size. See Consul catalog bloat.
• Add server capacity. If commit time stays elevated after disk and churn are addressed, the cluster is undersized for the write load. Adding read replicas relieves read pressure but does not help the write pipeline.

If DNS-only latency is the problem

• Set non-zero TTLs. The default TTL of 0 forces every query to the server. Configure node_ttl and service_ttl to balance freshness against load. Higher TTLs reduce server load but delay visibility into service changes.
• Tune downstream resolvers. dnsmasq, systemd-resolved, and Unbound all cache according to their own TTL logic and Consul’s advertised TTL. Confirm that the downstream layer’s cache size and TTL are appropriate for your service churn rate. Negative caching (NXDOMAIN) can keep a service marked as down long after recovery; tune the SOA min TTL in your Consul DNS configuration if your version exposes it.
• Reduce per-query cost. Services with hundreds or thousands of healthy instances produce large responses that can truncate over UDP and force TCP retry. Reduce the number of records returned per query where possible, or accept the TCP overhead for those services.

If the agent cannot reach servers

• Check port 8300 connectivity. A common pattern is gossip (port 8301) working while RPC (port 8300) is blocked by a firewall or security group change. See Consul client rpc failed.
• Check FD usage on servers. Servers at the FD limit refuse new RPC connections while existing ones continue to be served.
• Check TLS configuration. A renewed server certificate that has not been distributed to agents causes silent RPC failures. Gossip may remain healthy while RPC fails. See Consul anti-entropy not syncing.

Prevention

• Monitor DNS p99 latency continuously. PAGE at 500ms sustained or 1% failure rate. TICKET when trending toward 100ms. Single-digit milliseconds is the cached baseline.
• Track TTL configuration in version control. A change to TTL=0 can quadruple server load without an obvious signal in any other metric.
• Track Raft commit time and disk await proactively. These are the leading indicators for the most common DNS latency root cause.
• Track consul.dns.stale_queries on every agent. Non-zero values mean the agent cannot reach servers and is serving cached state.
• Capacity plan around catalog size. Catalog scans for DNS scale with the number of service instances and checks. Track these counts weekly.
• Validate downstream resolver caching behavior. Periodically compare direct agent queries against system-resolver queries to confirm caching layers behave as expected.

How Netdata helps

• Per-second DNS latency collection on consul.dns.domain_query.* exposes p99 and mean latency without aggregation windows that hide transients during incidents.
• Cross-signal correlation lets you compare DNS latency against Raft commit time, RPC failures, and disk I/O on the same timeline, making layer isolation a visual check instead of three separate commands.
• ML-based anomaly detection on DNS latency, stale query rate, and RPC failure rate surfaces slow drift before it crosses static thresholds.
• Per-agent staleness tracking on consul.dns.stale_queries and consul.client.rpc.failed catches silent catalog drift before consumers notice.

Related guides

• Consul catalog bloat: too many services and checks slowing everything down
• Consul registration storm: catalog churn overwhelming Raft
• Consul anti-entropy not syncing: local agent state and the catalog drifting apart
• Consul client rpc failed: agents alive but the catalog is going stale
• Consul on EBS: burst-credit exhaustion and the sudden latency cliff
• Consul gossip encryption key mismatch: a botched keyring rotation splits the pool
• Consul gossip flapping: nodes oscillating between alive, suspect, and failed
• Consul serf queue backlog: an agent falling behind on gossip
• Consul gossip storm after mass recovery: rejoin floods and anti-entropy spikes
• Consul health check flapping: the passing/critical oscillation that churns the catalog
• How Consul actually works in production: a mental model for operators
• Consul leader election storm: repeated elections and rolling write outages