Consul DNS SERVFAIL: service discovery is broken for your applications

SERVFAIL from Consul’s DNS interface on port 8600 means service discovery is broken for every application resolving names through it. Applications see connection refused, timeouts, and cascading failovers. The symptom is broad, but the root cause lives along one of four layers: the downstream resolver (if present), the Consul agent’s built-in DNS server, the agent-to-server RPC channel, or the server’s catalog state.

NXDOMAIN is a related but distinct signal. SERVFAIL means the resolver could not complete the query: server overload, timeout, unreachable upstream, or internal error. NXDOMAIN means the query completed but the name does not exist in the catalog. Both break discovery, but they point to different layers. Conflating them during an incident wastes time.

The diagnostic path matters because Consul DNS is frequently chained behind dnsmasq, systemd-resolved, or corporate DNS forwarders. These layers cache both successes and failures. A SERVFAIL that lasts 30 seconds at the Consul agent can persist for minutes at the downstream resolver, which may keep serving the cached failure long after Consul itself has recovered.

What this means

Consul’s DNS server listens on UDP and TCP port 8600 by default and answers queries for the consul. domain. Each query for <service>.service.consul triggers an internal catalog lookup that flows from the agent to a server over RPC, then into the replicated state store. If any link fails or is too slow, the agent returns SERVFAIL or serves a stale cached result.

The four failure layers and their symptoms:

1. Downstream resolver (dnsmasq, systemd-resolved, corporate DNS): caches responses, including failures. Amplifies a brief Consul blip into a sustained outage.
2. Consul agent DNS server: serves queries from local cache or forwards to servers. If it cannot reach any server, returns SERVFAIL or serves stale data depending on configuration.
3. Agent-to-server RPC: the channel for catalog lookups. If blocked, timed out, or rate-limited, DNS queries fail at the agent.
4. Server catalog and state store: the authoritative source. If the catalog is inconsistent, ACLs deny the query, or the scan is too slow due to cardinality, the server returns empty results or times out.

flowchart TD
    App[Application query] --> Resolver[Downstream resolver
dnsmasq / systemd-resolved]
    Resolver --> Agent[Consul agent DNS :8600]
    Agent --> Server[Consul server RPC]
    Server --> Catalog[Catalog / state store]
    Agent -.->|server unreachable| ServfailA[SERVFAIL or stale]
    Server -.->|overloaded or slow| ServfailB[SERVFAIL or timeout]
    Catalog -.->|empty result or ACL deny| Nxdomain[NXDOMAIN]
    Resolver -.->|caches SERVFAIL| CachedFail[persistent failure]

The key first distinction: if dig directly against 127.0.0.1 -p 8600 succeeds but the application still fails, the problem is in the downstream resolver or its caching layer, not in Consul itself.

Common causes

Quick checks

Run these on the node where the application is failing. All are read-only and safe.

# Direct DNS query to the Consul agent, bypassing downstream resolvers
dig @127.0.0.1 -p 8600 myservice.service.consul SRV +stats +time=2 +tries=1

# Check if the agent knows about any servers
consul info | grep -A5 "known_servers"

# Verify leader exists and is reachable
curl -s http://127.0.0.1:8500/v1/status/leader

# Check DNS-specific telemetry counters
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -E "dns\.|stale_queries"

# Check if the service exists in the catalog at all
curl -s "http://127.0.0.1:8500/v1/catalog/service/myservice" | jq length

# Check healthy instances specifically
curl -s "http://127.0.0.1:8500/v1/health/service/myservice?passing=true" | jq length

# Verify agent-to-server RPC health
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep "client.rpc"

# If using dnsmasq, check if it is forwarding correctly
dig @127.0.0.1 -p 53 myservice.service.consul SRV +time=2

# If using systemd-resolved, check its cache and forwarding
resolvectl statistics
resolvectl query myservice.service.consul

If the direct query to port 8600 succeeds but the downstream resolver query fails, the problem is in the caching or forwarding layer. If the direct query fails, the problem is in Consul.

How to diagnose it

1. Bypass downstream resolvers first. Always start with dig @127.0.0.1 -p 8600. This isolates Consul from dnsmasq, systemd-resolved, and corporate DNS infrastructure. If this works, skip to step 7.

2. Check the response code. SERVFAIL (rcode 2) means the query could not complete. NXDOMAIN (rcode 3) means the name does not exist. The status: line in dig output shows the rcode.

3. Verify server reachability from the agent. Run consul info and check known_servers. If it shows zero or the agent cannot reach the leader, the agent returns SERVFAIL or serves stale results. Check consul.client.rpc.failed for sustained non-zero rates.

4. Check leader existence and Raft health. Query curl http://127.0.0.1:8500/v1/status/leader. An empty string means no leader, which means no catalog lookups can complete. Check consul.raft.commitTime for sustained elevation above 100ms.

5. Verify the service exists in the catalog. Query curl http://127.0.0.1:8500/v1/catalog/service/<name>. If the catalog returns empty but the service is registered locally on the agent, anti-entropy sync is failing. Check the agent log for anti_entropy errors; current Consul telemetry has no dedicated anti-entropy failure-rate metric.

6. Check ACL resolution if ACLs are enabled. DNS queries for <service>.service.consul require the agent’s token to have service:read permission on the target service. A token without permission produces NXDOMAIN, not an explicit denial. Test with a management token to confirm.

7. If direct Consul DNS works, test the downstream resolver. Query through dnsmasq or systemd-resolved. If it returns SERVFAIL while direct Consul DNS works, the resolver is caching a stale failure or has a forwarding misconfiguration.

8. Check for recursor loops. If Consul has recursors configured that point back to a resolver that forwards to Consul, a DS consul. query can loop. dnsmasq’s concurrent query limit fills up, causing SERVFAIL for all requests. The documented --dns-forward-max default is 150.

9. Check consul.dns.stale_queries. This counter tracks queries served stale by more than 5 seconds. A rising value means the agent is serving cached results because it cannot reach servers fast enough.

Metrics and signals to monitor

Fixes

Agent cannot reach any server

The most common cause of persistent SERVFAIL. The agent is alive and serving DNS, but has no server to query.

• Verify network connectivity on port 8300 (RPC) between the agent and servers. A firewall change that blocks 8300 while leaving 8301 (gossip) open produces exactly this symptom: the agent appears healthy in gossip but cannot push or pull catalog state.
• Check server file descriptor usage. If servers are at their FD limit, they reject new RPC connections. The fix is raising the ulimit, not restarting Consul.
• If allow_stale is true (the default since Consul 0.7), the agent serves cached results instead of SERVFAIL when servers are unreachable. This masks the failure temporarily but produces increasingly stale data. Whether that is acceptable depends on how stale is tolerable for your consumers.

Server cluster overloaded

DNS queries require catalog lookups on servers. If the server cluster is saturated, DNS latency spikes and eventually times out.

• Check consul.raft.commitTime and disk I/O latency. Slow disks are the most common cause of Raft degradation.
• Check consul.runtime.num_goroutines and file descriptor usage for resource exhaustion.
• If the overload is from catalog churn (health check flapping, registration storms), the DNS symptom is secondary. Address the churn source first. See Consul registration storm.

Catalog inconsistency producing NXDOMAIN

The service is registered locally on the agent but not visible in the catalog. Anti-entropy sync is failing.

• Check the agent log for sustained anti_entropy errors and compare the agent’s registrations with the catalog.
• Verify the agent can reach servers on port 8300.
• If necessary, force a sync by restarting the agent. This is disruptive: the agent re-registers all services, causing a brief registration burst that adds load to the server cluster.

ACL denial producing NXDOMAIN

When ACLs are enabled, DNS queries require the agent’s token to have service:read permission on the target service. A token without permission returns NXDOMAIN, not an explicit error.

• Verify the agent token has a matching service_prefix or service ACL rule granting read permission. query_prefix governs prepared-query management, not ordinary service DNS lookup execution; even prepared-query execution still requires service read access (or uses the token captured when the query was created).
• Check ACL token resolution latency. High latency on every DNS query indicates ACL cache misses or an undersized token cache.

Downstream resolver caching SERVFAIL

systemd-resolved caches SERVFAIL responses for long periods. dnsmasq can enter a forwarding loop with Consul recursors.

• Flush caches: resolvectl flush-caches for systemd-resolved; restart dnsmasq if it is stuck in a loop.
• Current systemd resolved.conf documents address:port for DNS=; systemd issue #17858 records v245 rejecting this syntax. On older installations, verify the local behavior before relying on the port form.
• Consider setting use_cache and appropriate TTLs on the Consul agent to control caching behavior at the source.

Recursor SERVFAIL

If Consul has recursors configured, Consul 1.2.3 fixed the recursor path so a SERVFAIL or REFUSED response advances to the next recursor; releases before that could stop at the first failed recursor.

• Check dns_config.recursor_timeout (default 2s). If recursors are slow, queries time out and return SERVFAIL.
• Use dns_config.recursor_strategy = "random" to distribute load across recursors instead of always hitting the first one sequentially; this option was introduced in Consul 1.11.

Prepared query complexity

Prepared queries can add latency and interact with rate limits when the result set is large.

• Test the prepared query directly: curl -s "http://127.0.0.1:8500/v1/query/<name>/execute".
• If the query is slow, check the number of matching service instances. Large result sets increase execution time.

Prevention

• Monitor consul.dns.stale_queries and DNS latency. These are the earliest indicators that DNS resolution is degrading, before applications start failing.
• Monitor agent-to-server RPC health on every client agent. Sustained consul.client.rpc.failed is the leading indicator for catalog staleness, which manifests as NXDOMAIN or SERVFAIL.
• Track catalog size growth. Large catalogs slow DNS scans. Monitor total service instances and health check counts as capacity signals.
• Test DNS resolution as part of health checks. A synthetic query against a known service catches DNS degradation before real applications hit it.
• Document your resolver chain. If you run dnsmasq or systemd-resolved in front of Consul, ensure the team knows how to flush caches and diagnose forwarding loops during incidents.
• Verify ACL token permissions after policy changes. A token that loses read permission on a service produces NXDOMAIN silently with no error log.

How Netdata helps

• DNS latency and stale query tracking at per-second resolution. Netdata collects consul.dns.domain_query timers and consul.dns.stale_queries, making DNS degradation visible within seconds rather than relying on longer scrape intervals.
• Correlating DNS failures with Raft and RPC health. When DNS latency spikes, overlay consul.raft.commitTime, consul.client.rpc.failed, and RPC latency together to distinguish “agent cannot reach server” from “server is overloaded” from “catalog is stale.”
• Agent-to-server RPC failure detection. Netdata tracks consul.client.rpc.failed on every agent, surfacing the pattern where agents are alive in gossip but cannot push state to the catalog.
• Disk I/O monitoring on server volumes. Slow disks are the most common root cause of Raft degradation that cascades into DNS failures. Per-disk I/O latency metrics provide early warning before commit times approach the election timeout.

Related guides

• Consul catalog bloat: too many services and checks slowing everything down
• Consul registration storm: catalog churn overwhelming Raft
• Consul anti-entropy not syncing: local agent state and the catalog drifting apart
• Consul client rpc failed: agents alive but the catalog is going stale
• Consul on EBS: burst-credit exhaustion and the sudden latency cliff
• Consul gossip encryption key mismatch: a botched keyring rotation splits the pool
• Consul gossip flapping: nodes oscillating between alive, suspect, and failed
• Consul serf queue backlog: an agent falling behind on gossip
• Consul gossip storm after mass recovery: rejoin floods and anti-entropy spikes
• How Consul actually works in production: a mental model for operators
• Consul leader election storm: repeated elections and rolling write outages
• Consul monitoring checklist: the signals every production cluster needs