Consul cross-datacenter query failure: prepared-query failover masking a DC outage

A Consul prepared query with cross-DC failover does exactly what it was configured to do: it silently forwards lookups to a remote DC when local results are empty. Consumers receive valid service discovery results, applications keep connecting, and no error fires. The local DC may have zero healthy instances, and nobody on the consumer side knows.

The DNS response is identical whether nodes came from the local DC or a remote one. There is no EDNS flag, no source record, no indicator in the answer. Receiving results does not mean local health.

This article covers how to detect active failover, distinguish it from a genuine local outage, handle ACL replication lag in the failover target DC, and understand why Connect mesh traffic does not follow DNS prepared-query failover across datacenters.

What this means

Prepared queries are Consul’s mechanism for geo-failover service discovery. A query defines a service name plus an optional failover policy. When a consumer resolves the query, Consul first checks for healthy nodes in the local DC. If none exist, it applies the failover policy before returning an empty result.

Three failover mechanisms exist:

• NearestN: dynamic selection based on network coordinate RTT to the nearest N datacenters.
• Datacenters: a static, ordered list of DCs to try in sequence.
• Targets: for a sequential list of WAN-federated datacenters and cluster peers. Mutually exclusive with NearestN and Datacenters; this field was introduced in Consul 1.13.

The masking happens at the DNS layer. When a prepared query resolves via DNS on port 8600, the response contains A, AAAA, or SRV records with no field indicating which DC provided the nodes. An application that resolves my-service.query.consul and gets three IP addresses cannot tell whether those addresses are local or 200 milliseconds away in a remote DC.

The HTTP execute API exposes what DNS hides. The endpoint GET /v1/query/:uuid/execute returns two fields that reveal failover state:

• Datacenter: the DC that ultimately provided the nodes.
• Failovers: the count of remote DCs queried during resolution. Zero means results came from the local DC.

Most applications consume Consul via DNS, not the HTTP API. That is the core visibility gap. The failover is working as designed, but operators must actively poll the execute endpoint to know it is happening.

flowchart LR
  A[Consumer DNS query] --> B[Consul agent]
  B --> C{Local DC has
healthy nodes?}
  C -->|Yes| D[Return local nodes]
  C -->|No| E[Failover policy:
NearestN or Datacenters]
  E -->|WAN RPC| F[Remote DC server]
  F --> G[Remote healthy nodes]
  G --> H[DNS response:
no source indicator]
  H --> I[Consumer connects to
remote DC instance]

The “no” branch produces a valid response that looks identical to the “yes” branch from the consumer’s perspective. The only operators who can tell the difference are those polling the HTTP API or correlating DNS latency with health check state.

Common causes

Quick checks

Run these read-only checks to determine whether failover is active and why.

# Execute a prepared query via HTTP and inspect failover state
curl -s http://127.0.0.1:8500/v1/query/<query-uuid>/execute \
  | jq '{Datacenter, Failovers, NodeCount: (.Nodes | length)}'
# Failovers > 0 means results came from a remote DC

# Check local health for the service (are local instances actually dead?)
curl -s "http://127.0.0.1:8500/v1/health/service/<service>?passing=true" | jq length
# Zero means every local instance is unhealthy

# Check ACL replication status in secondary DC (run on a secondary DC server)
curl -s http://127.0.0.1:8500/v1/acl/replication | jq '{Enabled, Running, ReplicatedIndex, ReplicatedTokenIndex, LastSuccess, LastError, LastErrorMessage}'
# LastErrorMessage should be empty; compare ReplicatedTokenIndex with the X-Consul-Index from /v1/acl/tokens; a large gap indicates lag

# Check WAN gossip membership (is the remote DC reachable?)
curl -s "http://127.0.0.1:8500/v1/agent/members?wan=true" \
  | jq '[.[] | .Tags.dc] | group_by(.) | map({dc: .[0], servers: length})'
# Each federated DC should show its expected server count

# Time a DNS lookup for the prepared query name
dig @127.0.0.1 -p 8600 <query-name>.query.consul +stats | grep "Query time"
# Elevated latency suggests cross-DC resolution

# Check whether ACL filtering removed results from the query
curl -sI http://127.0.0.1:8500/v1/query/<query-uuid>/execute \
  | grep -i "X-Consul-Results-Filtered-By-ACLs"
# Header present and true means the query token lacks permissions for some nodes

How to diagnose it

1. Confirm failover is active. Execute the prepared query via the HTTP API. If Failovers is greater than zero, the query is forwarding to a remote DC. If Datacenter does not match the local DC name, results are coming from elsewhere.

2. Check whether local instances are genuinely unhealthy. Query /v1/health/service/<service>?passing=true for the service in the local DC. Zero results means every instance is failing health checks. The failover is masking a real local outage.

3. If local instances are healthy but failover is still active, check ACL filtering. The X-Consul-Results-Filtered-By-ACLs response header on the execute endpoint indicates whether the query’s token is filtering out nodes that should be visible. A misconfigured token can remove all local results, triggering failover to a remote DC unnecessarily.

4. If the failover target DC is returning errors, check ACL replication. When the query forwards to a remote DC, the query’s ACL token must be valid there. Run curl /v1/acl/replication on a server in the target DC. If ReplicatedTokenIndex lags behind the token index exposed by the target DC, LastErrorMessage is non-empty, or LastError is later than LastSuccess, the token or its policies may not be replicated yet. The server will reject the request with “ACL not found” or “permission denied.”

5. Verify WAN connectivity between DCs. Check Serf WAN membership. If the target DC is missing from the WAN pool entirely, the failover RPC has no destination. The query will fail rather than silently forward.

6. If using Connect service mesh, verify the traffic path. DNS prepared-query failover resolves remote IPs, but Consul does not support prepared queries for service-mesh failover; a standard prepared query does not give the sidecar proxy the mesh routing/identity configuration needed for the remote instance. Issue #8011 records this failure with ECONNRESET and remains open at review time.

Metrics and signals to monitor

Fixes

Local instances are genuinely all dead

The failover is working correctly. The problem is the local service, not Consul. Focus on recovering the local instances. Do not disable failover to “fix” the visibility problem. Removing failover means consumers get empty DNS responses and fail immediately instead of degrading to the remote DC.

The operational fix is detection: add monitoring that polls the execute API and alerts when Failovers exceeds zero for a query that should be locally served. That alert is your signal that local health has collapsed.

ACL replication lag causing authorization failures in target DC

When failover forwards a query to a remote DC, the query’s ACL token is evaluated in that DC. If replication has not yet delivered the token or its policies, the remote server returns “ACL not found” or “permission denied.” This is the composite failure pattern known as ACL replication staleness during failover.

Steps to resolve:

1. Check /v1/acl/replication on the target DC server. Confirm Enabled is true and inspect LastError.
2. Verify WAN connectivity between the primary and secondary DC. ACL replication traverses the same WAN links as gossip.
3. Check the replication token. If it has expired or its permissions changed, replication stalls silently. Rotate the token if needed.
4. Check primary DC ACL subsystem load. A burst of token creation in the primary can overwhelm replication before the secondary catches up.
5. If replication is broken and authorization is completely failing, consider the acl_down_policy setting as an emergency measure. Understand the security tradeoff before changing it.

WAN connectivity degradation

If the failover target DC is unreachable, the query fails entirely instead of silently forwarding. Consumers see DNS SERVFAIL or timeouts. This is arguably better than silent masking because the failure is visible, but the safety net is gone.

Check Serf WAN membership. If the remote DC servers are missing from the WAN pool, the cross-DC RPC has no destination. Common causes include WAN gossip port (8302) blocked by a firewall change, WAN gossip encryption key mismatch between DCs, or the remote DC being completely down.

Connect mesh does not follow prepared-query failover

If your service uses Consul Connect (mTLS sidecar proxies), prepared-query DNS failover resolves remote IPs, but the sidecar proxy lacks the service-resolver/routing configuration required for mesh failover. The DNS lookup succeeds, the application attempts to connect, and the connection can reset. This looks like a network error but is a capability boundary between DNS-based discovery and mesh-based routing.

For Connect-enabled services that need cross-DC failover, use service-resolver config entries with Failover instead of prepared queries. Service-resolver failover is designed for the mesh and handles identity and routing correctly across DC boundaries.

Prevention

Monitor the execute API, not just DNS. The DNS interface cannot tell you whether results are local or remote. Polling /v1/query/:uuid/execute and alerting on Failovers > 0 is the only reliable way to detect silent failover for queries expected to serve locally.

Track ACL replication lag in every secondary DC independently. “Works in primary” is not sufficient. Secondary DC ACL monitoring is one of the most common gaps in multi-DC deployments. Replication lag greater than 5 seconds, or any LastError, warrants investigation before it causes authorization failures during failover.

Test cross-DC lookup latency proactively. Do not wait for a failover event to discover that WAN RTT adds 200 milliseconds to every DNS resolution. Regular probing of cross-DC prepared query names reveals WAN degradation before it becomes an outage.

Document which prepared queries have failover enabled and what their target DCs are. During an incident, operators need to know immediately whether a given query can forward, where it forwards to, and what the ACL implications are. Undocumented failover policies are a common source of confusion during cross-DC events.

Know which services use Connect mesh versus plain DNS discovery. Prepared-query failover only helps services that consume DNS directly. Connect-enabled services need service-resolver failover, not prepared queries. Mixing the two without understanding the difference creates a class of bugs where DNS resolves but connections fail.

Separate WAN gossip encryption key management from LAN. WAN and LAN gossip use independent encryption keys. A key rotation that updates one but not the other silently breaks cross-DC communication. This is a common cause of silent WAN federation failure.

How Netdata helps

• Per-second DNS query latency surfaces the latency jump the moment lookups start crossing DC boundaries, before consumers notice degraded response times.
• Health check critical state ratio shows local instances failing in real time, directly correlatable with the moment Failovers transitions from zero to non-zero.
• ACL replication lag metrics expose stale tokens in secondary DCs before authorization failures reach applications during failover.
• Serf WAN member count per DC provides immediate visibility into cross-DC connectivity, showing whether failover targets are even reachable.
• Correlating DNS latency spikes with health check failures and WAN membership changes pinpoints whether the root cause is a local service outage, WAN degradation, or ACL replication lag, rather than guessing from a single signal.
• Tracking RPC request error rates with “ACL not found” patterns in secondary DCs catches replication-driven authorization failures before they cascade.

Related guides

• Consul blocking query accumulation: leaked watches that pile up goroutines
• Consul catalog bloat: too many services and checks slowing everything down
• Consul registration storm: catalog churn overwhelming Raft
• Consul anti-entropy not syncing: local agent state and the catalog drifting apart
• Consul client rpc failed: agents alive but the catalog is going stale
• Consul Connect CA rotation failure: a root roll that never finished
• Consul Connect certificate expired: mTLS handshakes failing across the mesh
• Consul DeregisterCriticalServiceAfter: instances vanishing from the catalog
• Consul DNS latency high: slow lookups stalling connections and failovers
• Consul DNS SERVFAIL: service discovery is broken for your applications
• Consul stale DNS queries: the agent is answering from cache
• Consul on EBS: burst-credit exhaustion and the sudden latency cliff