A Consul prepared query with cross-DC failover does exactly what it was configured to do: it silently forwards lookups to a remote DC when local results are empty. Consumers receive valid service discovery results, applications keep connecting, and no error fires. The local DC may have zero healthy instances, and nobody on the consumer side knows.

The DNS response is identical whether nodes came from the local DC or a remote one. There is no EDNS flag, no source record, no indicator in the answer. Receiving results does not mean local health.

This article covers how to detect active failover, distinguish it from a genuine local outage, handle ACL replication lag in the failover target DC, and understand why Connect mesh traffic does not follow DNS prepared-query failover across datacenters.

What this means

Prepared queries are Consul’s mechanism for geo-failover service discovery. A query defines a service name plus an optional failover policy. When a consumer resolves the query, Consul first checks for healthy nodes in the local DC. If none exist, it applies the failover policy before returning an empty result.

Three failover mechanisms exist:

  • NearestN: dynamic selection based on network coordinate RTT to the nearest N datacenters.
  • Datacenters: a static, ordered list of DCs to try in sequence.
  • Targets: for cluster peering connections. Mutually exclusive with NearestN and Datacenters.

The masking happens at the DNS layer. When a prepared query resolves via DNS on port 8600, the response contains A, AAAA, or SRV records with no field indicating which DC provided the nodes. An application that resolves my-service.query.consul and gets three IP addresses cannot tell whether those addresses are local or 200 milliseconds away in a remote DC.

The HTTP execute API exposes what DNS hides. The endpoint GET /v1/query/:uuid/execute returns two fields that reveal failover state:

  • Datacenter: the DC that ultimately provided the nodes.
  • Failovers: the count of remote DCs queried during resolution. Zero means results came from the local DC.

Most applications consume Consul via DNS, not the HTTP API. That is the core visibility gap. The failover is working as designed, but operators must actively poll the execute endpoint to know it is happening.

flowchart LR
  A[Consumer DNS query] --> B[Consul agent]
  B --> C{Local DC has
healthy nodes?}
  C -->|Yes| D[Return local nodes]
  C -->|No| E[Failover policy:
NearestN or Datacenters]
  E -->|WAN RPC| F[Remote DC server]
  F --> G[Remote healthy nodes]
  G --> H[DNS response:
no source indicator]
  H --> I[Consumer connects to
remote DC instance]

The “no” branch produces a valid response that looks identical to the “yes” branch from the consumer’s perspective. The only operators who can tell the difference are those polling the HTTP API or correlating DNS latency with health check state.

Common causes

CauseWhat it looks likeFirst thing to check
All local instances failedDNS returns valid IPs, latency slightly elevated, no consumer errorsQuery /v1/health/service/<name>?passing=true locally; zero confirms local outage
ACL replication lag in target DC“ACL not found” or “permission denied” in remote DC server logs during failoverRun curl /v1/acl/replication in the secondary DC, check ReplicatedIndex lag
WAN link degraded or partitionedFailover queries time out or return SERVFAIL instead of remote resultsCheck Serf WAN member count per DC; missing DC means failover target gone
Connect mesh and prepared query mismatchDNS resolves remote IPs but sidecar proxy connections resetDetermine if traffic is plain DNS discovery or Connect mTLS; cross-DC behavior differs

Quick checks

Run these read-only checks to determine whether failover is active and why.

# Execute a prepared query via HTTP and inspect failover state
curl -s http://127.0.0.1:8500/v1/query/<query-uuid>/execute \
  | jq '{Datacenter, Failovers, NodeCount: (.Nodes | length)}'
# Failovers > 0 means results came from a remote DC

# Check local health for the service (are local instances actually dead?)
curl -s "http://127.0.0.1:8500/v1/health/service/<service>?passing=true" | jq length
# Zero means every local instance is unhealthy

# Check ACL replication status in secondary DC (run on a secondary DC server)
curl -s http://127.0.0.1:8500/v1/acl/replication | jq '{Enabled, ReplicatedIndex, LastError}'
# LastError should be empty; large gap between ReplicatedIndex and primary indicates lag

# Check WAN gossip membership (is the remote DC reachable?)
curl -s "http://127.0.0.1:8500/v1/agent/members?wan=true" \
  | jq '[.[] | .Tags.dc] | group_by(.) | map({dc: .[0], servers: length})'
# Each federated DC should show its expected server count

# Time a DNS lookup for the prepared query name
dig @127.0.0.1 -p 8600 <query-name>.query.consul +stats | grep "Query time"
# Elevated latency suggests cross-DC resolution

# Check whether ACL filtering removed results from the query
curl -sI http://127.0.0.1:8500/v1/query/<query-uuid>/execute \
  | grep -i "X-Consul-Results-Filtered-By-ACLs"
# Header present and true means the query token lacks permissions for some nodes

How to diagnose it

  1. Confirm failover is active. Execute the prepared query via the HTTP API. If Failovers is greater than zero, the query is forwarding to a remote DC. If Datacenter does not match the local DC name, results are coming from elsewhere.

  2. Check whether local instances are genuinely unhealthy. Query /v1/health/service/<service>?passing=true for the service in the local DC. Zero results means every instance is failing health checks. The failover is masking a real local outage.

  3. If local instances are healthy but failover is still active, check ACL filtering. The X-Consul-Results-Filtered-By-ACLs response header on the execute endpoint indicates whether the query’s token is filtering out nodes that should be visible. A misconfigured token can remove all local results, triggering failover to a remote DC unnecessarily.

  4. If the failover target DC is returning errors, check ACL replication. When the query forwards to a remote DC, the query’s ACL token must be valid there. Run curl /v1/acl/replication on a server in the target DC. If ReplicatedIndex lags significantly behind the primary DC or LastError is non-empty, the token may not exist yet in the remote DC. The server will reject the request with “ACL not found” or “permission denied.”

  5. Verify WAN connectivity between DCs. Check Serf WAN membership. If the target DC is missing from the WAN pool entirely, the failover RPC has no destination. The query will fail rather than silently forward.

  6. If using Connect service mesh, verify the traffic path. DNS prepared-query failover resolves remote IPs, but the Connect sidecar proxy cannot route mTLS traffic to a remote DC instance over a standard prepared query.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Prepared query Failovers count (HTTP execute API)Non-zero means results came from a remote DC, masking local emptinessAny sustained non-zero value for a query expected to be locally served
Datacenter field on query executionIdentifies which DC is actually serving resultsField does not match local DC name
DNS query latency for prepared query namesCross-DC lookups incur WAN RTT on every resolutionSustained latency increase for a specific .query.consul name
consul_health_service_status critical count (local DC)Local instances dying is what triggers failover in the first placeSpike in critical checks for the affected service
consul_acl_replication lag (secondary DCs)Stale tokens cause authorization failures when queries forward remotelyLag greater than 5 seconds, or LastError non-empty
Serf WAN member count per DCMissing DC means failover has no targetAny federated DC showing fewer servers than expected
consul_rpc_request error rate with “ACL not found”Replication lag surfacing as authorization failures in the target DCErrors appearing only in secondary DC, not primary

Fixes

Local instances are genuinely all dead

The failover is working correctly. The problem is the local service, not Consul. Focus on recovering the local instances. Do not disable failover to “fix” the visibility problem. Removing failover means consumers get empty DNS responses and fail immediately instead of degrading to the remote DC.

The operational fix is detection: add monitoring that polls the execute API and alerts when Failovers exceeds zero for a query that should be locally served. That alert is your signal that local health has collapsed.

ACL replication lag causing authorization failures in target DC

When failover forwards a query to a remote DC, the query’s ACL token is evaluated in that DC. If replication has not yet delivered the token or its policies, the remote server returns “ACL not found” or “permission denied.” This is the composite failure pattern known as ACL replication staleness during failover.

Steps to resolve:

  1. Check /v1/acl/replication on the target DC server. Confirm Enabled is true and inspect LastError.
  2. Verify WAN connectivity between the primary and secondary DC. ACL replication traverses the same WAN links as gossip.
  3. Check the replication token. If it has expired or its permissions changed, replication stalls silently. Rotate the token if needed.
  4. Check primary DC ACL subsystem load. A burst of token creation in the primary can overwhelm replication before the secondary catches up.
  5. If replication is broken and authorization is completely failing, consider the acl_down_policy setting as an emergency measure. Understand the security tradeoff before changing it.

WAN connectivity degradation

If the failover target DC is unreachable, the query fails entirely instead of silently forwarding. Consumers see DNS SERVFAIL or timeouts. This is arguably better than silent masking because the failure is visible, but the safety net is gone.

Check Serf WAN membership. If the remote DC servers are missing from the WAN pool, the cross-DC RPC has no destination. Common causes include WAN gossip port (8302) blocked by a firewall change, WAN gossip encryption key mismatch between DCs, or the remote DC being completely down.

Connect mesh does not follow prepared-query failover

If your service uses Consul Connect (mTLS sidecar proxies), prepared-query DNS failover resolves remote IPs but the sidecar proxy cannot establish the mTLS session to the remote instance. The DNS lookup succeeds, the application attempts to connect, and the connection resets. This looks like a network error but is actually a capability gap between DNS-based discovery and mesh-based routing.

For Connect-enabled services that need cross-DC failover, use service-resolver config entries with Failover instead of prepared queries. Service-resolver failover is designed for the mesh and handles identity and routing correctly across DC boundaries.

Prevention

Monitor the execute API, not just DNS. The DNS interface cannot tell you whether results are local or remote. Polling /v1/query/:uuid/execute and alerting on Failovers > 0 is the only reliable way to detect silent failover for queries expected to serve locally.

Track ACL replication lag in every secondary DC independently. “Works in primary” is not sufficient. Secondary DC ACL monitoring is one of the most common gaps in multi-DC deployments. Replication lag greater than 5 seconds, or any LastError, warrants investigation before it causes authorization failures during failover.

Test cross-DC lookup latency proactively. Do not wait for a failover event to discover that WAN RTT adds 200 milliseconds to every DNS resolution. Regular probing of cross-DC prepared query names reveals WAN degradation before it becomes an outage.

Document which prepared queries have failover enabled and what their target DCs are. During an incident, operators need to know immediately whether a given query can forward, where it forwards to, and what the ACL implications are. Undocumented failover policies are a common source of confusion during cross-DC events.

Know which services use Connect mesh versus plain DNS discovery. Prepared-query failover only helps services that consume DNS directly. Connect-enabled services need service-resolver failover, not prepared queries. Mixing the two without understanding the difference creates a class of bugs where DNS resolves but connections fail.

Separate WAN gossip encryption key management from LAN. WAN and LAN gossip use independent encryption keys. A key rotation that updates one but not the other silently breaks cross-DC communication. This is a common cause of silent WAN federation failure.

How Netdata helps

  • Per-second DNS query latency surfaces the latency jump the moment lookups start crossing DC boundaries, before consumers notice degraded response times.
  • Health check critical state ratio shows local instances failing in real time, directly correlatable with the moment Failovers transitions from zero to non-zero.
  • ACL replication lag metrics expose stale tokens in secondary DCs before authorization failures reach applications during failover.
  • Serf WAN member count per DC provides immediate visibility into cross-DC connectivity, showing whether failover targets are even reachable.
  • Correlating DNS latency spikes with health check failures and WAN membership changes pinpoints whether the root cause is a local service outage, WAN degradation, or ACL replication lag, rather than guessing from a single signal.
  • Tracking RPC request error rates with “ACL not found” patterns in secondary DCs catches replication-driven authorization failures before they cascade.