Consul WAN federation down: a datacenter missing from the WAN pool

When an entire remote datacenter’s servers vanish from the WAN gossip pool, the symptom is usually downstream: cross-DC service lookups return empty results, prepared queries that failover across DCs stop working, and ACL or Connect CA replication between datacenters silently stalls. The Consul UI on the local DC may still look healthy because the local cluster is fine. The damage is in the federation layer.

The canonical diagnostic signal is consul members -wan. On any server in any federated DC, this should list every server in every federated DC. When a whole DC’s server rows are missing or stuck in failed state, WAN federation for that DC is broken. The equivalent API call is /v1/agent/members?wan=true.

This is a PAGE-level condition when an entire DC is gone: cross-DC functionality for that DC is broken, and prepared-query failover to it cannot work. A single remote server missing is a TICKET: cross-DC routing still works through the surviving servers of that DC, but redundancy for cross-DC RPC is reduced.

What this means

Consul’s WAN gossip pool runs over both UDP and TCP on port 8302 by default and connects only server agents across datacenters. Client agents never appear in the WAN pool. Each server maintains an independent membership list, and WAN gossip failure detection runs on its own timers with higher latency tolerance than LAN gossip. A missing DC means the local servers cannot reach any server in the remote DC over the gossip protocol, which breaks the RPC routing that cross-DC service discovery, prepared-query failover, ACL token replication, and Connect CA initialization all rely on.

Two subtleties matter before you start debugging. First, the gossip encryption key must be identical across every federated DC. If different DCs were initialized with different encrypt keys, the WAN pool silently partitions with no effect on local DC health. Second, in Consul 1.8.0 and later, WAN federation can be configured to route through mesh gateways instead of direct server-to-server gossip on 8302. The failure modes are different: mesh-gateway federation depends on primary_gateways, TLS certificate SANs, and ACL token replication, none of which matter for traditional direct-gossip federation. You must know which federation style your cluster uses before diagnosing.

flowchart TD
    A[Remote DC servers missing from consul members -wan] --> B{Is the remote DC up?}
    B -->|No| C[Recover remote DC first]
    B -->|Yes| D{Can port 8302 reach remote?}
    D -->|No| E[Fix firewall or network path]
    D -->|Yes| F{WAN key matches?}
    F -->|No| G[Reconcile gossip key]
    F -->|Yes| H{Using mesh gateways?}
    H -->|Yes| I[Check primary_gateways and TLS SANs]
    H -->|No| J[Verify retry_join_wan and advertise_addr_wan]

Common causes

Quick checks

# List WAN pool from any local server - the primary triage signal
consul members -wan

# Same data via API, useful for monitoring and when CLI is unavailable
curl -s "http://127.0.0.1:8500/v1/agent/members?wan=true" | jq '.[].Status'

# Group WAN members by datacenter to spot the missing DC fast
curl -s "http://127.0.0.1:8500/v1/agent/members?wan=true" | \
  jq -r 'group_by(.Tags.dc)[] | "\(.[0].Tags.dc): \(length) members"'

# Check gossip encryption keys across DCs - WAN section must list the same key in every DC
consul keyring -list

# Inspect WAN gossip state on this server
consul info | grep -A 20 serf_wan

# Test UDP connectivity to a specific remote server's WAN gossip port.
# NOTE: UDP connectivity tests are inherently unreliable. A failure may mean
# the port is blocked OR simply that no ICMP unreachable was returned.
# Also test TCP 8302, which serf uses for join messages.
nc -zuv <remote-server-ip> 8302
nc -zv <remote-server-ip> 8302

# Test cross-DC RPC reachability (port 8300) - distinguishes gossip from RPC failure
nc -zv <remote-server-ip> 8300

# Confirm which federation style is in use
<!-- TODO: verify whether consul info reliably exposes mesh gateway WAN federation status; checking the config file directly may be more reliable -->
consul info | grep -i mesh

How to diagnose it

1. Confirm the scope. Run consul members -wan from a server in the local DC and identify exactly which DC is missing. Run it again from a server in a third DC if one exists: a DC missing from all other DCs’ views is a problem with that DC; a DC missing from only one other DC’s view is a bilateral network issue.

2. Verify the remote DC is alive. Check the remote DC’s own consul members (LAN pool). If the remote DC has lost quorum or all servers are down, WAN membership is a downstream symptom. Recover the remote DC first.

3. Test the network path. From a local server, test both UDP and TCP on port 8302 to a remote server IP. Serf WAN needs both protocols on 8302. If both fail, the cause is network: a firewall rule, a security group, a routing change, or a true cross-DC outage. Test TCP 8300 as well: if 8302 fails but 8300 works, the problem is specifically the WAN gossip port.

4. Compare encryption keys. Run consul keyring -list on a server in each DC. The WAN: section must list the same key with full node coverage in every DC. A mismatch silently partitions the WAN pool. This is one of the most common causes and one of the easiest to miss, because local LAN health is unaffected.

5. Determine the federation style. If enable_mesh_gateway_wan_federation = true is set, the cluster uses mesh-gateway federation and direct gossip on 8302 is not the primary path. Check that primary_gateways is configured on secondary DCs and that retry_join_wan is omitted. If enable_mesh_gateway_wan_federation is not set, the cluster uses traditional direct-gossip federation and retry_join_wan should be configured.

6. For mesh-gateway clusters, check the gateway path. The most common mesh-gateway failures are TLS certificate SANs that do not cover the server DNS names and ACL token replication lag in secondary DCs that prevents mesh gateway registration. Look for “No path to datacenter” errors in server logs.

7. Check advertise_addr_wan and translate_wan_addrs. If consul members -wan lists remote nodes but cross-DC RPC fails with i/o timeout, the advertised WAN address may not be routable from the local DC. The advertised WAN address must be reachable across the WAN; a private bind_addr that works locally will not work for remote peers.

8. Check reconnect_timeout_wan. This defaults to 72 hours. A failed remote server stays in the WAN pool as failed for this long before being reaped. If you are seeing stale failed entries from a server that was decommissioned, this is expected behavior, not a live failure.

Metrics and signals to monitor

Fixes

Cross-DC network outage or firewalled 8302

Resolve the underlying network issue. If a firewall change is the cause, open both UDP and TCP on 8302 between all server IPs across the federated DCs. Once connectivity is restored, WAN gossip heals automatically: failed members return to alive within a few gossip intervals. You do not need to re-join unless the reconnect_timeout_wan window has expired and the members have been reaped, in which case run consul join -wan <remote-server-ip> from a local server.

Gossip encryption key mismatch

All federated DCs must use the same gossip encryption key. To reconcile, use consul keyring to install the correct key on every server in every DC and remove the divergent key. After reconciliation, WAN membership converges within gossip intervals. Verify with consul keyring -list in every DC.

Remote DC down

Recover the remote DC’s own cluster first. WAN membership is downstream of LAN health: if the remote DC has no leader or no quorum, its servers will not participate in WAN gossip regardless of network path. Once the remote DC recovers, WAN federation resumes without intervention unless members were reaped.

Mesh gateway path issues

For mesh-gateway federation , the failure surface is different. Verify that enable_mesh_gateway_wan_federation = true is set consistently on all servers, that secondary DCs use primary_gateways rather than retry_join_wan, and that primary gateway addresses are routable from secondary DCs. Do not decommission primary gateway IPs before secondary DCs have received updated gateway addresses, or the primary DC becomes unreachable. For TLS SAN issues, regenerate server certificates with -additional-dnsname covering the server DNS names. For ACL replication failures blocking mesh gateway registration, fix the replication token and wait for convergence.

advertise_addr_wan mismatch

Set advertise_addr_wan to an address routable from all federated DCs. If the cluster uses translate_wan_addrs, verify the translation mapping is correct for every server. Restarting the agent is required for advertise_addr_wan changes to take effect.

Clearing stale failed entries

If a remote server was permanently decommissioned but still appears as failed in the WAN pool, use consul force-leave -wan <node>.<datacenter> to remove it. The node name in the WAN pool is [node-name].[datacenter], not the bare node name. Stale entries do not break federation, but they pollute membership views and can confuse monitoring. Only run force-leave on nodes you have confirmed are permanently decommissioned.

Prevention

• Monitor consul.serf.wan.members per DC as a PAGE signal. An entire DC dropping out is high-severity; alert on it directly rather than waiting for downstream symptoms.
• Run consul keyring -list periodically across all DCs. Detect key drift before it partitions the pool. This matters most after key rotation.
• Test cross-DC 8302 connectivity (both UDP and TCP) proactively. A synthetic check from each DC to each other DC catches firewall and routing changes before they become outages.
• Document the federation style. Whether the cluster uses direct gossip or mesh-gateway federation determines the entire diagnostic path.
• Track ACL replication lag in secondary DCs. Rising lag is often the first visible symptom of WAN degradation, ahead of WAN membership changes.
• Plan key rotations carefully. Key rotation must be coordinated across all federated DCs. Verify convergence with consul keyring -list before declaring the rotation complete.

How Netdata helps

• Per-second consul.serf.wan.members tracking pinpoints the exact second a remote DC drops from the WAN pool and correlates it with network, disk, or process events on the local servers.
• Cross-DC correlation of WAN membership against ACL replication lag, cross-DC RPC latency, and inter-DC packet loss shortens the path from symptom to root cause: network problem, key mismatch, or remote DC down.
• Anomaly detection on WAN member counts flags a DC disappearing even when no static threshold has been set.
• Per-DC dashboards let you compare WAN pool views from servers in different DCs side by side, which is the bilateral-versus-global distinction the diagnosis requires.

Related guides

• Consul blocking query accumulation: leaked watches that pile up goroutines
• Consul catalog bloat: too many services and checks slowing everything down
• Consul registration storm: catalog churn overwhelming Raft
• Consul anti-entropy not syncing: local agent state and the catalog drifting apart
• Consul client rpc failed: agents alive but the catalog is going stale
• Consul Connect CA rotation failure: a root roll that never finished
• Consul Connect certificate expired: mTLS handshakes failing across the mesh
• Consul DeregisterCriticalServiceAfter: instances vanishing from the catalog
• Consul DNS latency high: slow lookups stalling connections and failovers
• Consul DNS SERVFAIL: service discovery is broken for your applications
• Consul stale DNS queries: the agent is answering from cache
• Consul on EBS: burst-credit exhaustion and the sudden latency cliff