Consul WAN link saturation: cross-DC latency triggering gossip suspicion

Remote datacenter servers start oscillating between alive and suspect in the WAN gossip pool. Cross-DC RPCs fail with “no path to datacenter” errors. ACL replication lag in secondary DCs climbs into seconds. Cross-DC prepared queries time out. LAN Consul signals look fine, but the federation is broken.

The usual cause is WAN link saturation. Consul’s WAN gossip pool uses a separate, more conservative set of timing defaults than LAN, but it is still vulnerable to high or spiking RTT. When the inter-DC link saturates, gossip probes time out, remote servers are marked suspect then failed, and WAN membership flaps. Each flap disrupts cross-DC RPC routing, which compounds the problem because RPC retries add traffic to the already-saturated link.

This article covers how to separate a saturated-but-up WAN link from a genuine partition, what drives the cross-DC traffic that saturates it (replication, prepared queries, mesh gateway federation), and which WAN gossip thresholds to revisit before assuming the protocol itself is broken.

What this means

WAN gossip is the membership and failure-detection protocol that runs between Consul servers in different datacenters. It is a separate Serf pool from LAN gossip and operates on the WAN port (default 8302, TCP and UDP). The WAN pool includes only server nodes; client agents do not participate.

Three classes of traffic cross the inter-DC link in a federated deployment:

1. WAN gossip: Serf/memberlist probes and event propagation over 8302/UDP, with TCP fallback.
2. Cross-DC RPC: server-to-server RPC multiplexed over 8300/TCP, used for cross-DC service lookups, prepared query failover, and ACL replication.
3. Mesh gateway traffic: when WAN federation is configured via mesh gateways, both gossip and RPC traverse the gateway’s Envoy proxy, which adds its own failure surface.

When the WAN link saturates, two things happen in parallel. WAN gossip probes (which have a configurable timeout) time out because RTT exceeds the probe timeout. Cross-DC RPCs queue behind the saturated link and either time out or surface as “no path to datacenter” errors in server logs.

The gossip layer reacts to missed probes by marking the unresponsive remote server suspect, then failed. Once a server is failed in the WAN pool, cross-DC RPC routing to that DC degrades or stops. As remote servers cycle alive/suspect/failed/alive, downstream consumers (load balancers, prepared queries, ACL replication) see flapping.

flowchart TD
    A["Cross-DC traffic spike:
replication, prepared queries, mesh gateway"] --> B[WAN link saturates]
    B --> C[RTT exceeds probe_timeout]
    C --> D[WAN gossip probes fail]
    C --> E[Cross-DC RPC queues / times out]
    D --> F["Remote servers marked suspect, then failed"]
    F --> G[WAN membership flaps]
    G --> H[Cross-DC RPC routing degrades]
    E --> H
    H --> I[RPC retries add load to saturated link]
    I --> B

A key gotcha: WAN gossip can show remote servers as alive (because UDP probes succeed in one direction) while TCP-based RPC forwarding on 8300 fails in the other direction. The most common cause is asymmetric routing or a bind_addr set to a private IP that is not routable across the WAN. Do not assume that consul members -wan showing alive means the federation is healthy.

Common causes

Current configuration references use nested gossip_lan / gossip_wan blocks; older configuration layouts referred to Serf LAN/WAN tuning. Verify the layout supported by your Consul version before changing either block.

Quick checks

# WAN gossip membership from this server's perspective
consul members -wan

# WAN membership grouped by datacenter, with status counts
curl -s "http://127.0.0.1:8500/v1/agent/members?wan=true" | \
  python3 -c "
import sys, json
from collections import Counter
by_dc = {}
for m in json.load(sys.stdin):
    dc = m.get('Tags', {}).get('dc', 'unknown')
    by_dc.setdefault(dc, Counter())[m['Status']] += 1
for dc, counts in sorted(by_dc.items()):
    print(f'DC {dc}: {dict(counts)}')"

# ACL replication status in a secondary DC (lag is a strong WAN-health proxy)
curl -s http://127.0.0.1:8500/v1/acl/replication | python3 -m json.tool

# WAN gossip queue depths on a server (sustained non-zero means falling behind)
curl -s http://127.0.0.1:8500/v1/agent/metrics | \
  python3 -c "
import sys, json
d = json.load(sys.stdin)
for section in ('Samples', 'Counters', 'Gauges'):
    for s in d.get(section, []):
        if 'serf.queue_' in s['Name'] or 'memberlist.' in s['Name']:
            print(s['Name'], section, 'Count:', s.get('Count'), 'Mean:', s.get('Mean'), 'Value:', s.get('Value'))"

# RTT and packet loss across the inter-DC link (run from each server; 50 cycles is slow)
mtr --report --report-cycles 50 <remote-server-wan-addr>

# Cross-DC RPC test: a lookup against a remote DC's catalog
time curl -s "http://127.0.0.1:8500/v1/catalog/service/<service>?dc=<remote-dc>"

# Recent WAN gossip flaps and cross-DC RPC errors in server logs
journalctl -u consul --since '30 min ago' | \
  grep -Ei 'suspect|failed|no path to datacenter|rpc failed' | tail -50

All of the above are read-only and safe to run during an incident.

How to diagnose it

The first decision is: saturated-but-up link, or genuine partition? They present similarly but require different responses.

1. Check both directions of the link. Run mtr from each DC to the other. Asymmetric packet loss or latency is the most common cause of “alive in gossip but RPC fails.” If only one direction is degraded, expect asymmetric routing, a half-open firewall state, or a misconfigured bind_addr.

2. Compare WAN gossip status with cross-DC RPC success. WAN gossip probes are UDP and may succeed where TCP RPC fails. If consul members -wan shows remote servers as alive but a cross-DC catalog lookup returns errors or times out, the link is degraded in a way that affects TCP but not yet UDP. This often precedes full gossip flap.

3. Look at WAN gossip queue depths. consul.serf.queue_Event and consul.serf.queue_Intent sustaining above zero on a server means it is receiving gossip faster than it can process. On a WAN pool, this typically means the remote side is reflooding events after membership changes. Sustained queue depth plus a rising flap count is a saturation signature, not a partition.

4. Correlate ACL replication lag with WAN gossip flaps. ACL replication runs over cross-DC RPC. If ACL replication lag spikes precede or coincide with WAN member flaps, cross-DC RPC load is the saturation driver, not the gossip protocol itself. That distinction determines whether you tune gossip or throttle RPC.

5. If using mesh gateway federation, check the gateway separately. When both gossip and RPC traverse a mesh gateway, an Envoy bottleneck or an ACL-induced gRPC stream closure looks exactly like WAN link saturation. Inspect Envoy /stats for upstream connection failures and Consul server logs for xDS errors before assuming the underlying network is the problem.

6. Verify the WAN gossip encryption key is consistent across DCs. consul keyring -list should show the same WAN key on every server. A botched key rotation can cause a continuous gossip partition that looks like link saturation.

Metrics and signals to monitor

Current telemetry exposes memberlist state/byte counters and Serf queue summaries; it does not document serf.wan.members or serf.member.flap. Confirm gossip-pool labels in your Prometheus/StatsD sink.

Fixes

Saturated inter-DC bandwidth

The driver is usually cross-DC RPC volume, not gossip itself. Identify the source: prepared queries with aggressive failover, ACL replication after a bulk token operation in the primary DC, or applications treating cross-DC Consul reads as local.

Short-term, rate-limit or shed the cross-DC caller. If ACL replication is the driver, throttle token creation in the primary DC and let the secondary catch up. If prepared queries are the driver, review their TTL and failover policy.

Longer-term, size headroom around replication lag and bandwidth utilization. A common operator target is inter-DC utilization under 70% with burst capacity for recovery sync, and replication lag safely below the RPC timeout. No universal HashiCorp percentage target was found; size the link to your measured recovery/reconciliation workload, not to average utilization.

WAN gossip thresholds too tight for the actual RTT

The WAN gossip block has separate defaults from the LAN block because WAN links are higher-latency. HashiCorp documents this block as “Advanced” and warns that improper tuning can cause unexpected failures. Before changing anything:

• Measure the real inter-DC RTT distribution, including tail latency. Defaults that look fine on the median will flap under p99.
• The suspicion timeout scales with cluster size and probe_interval. Increasing probe_interval lengthens the failure-detection window, which reduces false suspicions but slows detection of real failures. Trade deliberately.
• Increasing suspicion_mult makes the protocol more tolerant of transient probe failures. Do this only if you have confirmed the link is lossy-but-up rather than partitioned.
• Do not copy LAN gossip settings into the WAN block.

Bad tuning is worse than the original problem because it masks real failures. Change one parameter at a time and observe flap rate before and after.

Mesh gateway bottleneck

If WAN federation runs through mesh gateways, the gateway is a shared chokepoint for both gossip and RPC. Confirm the gateway Envoy has adequate CPU and file-descriptor headroom, and that its upstream connections to remote servers are stable. Envoy gRPC stream closures (often ACL-induced) manifest as flapping WAN membership even when the underlying network is healthy.

Note that retry_join_wan addresses are only for traditional WAN federation and must be omitted when federating servers via mesh gateways; secondaries use primary_gateways. Mixing the two mechanisms produces misleading connectivity errors that can look like saturation.

Asymmetric routing or non-routable bind_addr

The most common cause of “alive in WAN but cross-DC RPC fails” is bind_addr set to a private IP that is not routable across the WAN. Set advertise_addr_wan explicitly to the routable WAN address on every server. After fixing, verify with bidirectional mtr and a cross-DC RPC test from each server.

Gossip encryption key mismatch

WAN gossip uses a separate key from LAN. consul keyring -list should show the same WAN key on every server in every DC. Keyring mutations apply to LAN and WAN pools in lock-step, but verify both sections after every install/use/remove and stage rotations so no server is left without a key its peers accept.

Prevention

• Alert on WAN membership with WAN-specific thresholds. A remote server in suspect for more than one WAN probe_interval, or in failed at all, is page-worthy in a federated deployment. Use thresholds separate from your LAN gossip alerts.
• Baseline inter-DC RTT and track its p99. Average RTT is misleading; gossip failures are driven by tail latency. Alert when p99 RTT approaches the configured probe_timeout.
• Monitor ACL replication lag in every secondary DC independently. “Works in primary” is not sufficient. Lag is the earliest signal that cross-DC RPC is degraded.
• Track WAN gossip bytes per DC and graph against inter-DC interface utilization. The two should track each other; divergence indicates either queueing on the network or a processing backlog on a server.
• Periodically test cross-DC RPC, not just WAN membership. A scheduled cross-DC catalog lookup from each DC catches the asymmetric “gossip alive but RPC fails” failure mode before users do.
• Document the WAN gossip parameters in use per DC. Available parameters and their defaults have evolved across Consul versions. Re-baseline after every upgrade.

How Netdata helps

• Per-second metrics on serf.wan.members and flap counters let you see WAN membership oscillation at the timescale gossip actually operates, instead of a 60-second polling interval that hides sub-minute flaps.
• Correlating WAN gossip flaps with inter-DC interface utilization and RTT in a single view is the fastest path to confirming saturation as the root cause versus a partition or a misconfigured gateway.
• Anomaly detection on ACL replication lag and cross-DC RPC latency surfaces the slow degradation that precedes overt flapping, typically tens of minutes before the WAN pool starts cycling suspect/failed.
• Per-server gossip queue depth and bytes-sent/received let you pinpoint which server is falling behind and whether the backlog is processing-bound or network-bound.
• Composite dashboards across DCs make asymmetric degradation visible: a flap or lag spike present in only one direction is the signature of asymmetric routing or a half-open firewall, not a saturated link.

Related guides

• Consul blocking query accumulation: leaked watches that pile up goroutines
• Consul catalog bloat: too many services and checks slowing everything down
• Consul registration storm: catalog churn overwhelming Raft
• Consul anti-entropy not syncing: local agent state and the catalog drifting apart
• Consul client rpc failed: agents alive but the catalog is going stale
• Consul Connect CA rotation failure: a root roll that never finished
• Consul Connect certificate expired: mTLS handshakes failing across the mesh
• Consul DeregisterCriticalServiceAfter: instances vanishing from the catalog
• Consul DNS latency high: slow lookups stalling connections and failovers
• Consul DNS SERVFAIL: service discovery is broken for your applications
• Consul stale DNS queries: the agent is answering from cache
• Consul on EBS: burst-credit exhaustion and the sudden latency cliff