Consul gossip storm after mass recovery: rejoin floods and anti-entropy spikes

After a mass recovery event (AZ healing, partition recovery, fleet-wide rolling restart), dozens or hundreds of Consul agents rejoin the LAN gossip pool in a compressed window. Each returning agent generates member-join events that every other member processes, and triggers anti-entropy sync to reconcile its local service registrations against the server catalog. The coordinated spike hits Serf message queues, catalog registration writes, and Raft commits simultaneously.

Most clusters absorb the spike and recover within minutes. The risk is write volume pushing Raft commitTime toward the election timeout. Once followers cannot reach the leader within the election window, they start new elections, cascading a transient burst into leader thrashing and repeated write outages.

First diagnostic question: does the storm correlate with a known recovery event? A gossip storm with no corresponding infrastructure event points to a different root cause, most commonly a gossip encryption key mismatch causing rejoin loops.

What this means

The storm has three overlapping phases that hit simultaneously.

Rejoin flood. N agents rejoining at once means N member-join events disseminated to all members via Serf gossip. Each join triggers status updates, health check re-evaluations, and event propagation. Gossip queue depths spike as agents receive events faster than they can drain them.

Anti-entropy reconciliation. Each returning agent reconciles its local state against the server catalog. Every service and check that differs generates a catalog registration write through Raft. With hundreds of agents syncing at once, catalog registration rate jumps by orders of magnitude over baseline.

Raft pipeline pressure. The catalog writes flow through the Raft apply path. The leader sees a surge in apply operations, replication load, and commit latency. If commitTime stays well below the election timeout, the cluster processes the backlog and recovers. If commitTime approaches the election timeout, followers lose contact with the leader and trigger elections.

flowchart TD
    A[Mass recovery: AZ heal or rolling restart] --> B[Many nodes rejoin gossip pool]
    B --> C[Serf queues spike
member-join events flood]
    B --> D[Anti-entropy sync
catalog.register storms]
    C --> E[raft.apply rate spikes]
    D --> E
    E --> F[raft.commitTime rises]
    F --> G{Approaches election timeout?}
    G -->|No| H[Self-resolves in minutes]
    G -->|Yes| I[Leader thrashing
repeated elections]

The storm is self-limiting once rejoin events drain and anti-entropy converges. The danger window is the few minutes during which all three phases overlap.

Common causes

Quick checks

Read-only and safe during an active incident.

# Gossip queue depths (Event and Intent queues)
curl -s 'http://127.0.0.1:8500/v1/agent/metrics?format=json' | grep -E 'serf\.queue\.(Event|Intent)'

# Current leader
curl -s http://127.0.0.1:8500/v1/status/leader

# Serf LAN member count and statuses
curl -s http://127.0.0.1:8500/v1/agent/members | python3 -c "
import sys,json
members=json.load(sys.stdin)
alive=sum(1 for m in members if m['Status']==1)
failed=sum(1 for m in members if m['Status']==4)
print(f'Alive: {alive}, Failed: {failed}, Total: {len(members)}')
"

# Raft commit time (histogram in Samples)
curl -s 'http://127.0.0.1:8500/v1/agent/metrics?format=json' | python3 -c "
import sys,json
d=json.load(sys.stdin)
for s in d.get('Samples',[]):
    if 'raft.commitTime' in s['Name']:
        print(s['Name'], 'Count:', s.get('Count'), 'Mean:', s.get('Mean'))
"

# Catalog registration count
curl -s 'http://127.0.0.1:8500/v1/agent/metrics?format=json' | python3 -c "
import sys,json
d=json.load(sys.stdin)
for c in d.get('Counters',[]):
    if 'catalog.register' in c['Name']:
        print(c['Name'], 'Count:', c.get('Count'))
"

# Leader state (gauge: 1 = leader, 0 = follower)
# Poll repeatedly; 2+ flips in 10 min indicates thrashing
curl -s 'http://127.0.0.1:8500/v1/agent/metrics?format=json' | python3 -c "
import sys,json
d=json.load(sys.stdin)
for g in d.get('Gauges',[]):
    if 'raft.state.leader' in g['Name']:
        print(g['Name'], 'Value:', g.get('Value'))
"

# Raft peers and voter status
consul operator raft list-peers

# Gossip encryption key distribution
consul keyring -list

# Leader lastContact with followers (leader only)
curl -s 'http://127.0.0.1:8500/v1/agent/metrics?format=json' | grep -i 'lastContact'

How to diagnose

1. Confirm a recovery event occurred. Check cloud provider event logs, deployment pipelines, or network monitoring for an AZ recovery, partition heal, or mass restart within the last 5 to 15 minutes. If nothing matches, skip to step 6.

2. Check gossip queue depths. Spiking but trending downward means the rejoin flood is draining. Growing unbounded means agents are falling behind and may be marked failed by peers, which generates more gossip events and creates a positive feedback loop.

3. Check catalog registration rate. A spike in consul.catalog.register confirms anti-entropy is the write source. A 10x spike over baseline during recovery is expected. Sustained elevation after the recovery window suggests ongoing state drift.

4. Check Raft commit time. This is the critical cascade indicator. If consul.raft.commitTime stays well below the election timeout (ideally below one-tenth of it), the cluster will self-recover. If it approaches the election timeout, you are at risk of leader thrashing.

5. Check for leadership transitions. Poll consul.raft.state.leader or compare consul operator raft list-peers output over time. More than 2 leadership transitions per 10 minutes means the storm has cascaded into leader thrashing and an active write outage loop. See the leader election storm guide for that phase.

6. If no recovery event: check gossip key distribution. Run consul keyring -list. If different nodes show different keys, a partial key rotation is splitting the cluster into sub-groups that cannot authenticate gossip messages. Agents reconnect, fail to decrypt, drop, and retry in a loop that looks like a recovery storm but never converges.

7. Check for semi-detached nodes. A node that can reach some peers but not others keeps participating in gossip while generating suspect and failure events across the partition boundary. Compare consul members output from multiple servers. Persistent asymmetry in member lists points to a network-level problem, not a recovery storm.

Metrics and signals to monitor

Fixes

If commitTime is climbing but no elections yet

The cluster is absorbing the spike. Monitor closely and do not add load. Every manual registration, KV write, or config change during this window competes with the recovery backlog for Raft pipeline capacity. The storm should self-resolve as rejoin events drain and anti-entropy converges.

If the storm has cascaded into leader thrashing

The write spike has overwhelmed the Raft pipeline. The priority is reducing write load to give the leader breathing room.

• Pause the trigger. If a mass rolling restart is in progress, pause the deployment pipeline to stagger rejoining agents. If automated scaling is adding many agents at once, rate-limit the scaling operation.
• Stop voluntary writes. Hold deployment registrations, KV bulk operations, and config changes until commitTime drops below one-tenth of the election timeout.
• Increase deregister_critical_service_after on flapping services in the next config deployment. Each check state transition is a Raft write; longer deregister windows prevent cascading deregistrations from compounding the write load.

If the storm is a gossip key mismatch

This is not a recovery storm. Agents using different encryption keys cannot authenticate gossip messages to each other.

• Run consul keyring -list to identify which keys are in use and on how many nodes.
• Ensure all nodes have the current primary key: consul keyring -install=<key>.
• Remove stale keys with consul keyring -remove=<key> only after confirming all nodes have the new key.
• Verify convergence: gossip queue depths should return to zero and member counts should stabilize without further oscillation.

If semi-detached nodes are driving suspect storms

A node with partial network connectivity generates repeated suspect and failure events across the partition boundary. This creates ongoing gossip churn even after the primary recovery event has passed.

• Compare consul members output from multiple servers for member list discrepancies.
• Verify network connectivity between the suspected node and all peers, not just the leader. Asymmetric partitions are common.
• If the node has persistent connectivity problems, drain its services and restart it with corrected network configuration.

Prevention

Stagger mass restarts. Bring agents back in waves of 5 to 10 percent of the fleet, separated by 30 to 60 seconds, to prevent the synchronized rejoin flood. Most deployment and orchestration systems support rate limiting or canary percentages.

Monitor gossip queue depth proactively. Any sustained non-zero value in consul.serf.queue.Event or consul.serf.queue.Intent during normal operations means the cluster is near gossip processing saturation. A recovery event on top of a saturated gossip pool cascades faster and recovers slower.

Track catalog registration baseline. Know your normal catalog registration rate. A recovery storm producing a 50x spike over an already high baseline is more dangerous than the same multiplier over a low baseline, because the Raft pipeline has less headroom.

Validate gossip key rotations in staging. Partial key rotation is a common trigger for rejoin storms. Test the full install, verify, remove cycle in a non-production environment before running it against the production fleet.

Keep Raft commit time headroom. If commitTime is already elevated before a recovery event, the cluster has less margin before the write spike pushes it past the election timeout. Slow server disk I/O is the most common root cause of elevated commitTime; monitor disk latency proactively.

How Netdata helps

• Per-second gossip queue metrics show consul.serf.queue.Event and consul.serf.queue.Intent spiking in real time during a recovery event, rather than minutes later in aggregated telemetry.
• Cross-subsystem correlation lets you overlay gossip queue depth, catalog registration rate, Raft apply rate, and commitTime on a single timeline. The three-phase cascade (rejoin flood, anti-entropy, Raft pressure) is visible as overlapping spikes.
• Anomaly detection on consul.raft.commitTime flags the moment the write spike deviates from the cluster’s normal pattern, even before it crosses a fixed threshold. This helps distinguish a normal recovery burst from one trending toward leader thrashing.
• Leader election tracking shows consul.raft.state.leader transitions across servers, confirming whether the storm has cascaded into thrashing.
• Historical comparison lets you compare this recovery event against previous ones to determine whether the storm is converging normally or taking longer than expected.

Related guides

• Consul leader election storm: repeated elections and rolling write outages
• Consul raft commitTime high: the write pipeline is slowing down
• Consul raft lastContact rising: followers drifting toward an election
• Consul server in failed state: reading consul members during an incident
• Consul monitoring checklist: the signals every production cluster needs
• How Consul actually works in production: a mental model for operators