$ guides / consul / consul-gossip-encryption-key-mismatch ▌

Operations Guides

Consul gossip encryption key mismatch: a botched keyring rotation splits the pool

You rotated the gossip encryption key. Nodes are dropping from the member list, showing as “failed” or “suspect” to peers, and service discovery is degrading. The network is fine. The agents are running. Raft may even be stable. But the gossip pool has split.

The root cause is at the protocol layer. Consul’s Serf gossip protocol encrypts every message with a symmetric key. When nodes hold different keys, they cannot decrypt each other’s gossip messages. Each side sees the other as unresponsive, and failure detection kicks in. This looks exactly like a network partition, but pings between the affected hosts succeed.

This article covers how to detect the split with consul keyring -list, how to distinguish a key mismatch from a real network partition, and how to recover without making the partition worse.

What this means

Consul runs two independent gossip pools: LAN gossip on port 8301 (all agents within a datacenter) and WAN gossip on port 8302 (server-to-server across datacenters). Each pool has its own encryption key and its own keyring file ($DATA_DIR/serf/local.keyring for LAN, $DATA_DIR/serf/wan.keyring for WAN). When encryption is enabled, every gossip message is encrypted with AES-GCM using the pool’s primary key.

During a keyring rotation, you install a new key, promote it to primary, then remove the old key. The window between “new key installed” and “old key removed” is the danger zone. If any node misses a step, it ends up with a key that its peers no longer share, or lacks a key that its peers now use as primary.

The symptom is mutual: nodes on key A send encrypted gossip that nodes on key B cannot decrypt. Both sides mark the other as suspect, then failed. The gossip membership list diverges, anti-entropy sync stops between the affected nodes, and service discovery starts returning stale or missing results.

flowchart TD
    A[3 nodes on old key] --> B[New key installed on 2 of 3]
    B --> C[keyring -use promotes new key]
    C --> D[Node 3 lacks new key]
    D --> E[Node 3 cannot decrypt gossip from peers]
    E --> F[Peers mark Node 3 suspect then failed]
    E --> G[Node 3 marks peers suspect then failed]
    F --> H[Logical gossip partition]
    G --> H

The tell in the agent logs: Failed to join <addr>: No installed keys could decrypt the message. If you see this, the network path is fine but the keys do not match.

Common causes

Cause	What it looks like	First thing to check
Partial install of new key	Some nodes show `[n-1/total]` for the new key in `consul keyring -list`	Run `consul keyring -list` and compare key counts across the pool
Premature `-use` before all nodes have the new key	Nodes missing the new key appear failed immediately after the `-use` command	Check which nodes are missing the new key in `-list` output
Premature `-remove` of old key	Nodes that only held the old key go silent after removal	Check if the old key count dropped to zero before all nodes had the new key
LAN and WAN keys rotated out of sync	LAN pool healthy but WAN pool split, or vice versa	Check both pools in `consul keyring -list` output separately
`encrypt` config changed but agent still uses old key	Agent log shows a warning about conflicting keys; `local.keyring` holds the old key	Check `$DATA_DIR/serf/local.keyring` on the affected node
Gossip encryption disabled entirely	`consul keyring -list` returns no keys or errors	Check the `encrypt` config field and whether the agent started with encryption

Quick checks

# Primary diagnostic: list all keys and per-node counts for both pools
consul keyring -list

# Check gossip membership from a server (LAN pool)
consul members

# Check WAN gossip membership (federated clusters only)
consul members -wan

# Search agent logs for the decryption error
journalctl -u consul --since '30 min ago' | grep "No installed keys could decrypt"

# Check gossip protocol state on the local agent
consul info | grep -A 20 serf_lan

# Verify network connectivity to an affected node (rule out real partition)
ping <affected-node-ip>
nc -uv <affected-node-ip> 8301

# Check the local keyring file on a suspect node
cat $DATA_DIR/serf/local.keyring

# List the Raft peer set to check consensus is unaffected
consul operator raft list-peers

All of these are read-only except ping and nc, which are diagnostic network probes.

How to diagnose it

Run consul keyring -list from any server. The output shows each pool (LAN per datacenter, WAN) with each installed key and a [count/total] indicator. In a healthy cluster, every node in a pool shows the same key with the same count. A key showing [2/5] means three nodes are missing it.
Identify the divergence. If the LAN pool shows the new key at [4/5] while the old key is at [5/5], one node is missing the new key. If the old key shows [0/5] but a node is still failing, that node may have been isolated by a premature removal and cannot report its keyring state (keyring queries are relayed via Serf gossip, which the isolated node cannot decrypt).
Check agent logs on the failing node. The message No installed keys could decrypt the message confirms a key mismatch. Absence of this message with members still showing failed points toward a genuine network issue.
Verify network connectivity. Ping the affected node and test the gossip port (8301 for LAN, 8302 for WAN). If the network is reachable but gossip is failing, the key mismatch is confirmed.
Check the keyring file on the affected node. If the operator changed the encrypt config parameter but the agent is still using the old key, the keyring file is the source of truth. The encrypt parameter is only read on first startup; after that, the keyring file governs. The path depends on your data_dir configuration (commonly /opt/consul/data/serf/local.keyring for LAN, /opt/consul/data/serf/wan.keyring for WAN).
Compare LAN and WAN pools. If only one pool is split, you may have a rotation that succeeded on one pool but failed on the other. The two pools must be rotated independently and verified separately.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`consul keyring -list` output	Definitive view of which keys are on which nodes	Any key with `[count/total]` where count is less than total
`consul.serf.lan.members` gauge	Count of LAN members known to this agent	Sudden member count drop that does not correlate with network events
`consul.serf.wan.members` gauge	Count of WAN members known to this server	Remote DC servers disappearing from WAN pool
Agent log error rate for “No installed keys could decrypt”	Protocol-level confirmation of key mismatch	Any non-zero rate during or after a rotation
RPC errors to specific nodes	Distinguishes a gossip-only partition from broader connectivity loss	RPC succeeds to a node that gossip marks as failed
Raft leader stability	Confirms consensus is unaffected by the gossip split	If Raft also destabilizes, the problem is broader than gossip

Fixes

Nodes missing the new key (partial install)

Install the new key on all nodes before promoting it. The key must be present on every node in the pool before it can be safely set as primary.

# Install the new key (requires keyring:write ACL)
consul keyring -install <new-key>

# Verify all nodes now have both keys
consul keyring -list

Once all nodes show both keys at [total/total], proceed with -use to promote the new key.

Premature `-use` promoted the new key before install completed

If the new key is already primary on some nodes but not installed on others, the affected nodes cannot decrypt gossip. Re-install the new key on the missing nodes. The -install operation is relayed via Serf queries over the gossip mesh, so if the mesh is still partially functional, the key will propagate. If the partition is complete and the node cannot decrypt any gossip including Serf queries, you must correct the local.keyring file manually and restart the agent (see below).

# On the isolated node, or targeting it specifically
consul keyring -install <new-key>

Premature `-remove` isolated nodes on the old key

If the old key was removed before all nodes had the new key, the isolated nodes hold a key that no peer recognizes. They can no longer communicate with the cluster.

Re-install the old key to restore communication, then redo the rotation correctly:

# Re-install the old key to restore the bridge
consul keyring -install <old-key>

# Verify all nodes can communicate again
consul members

# Now install the new key on ALL nodes, verify, then use, then remove
consul keyring -install <new-key>
consul keyring -list  # verify all nodes have both keys
consul keyring -use <new-key>
consul keyring -list  # verify new key is primary everywhere
consul keyring -remove <old-key>
consul keyring -list  # verify only new key remains

consul keyring -remove on the currently active primary key is rejected with Removing the primary key is not allowed. You must -use the new key before removing the old one.

The `encrypt` config parameter is being ignored

The encrypt field in the Consul configuration is only read on first startup. After that, the key is stored in the keyring file and subsequent restarts read from it. Changing encrypt in the config after initial startup produces a warning and has no effect.

If a node is using the wrong key because of a stale keyring file, the fix is to stop the agent, remove the keyring file, correct the encrypt config, and restart.

Warning: this is destructive. Removing the keyring file and restarting means the node starts with only the key from the encrypt config. If the cluster has already rotated to a different key, the node will still be broken. Only do this when the node is already isolated or the cluster is healthy on a single key that matches the config.

# Stop the agent
systemctl stop consul

# Remove the stale keyring file
rm $DATA_DIR/serf/local.keyring

# Verify the encrypt config matches what the cluster currently uses as primary
grep encrypt /etc/consul/consul.hcl

# Restart the agent
systemctl start consul

The agent reads the encrypt parameter from config and creates a fresh keyring file. This only works if the key in the config matches what the rest of the cluster expects.

Gossip encryption is disabled

If consul keyring -list returns no keys or errors, gossip encryption may not be enabled. All gossip traffic including membership data, node metadata, and event propagation is plaintext on the network. This is a security posture issue, not a partition issue.

Prevention

Verify between every step. After -install, confirm all nodes show the new key at [total/total]. After -use, confirm the new key is primary everywhere. After -remove, confirm only one key remains.
Rotate LAN and WAN independently. Each pool has its own key. Verify each pool separately in the -list output.
Understand the encrypt parameter lifecycle. It is a first-boot setting. After that, the keyring file is the source of truth. Document this in your rotation runbook so operators do not assume a config change will propagate.
Automate with verification gates. The rotation procedure is mechanical: consul keygen, then install, verify, use, verify, remove, verify. Script it with checks that abort if any step does not reach [total/total].
Check ACL permissions before rotating. List operations require keyring:read; install, use, and remove require keyring:write. Verify the operator token has both before starting.
Monitor consul keyring -list as a scheduled check. Run it periodically and alert if any key shows a count less than total. This catches partial installs before they become partitions.

How Netdata helps

Per-second Serf member tracking catches the partition as it forms. A sudden drop in alive members that does not correlate with network errors is a strong signal of a protocol-level split, not a network outage.
Correlate member status with agent log errors. If the log collector surfaces “No installed keys could decrypt” at the same timestamp as member status changes, the diagnosis is immediate.
Gossip queue depth visibility (consul.serf.queue.Event, consul.serf.queue.Query) shows whether the gossip processing pipeline is backing up as a side effect of the partition.
RPC error tracking distinguishes a gossip-only partition from a broader connectivity issue. If RPC to a node succeeds while gossip marks it failed, the keys are the problem.
Raft stability signals (consul.raft.state.leader, consul.raft.commitTime) confirm whether the gossip split has propagated to consensus. A gossip partition that does not affect Raft is easier to recover from.