Consul stale Raft peer: removing a failed server from the configuration

A Consul server has crashed, been decommissioned, or been replaced, but its entry still sits in the Raft configuration as a voting peer. Serf gossip already marks the node failed or left, yet GET /v1/operator/raft/configuration still lists it as a voter. The dead peer keeps counting toward quorum even though it will never cast another vote.

This is not always an emergency. A healthy cluster with autopilot enabled usually self-heals within tens of seconds. It becomes an emergency when autopilot is disabled, when the cleanup window has not elapsed, or when a second failure tips the cluster below the now-inflated quorum threshold.

This article covers how to confirm a peer is genuinely stale rather than transiently partitioned, when to wait for autopilot, when to act manually with consul operator raft remove-peer, and what to do when the cluster has already lost quorum and remove-peer cannot run at all.

What this means

Consul tracks cluster membership in two independent subsystems: Serf gossip and the Raft configuration. They normally agree, but they can diverge. Serf can declare a node failed while Raft still considers it a voting peer, because Raft configuration changes are themselves Raft writes that require a leader and quorum to commit.

A stale peer is dangerous because quorum is calculated against the configured voter set, not the alive voter set. In a 3-server cluster you need 2 voters to elect a leader. If one server is permanently gone but still listed as a voter, you still need 2 of the remaining 2 to maintain quorum. The cluster has zero failure tolerance. The next restart, network blip, or disk stall on a surviving server drops the cluster below quorum and blocks every write.

Telltale signs:

• consul members shows the node as failed or left.
• consul operator raft list-peers still lists it as a voter, often with the Node column showing (unknown).
• The dead node’s own logs (if you still have them) show failed to join Raft or not part of configuration.
• The surviving cluster’s leader election count stays at zero, but consul.raft.peers reports a higher count than the number of reachable servers.

For the broader mental model of how Raft, Serf, and the catalog interact, see How Consul actually works in production.

flowchart TD
    A[Suspect stale peer] --> B{Listed as voter in Raft?}
    B -->|No| Z[Not stale]
    B -->|Yes| C{Reachable on port 8300?}
    C -->|Yes| D[Partitioned - do NOT remove]
    C -->|No| E{Leader exists?}
    E -->|No| F[peers.json recovery]
    E -->|Yes| G{Autopilot on?}
    G -->|Yes| H[Wait ServerStabilizationTime]
    G -->|No| I[force-leave then remove-peer]
    H --> J{Still listed?}
    J -->|Yes| I
    J -->|No| Z

Common causes

Quick checks

Run these read-only. None mutate cluster state.

# List Raft peers from the leader
consul operator raft list-peers

# Read the configuration from any server, even with no leader
consul operator raft list-peers -stale

# Compare to gossip membership
consul members -status=failed
consul members -status=left

# HTTP API view of the same Raft configuration
curl -s http://127.0.0.1:8500/v1/operator/raft/configuration | jq '.Servers[] | {Node, Address, Voter, Leader}'

# Confirm who the current leader is (empty string = no leader)
curl -s http://127.0.0.1:8500/v1/status/leader

# Check autopilot configuration on the cluster
consul operator autopilot get-config

# Inspect the suspected dead node's last known state if its host is still up
consul info | grep -A5 last_contact

A stale peer is confirmed when list-peers lists a voter whose address does not respond on the Raft port (8300 by default) and whose node is failed or left in consul members.

How to diagnose it

1. Confirm divergence. Run consul operator raft list-peers and consul members from the same server. The peer set and the alive server list should match. Any server present in the first and absent (or failed/left) in the second is a candidate stale peer.

2. Prove the node is unreachable, not just slow. Before removing anything, test the Raft RPC port directly from each surviving server.

   # From a surviving server, test port 8300 on the suspected dead peer
   nc -vz <suspect-ip> 8300
   

   If even one surviving server can reach the suspect on 8300, do not remove it. You likely have an asymmetric partition, and forcing removal risks split-brain. See Consul lost quorum: Raft peers below the majority needed to elect a leader for the partition case.

3. Check autopilot state. consul operator autopilot get-config reports CleanupDeadServers, LastContactThreshold, and ServerStabilizationTime. If CleanupDeadServers = true, autopilot is supposed to be cleaning up. Note that cleanup is triggered by a stable replacement joining, not by the death itself. If no replacement has joined, autopilot may never act, and the long-running non-autopilot reaper is the only automatic path.

4. Check whether a replacement is expected. If you run an autoscaler or a replacement workflow, a stale peer that lingers for a minute or two while the replacement bootstraps is normal. Stale for tens of minutes with no replacement activity is not.

5. Check for the 1.13+ livelock signature. If the cluster is also experiencing repeated leader elections and the supposedly dead server has come back online, you may be hitting hashicorp/raft issue #524. In that case the fix is to restart the returning node cleanly so it does not rejoin with a stale higher term, not to remove it as a peer.

6. Verify a leader exists. consul operator raft remove-peer requires an active leader. If GET /v1/status/leader returns an empty string, you are in outage-recovery territory, not stale-peer-removal territory. Skip to the peers.json procedure in the Fixes section.

Metrics and signals to monitor

Fixes

Wait for autopilot

If CleanupDeadServers = true and a replacement server has joined, autopilot removes the dead peer after LastContactThreshold has been exceeded and the replacement has been stable for ServerStabilizationTime. Defaults are 200ms and 10s respectively, but verify against your own config. This is the lowest-risk path: let the cluster converge.

If autopilot is on but no replacement is joining, autopilot has nothing to trigger on. You either need to start a replacement or move to manual removal.

consul force-leave first

If the stale node still appears in consul members output (even in failed state), the official guidance is to start with force-leave rather than reaching for remove-peer directly.

# Mark the node as gracefully gone in gossip
consul force-leave <node-name>

force-leave is safer because it goes through the normal leave path. Caveat: if the node has been reconfigured from server mode to client mode, force-leave exits 0 but does not remove it from the Raft configuration. In that case you must use remove-peer.

consul operator raft remove-peer

This is the manual override when autopilot has not cleaned up and force-leave did not remove the peer from Raft. It requires an active leader.

# Remove by Raft address (works on all Raft protocol versions)
consul operator raft remove-peer -address="<ip>:8300"

# Remove by server ID (Raft protocol 3+, preferred for protocol 3 clusters)
consul operator raft remove-peer -id="<node-id>"

The equivalent HTTP API call is DELETE /v1/operator/raft/peer?address=<ip>:8300 or ?id=<node-id>. It requires the operator:write ACL permission.

Tradeoffs and warnings:

• Removing a peer that is actually alive and partitioned can cause split-brain. Only do this after step 2 of the diagnosis (direct port test from every surviving server).
• Removing a peer reduces the quorum requirement. In a 3-server cluster with one stale peer removed, you now have 2 voters and still need 2 to maintain quorum, which leaves you with zero failure tolerance until a replacement joins. Plan to add one quickly.
• This command does not work without a leader. If quorum is already lost, use peers.json recovery.

peers.json recovery (no leader)

When the cluster has already lost quorum and remove-peer cannot run, the only path is the peers.json outage recovery procedure.

Warning: this is destructive and disruptive. It rewrites the Raft configuration from a hand-written file. Any server not listed in peers.json is permanently removed from Raft, even if it is still alive. The cluster is unavailable for writes while every surviving server is stopped and restarted. Treat this as a last resort, and only after you have confirmed which servers hold the most recent committed log entries.

# Stop Consul on each surviving server you intend to keep.
# The cluster cannot process writes while servers are stopped.
systemctl stop consul

# On each surviving server, write a peers.json with ONLY the nodes you want to keep.
# Path is <data_dir>/raft/peers.json
cat > /opt/consul/data/raft/peers.json <<'EOF'
[
  {"id": "<node-id-1>", "address": "10.0.0.1:8300"},
  {"id": "<node-id-2>", "address": "10.0.0.2:8300"}
]
EOF

# Start Consul on each surviving server
systemctl start consul

Consul reads peers.json on startup, uses it to reconstruct the Raft configuration, and deletes the file. Constraints:

• This only works on servers that previously had a valid Raft data directory. Placing peers.json on a brand-new node that has never participated in Raft causes startup failure.
• On Raft protocol 3 clusters, id must be the server’s node ID (UUID), not its IP address. Using the wrong identifier silently fails to form a quorum.
• After recovery, the cluster has a fresh peer set. Any server not listed in peers.json is gone from Raft, even if it is still alive elsewhere.

For the broader write-pathology context when commit times climb during these incidents, see Consul raft commitTime high: the write pipeline is slowing down.

Prevention

• Leave autopilot enabled. CleanupDeadServers = true (the default) is the single most effective control against stale peer accumulation. Disabling it is almost always a mistake outside of a controlled debugging session.
• Monitor consul.raft.peers against expected cluster size. The gauge reports configured voters, not alive voters. Any mismatch with the number of reachable servers is the earliest signal.
• Monitor consul.serf.lan.member.status for servers. A server in failed or left state for more than ServerStabilizationTime plus a small margin should page, regardless of whether Raft has caught up.
• Use server IDs, not IPs, in any automation. Protocol 3 clusters identify servers by UUID. IP-based removal in a cluster where addresses have changed (DHCP, k8s pod churn) silently fails.
• Do not switch server nodes to client mode without force-leaving first. A server that becomes a client while still in the Raft configuration produces a peer that force-leave will not remove.
• Treat peers.json recovery as a documented runbook, not an ad-hoc fix. Practice it in staging. The first time you reach for it should not be during a real outage.

For a full checklist of the signals a production Consul deployment should track, see Consul monitoring checklist and Consul monitoring maturity model.

How Netdata helps

• Correlate consul.raft.peers with Serf member status. Per-second resolution makes the divergence between configured voters and alive members visible within seconds, before the next failure tips the cluster into quorum loss.
• Surface leader election count spikes. A stale peer that reappears with a higher term (the 1.13+ livelock signature) shows up as a sharp increase in leadership transitions. Correlating that with the reappearance of a previously-failed server tells you to restart the returning node, not remove it.
• Track last_contact trends on each follower. Per-server views distinguish “one follower drifting” from “all followers drifting,” which separates a stale-peer problem from a leader-disk problem.
• Alert on Raft peer count mismatch. A comparison between consul.raft.peers and the count of alive server members is a direct stale-peer detector.
• Combine with disk I/O metrics on the Raft volume. Slow disk is the most common cause of leader instability during and after peer removal. Disk latency charts sit next to the Consul metrics, so the correlation is immediate.

Related guides

• How Consul actually works in production: a mental model for operators
• Consul leader election storm: repeated elections and rolling write outages
• Consul monitoring checklist: the signals every production cluster needs
• Consul monitoring maturity model: from survival to expert
• Consul “No cluster leader”: every write is failing
• Consul raft commitTime high: the write pipeline is slowing down
• Consul lost quorum: Raft peers below the majority needed to elect a leader