Consul lost quorum: Raft peers below the majority needed to elect a leader

A Consul cluster has lost quorum when the number of voting Raft peers drops below the majority required to elect a leader. For a 3-server cluster that means fewer than 2 voters. For a 5-server cluster, fewer than 3. With no leader, every write fails: service registrations, health-check state updates, KV writes, session creation, ACL token creation. Reads served in stale mode still return data, but that data is frozen at the moment quorum was lost.

The defining signal is an empty /v1/status/leader response combined with a voter count from /v1/operator/raft/configuration below the quorum threshold. Quorum is computed from voters only. Non-voter read replicas (an Enterprise feature) and servers mid-snapshot-join do not count. The Voter column of /v1/operator/raft/configuration is the number that matters, not the count of consul members rows.

The recovery path depends on whether a leader still exists anywhere. If at least one voter can reach a leader, you can remove a dead peer through the normal Raft API. If no leader can be elected, the only path is the manual peers.json recovery procedure, which is destructive and must be executed with care.

What this means

Raft requires a majority of voters to commit entries and to elect a leader. When voter count drops below quorum:

• No new leader can be elected, because every election fails to reach a majority.
• No follower can be promoted or re-added, because adding or removing peers requires a leader to commit the configuration change.
• All writes return “no cluster leader” errors.
• Stale reads continue to work, but they return state frozen at quorum loss.

The cluster cannot self-heal. The dead peers cannot be re-added because adding peers requires a leader, and no leader can be elected because the dead peers are gone.

The crucial distinction is between “below expected count but quorum maintained” and “below quorum.” A 5-server cluster with 4 voters has lost redundancy but is fully functional. A 5-server cluster with 2 voters is in a catastrophic failure. Count voters, not servers.

Common causes

Quick checks

# Confirm leaderless state from a server
curl -s http://127.0.0.1:8500/v1/status/leader
# Empty string ("") means no leader

# Inspect the Raft peer set, tolerating leaderless state
curl -s "http://127.0.0.1:8500/v1/operator/raft/configuration?stale" \
  | jq '.Servers[] | {Node, Address, Voter, Leader}'

# Same thing via CLI; -stale permits reading without a leader
consul operator raft list-peers -stale

# Count voters explicitly
curl -s "http://127.0.0.1:8500/v1/operator/raft/configuration?stale" \
  | jq '[.Servers[] | select(.Voter == true)] | length'

# Cross-check with gossip membership (alive does NOT mean voter)
consul members

# Check the peers telemetry gauge and election state
curl -s http://127.0.0.1:8500/v1/agent/metrics \
  | grep -E 'consul\.raft\.peers|consul\.raft\.state\.(leader|candidate)'

# Look for election failures and consensus errors
journalctl -u consul --since '30 min ago' \
  | grep -iE 'raft|election|quorum|leader'

# Verify pairwise RPC reachability between server pairs (port 8300)
for peer in 10.0.0.2 10.0.0.3; do
  nc -zv -w 2 "$peer" 8300
done

Only count voters. consul members shows nodes as alive even when they are not in the Raft configuration. Gossip and Raft are independent subsystems and routinely diverge, which is a common source of confusion during quorum incidents.

The ?stale query parameter on /v1/operator/raft/configuration is essential here. Without it, the endpoint tries to consult a leader, which by definition does not exist. ?stale lets the endpoint return the cached configuration directly.

The CLI flag is a Boolean HTTP flag: -stale (or the equivalent -stale=true); it sets AllowStale, so the command can read the local/stale Raft configuration without a leader.

How to diagnose it

1. Confirm leaderless state. Hit /v1/status/leader from each surviving server. An empty response on all of them confirms no leader exists cluster-wide. A non-empty response on one server but empty on others suggests a stale read or an asymmetric partition, not a true leaderless state.
2. Enumerate voters using ?stale. This is the only reliable way to read the configuration when the cluster has no leader. Note each entry’s Node, Address, Voter, and Leader fields.
3. Compare voter count to expected quorum. Quorum is (N / 2) + 1. For 3 servers, that is 2. For 5 servers, that is 3. If the voter count is at or above quorum but /v1/status/leader is empty, suspect a transient election, slow disk, or partition. Do not treat that as permanent quorum loss.
4. Classify each missing voter. For each voter in the expected set that is absent from the configuration, determine whether the underlying host is permanently gone (terminated, decommissioned, disk failed) or temporarily unreachable (rebooting, network blip, snapshot-join in progress). The recovery path differs sharply between the two.
5. Check Autopilot. If CleanupDeadServers is enabled (the default), Autopilot may have removed dead voters automatically. A missing or zero min_quorum setting lets Autopilot remove voters down to whatever it considers dead, including below quorum. This is one of the most common ways clusters silently lose redundancy during rolling restarts.
6. Verify pairwise connectivity. Network partitions present exactly like quorum loss. Confirm that every surviving server can reach every other surviving server on the Raft RPC port (8300) and the Serf LAN port (8301). Asymmetric partitions are common and easy to miss.
7. Decide on a recovery path using the decision tree below.

flowchart TD
  A["/v1/status/leader empty?"] -->|Yes| B["Read raft/configuration?stale"]
  B --> C{"Voter count >= quorum?"}
  C -->|Yes| D["Transient: disk, network, or election.
Do not remove peers."]
  C -->|No| E["Classify missing voters"]
  E --> F{"Any voter can reach a leader?"}
  F -->|Yes| G["consul operator raft remove-peer,
then add replacements."]
  F -->|No| H{"Partition suspected?"}
  H -->|Yes| I["Fix the network first.
Raft self-heals on heal."]
  H -->|No| J["Manual peers.json recovery.
Destructive: data loss possible."]

Metrics and signals to monitor

Fixes

If a leader still exists somewhere

If at least one voter can still reach a leader, the safe path is consul operator raft remove-peer. The change is committed through Raft itself, preserving log consistency.

# Remove a permanently-dead peer by address
consul operator raft remove-peer -address="10.0.0.3:8300"

Only remove peers you are certain are permanently gone. Premature removal of a peer that is still reachable can cause split-brain if that peer later rejoins with divergent state.

After removal, add replacement servers one at a time. Each new server must complete the snapshot-join before it is promoted to voter. Autopilot will promote it automatically once server_stabilization_time is satisfied. Do not promote manually unless you understand the consequences.

If no leader can be elected: peers.json recovery

When the cluster truly has no leader and cannot elect one, consul operator raft remove-peer does not work: the command requires a leader to commit the change. The only recovery path is the manual peers.json procedure, which rewrites the Raft configuration directly on disk.

Warning: this procedure is destructive. It implicitly commits outstanding Raft log entries, including uncommitted ones, and can cause data loss. Multiple servers being lost is the usual reason you are in this state, which means committed entries may already be incomplete. Treat this as a last resort.

Procedure:

1. Pick a single surviving server to seed the new configuration. Its Raft log becomes the authoritative state for the recovered cluster. Choose the server with the most complete log if you can determine it.
2. Stop Consul on every server. The cluster must be cold during recovery.
3. On the chosen seed server, locate the Raft data directory (commonly <data_dir>/raft/).
4. Read the node ID for each surviving voter you want in the new configuration. The ID lives in the node-id file at the root of the data directory (for example, <data_dir>/node-id), not inside the raft subdirectory.
5. Write a peers.json file into the Raft directory. The format is a JSON array of objects, one per surviving voter:

[
  {"id": "<node-id-of-seed>", "address": "10.0.0.1:8300", "non_voter": false},
  {"id": "<node-id-of-second-survivor>", "address": "10.0.0.2:8300", "non_voter": false}
]

Only include servers that have valid Raft data on disk. Including a server with an empty data directory causes Consul to refuse to start, with an error indicating it will not recover a cluster with no initial state.

6. Start Consul on the listed servers. Consul ingests peers.json, recovers the Raft configuration, and deletes the file on successful startup. Keep all listed servers stopped until the file is in place, and ensure every surviving server receives the same configuration before startup.
7. Confirm a leader exists via /v1/status/leader.
8. Once quorum is healthy, add any additional replacement servers through the normal consul join flow and let Autopilot promote them.

For Raft protocol v3 (the current default), ReadConfigJSON accepts the id, address, and optional non_voter fields shown above; non_voter defaults to false when omitted. Older protocol v2 clusters used the flat address-string format, and current v3 clusters parse only the object format.

If the cause is a network partition

Do not run remove-peer or peers.json recovery until you understand the partition geometry. A partition that heals will let Raft reconcile automatically: the minority side discards its state and resyncs from the majority. Manual peer manipulation during a transient partition can permanently split the cluster or discard committed entries.

Confirm the partition is real and persistent before treating it as permanent server loss. Asymmetric partitions are common: server A may reach B but not C, while B reaches C but not A. Test every pair, not just paths to the leader.

Prevention

• Set autopilot.min_quorum. The default of 0 lets Autopilot prune without a quorum floor. Set it to the number of voters you must retain for quorum—3 for both 3- and 5-server clusters; use a higher value only if you want a stricter maintenance floor. This single change prevents the most common cause of cascading quorum loss during rolling restarts.
• Wait for Voter: true between restarts. During rolling restarts, confirm each restarted server has rejoined as a voter in consul operator raft list-peers before restarting the next one. Autopilot can remove a temporarily-down server before it rejoins, and if the next server fails before the first re-promotes, quorum is lost.
• Use odd server counts. A 4-server cluster has the same failure tolerance as a 3-server cluster (tolerates 1 loss) but loses quorum on the second failure just as fast. Use 3 or 5.
• Spread servers across failure domains. Three servers in two availability zones guarantees that a single AZ failure can lose quorum. Spread servers so that no single AZ, rack, or power domain holds a majority.
• Keep the Raft data directory on dedicated fast storage. Slow disks cause election timeouts that look identical to quorum loss in the metrics. SSDs are not optional for Consul servers.
• Rehearse peers.json recovery. The procedure is destructive and easy to get wrong under pressure. Run it in a staging cluster at least once before you need it for real.
• Page on voter count, not just leader absence. A cluster can lose a voter and continue operating with no alerts until the next failure takes it below quorum. Alert on voter count below expected, not just on an empty /v1/status/leader.

How Netdata helps

• Per-second consul.raft.peers collection catches a voter drop the moment it happens, before the next failure pushes the cluster below quorum. A 1-minute polling interval can miss the entire window between Autopilot removal and the next failure.
• Correlate consul.raft.peers with consul.raft.state.candidate and /v1/status/leader on a single timeline. The shape of the divergence tells you whether you are dealing with permanent quorum loss, a transient election, or a network partition.
• ML anomaly detection on consul.raft.leader.lastContact and disk write latency surfaces the slow-disk pattern that precedes many quorum losses. Elections triggered by fsync latency look identical to elections triggered by server loss in raw metrics, but the preceding disk-latency anomaly distinguishes them.
• Cross-server views of consul.serf.lan.members against the Raft configuration surface the gossip/Raft divergence that signals a phantom node or a stuck snapshot-join before it becomes a quorum incident.

Related guides

• How Consul actually works in production: a mental model for operators
• Consul monitoring checklist: the signals every production cluster needs
• Consul monitoring maturity model: from survival to expert
• Consul “No cluster leader”: every write is failing