Consul "No cluster leader": every write is failing

When Consul cannot elect a Raft leader, every write fails. The error surfaces to clients as rpc error making call: No cluster leader, sometimes as no known leader. Service registrations hang, KV writes return errors, sessions and ACL tokens cannot be created, and health check state stops reaching the catalog.

Only stale reads continue to work: stale-mode API queries and DNS lookups (which default to allow_stale) return whatever the responding server last committed. Default and consistent reads fail because they require a leader, which can surface differently to callers depending on their read mode.

A brief leaderless window during a rolling restart is expected. Healthy failover takes roughly 1-5 seconds. If /v1/status/leader has been empty for more than 15-30 seconds, quorum is almost certainly lost and you are in an active incident. Page on sustained absence, not on the 2-second blip during a planned deploy.

The rest of this article covers distinguishing true quorum loss from leader churn, isolating the cause (network, disk, GC, corruption), and recovering safely without split-brain.

What this means

Consul servers run Raft. Only the leader accepts Apply RPCs (writes); followers forward writes and serve reads. A leader can only be elected if a majority of voting peers are mutually reachable and able to durably append log entries. Quorum size is (N/2)+1: a 3-server cluster needs 2 voters, a 5-server cluster needs 3.

When no leader exists, the cluster cannot commit. Existing catalog state is intact on disk, but every mutation path is blocked. Stale reads and DNS may keep working, which can mask the severity of the outage: callers assume the cluster is healthy because discovery is returning results.

Two distinct failure shapes share this symptom. The first is true quorum loss: too few voters are reachable to elect anyone. The second is leader churn (thrashing): leaders keep getting elected and immediately deposed, so consul.raft.state.leader increments but no leader survives long enough to commit. Both produce “No cluster leader” errors to clients, but the diagnostics and fixes differ.

Common causes

Quick checks

Run these read-only. They are safe.

# Leader identity from THIS server (may be stale if this server lost the election)
curl -s http://127.0.0.1:8500/v1/status/leader

# Cross-check from every server. A losing server can report a stale leader address.
for s in server1 server2 server3; do
  echo -n "$s: "; curl -s --max-time 2 http://$s:8500/v1/status/leader; echo
done

# Voter count and leader in Raft configuration.
# ?stale is REQUIRED when there is no leader: the non-stale call itself fails
# with "No cluster leader" because it requires a leader to answer.
curl -s "http://127.0.0.1:8500/v1/operator/raft/configuration?stale" \
  | jq '.Servers[] | {Node, Address, Voter, Leader}'

# Same thing via CLI; -stale=true is the equivalent flag
consul operator raft list-peers -stale=true

# Gossip view of server membership, independent of Raft
consul members

# Raft last contact (followers only) and commit time (leader only)
curl -s http://127.0.0.1:8500/v1/agent/metrics \
  | grep -E "consul.raft.leader.lastContact|consul.raft.commitTime|consul.raft.state.candidate"

# Disk latency on the Raft data directory volume
iostat -x 1 5

# Recent election activity
journalctl -u consul --since "10 minutes ago" \
  | grep -E "starting election|entering leader state|Failed to make RequestVote RPC"

One gotcha worth flagging twice: /v1/status/leader can return a stale leader address from a server that has not yet realized the leader stepped down. Always cross-check from at least two servers, and always reach for /v1/operator/raft/configuration?stale (or the CLI’s -stale=true) because the non-stale variant of those calls requires a leader and will fail with the exact error you are debugging.

How to diagnose it

1. Confirm the symptom is sustained, not a deploy blip. Empty leader for 1-5s during a rolling restart is expected; sustained leaderlessness beyond 15-30s is an incident.
2. Enumerate voters with consul operator raft list-peers -stale=true. Count Voter: true rows.
3. Compare voter count to expected cluster size and to quorum (N/2)+1. If voters are below quorum, you have lost servers. Recovery is to bring them back, not to manually edit the peer set.
4. Check pairwise connectivity between every server pair, not just to the leader. Asymmetric partitions (A sees B, B does not see A) are common and easy to miss.
5. If a leader is briefly elected, check its disk I/O latency, CPU, and Go GC pause. consul.raft.commitTime near the heartbeat timeout is the smoking gun for slow disk.
6. If elections keep happening (consul.raft.state.leader counter rising), you have leader churn, not steady quorum loss. The fix targets the root cause, not the election count.

flowchart TD
    A["/v1/status/leader empty for >15-30s"] --> B["list-peers -stale=true
count voters"]
    B --> C{"Voters >= quorum?"}
    C -- "No" --> D["Quorum loss
bring back failed servers"]
    C -- "Yes" --> E["consul members
on each server"]
    E --> F{"All servers
reachable pairwise?"}
    F -- "No" --> G["Network partition
fix L3/L4"]
    F -- "Yes" --> H["Check commitTime
and disk await on leader"]
    H --> I{"commitTime near
heartbeat timeout?"}
    I -- "Yes" --> J["Slow disk I/O"]
    I -- "No" --> K["Check GC pause
and CPU on leader"]

Do not jump to peer-set surgery. Removing a peer lowers the quorum size and is hard to reverse cleanly. Only consider consul operator raft remove-peer when you are certain the peer is permanently gone.

Metrics and signals to monitor

The exact default election_timeout varies by version and is not consistent across the source material; what matters operationally is your configured value and the leader-lease multiplier.

Fixes

Quorum loss: bring back failed servers

If voters are below quorum because servers are actually down or partitioned, the only safe fix is to restore them. In a 3-server cluster that has lost two of three, bringing back one restores quorum (2 of 3 voters reachable).

If the lost servers cannot be recovered (hardware gone, data corrupt) and the cluster is permanently below quorum, the last-resort recovery is the peers.json procedure. This is destructive and can cause split-brain if run on more than one server or on the wrong server. Read the official HashiCorp recovery guide before doing it.

With raft protocol v3 (default since Consul 1.0), peers.json is a JSON array of objects with id, address, and non_voter fields, where id must match the server’s persisted node ID. The older v2 format (a flat array of ip:port strings) does not work on v3 clusters.

A common operator error: placing peers.json on a server with an empty data directory causes Consul to refuse startup with “refused to recover cluster with no initial state, this is probably an operator error”. Only use peers.json on servers that still have their existing data directory.

Network partition

Resolve at the network layer. Do not manipulate the Raft peer set to “work around” a partition. When the partition heals, Raft reconciles automatically: the minority side discards its divergent state and follows the majority. Any writes accepted by a minority side during the partition are lost by design.

Slow disk I/O

The most common root cause of leader churn. The leader cannot fsync Raft log entries fast enough, heartbeats stall, followers trigger elections.

• Migrate the Raft data directory to a dedicated SSD. Never colocate it with logs or another database.
• On AWS, do not run the Raft volume on gp2 with exhausted burst credits. Use io1/io2 with provisioned IOPS, or gp3 with provisioned IOPS.
• Reduce write load: find the source of catalog churn (flapping health checks, runaway service registration, KV write storms) and shed it.

GC pauses on the leader

Large Go heaps with high allocation rates cause stop-the-world pauses that exceed the election timeout. Increase GOGC (default 100), reduce heap pressure, or raise election_timeout as a stopgap while you address the underlying cause.

Manual leader stability

consul operator raft transfer-leader can move leadership to a known-healthy server when one server keeps winning and losing elections. This is a mitigation, not a fix.

Prevention

• Run an odd number of servers. 3 or 5, never 4. A 4-server cluster has the same fault tolerance as 3 (quorum size 3, tolerates one failure) but is more likely to lose quorum during a partition.
• Dedicated SSD for the Raft data directory. Nothing else on the volume. Monitor disk write latency (await), not just utilization.
• Set FD limits high. Consul documentation recommends at least 65536 for servers. Default OS ulimits (1024) are catastrophically low.
• Spread servers across failure domains. A 3-server cluster split across only 2 AZs loses quorum when the AZ holding 2 servers fails its network. Use 3 AZs for a 3-server cluster.
• Alert on the consul.raft.state.leader counter, not on /v1/status/leader liveness alone. Leader existence blips during every rolling restart; sustained absence or rising election count is the real signal.
• Pairwise server connectivity checks. Asymmetric partitions are missed by leader-only checks.
• Track consul.raft.commitTime as a leading indicator. It degrades before elections start.

Monitoring and correlation with Netdata

• Per-second consul.raft.state.leader, consul.raft.state.candidate, consul.raft.leader.lastContact, and consul.raft.commitTime. Per-second sampling matters because Raft elections can complete between the 10-15s scrape intervals used by other tools, hiding the actual transition.
• Correlate Raft metrics with disk await on the same host. The single highest-value correlation for leader churn is commitTime rising alongside disk write latency on the leader’s Raft volume. Surfacing both on one timeline shortens root cause from “Consul is broken” to “the leader’s disk is saturated”.
• Cross-server view of consul.raft.lastLog.index and Serf member status. Divergence between servers and gossip-vs-Raft mismatch are the signals most teams miss until an incident.
• Anomaly detection on consul.runtime.total_gc_pause_ns and goroutine count. These leak slowly and only matter at election time; anomaly detection catches the trend before the threshold alert fires.
• Anomaly-aware alerting on consul.client.rpc.failed on client agents. Silent catalog staleness, where clients are alive in gossip but unable to push to servers, is the most underdiagnosed precursor to a leaderless incident.

Related guides

• Consul leader election storm: repeated elections and rolling write outages
• Consul lost quorum: Raft peers below the majority needed to elect a leader
• Consul raft commitTime high: the write pipeline is slowing down
• Consul raft lastContact rising: followers drifting toward an election
• Consul leader stable but commits stalled: writes silently failing
• Consul stale Raft peer: removing a failed server from the configuration
• Consul Raft log divergence: catching a corrupt follower before it wins an election
• Consul monitoring checklist: the signals every production cluster needs
• How Consul actually works in production: a mental model for operators
• Consul monitoring maturity model: from survival to expert
• Consul server in failed state: reading consul members during an incident
• Consul gossip flapping: nodes oscillating between alive, suspect, and failed