Consul leader election storm: repeated elections and rolling write outages

Your Consul cluster has a leader. Writes are still failing. The leader keeps changing. Applications see intermittent “no cluster leader” errors, watches reconnect, DNS returns stale results, and every few seconds a different server wins an election only to lose it again.

consul.raft.state.leader and consul.raft.state.candidate are counters: the former increments when a server becomes leader, and the latter when it starts an election. consul.server.isLeader is the separate 1/0 gauge.

This is a leader election storm. Unlike a clean one-time failover, the cluster never stabilizes. Each election blocks all writes for one to several seconds. When elections recur faster than the recovery window, the cluster is effectively write-unavailable while technically always having “a leader.” A naive alert on “no leader” stays silent. The real signal is recurrence: leadership transitions accumulating over time, each one a brief but real outage.

What this means

Each Raft leader election is a write outage window. During the election, no server accepts writes. Existing reads may continue in stale mode, but consistent reads and writes block. When the election completes, writes resume until the next election starts.

In a storm, this cycle repeats. The cluster oscillates between “briefly functional with a new leader” and “no leader, election in progress.” At one election every 30 seconds, most operations eventually succeed with elevated latency. At several elections per minute, most writes time out.

The root mechanism is almost always the same: something is preventing the leader from sending heartbeats to followers fast enough. Raft followers start an election if they do not receive a heartbeat within the election timeout. If the underlying cause (slow disk, CPU starvation, network latency) affects all servers equally, each new leader hits the same wall and loses leadership again.

flowchart TD
    A[Slow disk / CPU / network] --> B[Leader cannot fsync Raft log fast enough]
    B --> C[Heartbeat to followers delayed]
    C --> D[Follower election timeout fires]
    D --> E[Follower starts election]
    E --> F[New leader elected]
    F --> G[Same underlying cause persists]
    G --> B
    F --> H[Brief write outage during each election]

The operational threshold: more than 2 elections in 10 minutes outside a maintenance window indicates a systemic problem.

Common causes

Slow disk is the most common cause by a wide margin. The Raft log is persisted with fsync on every write. If the disk cannot complete fsync fast enough, heartbeats queue behind log writes. EBS gp2 volumes with exhausted burst credits, network-attached storage, and spinning disks are the usual suspects.

Quick checks

Run these read-only checks to confirm the storm and identify the current leader.

# Confirm current leader identity (empty string means no leader)
curl -s http://127.0.0.1:8500/v1/status/leader

# Check leadership state on this server
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep "raft.state.leader"

# Watch for election log lines in real time
journalctl -u consul -f | grep -E "entering leader state|heartbeat timeout reached, starting election"

# Full Raft peer configuration
curl -s http://127.0.0.1:8500/v1/operator/raft/configuration | python3 -m json.tool

# Last contact times (reported on the leader about each follower)
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep "raft.leader.lastContact"

# Raft commit time (leader only)
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep "raft.commitTime"

# Disk I/O on the Raft data volume
iostat -x 1 5

# GC pause duration
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep "gc_pause"

Do not restart servers during an active storm. Restarting removes one voter from the peer set and may push the cluster below quorum, converting a storm into total unavailability.

How to diagnose it

1. Confirm recurrence. Check whether consul.raft.state.leader is changing repeatedly. A single transition is a failover. Multiple transitions within 10 minutes is a storm. Cross-check with server logs for repeated “entering leader state” lines.

2. Identify the current leader. Query /v1/status/leader. Note which server holds leadership and whether it changes between checks.

3. Check disk I/O on the leader. Run iostat -x 1 5 on the server that is currently leader. Look at await (write latency) and %util. If await is above 10ms sustained, or %util is pinned at 100%, disk I/O is the likely cause. Also check whether you are on EBS gp2 with exhausted burst credits.

4. Check lastContact for all followers. The metric is leader-side and reports contact with followers. If all followers show rising lastContact before each election, the leader is struggling. If only one follower shows it, the problem is the network path to that specific follower.

5. Check CPU and GC. If disk I/O is healthy, check whether the server is CPU-starved (pinned at 100%) or experiencing GC pauses. Both prevent the Raft goroutine from sending heartbeats on time.

6. Check pairwise network connectivity. Run consul info on each server and compare the Serf LAN sections. Asymmetric partitions, where server A sees B but not C, are a common cause of repeated elections.

7. Correlate with recent changes. Did the storm start after a deploy, a scaling event, a Consul upgrade, or a storage change? Version-specific regressions in Raft behavior exist and should be ruled out.

Metrics and signals to monitor

The consul.raft.commitTime signal is only reported on the leader. When leadership changes, this metric disappears and reappears on a different server. Your monitoring must track which server is leader to interpret it correctly.

Fixes

Slow disk I/O

This is the most common cause. Investigate it first.

Immediate: Identify and reduce write load. If a health check thundering herd or runaway registration loop is generating excessive Raft writes, shedding that load reduces fsync pressure. Check the consul.catalog.register latency-timer sample rate for churn.

Short-term: If the leader’s disk is slow due to burst credit exhaustion on EBS gp2, credits will refill over time but the storm will continue until they do. You cannot wait this out during an active incident. Migrate the Raft data directory to faster storage: provisioned IOPS volumes, local NVMe, or a dedicated SSD volume.

Permanent: Never colocate the Raft data directory with other I/O-heavy workloads. HashiCorp production guidance calls for dedicated SSD storage with sufficient IOPS headroom. Rotational disks and network-attached storage are not suitable for Raft persistence.

CPU starvation

If the server is containerized, check CPU limits. Consul servers under-provisioned on CPU cannot process Raft, gossip, and RPC concurrently. The Raft goroutine gets starved and misses heartbeat deadlines.

Fix: Increase CPU allocation. HashiCorp’s production starting point is 8–16 CPU cores. If you cannot resize immediately, reduce non-critical load by disabling non-essential health checks or reducing DNS recursion.

Network latency between servers

Consul servers must maintain low-latency connectivity. The election timeout assumes sub-second round trips between servers. If inter-server latency approaches the election timeout, followers will repeatedly trigger elections.

Fix: Verify network connectivity between all server pairs, not just to and from the leader. Asymmetric partitions are common. HashiCorp’s production starting point is average RTT below 50ms and 99th-percentile RTT below 100ms. Verify inter-server RTT is within tolerance, especially if servers span availability zones.

Go GC pauses on large heaps

Consul servers with multi-GB heaps can experience stop-the-world GC pauses long enough to trigger election timeouts. The consul.runtime.gc_pause_ns metric will show spikes correlating with elections.

Fix: Reduce heap pressure. Identify what is consuming memory (large KV values, catalog bloat, watch accumulation) and address it. Tuning GOGC upward reduces GC frequency at the cost of higher peak memory. This is a stopgap, not a solution.

Version-specific regressions

Consul 1.13.x introduced a regression where evicting a single server node, even a non-leader follower, caused the cluster to cycle through leaders for 3-15 minutes before stabilizing. This was not present in 1.12.x where recovery took 2-10 seconds. The regression was reported against 1.13.1 through 1.14.0-beta1.

If your storm started immediately after an upgrade and the recovery behavior matches this pattern, check your Consul version against the known regression.

The regression was fixed in Consul 1.13.4; the corresponding Raft update was already present in Consul 1.14.0 (issue 15231).

Asymmetric partitions

If only some followers consistently lose contact with the leader, you may have an asymmetric network partition. Server A can reach B, B can reach C, but A cannot reach C. This causes elections that succeed from one perspective but fail from another.

Fix: Run consul members from each server and compare the views. Use mtr or ping to test all server pairs in both directions. Firewall changes, security group updates, and routing table drift are common causes.

Prevention

Monitor disk write latency as a page-level signal. This is the single most important preventive measure. Disk write latency (await) on the Raft data volume should stay below 10ms. Alert before it approaches the election timeout. Most storms are preventable if disk latency is caught early.

Alert on election count, not just leader absence. A naive “is there a leader” check will not catch a storm. Alert when leadership transitions exceed 2 in 10 minutes outside a maintenance window.

Suppress expected elections during rolling upgrades. During rolling server restarts, some elections are expected. Your alerting should account for maintenance windows or use suppression during planned operations.

Run 3 or 5 servers, never an even number. Even server counts can split quorum evenly during a partition (2-2 with 4 servers), preventing any leader election from succeeding. Always use odd numbers of voting servers.

Track consul.raft.commitTime trends over days. Commit time creeping upward is the early warning before a storm. If commit time p99 trends toward a significant fraction of the election timeout, investigate before it becomes an incident.

Validate Consul upgrades in staging. Version-specific regressions in Raft behavior exist. Test upgrades with realistic load before applying to production. Monitor election behavior specifically during and after upgrades.

How Netdata helps

• Per-second metric resolution captures election transitions that coarser polling misses. A storm with elections every few seconds is visible as a steep staircase on consul.raft.state.leader rather than a single blurred sample.
• Correlate consul.raft.leader.lastContact with disk latency on the same dashboard. When both spike together, the root cause is disk I/O. When lastContact spikes without disk latency, the cause is network or CPU.
• Anomaly detection on consul.raft.commitTime surfaces gradual degradation before it crosses a static threshold.
• Per-container CPU and memory metrics identify CPU starvation or GC pressure on Consul servers running in containers, where the problem is invisible to host-level metrics.

Related guides

• How Consul actually works in production: a mental model for operators
• Consul monitoring checklist: the signals every production cluster needs
• Consul monitoring maturity model: from survival to expert
• Consul “No cluster leader”: every write is failing