$ guides / consul / consul-slow-disk-raft-timeouts ▌

Operations Guides

Consul slow disk causing Raft timeouts: the number-one cause of leader instability

Your Consul cluster is cycling through leaders. Writes fail with “no cluster leader” errors. The cluster technically has a leader at any given moment, but constant re-elections make writes effectively unavailable. The logs show [WARN] raft: heartbeat timeout reached, starting election repeating across all servers.

The root cause is almost always the same: slow disk I/O on the volume backing the Raft data directory. Consul’s Raft log store performs an fsync on every appended entry. When the disk cannot complete those syncs quickly enough, heartbeats and commits slow down. Followers miss their heartbeat window and start new elections. The new leader lands on the same slow storage. The cycle repeats.

The fix is dedicated, fast storage for the Raft data directory, and proactive monitoring of disk write latency (await) at PAGE severity before it reaches the election threshold.

What this means

Every write to Consul (service registration, health check update, KV write, session creation) becomes a Raft log entry that the leader must append, fsync to disk, replicate to a quorum of followers, and apply to the in-memory state machine. The fsync is on the critical path of every write.

When the disk behind the data_dir volume gets slow, the Raft pipeline backs up. The leader cannot commit entries or send heartbeats fast enough. Followers cross the election timeout threshold and call a new election. Each election blocks all writes for the duration of the vote. If elections happen faster than the cluster can stabilize, the result is sustained write unavailability.

flowchart TD
    A[Slow disk on Raft volume] --> B[fsync latency spikes]
    B --> C[commitTime rises]
    C --> D[Heartbeats delayed]
    D --> E[Follower election timers fire]
    E --> F[New election, writes blocked]
    F --> G[New leader, same slow disk]
    G --> C

This is a cliff-edge failure with a warning ramp. Raft commit time increases gradually as disk latency worsens. But the transition from “slow but functional” to “leader elections and write outages” is instantaneous once commit time crosses the heartbeat timeout.

Common causes

Cause	What it looks like	First thing to check
EBS gp2 burst credit exhaustion	Sudden latency cliff with no change in write volume. Works fine for hours, then collapses.	Check burst credit balance on the EBS volume.
Shared volume (NFS, shared block storage)	Intermittent latency spikes correlating with other tenants’ activity. Works fine in staging, fails in production.	Identify what else is writing to the same volume.
Spinning disk (HDD)	Consistently elevated commitTime. Elections are frequent but not always catastrophic.	Check disk type: `lsblk -d -o NAME,ROTA,SIZE,MODEL`
Noisy neighbor on the same volume	Application logs, another database, or monitoring data competing for disk I/O on the Raft volume. Spikes correlate with log rotation or batch jobs.	`pidstat -d 1` to see which processes are doing I/O.
Snapshot creation competing for I/O	Periodic commitTime spikes that correlate with snapshot intervals. Each spike may trigger an election.	Check Raft snapshot timing and snapshot size.

Quick checks

Run these read-only checks on the server that is currently the leader, or the server you suspect is causing the problem.

# Identify the current leader
curl -s http://127.0.0.1:8500/v1/status/leader

# Check Raft commit-time samples (only reported on the leader)
curl -s http://127.0.0.1:8500/v1/agent/metrics | jq '.Samples[] | select(.Name | contains("raft.commitTime"))'

# Check follower last-contact samples
curl -s http://127.0.0.1:8500/v1/agent/metrics | jq '.Samples[] | select(.Name | contains("raft.leader.lastContact"))'

# Check how many times this node has started an election
curl -s http://127.0.0.1:8500/v1/agent/metrics | jq '.Counters[] | select(.Name | contains("raft.state.candidate"))'

# Disk I/O latency on the Raft volume (find the device first)
iostat -x 1 5

# Confirm Consul is the process driving I/O
pidstat -d 1

# Raft data directory size and contents
du -sh /opt/consul/data/raft/
ls -lh /opt/consul/data/raft/

# Full Raft peer list and voter status
consul operator raft list-peers

commitTime and lastContact are sample summaries; candidate is a counter. Check consul.raft.applied_index/last_index gauges separately when you need index progression.

How to diagnose it

Identify the current leader. Run curl -s http://127.0.0.1:8500/v1/status/leader. If the response is empty, there is no leader and you are in an active outage. If the address keeps changing between checks, you are in a leader thrashing loop.
Check disk I/O latency on the leader’s server. Run iostat -x 1 5 and look at the await column for the device backing the data_dir. Sustained values above 10ms are a problem. Values in the hundreds of milliseconds will cause Raft timeouts. Check %util for additional context on device saturation.
Confirm Consul is the process causing I/O. Run pidstat -d 1 to see per-process disk I/O. If Consul is the top consumer, the Raft pipeline is the workload. If another process is saturating the disk, you have a noisy neighbor problem on a shared volume.
Correlate commitTime with disk latency. Check consul.raft.commitTime from the metrics endpoint. If commitTime spikes track with disk await spikes, disk I/O is the bottleneck. If commitTime is high but disk await is low, the problem may be network replication latency, FSM apply bottleneck, or CPU and GC pressure.
Check follower last contact times. If consul.raft.leader.lastContact is trending upward on all followers simultaneously, the leader is struggling. If only one follower shows high last contact, that follower has a local disk or network problem.
Rule out non-disk causes. Check Go GC pause times (consul.runtime.gc_pause_ns or equivalent in your Consul version). A large heap with high allocation rates can cause multi-millisecond stop-the-world pauses that look like network or disk problems. Check CPU utilization. CPU starvation in containerized deployments can slow Raft processing without any disk involvement.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`consul.raft.commitTime`	The single best indicator of Raft write pipeline health. Captures disk write latency, network replication, and FSM apply time in one number.	Sustained above 100ms is degraded. Above 500ms risks heartbeat timeouts.
`consul.raft.leader.lastContact`	Measures how recently each follower heard from the leader. Gives a predictive window before an election triggers.	Trending upward on all followers indicates leader struggle. Approaching election timeout is imminent election.
`consul.raft.state.candidate` (counter)	Counts how many times this node has started an election. Each increment correlates with a write outage window.	More than 2 increments per 10 minutes outside maintenance is a systemic problem.
Disk write latency (await)	The root cause signal. Raft log writes are fsync-heavy. This is the leading indicator.	Sustained above 10ms. PAGE at this threshold before elections begin.
`%util`	Context for disk saturation. High `%util` with low throughput indicates a slow device.	Consistently in the 90s on a dedicated Raft volume.
GC pause time	Stop-the-world pauses affect Raft timing. Can mimic disk latency symptoms.	Pauses above 100ms on servers with large heaps.
Raft snapshot size	Large snapshots compete for disk I/O during creation and take longer to restore on restart. Can trigger elections during snapshot creation.	Growing over weeks indicates catalog or KV growth.

Fixes

Migrate to dedicated, fast storage

This is the permanent fix. HashiCorp’s long-standing guidance is explicit: dedicated SSD, nothing else on the volume. Never colocate Raft data with application logs, another database, or monitoring data.

On AWS, this means provisioned IOPS volumes (io1, io2) or gp3 with sufficient baseline IOPS. Do not rely on gp2 burst credits for a production Raft volume. The burst exhaustion cliff gives no warning in Consul metrics until elections begin.

On other platforms, the equivalent is locally-attached NVMe or SSD with no other workload sharing the device.

Migration steps (rolling, one server at a time, maintaining quorum throughout):

Provision the new volume on one server.
Stop the Consul process gracefully (systemctl stop consul or equivalent). Do not run consul leave; that deregisters the node from the cluster.
Copy the contents of data_dir to the new volume preserving all attributes: rsync -aHAX /opt/consul/data/ /new-volume/consul/data/
Update data_dir in the Consul configuration to point to the new mount.
Start Consul and verify it rejoins the cluster and catches up: consul members and consul operator raft list-peers.
Confirm quorum is healthy before proceeding to the next server.
Repeat for each server.

Reduce write volume

If you cannot immediately migrate storage, reduce the write load on the Raft pipeline. Identify what is generating excessive catalog churn:

Flapping health checks oscillating between passing and critical. Each transition is a Raft write.
Excessive service registration and deregistration from deployment churn or buggy automation.
KV write storms from applications treating Consul KV as a high-throughput database.
Anti-entropy sync storms after mass node recovery.

Temporarily disabling non-critical health checks or increasing check intervals reduces Raft write volume and may stabilize the cluster enough to plan a storage migration.

Move the Raft data directory off a shared volume

If the Raft data directory shares a volume with other workloads (logs, another database, monitoring data), isolate it. The Raft write pattern is fsync-heavy and latency-sensitive. Any other I/O workload on the same device introduces unpredictable latency spikes that translate directly into Raft timeouts.

If commitTime spikes correlate with snapshot creation intervals, the snapshot is competing with Raft log writes for disk I/O. Options:

Move to faster storage (permanent fix).
Tune snapshot parameters. raft_snapshot_threshold and raft_snapshot_interval control when and how often snapshot checks/writes occur; both have been reloadable with consul reload or SIGHUP since Consul 1.10.0. Increasing the threshold/interval reduces snapshot I/O but increases log accumulation and recovery replay.
Track snapshot size over time. Growing snapshots indicate catalog or KV growth, which makes snapshot-I/O contention worse.

Prevention

Monitor disk write latency at PAGE severity. This is the single most important preventive measure. Track await on the device backing the data_dir volume. Alert at sustained values above 10ms. The goal is to catch disk degradation before it reaches the election threshold.

Do not monitor throughput or utilization as the primary disk signal. A disk can show low throughput and low utilization while still having high per-operation latency. A single slow fsync is enough to stall a Raft heartbeat. Await is the metric that matters.

Dedicate the Raft volume. Nothing else writes to it. Not logs, not another database, not monitoring data. This eliminates the noisy neighbor failure mode entirely.

Track commitTime trends over time. Even if commitTime is below the danger zone, a steady upward trend over days or weeks indicates the disk is approaching its capacity ceiling. Project the trend forward. If the linear extrapolation crosses 50% of the election timeout within two weeks, act immediately.

Track snapshot size. Growing snapshot size means catalog or KV growth. Larger snapshots take longer to create, competing for disk I/O, and longer to restore on server restart. Both can trigger elections.

Size for write bursts, not averages. Disk I/O is not uniform. Health check storms, mass node recovery, and deployment bursts generate write spikes several times the steady-state rate. Provision disk IOPS capacity for peak burst, not average load.

How Netdata helps

Netdata collects per-second disk latency metrics (await, %util) on every device, giving you the resolution to catch the sub-minute latency spikes that trigger Raft timeouts.
The Consul integration surfaces consul.raft.commitTime, consul.raft.leader.lastContact, and election counters alongside system-level disk metrics in the same dashboard, so you can correlate a disk await spike with a commitTime spike and a leader transition without switching tools.
Per-second granularity matters for Raft because the failure mode is a cliff edge: commit time is fine until it crosses the heartbeat timeout, and the transition is instantaneous. Coarse polling intervals can miss the entire ramp.
Disk I/O metrics are collected automatically on every node with zero configuration, so the leading indicator (await on the Raft volume) is available from the moment Netdata is installed.