Consul on EBS: burst-credit exhaustion and the sudden latency cliff

Consul is stable. Raft commit times under 10ms, leadership steady, discovery in single-digit milliseconds. Then, with no deploy, no config change, and no traffic spike, writes start timing out. consul.raft.commitTime jumps from 5ms to 500ms. Followers report lastContact climbing toward the election timeout. Within seconds, the cluster elects a new leader, then another, then another. Every write fails with “no cluster leader.”

The root cause is under Consul, in the block storage layer. AWS gp2 EBS volumes accumulate I/O burst credits when idle and spend them to sustain burst performance up to 3000 IOPS. When credits deplete, the volume drops to its baseline rate of 3 IOPS per GiB of volume size. A 100 GB gp2 volume falls from 3000 IOPS to 300 IOPS. Consul has no metric for burst credit depletion. The disk simply gets slow, and Raft, which fsyncs every log entry to disk before acknowledging it, hits a wall.

This article covers how to recognize the pattern, correlate Consul signals with CloudWatch storage metrics, size the Raft volume correctly, and prevent recurrence.

What this means

Raft persists every log entry to disk with an fsync before the leader acknowledges the write. Durability requires that committed entries survive a crash. The consequence: Raft write throughput and stability are bounded by disk I/O latency. When disk latency spikes, commit time spikes. When commit time approaches the election timeout, the leader cannot send heartbeats fast enough. Followers interpret the silence as leader failure and start an election.

On gp2 volumes, the latency cliff is not caused by wear, failure, or contention. It is caused by the burst credit balance reaching zero. AWS provisions gp2 volumes with an initial credit balance and refills it at 3 IOPS per GiB per second. Volumes smaller than 1 TiB can burst to 3000 IOPS by spending credits. Once exhausted, the volume is throttled to baseline. The transition is abrupt, with no gradual degradation in the disk’s own metrics, and Consul has no visibility into EBS credit state.

flowchart TD
    A["gp2 burst credits deplete"] --> B["Disk I/O drops to baseline"]
    B --> C["Raft fsync latency spikes"]
    C --> D["commitTime cliffs to hundreds of ms"]
    D --> E["Leader cannot send heartbeats in time"]
    E --> F["Followers start elections"]
    F --> G["Write outage during each election"]
    G --> H["New leader on same slow volume"]
    H --> E

The cycle repeats because the new leader typically runs on infrastructure with the same storage profile. If all three or five servers use gp2 volumes of the same size, they may exhaust credits at similar times, and the problem propagates across the cluster.

Common causes

Quick checks

Read-only and safe to run during an incident.

# Check if a leader exists and who it is
curl -s http://127.0.0.1:8500/v1/status/leader

# Check Raft peer configuration and current leader
curl -s http://127.0.0.1:8500/v1/operator/raft/configuration | jq '.Servers[] | {Node, Address, Leader, Voter}'

# Check commit time on the leader (run on the leader node)
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -A5 'raft.commitTime'

# Check last contact on followers (run on each follower)
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -A5 'raft.leader.lastContact'

# Check disk I/O latency on the Raft volume (await column, milliseconds)
iostat -x 1 5

# Check what filesystem backs the Raft data directory
df /opt/consul/data/raft/
# Adjust path if your data_dir differs

# Check block devices and mount points
lsblk -o NAME,SIZE,TYPE,MOUNTPOINT

# Check leadership state (gauge: 1 if this node is leader, 0 otherwise)
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -A5 'raft.state.leader'

If iostat shows await jumping from single digits to hundreds of milliseconds, and CloudWatch BurstBalance is near zero, the diagnosis is confirmed.

How to diagnose it

1. Identify the current leader. Run curl -s http://127.0.0.1:8500/v1/status/leader. The response is the leader’s address, or empty if there is no leader. If empty, the cluster has lost quorum or is mid-election.

2. Check commit time on the leader. SSH to the leader and query the metrics endpoint. A healthy cluster shows consul.raft.commitTime mean well under 50ms. Above 100ms is degraded. Above 500ms means elections are imminent or already happening.

3. Check disk latency on the leader. Run iostat -x 1 5 and focus on the device backing the Raft data directory. The await column is average I/O latency in milliseconds. Above 10ms is concerning. Above 100ms is a storage problem Raft cannot tolerate.

4. Correlate with CloudWatch. Check these metrics on the EBS volume backing the Raft data directory:

   • BurstBalance: percentage of burst credits remaining. Near 0% confirms gp2 throttling.
   • VolumeReadOps and VolumeWriteOps: actual IOPS delivered. Compare against baseline (3 IOPS per GiB for gp2).
   • VolumeTotalReadTime and VolumeTotalWriteTime: per-operation latency from the EBS side.

5. Check the EC2 instance-level burst balance. Even if the EBS volume has credits, some EC2 instance types enforce a separate burst budget for EBS I/O. Check EBSIOBalance% and EBSByteBalance%.

6. Verify the Raft data directory is on a dedicated volume. If the Raft directory shares a volume with application logs, other databases, or the OS, contention from those workloads can exhaust burst credits faster than Consul alone would. Consul’s disk I/O profile is latency-sensitive and bursty during snapshots; sharing the volume multiplies the problem.

7. Check the pattern across all servers. If all servers use identically sized gp2 volumes, they may exhaust credits at the same time. Check BurstBalance on every server’s volume, not just the current leader’s. The next leader could be on a volume about to hit the same wall.

Metrics and signals to monitor

Fixes

Immediate: reduce write pressure

If you cannot change storage immediately, reduce the write load on Raft to slow credit consumption:

• Identify and stop sources of excessive catalog churn: flapping health checks, runaway service registration loops, misconfigured watches generating constant updates.
• Temporarily increase health check intervals to reduce state transitions.
• Disable non-critical health checks that are generating frequent catalog writes.
• Stop any application using Consul KV as a high-throughput data store.

This buys time. It does not solve the storage problem.

Short-term: modify the volume in place

AWS allows modifying gp2 volumes to gp3 without detaching them on most current instance types. A modification takes effect within minutes to hours depending on size. gp3 volumes do not use burst credits: they sustain their provisioned IOPS and throughput indefinitely.

# Modify a gp2 volume to gp3 (AWS CLI)
# Replace vol-xxxxxxxx with your volume ID
aws ec2 modify-volume --volume-id vol-xxxxxxxx \
  --volume-type gp3 --iops 3000 --throughput 125

# Check modification progress
aws ec2 describe-volumes-modifications --volume-ids vol-xxxxxxxx

The 3000 IOPS and 125 MB/s values are the gp3 baseline included at no extra cost over the volume price. HashiCorp’s production recommendation is higher. See sizing guidance below.

Correct sizing: provision for sustained Consul write load

HashiCorp recommends gp3 volumes with at least 10000 IOPS and 250 MB/s throughput for Consul server data directories in production. This is the floor for a production cluster, not a generous allocation. The Raft write pipeline is fsync-bound, and every log entry, every snapshot, and every log compaction operation competes for disk I/O on the same volume.

For clusters with large catalogs (thousands of services, frequent health check updates) or frequent snapshot operations, consider io2 volumes. io2 provides provisioned IOPS with sub-millisecond latency and no burst behavior. gp3 can exhibit higher latency variation than io2 at low queue depths, which matters for Raft’s synchronous single-threaded write pattern.

Sizing rules that prevent recurrence:

• Never use gp2 for the Raft data directory in production. The burst credit model is fundamentally incompatible with a sustained write workload. Credits accumulate during quiet periods and deplete during write bursts, creating unpredictable latency cliffs.
• Never share the Raft volume with logs, other databases, or the OS. Consul’s disk I/O profile is latency-sensitive and bursty during snapshots. Any co-located workload can steal burst credits or introduce latency spikes at the worst moment.
• Provision headroom for snapshot creation. Snapshots are large sequential writes that compete with the fsync-heavy Raft log. Undersized IOPS causes commit time to spike during every snapshot cycle, and if the spike exceeds the election timeout, it triggers an election.
• Use dedicated SSD. Rotational disks cause Raft timeouts even without burst credit exhaustion.

Prevention

• Monitor BurstBalance alongside Consul metrics if you are still on gp2. Alert when it drops below 20%. The better fix is to migrate to gp3 or io2 and eliminate burst credits entirely, but until you do, this is the only signal that gives advance warning of the cliff.

• Alert on disk write latency (await) on the Raft volume. Page at 10ms sustained. This is the leading indicator. By the time commitTime spikes, you have already lost the race between detection and election.

• Track consul.raft.commitTime as a primary SLO. Page on sustained values above 500ms. Commit time should stay well below the election timeout.

• Correlate leader elections with storage events. A single election during planned maintenance is expected. Elections that cluster together and correlate with CloudWatch burst balance drops or disk latency spikes are a storage problem masquerading as a Consul problem.

• Size all server volumes identically and provision adequately. If one server has a smaller volume (lower baseline IOPS on gp2, or lower provisioned IOPS on gp3), it will be the first to hit the wall and the most frequent election trigger. Heterogeneous storage across servers creates a weakest-link topology.

How Netdata helps

• Netdata’s per-second disk metrics surface await and %util on the Raft volume in real time. When burst credits deplete, the latency cliff appears in disk metrics before it appears in Consul metrics, giving seconds to minutes of lead time before elections begin.

• The Consul collector exposes consul.raft.commitTime, consul.raft.leader.lastContact, leadership state, and Raft peer state at per-second resolution. Correlating a commit time spike with a disk latency spike on the same node identifies storage as the root cause without switching tools.

• Composite dashboards let you overlay OS-level disk I/O metrics with Consul Raft metrics on the same timeline. For multi-node clusters, side-by-side node views let you compare disk latency across all servers simultaneously, identifying the weakest node before it triggers an election cascade.

Related guides

• Consul gossip encryption key mismatch: a botched keyring rotation splits the pool
• Consul gossip flapping: nodes oscillating between alive, suspect, and failed
• Consul serf queue backlog: an agent falling behind on gossip
• Consul gossip storm after mass recovery: rejoin floods and anti-entropy spikes
• How Consul actually works in production: a mental model for operators
• Consul leader election storm: repeated elections and rolling write outages
• Consul monitoring checklist: the signals every production cluster needs
• Consul monitoring maturity model: from survival to expert
• Consul “No cluster leader”: every write is failing
• Consul raft commitTime high: the write pipeline is slowing down
• Consul Raft data directory full: the server that can no longer write
• Consul raft lastContact rising: followers drifting toward an election