$ guides / consul / consul-stalled-commit-index ▌

Operations Guides

Consul leader stable but commits stalled: writes silently failing

You are paged for “Consul writes failing.” You check /v1/status/leader: it returns an address. You check consul members: all servers alive. Leadership has been stable for hours. Your “is there a leader?” alert is silent. Yet KV writes time out, service registrations hang, and sessions cannot be created.

The leader is alive, holds leadership, and answers health probes. But its commit index is not advancing. Somewhere between the leader receiving a write and that write becoming visible in the FSM, the pipeline is stalled. The cluster is write-dead.

Most monitoring checks whether a leader exists, whether servers are alive in gossip, and whether the HTTP API responds. None of those answer the question that matters: are writes actually committing? This guide covers how to confirm a stalled commit index, identify where in the pipeline the stall sits, and recover without making things worse.

What this means

Consul writes flow through a Raft pipeline. The leader receives the write, appends it to its log, persists the log entry to disk via fsync, replicates it to followers, and once a quorum of followers acknowledge persistence, the leader advances its commit index. Committed entries are then applied to the FSM, which is the actual catalog, KV store, and session table.

Each step is a possible stall point. If the leader cannot persist, the commit cannot proceed. If followers cannot persist, quorum is never reached. If the FSM apply path blocks (large transaction, wedged state machine, apply channel backing up), the pipeline back-pressures into the commit path.

The signal that catches all of these is the commit index itself. When consul.raft.commitIndex stops advancing while the leader stays the same, the cluster is write-dead regardless of what /v1/status/leader reports.

flowchart LR
  Client[Client write] --> Leader[Leader receives]
  Leader --> Append[Append to log]
  Append --> Persist[Persist to disk]
  Persist --> Replicate[Replicate to followers]
  Replicate --> Quorum[Quorum ack]
  Quorum --> Commit[Advance commitIndex]
  Commit --> Apply[Apply to FSM]
  Apply --> Visible[State visible to reads]

  Persist -.->|disk full / I/O error| Stall1[Stall]
  Replicate -.->|followers cannot persist| Stall2[Stall]
  Apply -.->|FSM wedged / channel full| Stall3[Stall]

The diagnostic question is not “is there a leader?” but “is the commit index moving?” Once you confirm it is not, identify which of those three stall points is the cause.

Common causes

Cause	What it looks like	First thing to check
Disk full or nearly full on `data_dir`	Writes return “timed out enqueuing operation”; commit index frozen; logs show fsync errors	`df -h` on the data_dir volume
Disk I/O saturation (not full, just slow)	`consul.raft.commitTime` rising toward heartbeat timeout; eventual leader step-down or stall	`iostat -x 1` on the data_dir device
FSM apply deadlock or wedged state machine	Apply latency spiking or stuck; apply channel backed up; commit index stalls after quorum reached	Apply latency percentile trend from metrics
Followers cannot reach quorum	Commit index frozen even though leader is healthy; replication lag growing on all followers	`consul operator raft list-peers` commit index per server
Snapshot creation contention	Commit time spikes during snapshot; apply latency bursty; logs show snapshot operations	Snapshot events in leader logs

Confirmed: consul.raft.fsm.apply (summary, ms) matches official telemetry docs

Quick checks

Run these on the leader. All are read-only except the final PUT probe, which writes a throwaway key.

# Confirm a leader exists (this check passes during this incident)
curl -s http://127.0.0.1:8500/v1/status/leader

# Confirm this agent is a server (compare its address to the leader above)
curl -s http://127.0.0.1:8500/v1/agent/self | grep -A2 '"server"'

# Compare per-server commit index and last log index
consul operator raft list-peers

# There is no /v1/status/raft endpoint. The status API exposes only
# /v1/status/leader and /v1/status/peers; per-server indexes come from the
# agent telemetry gauges below or `consul operator raft list-peers` (per-peer
# LastIndex).
curl -s http://127.0.0.1:8500/v1/agent/metrics | \
  jq '.Gauges | with_entries(select(.Key | test("raft|commit|lastLog"; "i")))'

# Check disk space on the data_dir
df -h /opt/consul/data

# Check disk I/O latency on the underlying device (let it run several seconds)
iostat -x 1 5

# Pull the leader-only Raft timers from metrics
curl -s http://127.0.0.1:8500/v1/agent/metrics | \
  grep -E "raft.commitTime|raft.apply"

# Confirm writes are actually failing (WRITE to a throwaway key)
time curl -s -X PUT -d 'probe' http://127.0.0.1:8500/v1/kv/__commit_probe

The two most revealing checks are the commit index snapshot and the PUT probe. If LastLogIndex is growing while CommitIndex is frozen, the leader is receiving writes but cannot commit them. If a simple KV PUT takes seconds or fails outright while /v1/status/leader answers immediately, you have confirmed the stall.

How to diagnose it

Confirm the stall. Pull commit and applied indices twice, 10 seconds apart. If CommitIndex does not advance while there is active write traffic, the cluster is stalled. Cross-check with consul operator raft list-peers to see whether followers are also behind.
Identify which server is leader. curl -s http://127.0.0.1:8500/v1/status/leader returns the leader address. SSH to that host. Leader-only metrics (consul.raft.commitTime, FSM apply latency) only report there.
Check disk first. It is the most common cause. Run df -h on the data_dir volume. If the volume is above roughly 95%, Raft cannot reliably fsync new log entries. Check I/O latency with iostat -x 1 5 and look at await on the underlying device. Anything sustained above 10ms puts you at risk; above 100ms is consistent with a stall.
Check FSM apply. Pull apply latency percentiles from /v1/agent/metrics. If p99 is above 500ms or trending sharply upward, the state machine is the bottleneck. Common drivers: a large KV transaction, a Connect CA rotation that deserializes the full certificate chain, or an anti-entropy sync storm after a partition heal.
Check follower replication. If consul.raft.commitTime is normal but the commit index still will not advance, the leader may not be reaching quorum. Look at consul.raft.leader.lastContact reported per follower. If any follower’s last contact is climbing toward the election timeout, that follower is not acknowledging replication.
Check the incoming apply rate. Confirmed: consul.raft.apply (counter, transactions/interval) exists in official telemetry docs If the apply rate is high but the commit index is not moving, writes are arriving but not landing. If the apply rate has dropped to near zero, clients have likely backed off and stopped retrying, which is a symptom rather than a cause.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`consul.raft.commitIndex` on leader	Direct measure of write progress	No advancement for more than 30s under write load
`consul.raft.commitTime` (leader only)	End-to-end Raft commit latency	Sustained above 500ms, or trending toward heartbeat timeout
Apply latency p99 (leader only)	State machine apply bottleneck	Sustained above 500ms, or 2x increase over baseline
`LastLogIndex - CommitIndex` on leader	Where the stall sits	Growing gap means entries appended but not committed
`consul.raft.leader.lastContact` per follower	Replication channel health	Any follower approaching election timeout
Disk space on `data_dir`	Whether Raft can persist at all	Above 90% is dangerous; above 95% is critical
Disk write latency on data_dir device	Leading indicator for Raft issues	`await` sustained above 10ms

Fixes

Recovery depends on the cause. The unifying rule: do not restart the leader as a first move. A restart forces a leader election, loses in-flight state, and may not recover if the underlying cause (disk full, wedged FSM) is still present on restart.

Disk full on the data_dir

Free space on the data_dir volume. The safest source of space is old, superseded snapshots.

In older Consul releases, disk-full conditions can leave corrupt .tmp service files in <data_dir>/services/ that prevent restart even after space is freed. If Consul fails to restart with “unexpected end of JSON input” after you free space, remove the offending service files manually.

Never truncate or delete files inside <data_dir>/raft/ to free space. The Raft log and snapshots there are required for cluster membership. If you must reduce snapshot footprint, lower the snapshot threshold and let Consul compact naturally.

Disk I/O saturation without full disk

If iostat shows high await but the volume has space, the underlying storage is too slow. Rotational disks, network-attached storage, and burst-credit-exhausted cloud volumes are the most common cause of this failure mode.

Short-term, shed write load. Disable non-critical health checks, pause anti-entropy on noisy agents, or rate-limit KV writers. Long-term, migrate the data_dir to dedicated SSD-backed storage on its own volume.

FSM apply deadlock

If FSM apply latency is the bottleneck, identify what is generating the apply load. Common culprits: a large KV transaction, a Connect CA root rotation, a health check thundering herd, or an anti-entropy sync storm after a partition heal. Reduce the input rate and let the FSM drain.

The FSM apply channel has a fixed buffer. Once it fills, the entire commit pipeline backs up.

If the FSM is truly wedged (apply latency is effectively infinite, not just high), a leader restart may be unavoidable.

Confirmed: consul operator raft transfer-leader exists per official CLI docs Try consul operator raft transfer-leader first to hand leadership to a follower with a healthier state machine. Only restart the process if that fails.

Followers cannot reach quorum

If the leader is healthy but the commit index will not advance, a quorum of followers may be unable to acknowledge replication. Check pairwise network connectivity between the leader and each follower.

Do not remove peers from the Raft configuration unless you are certain they are permanently gone. Premature peer removal risks split-brain and data loss. If you must remove a peer, use consul operator raft remove-peer and only after confirming the node is decommissioned.

Prevention

Alert on commit index progression, not just leader existence. A “leader exists” check returns true during this entire incident. The commit index tells you writes are landing.
Alert on consul.raft.commitTime sustained above 100ms. Sustained commit time above 500ms risks leader timeout.
Disk space alerts on the data_dir at 80% and 90%. Page at 90%. Consul cannot tolerate a full data_dir.
Disk I/O latency alerts on the data_dir device. Page if await is sustained above 10ms. Disk I/O is the single most common cause of Raft stalls.
Track FSM apply p99 over time. A creeping baseline signals catalog growth or pathological write patterns long before they cause a stall.
Distinguish leader-only metrics in dashboards. consul.raft.commitTime and apply latency metrics only report on the leader. Your monitoring must follow leadership changes or you will have blind spots during failovers.

How Netdata helps

Netdata collects consul.raft.commitIndex per second, so a stall is visible within seconds of onset rather than after the next scrape interval. ML-based anomaly detection on consul.raft.commitTime and FSM apply latency flags deviations from each server’s own baseline, which matters because healthy commit time is workload-dependent and absolute thresholds miss slow drift.

Disk metrics (await, utilization, queue depth) on the data_dir device appear alongside Raft metrics in the same dashboard, so I/O saturation is visible in the same view as the commit stall it causes. Leader-aware collection keeps leader-only metrics flowing across leadership changes.

A composite alert combining leadership stability with commit index progression catches this specific failure mode: leader unchanged, commit index flat under write load, write probe timing out.