Consul raft commitTime high: the write pipeline is slowing down

consul.raft.commitTime is the single best indicator of Raft health on a Consul server. It is a leader-only timer that folds disk write latency, replication latency, and FSM apply time into one number. When it climbs, every Consul write slows: service registrations queue, KV writes block, health check updates lag, and consumers downstream of the catalog start timing out. The disk-bound leader cannot get log entries committed fast enough, followers lose touch, and the next step is leader elections, during which the cluster cannot commit writes at all.

This article assumes you have already confirmed a leader exists. If /v1/status/leader returns empty, you are in a different incident: see Consul “No cluster leader”: every write is failing.

What this means

consul.raft.commitTime (milliseconds, timer/histogram) is reported only by the leader. It measures end-to-end latency from log entry append to quorum commit plus FSM apply.

Two properties shape triage:

• Leader-only emission. When leadership changes, the time series disappears from the old leader and reappears on the new one. A sudden drop to zero often signals a leadership change rather than improvement. Your monitoring must follow the leader across nodes.
• Composite cost. Disk write latency, network replication latency, and FSM apply time all flow into this one number. To isolate the bottleneck you need the components: consul.raft.leader.lastContact on followers, consul.raft.fsm.apply on the leader, and OS-level disk latency on the leader’s Raft volume.

The safe rule: keep consul.raft.commitTime below 1/10th of the election timeout. Consul’s performance.raft_multiplier scales the Raft election, heartbeat, and leader-lease timeouts together; the default multiplier is 5, while setting 1 uses the high-performance timing often used in production. Confirm the effective election timeout on your cluster before fixing the absolute millisecond ceiling.

Snapshot creation causes brief, normal spikes. They become pathological when they approach the heartbeat timeout.

Common causes

Quick checks

All read-only and safe to run during an incident.

# Confirm the current leader
curl -s http://127.0.0.1:8500/v1/status/leader

# Confirm peer count and voter status
consul operator raft list-peers

# Pull commitTime from the leader's telemetry (run on the leader)
curl -s http://127.0.0.1:8500/v1/agent/metrics | \
  python3 -c "import sys,json; [print(s['Name'], s.get('Mean'), s.get('Count')) for s in json.load(sys.stdin).get('Samples',[]) if 'raft.commitTime' in s['Name']]"

# Pull lastContact from each follower (run on each server)
curl -s http://127.0.0.1:8500/v1/agent/metrics | \
  python3 -c "import sys,json; [print(s['Name'], s.get('Mean')) for s in json.load(sys.stdin).get('Samples',[]) if 'raft.leader.lastContact' in s['Name']]"

# FSM apply latency on the leader
curl -s http://127.0.0.1:8500/v1/agent/metrics | \
  python3 -c "import sys,json; [print(s['Name'], s.get('Mean')) for s in json.load(sys.stdin).get('Samples',[]) if 'raft.fsm.apply' in s['Name']]"

# Catalog register/deregister operation samples
curl -s http://127.0.0.1:8500/v1/agent/metrics | \
  python3 -c "import sys,json; [print(s['Name'], s.get('Count'), s.get('Mean')) for s in json.load(sys.stdin).get('Samples',[]) if 'catalog.register' in s['Name'] or 'catalog.deregister' in s['Name']]"

# Disk I/O on the leader (await and %util on the Raft volume)
iostat -x 1 5

# Raft directory and snapshot sizes
du -sh /opt/consul/data/raft/
ls -lh /opt/consul/data/raft/snapshots/

# GC pause on the leader
curl -s http://127.0.0.1:8500/v1/agent/metrics | \
  python3 -c "import sys,json; [print(s['Name'], s.get('Mean')) for s in json.load(sys.stdin).get('Samples',[]) if 'gc_pause' in s['Name']]"

Adjust the data_dir path if your install is not in /opt/consul.

How to diagnose it

The fastest fork is whether consul.raft.leader.lastContact on followers is also elevated. That single correlation tells you whether the problem is disk-bound writes with heartbeats still flowing, or the whole Raft pipeline saturated.

flowchart TD
  A["commitTime elevated"] --> B{"lastContact on followers also elevated?"}
  B -- "Yes, all followers" --> C["Leader is struggling\nCheck disk I/O and GC pauses"]
  B -- "No, normal" --> D["Writes slow, heartbeats OK\nDisk I/O bottleneck"]
  B -- "Only one follower" --> E["Asymmetric network\nor that follower's disk"]
  C --> F["iostat -x 1 on leader"]
  C --> G["Check GC pause duration"]
  D --> F
  E --> H["Ping pairwise between servers"]
  F --> I{"await > 10ms?"}
  I -- "Yes" --> J["Disk is the cause\nMigrate or shed write load"]
  I -- "No" --> K["Check FSM apply latency\nand catalog churn"]

1. Identify the leader. curl -s http://127.0.0.1:8500/v1/status/leader. All subsequent leader-side checks run on that host. If you ran the metrics query on a follower and got nothing for commitTime, that is why.
2. Confirm commitTime elevation on the leader. A value of zero or an absent series means you are not on the leader.
3. Check lastContact on every follower. Uniform elevation means the leader is the bottleneck. One outlier means a network or single-host issue.
4. Check disk I/O on the leader. iostat -x 1 5. Sustained await above 10ms on the Raft volume is a serious problem.
5. Check FSM apply latency. consul.raft.fsm.apply should be a small fraction of commitTime. If apply latency dominates, the bottleneck is the state machine, not the disk.
6. Check apply rate and catalog churn. A spike in consul.catalog.register or consul.catalog.deregister indicates health check flapping or service registration storms.
7. Check GC pauses on the leader. Sustained pauses above 100ms can stall Raft entirely. Correlate with heap size.
8. Check snapshot timing. If spikes correlate with snapshot creation, confirm they stay bounded. Snapshots that push commitTime past the heartbeat timeout mean the disk is too slow for your state size.
9. Verify Raft directory and snapshot size. Growing snapshots indicate growing catalog or KV state, which raises I/O cost per snapshot and memory cost per restore.

Metrics and signals to monitor

Use consul.runtime.gc_pause_ns for individual GC pause events (a nanosecond summary); consul.runtime.total_gc_pause_ns is the cumulative pause gauge since process start.

For leader-only metrics, your monitoring must follow leadership changes. A per-host threshold on commitTime will silently miss the problem if leadership has moved.

Fixes

Do not restart Consul as a first move. That throws away state, resets metric history, and may make diagnosis harder if leadership lands on a different host.

Disk I/O on the leader’s Raft volume

This is the most common cause, by a wide margin. EBS gp2 with exhausted burst credits, shared volumes, NFS, and rotational disks all produce this pattern. The fix is structural, not configurable.

• Reduce write load to buy time. Disable non-critical health checks, throttle catalog writes, pause KV writes. Stopgap only; reduces observability.
• Move the Raft data directory to a dedicated SSD volume. Nothing else on that volume: not logs, not other databases, not the OS.
• On AWS, switch to provisioned IOPS (io1 or io2) if gp2 burst credits are the problem. Burst exhaustion is sharp and gives no warning in Consul metrics.
• Do not colocate the Raft data directory with anything I/O-heavy.

FSM apply bottleneck

The state machine itself is slow. Common drivers are large KV transactions, complex Connect CA operations, and certificate rotations.

• Identify offending operations from consul.raft.fsm.apply latency patterns and recent Raft log entries.
• Large KV values are expensive: every value is replicated to every server and included in every snapshot. Move bulk data out of Consul KV.
• Connect CA certificate rotation storms produce predictable spikes. Correlate with CA-related metrics.

Catalog churn from health check flapping

A downstream dependency failure causes mass health check failures, each generating catalog writes.

• Identify the shared failing dependency from health check output (the Output field, not just the status).
• Temporarily increase deregister_critical_service_after to slow mass deregistration. This also delays removal of genuinely dead services.
• Disable non-critical health checks to reduce write load.
• Watch the gap between commitIndex and lastAppliedIndex. If it grows, shed read load as well.

Network latency between servers

Asymmetric partitions are common: server A reaches B, but B cannot reach A. Pairwise checks matter more than leader-to-each checks.

• Run ping and mtr from each server to each other server, in both directions.
• Check for packet loss on the RPC and gossip ports (8300, 8301, 8302).
• If lastContact is elevated on only one follower, focus on that network path or that follower’s disk.
• performance.raft_multiplier scales key Raft timing parameters, including election, heartbeat, and leader-lease timeouts, and can stabilize leadership on higher-latency networks at the cost of longer failover. The allowed range is 1–10 and Consul’s current default timing is equivalent to multiplier 5; docs recommend multiplier 1 for production servers. Use it carefully and only after confirming the network issue cannot be resolved.

Snapshot-induced spikes

Brief spikes during snapshot creation are normal. Spikes approaching the heartbeat timeout are not.

• Track snapshot size over time. Growing snapshots raise I/O and memory cost per snapshot.
• If snapshot creation is causing elections, the disk is too slow for your state size. Fix is faster storage or smaller state, not tuning snapshot frequency.
• Verify the Raft volume has free space. A full disk prevents snapshot finalization and cascades into Raft degradation.

Prevention

• Monitor disk write latency on server volumes as a PAGE-level signal. Sustained await above 10ms on the Raft volume is the leading indicator for most commitTime incidents.
• Track consul.raft.commitTime against 1/10th of your election timeout, not against an absolute threshold alone. If you raise Raft timing via performance.raft_multiplier, raise the commit time ceiling proportionally.
• Track catalog size and snapshot size weekly. Slow growth compounds.
• Monitor leader-only metrics with leader-following logic. Per-host thresholds miss the metric when leadership moves.
• Watch health check transition rate, not just status distribution. Flapping checks generate write storms invisible if you only watch critical count.
• Track GC pauses on servers. Pauses above 100ms affect Raft timing and produce commit time spikes with no disk involvement.

How Netdata helps

• Per-second consul.raft.commitTime and its components (lastContact, fsm.apply) on the same timeline. Sub-second spikes during snapshot creation are visible, not averaged away.
• Disk I/O latency per device (await, %util, iowait) correlates directly with Raft write latency on the leader.
• Leader-following dashboards keep leader-only metrics visible across failovers.
• Anomaly detection on commitTime flags gradual creep before it crosses static thresholds.

Related guides

• How Consul actually works in production: a mental model for operators
• Consul monitoring checklist: the signals every production cluster needs
• Consul monitoring maturity model: from survival to expert
• Consul “No cluster leader”: every write is failing