$ guides / consul / consul-kv-write-latency-high ▌

Operations Guides

Consul KV write latency high: every write is a Raft commit

A KV PUT that used to return in 20ms is now taking 500ms, 2s, or timing out entirely. Applications stall on locks, configuration writes queue, and leader election alerts may be firing. Consul’s KV is not a standalone subsystem. Every KV write is a Raft log entry that must be replicated to a quorum of servers, fsynced to disk on each, and applied to the in-memory state machine before the API call returns. KV write latency is a direct reflection of Raft commit health.

The metric that captures this is consul.kvs.apply, a timer reported only on the leader. It measures end-to-end write latency for KV operations. When it tracks closely with consul.raft.commitTime, the problem is the Raft pipeline itself: disk I/O, network replication, or FSM apply cost. When KV latency is far above commit time, something between the client and Raft is queueing: RPC admission control, connection limits, or goroutine saturation.

Sustained consul.kvs.apply above 200ms warrants investigation. Above 1 second, Raft health itself is questionable and you may be one disk spike away from leader elections.

What this means

Consul’s KV store lives inside the Raft finite state machine. There is no separate write path. A PUT /v1/kv/<key> does the following:

The request lands on whichever server the client contacted. If that server is not the leader, it forwards the write.
The leader appends the KV mutation as a new Raft log entry.
The leader replicates that entry to all followers in parallel and waits for a quorum to acknowledge.
Each server in the quorum fsyncs the log entry to disk in the Raft data directory.
Once committed, the entry is applied to the in-memory FSM. Consul blocks the write response until both commit and apply are complete.
The API call returns to the client.

Every step is on the critical path. A slow disk on any quorum member inflates step 4. A saturated network between leader and a follower inflates step 3. A large KV value or a complex transaction inflates step 5 because the FSM must deserialize and index the new state. And because all state mutations share this pipeline, KV writes compete for Raft throughput with service registrations, health check updates, session operations, and ACL changes.

flowchart TD
    A[Client KV PUT] --> B{On leader?}
    B -- No --> C[Forward to leader]
    C --> D
    B -- Yes --> D[Append Raft log entry]
    D --> E[Replicate to followers]
    E --> F[Fsync to disk: leader + quorum]
    F --> G[Apply to FSM: KV state store]
    G --> H[Return to client]
    F -.->|slow disk| X[Inflated commitTime]
    E -.->|network latency| X
    G -.->|large values / deep trees| Y[Inflated fsm.apply]
    D -.->|RPC admission / conn limits| Z[Queueing before Raft]

Common causes

Cause	What it looks like	First thing to check
Disk I/O latency on the Raft volume	`consul.raft.commitTime` tracks KV latency closely; `iostat await` elevated; possible leader elections	`iostat -x 1` on the leader’s Raft volume
Raft pipeline saturation from all writes	KV latency tracks commit time; `consul.raft.apply` rate elevated; health check or catalog churn visible	`consul.raft.apply` counter trend
Large KV values	KV latency and `consul.raft.fsm.apply` both elevated; snapshot size growing; specific keys responsible	Enumerate key sizes, check for values approaching 512KB
RPC admission or connection queueing	KV latency far above `consul.raft.commitTime`; goroutines or FDs near limit; writes slow but stale reads fast	Compare `consul.kvs.apply` to `consul.raft.commitTime` directly
Leader instability	Intermittent spikes correlated with leader transitions; “no cluster leader” errors	`consul.raft.state.leader` gauge transitions and `lastContact`
Deep or wide key trees	Prefix scans and blocking queries slow; writes to high-churn prefixes expensive	`GET /v1/kv/?keys` and check tree shape

Quick checks

Run these on the leader first. All are read-only except the latency probe, which writes a small value to an isolated diagnostic key.

# Identify the leader
curl -s http://127.0.0.1:8500/v1/status/leader

# Time a single KV write (writes a small probe value to a diagnostic key)
time curl -s -X PUT -d 'probe' http://127.0.0.1:8500/v1/kv/__diag__/latency-probe

# Time a stale read (should be sub-ms locally on the leader)
time curl -s http://127.0.0.1:8500/v1/kv/__diag__/latency-probe

# Time a consistent read (costs a leader verification round-trip)
time curl -s "http://127.0.0.1:8500/v1/kv/__diag__/latency-probe?consistent"

# Check consul.kvs.apply telemetry on the leader
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -A3 "kvs.apply"

# Check consul.raft.commitTime on the leader
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -A3 "raft.commitTime"

# Check lastContact on a follower (run on a follower server)
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -A3 "raft.leader.lastContact"

# Enumerate KV key count
curl -s "http://127.0.0.1:8500/v1/kv/?keys" | python3 -c "import sys,json; print('Keys:', len(json.load(sys.stdin)))"

# Disk I/O on the Raft volume (identify the backing device first)
iostat -x 1 5

The critical comparison is between consul.kvs.apply and consul.raft.commitTime. If they track each other, the Raft pipeline is the bottleneck. If KV latency is meaningfully higher, look upstream at RPC handling, connection saturation, or goroutine pressure.

How to diagnose it

1. Confirm whether the problem is Raft or KV-specific.

Pull both consul.kvs.apply and consul.raft.commitTime from the leader. If commit time is the dominant cost (KV latency tracks commit time within a small margin), the Raft pipeline is saturated or slow. If KV latency exceeds commit time by a wide margin, the delay is in RPC admission, connection handling, or goroutine pressure before the write reaches Raft.

2. Check disk I/O on the leader and all quorum members.

Raft commit time is dominated by fsync latency. Run iostat -x 1 on each server and watch the await column for the device backing the Raft data directory (typically <data_dir>/raft/). Sustained write latency above 10ms on the Raft volume is a problem. Common causes: EBS gp2 burst credit exhaustion, shared or network-attached storage, spinning disks, noisy neighbors on the volume. This is the single most common root cause of high KV write latency in production.

3. Check the Raft apply rate and what is driving it.

KV writes are not the only thing flowing through Raft. Service registrations, health check state changes, sessions, and ACL operations all share the pipeline. Check consul.raft.apply and consul.catalog.register / consul.catalog.deregister counters. If catalog churn is high, health check flapping or a registration storm is consuming Raft throughput that KV writes need. See the related guides on catalog bloat, registration storms, and gossip flapping.

4. Inspect the KV store for large values and key tree shape.

Large values inflate the Raft log entry, slow the fsync, increase FSM apply cost, and grow snapshot size. The default kv_max_value_size is 512KB, but even values in the tens of KB written frequently degrade performance. Enumerate keys and check sizes:

# List keys sorted by value size, largest first
curl -s "http://127.0.0.1:8500/v1/kv/?recurse" | \
  python3 -c "
import sys,json
items=json.load(sys.stdin)
for i in sorted(items, key=lambda x: len(x.get('Value','') or ''), reverse=True)[:20]:
    print(f'{len(i.get(\"Value\",\"\") or \"\"):>8} bytes  {i[\"Key\"]}')
"

Also check whether any single prefix has an unusually deep or wide tree. Blocking queries on high-churn prefixes and prefix scans over wide trees are expensive even when individual writes are fast.

5. Check leader stability and lastContact on followers.

If KV latency spikes correlate with leader transitions, the leader is unstable. Check consul.raft.state.leader for transitions (the gauge is 1 on the leader, 0 otherwise) and consul.raft.leader.lastContact on followers. Rising lastContact toward the election timeout indicates the leader is struggling to heartbeat. This often co-occurs with disk I/O problems but can also be caused by Go GC pauses on large heaps.

6. Check RPC admission and resource saturation.

If KV latency exceeds commit time, check goroutine count (consul.runtime.num_goroutines) and file descriptor usage on the server. RPC rate limiting, connection pool exhaustion, and goroutine accumulation from leaked blocking queries all add latency before the write reaches Raft.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`consul.kvs.apply`	End-to-end KV write latency, leader only. Primary symptom metric.	Sustained above 200ms (TICKET) or 1s (PAGE)
`consul.raft.commitTime`	Time to commit a Raft entry to disk on the leader. Raft health proxy.	Tracking upward with kvs.apply, or above 100ms sustained
`consul.raft.fsm.apply`	Time to apply committed entries to the state machine.	Spikes on large values or complex transactions
`consul.raft.leader.lastContact`	Time since follower heard from leader. Predictive of elections.	Trending toward election timeout
`consul.raft.apply` (counter)	Raft log entries applied per interval. Write throughput.	Sudden 3x+ spike without new services (churn source)
Disk write latency (`await`)	fsync is the dominant cost in Raft commit.	Sustained above 10ms on the Raft volume
`consul.runtime.num_goroutines`	Proxy for concurrent load and leak detection.	Monotonic growth without load increase
Raft snapshot size	Tracks total state size including KV.	Growing week over week without explanation
`consul.raft.state.leader` (gauge)	Leader state (1=leader, 0=follower). Transitions indicate elections.	Multiple 0->1 transitions per 10 minutes outside maintenance

Fixes

Disk I/O is the bottleneck

This is the most common cause. Move the Raft data directory to a dedicated, faster volume. For cloud deployments, use provisioned IOPS (io1 or io2 on AWS) rather than gp2, which can exhaust burst credits and hit a sudden latency cliff. Never colocate the Raft volume with other I/O-heavy workloads like application logs or databases. On bare metal or VMs, dedicated SSDs are the baseline expectation, not an optimization.

If you cannot immediately migrate storage, reduce write pressure on Raft by throttling the heaviest write sources: flapping health checks, excessive service registration churn, or applications writing to KV at high frequency.

Raft pipeline is saturated by non-KV writes

KV latency is a victim of overall Raft throughput. If catalog churn is the driver, address the source. See the related guides on catalog bloat, registration storms, and gossip flapping. Temporary mitigations include increasing health check intervals, reducing the number of registered checks, and ensuring deregister_critical_service_after is set to prevent stale entries from accumulating.

Large KV values or KV-as-database usage

Consul KV is designed for configuration, coordination, and small metadata. It is not a general-purpose datastore. If applications are writing large values (approaching even tens of KB) or writing at high frequency, move that workload to an appropriate datastore. If you cannot move it immediately:

Reduce value size by storing only references or pointers in KV.
Clean up ephemeral or temporary keys that accumulate over time.
Consider batching related writes using Consul transactions (PUT /v1/txn), though each transaction is still a single Raft commit.

RPC admission or connection saturation

If KV latency exceeds commit time, the queueing is before Raft. Check and raise rpc_rate_limit and rpc_max_burst if they are too restrictive for your cluster size. Raise file descriptor limits via ulimit -n or LimitNOFILE in systemd. Investigate goroutine leaks from abandoned blocking queries or watches. Each blocking query holds a goroutine and a connection for its entire duration, and leaks compound over weeks.

Leader instability

If leader elections are recurring, the root cause is almost always disk I/O or resource starvation on the leader, not a Raft configuration problem. Fix the disk first. As a last resort during an active incident, you can transfer leadership to a known-healthier server with consul operator raft transfer-leader, but this only buys time if the underlying resource issue persists on the new leader.

Prevention

Monitor disk write latency on all server Raft volumes as a PAGE-level signal. This is the leading indicator. Do not wait for leader elections to tell you the disk is slow.
Track consul.kvs.apply and consul.raft.commitTime together. Their relationship is the fastest diagnostic. Alert when either crosses the TICKET threshold.
Track Raft snapshot size over time. Growing snapshots indicate state growth (KV, catalog, or checks) that will eventually degrade commit and restore performance.
Establish a KV usage policy. Define maximum value sizes, maximum key counts, and prohibited use cases. Enforce it in code review.
Audit goroutine count and FD usage weekly. Slow leaks from blocking queries and watches compound over weeks and inflate KV latency through RPC queueing.
Review kv_max_value_size if you need to cap large values. The default is 512KB. Lowering it can prevent accidental large-value writes from destabilizing the cluster.

How Netdata helps

Per-second consul.kvs.apply and consul.raft.commitTime surface the correlation between KV latency and Raft commit latency at the resolution needed to catch transient spikes that minute-aggregated metrics miss.
Anomaly detection on Raft commit time catches gradual drift that static thresholds miss, before KV latency becomes user-visible.
Disk I/O metrics per device (await, %util, read/write latency) alongside Consul metrics confirm or rule out disk saturation in a single view.
Leader election events correlated with KV latency spikes distinguish “Raft is slow” from “leadership is unstable” without manual log correlation.
Goroutine count and file descriptor trends reveal slow leaks from blocking queries that inflate KV latency through RPC queueing rather than Raft itself.