$ guides / consul / consul-kv-store-saturation ▌

Operations Guides

Consul KV store saturation: using the KV store as a database

Consul KV writes are slow. Raft commit times are climbing. Service registration and health check updates lag. DNS queries for service discovery take longer than usual. This combination often points to an application treating Consul’s KV store as a general-purpose database: high-frequency writes, large values, or deep key trees.

The problem is architectural, not a tuning issue. The KV store is part of Consul’s Raft finite state machine. Every KV write is a Raft log entry that must be committed through the leader, replicated to a quorum of servers, and applied to the in-memory state store on every server. Every KV value is included in every Raft snapshot. When an application treats the KV store as a database, the write load backs up the entire Raft pipeline, and every Consul subsystem that depends on Raft degrades with it.

This article covers how to identify KV-driven Raft saturation, distinguish it from catalog churn, and move the offending workload to an appropriate datastore.

What this means

KV is designed for configuration, coordination, and small metadata. When an application writes at high frequency, stores large values, or accumulates keys without cleanup, the Raft pipeline becomes the bottleneck. The failure mode is gradual at first, then cliff-edge:

KV write latency rises as Raft commit time increases.
Raft apply rate stays elevated because every KV write is an FSM mutation.
Server memory grows as the KV state accumulates in the in-memory state store.
Snapshot size grows, making snapshot creation slower and more I/O-intensive.
As commit time approaches the Raft heartbeat timeout, leader elections begin.
During each election, all writes fail, including service registrations and health check updates.

The blast radius extends beyond the application doing the heavy KV writes. Every Consul consumer experiences slower service discovery, delayed health updates, and eventually write unavailability during leader elections.

flowchart TD
    A[App writes KV at high frequency] --> B[Every write enters Raft log]
    B --> C[Leader replicates to quorum]
    C --> D[FSM applies entry on every server]
    D --> E[Commit time rises]
    E --> F[Snapshot size grows]
    F --> G[Memory and disk I/O pressure]
    E --> H[All Consul writes slow down]
    H --> I[Service discovery and health degrade]

Common causes

Cause	What it looks like	First thing to check
Application using KV as primary datastore	High `consul.kvs.apply` count, steady KV write rate, growing key count	Identify which application owns the busiest key prefixes
Large values in KV	High FSM apply latency per entry, growing server memory, slow snapshots	Enumerate keys and check value sizes
Configuration management writing full configs at high frequency	Periodic spikes in KV write rate correlating with config refresh interval	Check config management tool logs for KV write patterns
Service mesh control plane storing per-endpoint state in KV	Growing key count proportional to endpoint count, steady write rate from mesh components	Check which services or mesh components are writing to KV
Missing cleanup of temporary or ephemeral keys	Key count grows monotonically, memory grows without service growth	Count keys over time and look for stale prefixes

Quick checks

These commands are read-only and safe to run on any Consul server, except where noted.

# Identify the leader
curl -s http://127.0.0.1:8500/v1/status/leader

# Raft commit time (most useful on the leader)
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -E "raft.commitTime"

# KV apply latency
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -E "kvs.apply"

# Count total KV keys
curl -s http://127.0.0.1:8500/v1/kv/?keys | python3 -c "import sys,json; print('Total keys:', len(json.load(sys.stdin)))"

# FSM apply latency across all state types
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -E "raft.fsm"

# Server RSS (use pgrep -x consul | head -1 if multiple processes)
cat /proc/$(pgrep -xo consul)/status | grep VmRSS

# Raft log size (adjust path to match your -data-dir)
du -sh /opt/consul/data/raft/

# Snapshot files
ls -lh /opt/consul/data/snapshots/

# Time a single KV write to measure end-to-end latency.
# WARNING: this is a write. It creates a key called _health_probe.
# Clean up afterward: curl -s -X DELETE http://127.0.0.1:8500/v1/kv/_health_probe
time curl -s -X PUT -d 'test' http://127.0.0.1:8500/v1/kv/_health_probe

How to diagnose it

The goal is to confirm that KV operations are the dominant driver of Raft load, and to identify the specific workload causing it.

Confirm Raft is under write pressure. Check consul.raft.commitTime on the leader. If p99 is above 100ms sustained, the write pipeline is degraded. Above 500ms, you are at risk of leader elections.
Determine whether KV or catalog churn dominates. Compare consul.kvs.apply rate against consul.catalog.register and consul.catalog.deregister rates. If KV apply operations are the bulk of Raft writes, the KV store is the bottleneck. If catalog churn dominates, the problem is elsewhere.
Measure the KV key space. Run curl -s http://127.0.0.1:8500/v1/kv/?keys and count the entries. A growing key count without a corresponding business reason is a red flag.

Identify large values. Iterate over keys in the busiest prefixes and check individual value sizes. The default per-key limit is 512KB, but values approaching that size are expensive because each one must be deserialized, replicated, and snapshotted on every server.
Identify the heaviest KV writers. The KV API does not expose per-client write attribution. Correlate from the application side: check access logs, application metrics, or network-level analysis to determine which applications are writing to which key prefixes.
Check snapshot size and frequency. Raft snapshots include the full FSM state, including all KV data. If snapshot sizes are growing week over week, KV data is a likely contributor. Large snapshots slow down server restarts (full state restore) and increase memory pressure during snapshot creation.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`consul.raft.commitTime`	End-to-end Raft write latency. Captures disk, network, and FSM apply time in one number.	Sustained above 100ms (ticket), above 500ms (page, election risk)
`consul.kvs.apply`	Latency of KV write/delete operations through Raft.	Rising trend, or values exceeding Raft commit time (indicating queuing)
`consul.raft.apply`	Total FSM apply rate across all state types (KV, catalog, sessions, ACLs).	Sustained rate well above baseline without known cause
`consul.raft.fsm.apply`	Latency of individual FSM apply operations.	Spikes during KV writes with large values
`consul.runtime.alloc_bytes`	Current Go heap allocation. Proxy for state store size.	Monotonic growth without corresponding service growth
`consul.state.kv_entries`	Gauge of total KV entries on servers. Direct measure of KV store size.	Growth that does not match expected application behavior
Raft snapshot size	Size of FSM snapshots on disk. Correlates with KV and catalog volume.	Growing over weeks without service growth
Server RSS (`/proc/<pid>/status`)	OS-level memory usage. Includes state store and snapshot buffers.	Growth that tracks KV size growth
Disk write latency (`iostat await`)	Raft log writes are fsync-heavy. Slow disk is the most common cause of leader instability.	Sustained above 10ms on the Raft data volume

Fixes

All fixes share the same principle: move heavy KV workloads to a datastore designed for them.

Move application state out of KV

If an application is using Consul KV as its primary datastore, the only durable fix is migration. Workloads that need high write throughput, large values, or complex queries belong in a database.

Identify the key prefixes the application owns.
Migrate data to an appropriate datastore (a relational database, Redis, DynamoDB, or any system designed for the workload).
Update the application to use the new datastore.
Delete the migrated keys from Consul KV.

This is a coordinated change, not a quick fix. Leaving application state in KV guarantees the problem will recur.

Reduce value sizes

If values are large but write frequency is moderate, the bottleneck is per-operation cost. The default per-key limit is 512KB, but values approaching that size are expensive to deserialize, replicate, and snapshot.

The kv_max_value_size configuration option can increase the per-key limit, but raising it makes the problem worse, not better. Large values consume more memory per server, slow down snapshot creation, and increase Raft replication time. Store large blobs externally (object storage, a blob database) and keep only a reference in KV.

Reduce write frequency

If values are small but writes are frequent, the bottleneck is Raft throughput. Every write is a Raft commit regardless of value size. Options:

Batch writes using the transaction API (/v1/txn), which allows multiple KV operations in a single Raft commit. The transaction payload limit applies to the entire request body, not per-key.
Increase the write interval on the application side. Most configuration and coordination data does not need sub-second freshness.
Use stale reads (?stale consistency mode) for read-heavy paths to reduce load on the leader. Stale reads can be served by any server.

Clean up ephemeral keys

Consul does not automatically expire KV entries. Applications that create temporary keys must explicitly delete them. If the problem is unbounded key growth, audit the key space for stale prefixes:

Lock-related keys from sessions that were destroyed without releasing their keys.
Deployment artifact keys that were never cleaned up.
Feature flag or canary keys from experiments that ended.

Prevention

Enforce a KV usage policy. Document which key prefixes are sanctioned for KV use and what types of data are allowed. Review new KV usage in code review.
Monitor consul.state.kv_entries and snapshot size as trend signals. Growth in either without a known cause is an early indicator.
Track consul.raft.commitTime and consul.kvs.apply as production-critical signals. Commit time above 500ms is election territory.
Set per-application KV write budgets. If an application needs more than a handful of writes per second to KV, it is likely using KV for the wrong purpose.
Periodically audit the KV key space. Stale keys accumulate silently. Nobody notices until snapshot size or memory triggers an alert.
Distinguish KV from the catalog in your monitoring. Both drive Raft load, but the remediation is different. KV saturation requires moving data out. Catalog churn requires fixing registration behavior.

How Netdata helps

Netdata’s per-second metric collection lets you catch KV-driven Raft saturation before it triggers leader elections.

Correlate consul.raft.commitTime with consul.kvs.apply to confirm KV writes are the Raft bottleneck, not catalog churn. Per-second resolution means you see the spike as it happens.
Watch consul.runtime.alloc_bytes and server RSS together to distinguish KV-driven memory growth from goroutine leaks or catalog bloat.
Track Raft apply rate against catalog registration rate to quantify what fraction of Raft load comes from KV versus the service catalog.
Set anomaly-based alerts on consul.raft.commitTime to catch gradual degradation before it crosses the election timeout threshold.
Monitor disk write latency on the Raft data volume alongside commit time. If disk I/O is the amplifier for KV write load, Netdata’s system-level disk metrics show it in the same dashboard as Consul metrics.