$ guides / consul / consul-kv-value-too-large ▌

Operations Guides

Consul KV value too large: the 512KB limit and why approaching it hurts

You pushed a value into Consul KV and got HTTP 413, or your server logs are filling with lines like Request body(524401 bytes) too large, max size: 524288 bytes. That is the obvious failure: a single value crossed the default 512KB ceiling. The less obvious failure is that values well under the limit are still expensive, because every KV byte is replicated through Raft to every server and re-emitted in every snapshot.

If you are hitting rejections, watching snapshot sizes creep upward, or seeing Raft commit time drift and wondering whether KV usage is the cause, the answer is usually yes. The fix is architectural: move blobs out of KV and keep only small config or pointers.

The 512KB limit exists to stop you from accidentally turning the Raft FSM into a blob store. Raising it via kv_max_value_size is possible, but the cost model that made 512KB the default does not go away when you raise the ceiling.

What this means

Consul enforces a default maximum value size of 512KB (524288 bytes) per KV entry. The check runs on the HTTP API write path and returns HTTP 413 to the client. The limit is configurable via kv_max_value_size introduced in Consul 1.7.2, which defaults to the Raft suggested maximum of 512KB. A separate txn_max_req_len also introduced in Consul 1.7.2, also defaulting to 512KB, bounds the /v1/txn endpoint request body.

Two subtleties matter:

The txn endpoint limit applies to the whole envelope, not per-value. A transaction packing several legitimate-size values plus JSON metadata can trip the limit even though no single value is near 512KB. This is the classic pattern where a secret store writes a payload that grows once wrapped in the txn structure.
Cross-DC replication breaks when limits diverge. If the source datacenter runs a larger kv_max_value_size than the destination, replication fails with Request body too large errors. The mismatch is easy to miss because the primary DC looks healthy.

The reason raising the limit is usually wrong is the replication cost model:

Every KV write is a Raft log entry. The leader fsyncs it, then replicates it to a quorum.
Every KV value is held in the FSM on every server. A value stored once costs N copies of memory, where N is the server count.
Every KV value is included in every Raft snapshot. Snapshot creation loads the FSM into memory and writes it to disk; large values inflate both the memory spike and the on-disk snapshot size.
Snapshot restore on server restart reads the whole snapshot back. Larger snapshots mean longer cold starts.

So the symptom you are debugging might not be the 413 at all. It might be snapshot size growing week over week, commit time creeping up, or a server taking minutes to rejoin after a restart because it has to pull and apply a huge snapshot. The KV store is the hidden driver.

flowchart TD
    A[Large KV value written] --> B[Raft replicates value to N servers]
    B --> C[Value persists in every FSM]
    C --> D[Value re-emitted in every snapshot]
    D --> E[Snapshot I/O and memory spike on every cycle]
    E --> F[Raft commit time rises]
    F --> G[Heartbeat timeout risk]
    G --> H[Leader elections and write outage]

Common causes

Cause	What it looks like	First thing to check
Single value exceeds 512KB	HTTP 413 on write; `Request body(N bytes) too large` in server logs	Confirm payload size client-side before the write
Transaction envelope exceeds limit (multiple small values plus metadata)	413 on `/v1/txn` even though each value is small	Sum value sizes plus txn overhead; compare to `txn_max_req_len`
Accumulated large-ish values strain the cluster	No 413, but snapshot size and commit time trending up	Enumerate KV keys and sample value sizes
Cross-DC replication mismatch	`Request body too large` in secondary DC, primary is fine	Compare `kv_max_value_size` on both sides

Quick checks

# Check whether a leader exists and writes are flowing
curl -s http://127.0.0.1:8500/v1/status/leader

# Enumerate the KV key space size
curl -s http://127.0.0.1:8500/v1/kv/?keys | jq 'length'

# Sample value sizes for the first 50 keys (read-only, safe)
for k in $(curl -s "http://127.0.0.1:8500/v1/kv/?keys" | jq -r '.[]' | head -50); do
  size=$(curl -s "http://127.0.0.1:8500/v1/kv/$k" | jq -r '.[0].Value' | base64 -d 2>/dev/null | wc -c)
  echo "$size $k"
done | sort -rn | head -20

# KV write latency and Raft commit time from telemetry (leader-only metrics)
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -E 'kvs.apply|raft.commitTime'

# Snapshot directory size (path is deployment-specific; /opt/consul is a common default)
du -sh /opt/consul/data/raft/
ls -lh /opt/consul/data/raft/snapshots/ 2>/dev/null | tail -5

The data directory path depends on your data_dir setting. The /opt/consul/data/raft/ path above is the playbook example; substitute your own. These checks are read-only and safe to run on a production server.

How to diagnose it

Confirm the rejection is size-based, not permissions. A 413 is unambiguous. If you see 403, you have an ACL problem, not a size problem.
If you have a 413, measure the offending value client-side. The error message includes the byte count. Compare it to 524288 (512KB), or to your configured kv_max_value_size if you raised it.
If the symptom is cluster-wide degradation without 413s, profile the KV store. Run the size-sampling loop from the quick checks against the whole key space, not just the first 50 keys. Sort descending. Anything over roughly 50KB is suspicious; anything over roughly 200KB is a problem even though it fits.
Correlate KV size with Raft and snapshot metrics. Look at consul.raft.commitTime, snapshot directory growth, and server RSS together. If all three trend upward and KV is the largest contributor to FSM size, you have found the cause.
Check whether the txn endpoint is the culprit. If your client uses /v1/txn to batch writes, the limit applies to the whole request body. A batch of small values plus JSON overhead can exceed 512KB even when each value is small.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`consul.kvs.apply` latency	End-to-end KV write cost; tracks Raft commit	Sustained p99 above ~200ms
`consul.raft.commitTime`	Health of the Raft write pipeline	Trending toward a meaningful fraction of the election timeout
Raft snapshot directory size	Snapshot bloat from accumulated KV state	Growth week over week without catalog growth
Server RSS and `consul.runtime.alloc_bytes`	FSM memory cost; spikes during snapshot creation	RSS climbing without service growth
HTTP 413 count on KV writes	Direct evidence of limit rejections	Any non-zero rate
`consul.raft.state.leader` transitions	Leadership stability; large values can destabilize Raft	More than one transition outside maintenance

Fixes

Move blobs out of KV (the real fix)

Stop using KV for large blobs. Store the payload in an object store (S3, GCS, MinIO), a document store, or a database, and put a pointer in KV. Terraform state is the textbook example: a JSON blob that belongs in S3 with a KV entry holding the S3 key.

Tradeoff: clients now do a KV read plus a blob fetch. That is almost always acceptable, because large values are rarely on the hot read path, and the win is a smaller FSM, smaller snapshots, and lower commit time for everything else.

Split large configs

If the payload is genuinely configuration (a large JSON or YAML document), split it across multiple keys under a prefix. Consumers read the prefix and assemble. Each key stays small, writes are cheaper, and you avoid the 413.

Tradeoff: atomicity. A multi-key update is not atomic without a transaction, and transactions have their own size limit. If you need atomicity, use a session and Check-And-Set semantics, or accept eventual consistency.

Compress in-band

If you control both writer and reader, gzip the value before storing and decompress on read. This can turn a 400KB JSON config into a 60KB KV value. It does not fix the architectural smell, but it buys time and reduces snapshot size immediately.

Tradeoff: compression makes KV values opaque to consul kv and the UI. Confirm every reader decompresses.

Raise the limit (usually wrong, sometimes necessary)

Set kv_max_value_size higher than the default. This is a valid escape hatch when you have an unavoidable large value and cannot immediately refactor. HashiCorp’s own warning is blunt: tuning improperly can cause Consul to fail in unexpected ways, potentially affecting leadership stability and preventing timely heartbeat signals by increasing RPC IO duration.

Before you raise it:

Treat the change as a capacity decision, not a config tweak. You are increasing Raft log entry sizes, snapshot sizes, and memory pressure on every server.
Make sure every server in every federated datacenter has the same setting. Divergent limits break cross-DC replication.
Set a deadline to remove the large value. Raising the limit does not make the cost model disappear; it just moves the cliff.

Since Consul 1.10.0, raft_snapshot_threshold, raft_snapshot_interval, and raft_trailing_logs are reloadable via consul reload or SIGHUP. kv_max_value_size is not listed as reloadable and likely requires a rolling restart Use that to tune snapshot cadence if large values are forcing more frequent snapshots, but remember that bigger snapshots are the core problem, not snapshot frequency alone.

Patch if you are below the CVE-2025-11374 fix line

Confirmed: HCSEC-2025-29; fixed in CE 1.22.0; Enterprise 1.22.0, 1.21.6, 1.20.8, 1.18.12

CVE-2025-11374 is described as a KV endpoint DoS caused by incorrect Content-Length header validation. An attacker who can omit the Content-Length header and send an arbitrarily large payload can force Consul to allocate a buffer proportional to the incoming data and exhaust memory. This is not the same as the 512KB value limit, but it interacts with the same code path. Any operator dealing with KV size issues should confirm they are on a fixed release.

Prevention

Treat KV as config and coordination, not storage. If a value is a blob, it does not belong in KV; the replication and snapshot cost makes every server pay for it forever.
Add a CI check on value size. For anything written by automation, assert the payload is under a conservative ceiling (say 64KB) before the write. Catch the problem at the writer, not in the server logs.
Track snapshot size as a first-class metric. Snapshot growth is the slow-moving signal that tells you KV or catalog bloat is accumulating before commit time suffers.
Keep kv_max_value_size consistent across federated DCs. Document the value and treat changes like a schema migration.
Watch for the txn envelope trap. Clients batching writes through /v1/txn should sum payload plus overhead and stay well under txn_max_req_len.

How Netdata helps

Per-second consul.raft.commitTime and consul.raft.fsm.apply latency let you see the write-pipeline cost of large KV values in real time, not at minute granularity where spikes wash out.
Snapshot size and disk I/O on the Raft data directory correlate directly with KV bloat; Netdata surfaces both alongside each other so you do not have to join them by hand.
Go runtime metrics (consul.runtime.alloc_bytes, consul.runtime.total_gc_pause_ns) show the memory and GC cost of deserializing large values, and whether snapshot creation is doubling RSS.
Leader transition counters and consul.raft.leader.lastContact catch the downstream consequence of KV-driven Raft saturation before it becomes a write outage.
ML anomaly detection on commit time and apply latency flags the slow drift that precedes a cliff, which is exactly the signature of an accumulating KV store.