You pushed a value into Consul KV and got HTTP 413, or your server logs are filling with lines like Request body(524401 bytes) too large, max size: 524288 bytes. That is the obvious failure: a single value crossed the default 512KB ceiling. The less obvious failure is that values well under the limit are still expensive, because every KV byte is replicated through Raft to every server and re-emitted in every snapshot.

If you are hitting rejections, watching snapshot sizes creep upward, or seeing Raft commit time drift and wondering whether KV usage is the cause, the answer is usually yes. The fix is architectural: move blobs out of KV and keep only small config or pointers.

The 512KB limit exists to stop you from accidentally turning the Raft FSM into a blob store. Raising it via kv_max_value_size is possible, but the cost model that made 512KB the default does not go away when you raise the ceiling.

What this means

Consul enforces a default maximum value size of 512KB (524288 bytes) per KV entry. The check runs on the HTTP API write path and returns HTTP 413 to the client. The limit is configurable via kv_max_value_size , which defaults to the Raft suggested maximum of 512KB. A separate txn_max_req_len , also defaulting to 512KB, bounds the /v1/txn endpoint request body.

Two subtleties matter:

  1. The txn endpoint limit applies to the whole envelope, not per-value. A transaction packing several legitimate-size values plus JSON metadata can trip the limit even though no single value is near 512KB. This is the classic pattern where a secret store writes a payload that grows once wrapped in the txn structure.
  2. Cross-DC replication breaks when limits diverge. If the source datacenter runs a larger kv_max_value_size than the destination, replication fails with Request body too large errors. The mismatch is easy to miss because the primary DC looks healthy.

The reason raising the limit is usually wrong is the replication cost model:

  • Every KV write is a Raft log entry. The leader fsyncs it, then replicates it to a quorum.
  • Every KV value is held in the FSM on every server. A value stored once costs N copies of memory, where N is the server count.
  • Every KV value is included in every Raft snapshot. Snapshot creation loads the FSM into memory and writes it to disk; large values inflate both the memory spike and the on-disk snapshot size.
  • Snapshot restore on server restart reads the whole snapshot back. Larger snapshots mean longer cold starts.

So the symptom you are debugging might not be the 413 at all. It might be snapshot size growing week over week, commit time creeping up, or a server taking minutes to rejoin after a restart because it has to pull and apply a huge snapshot. The KV store is the hidden driver.

flowchart TD
    A[Large KV value written] --> B[Raft replicates value to N servers]
    B --> C[Value persists in every FSM]
    C --> D[Value re-emitted in every snapshot]
    D --> E[Snapshot I/O and memory spike on every cycle]
    E --> F[Raft commit time rises]
    F --> G[Heartbeat timeout risk]
    G --> H[Leader elections and write outage]

Common causes

CauseWhat it looks likeFirst thing to check
Single value exceeds 512KBHTTP 413 on write; Request body(N bytes) too large in server logsConfirm payload size client-side before the write
Transaction envelope exceeds limit (multiple small values plus metadata)413 on /v1/txn even though each value is smallSum value sizes plus txn overhead; compare to txn_max_req_len
Accumulated large-ish values strain the clusterNo 413, but snapshot size and commit time trending upEnumerate KV keys and sample value sizes
Cross-DC replication mismatchRequest body too large in secondary DC, primary is fineCompare kv_max_value_size on both sides

Quick checks

# Check whether a leader exists and writes are flowing
curl -s http://127.0.0.1:8500/v1/status/leader

# Enumerate the KV key space size
curl -s http://127.0.0.1:8500/v1/kv/?keys | jq 'length'

# Sample value sizes for the first 50 keys (read-only, safe)
for k in $(curl -s "http://127.0.0.1:8500/v1/kv/?keys" | jq -r '.[]' | head -50); do
  size=$(curl -s "http://127.0.0.1:8500/v1/kv/$k" | jq -r '.[0].Value' | base64 -d 2>/dev/null | wc -c)
  echo "$size $k"
done | sort -rn | head -20

# KV write latency and Raft commit time from telemetry (leader-only metrics)
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -E 'kvs.apply|raft.commitTime'

# Snapshot directory size (path is deployment-specific; /opt/consul is a common default)
du -sh /opt/consul/data/raft/
ls -lh /opt/consul/data/raft/snapshots/ 2>/dev/null | tail -5

The data directory path depends on your data_dir setting. The /opt/consul/data/raft/ path above is the playbook example; substitute your own. These checks are read-only and safe to run on a production server.

How to diagnose it

  1. Confirm the rejection is size-based, not permissions. A 413 is unambiguous. If you see 403, you have an ACL problem, not a size problem.

  2. If you have a 413, measure the offending value client-side. The error message includes the byte count. Compare it to 524288 (512KB), or to your configured kv_max_value_size if you raised it.

  3. If the symptom is cluster-wide degradation without 413s, profile the KV store. Run the size-sampling loop from the quick checks against the whole key space, not just the first 50 keys. Sort descending. Anything over roughly 50KB is suspicious; anything over roughly 200KB is a problem even though it fits.

  4. Correlate KV size with Raft and snapshot metrics. Look at consul.raft.commitTime, snapshot directory growth, and server RSS together. If all three trend upward and KV is the largest contributor to FSM size, you have found the cause.

  5. Check whether the txn endpoint is the culprit. If your client uses /v1/txn to batch writes, the limit applies to the whole request body. A batch of small values plus JSON overhead can exceed 512KB even when each value is small.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
consul.kvs.apply latencyEnd-to-end KV write cost; tracks Raft commitSustained p99 above ~200ms
consul.raft.commitTimeHealth of the Raft write pipelineTrending toward a meaningful fraction of the election timeout
Raft snapshot directory sizeSnapshot bloat from accumulated KV stateGrowth week over week without catalog growth
Server RSS and consul.runtime.alloc_bytesFSM memory cost; spikes during snapshot creationRSS climbing without service growth
HTTP 413 count on KV writesDirect evidence of limit rejectionsAny non-zero rate
consul.raft.state.leader transitionsLeadership stability; large values can destabilize RaftMore than one transition outside maintenance

Fixes

Move blobs out of KV (the real fix)

Stop using KV for large blobs. Store the payload in an object store (S3, GCS, MinIO), a document store, or a database, and put a pointer in KV. Terraform state is the textbook example: a JSON blob that belongs in S3 with a KV entry holding the S3 key.

Tradeoff: clients now do a KV read plus a blob fetch. That is almost always acceptable, because large values are rarely on the hot read path, and the win is a smaller FSM, smaller snapshots, and lower commit time for everything else.

Split large configs

If the payload is genuinely configuration (a large JSON or YAML document), split it across multiple keys under a prefix. Consumers read the prefix and assemble. Each key stays small, writes are cheaper, and you avoid the 413.

Tradeoff: atomicity. A multi-key update is not atomic without a transaction, and transactions have their own size limit. If you need atomicity, use a session and Check-And-Set semantics, or accept eventual consistency.

Compress in-band

If you control both writer and reader, gzip the value before storing and decompress on read. This can turn a 400KB JSON config into a 60KB KV value. It does not fix the architectural smell, but it buys time and reduces snapshot size immediately.

Tradeoff: compression makes KV values opaque to consul kv and the UI. Confirm every reader decompresses.

Raise the limit (usually wrong, sometimes necessary)

Set kv_max_value_size higher than the default. This is a valid escape hatch when you have an unavoidable large value and cannot immediately refactor. HashiCorp’s own warning is blunt: tuning improperly can cause Consul to fail in unexpected ways, potentially affecting leadership stability and preventing timely heartbeat signals by increasing RPC IO duration.

Before you raise it:

  • Treat the change as a capacity decision, not a config tweak. You are increasing Raft log entry sizes, snapshot sizes, and memory pressure on every server.
  • Make sure every server in every federated datacenter has the same setting. Divergent limits break cross-DC replication.
  • Set a deadline to remove the large value. Raising the limit does not make the cost model disappear; it just moves the cliff.

Since Consul 1.10.0 , raft_snapshot_threshold, raft_snapshot_interval, and raft_trailing_logs are reloadable via consul reload or SIGHUP. Use that to tune snapshot cadence if large values are forcing more frequent snapshots, but remember that bigger snapshots are the core problem, not snapshot frequency alone.

Patch if you are below the CVE-2025-11374 fix line

CVE-2025-11374 is described as a KV endpoint DoS caused by incorrect Content-Length header validation. An attacker who can omit the Content-Length header and send an arbitrarily large payload can force Consul to allocate a buffer proportional to the incoming data and exhaust memory. This is not the same as the 512KB value limit, but it interacts with the same code path. Any operator dealing with KV size issues should confirm they are on a fixed release.

Prevention

  • Treat KV as config and coordination, not storage. If a value is a blob, it does not belong in KV; the replication and snapshot cost makes every server pay for it forever.
  • Add a CI check on value size. For anything written by automation, assert the payload is under a conservative ceiling (say 64KB) before the write. Catch the problem at the writer, not in the server logs.
  • Track snapshot size as a first-class metric. Snapshot growth is the slow-moving signal that tells you KV or catalog bloat is accumulating before commit time suffers.
  • Keep kv_max_value_size consistent across federated DCs. Document the value and treat changes like a schema migration.
  • Watch for the txn envelope trap. Clients batching writes through /v1/txn should sum payload plus overhead and stay well under txn_max_req_len.

How Netdata helps

  • Per-second consul.raft.commitTime and consul.raft.fsm.apply latency let you see the write-pipeline cost of large KV values in real time, not at minute granularity where spikes wash out.
  • Snapshot size and disk I/O on the Raft data directory correlate directly with KV bloat; Netdata surfaces both alongside each other so you do not have to join them by hand.
  • Go runtime metrics (consul.runtime.alloc_bytes, consul.runtime.total_gc_pause_ns) show the memory and GC cost of deserializing large values, and whether snapshot creation is doubling RSS.
  • Leader transition counters and consul.raft.leader.lastContact catch the downstream consequence of KV-driven Raft saturation before it becomes a write outage.
  • ML anomaly detection on commit time and apply latency flags the slow drift that precedes a cliff, which is exactly the signature of an accumulating KV store.