Consul snapshot size growing: state bloat, slow restores, and commit spikes

Consul snapshots grow with the size of the FSM state: every registered service instance, every health check, every KV entry, every ACL token, every session. None of those individual writes looks expensive, so a team that registers a few new services per week will not see a spike in any single metric. What they get instead is a slow, monotonic climb in snapshot size that compounds across months until one day a snapshot save takes long enough to push Raft commit times past the heartbeat timeout, or a rejoined follower sits in snapshot restore long enough to fall out of the replication loop entirely.

This is a slow-burn failure. No single operation looks expensive, and the cost shows up at the worst possible moment: during leadership churn, a rolling restart, or a failover when you most need followers to catch up quickly. The mental model for how Consul’s Raft and catalog interact is covered in How Consul actually works in production; this article assumes that background and focuses on the snapshot-growth symptom.

The fix is not reactive. By the time a snapshot save is slow enough to page you, the catalog or KV bloat has been building for weeks. Track snapshot size as a PLAN-level weekly trend, correlate it with catalog and KV growth, and catch the slope before the cliff.

What this means

A snapshot is the serialized form of the in-memory FSM: catalog, KV, sessions, ACLs, intentions, prepared queries. Consul writes snapshots periodically (controlled by raft_snapshot_interval, default 30s in modern versions) once the Raft log accumulates enough entries since the last snapshot (controlled by raft_snapshot_threshold, default 16384).

Every server restores the same snapshot when it joins, rejoins after a restart, or falls behind the leader’s trailing log window and must be caught up via InstallSnapshot RPC.

Three costs scale linearly, or worse, with snapshot size:

• Memory during creation. Snapshot save serializes in-memory state and temporarily inflates the working set. On a large FSM, this spike drives GC pause time up at the exact moment Raft needs the CPU.
• Disk I/O during save and restore. Snapshots are large sequential writes on save and large sequential reads on restore. On shared or burst-credit-bounded storage (EBS gp2), a multi-gigabyte snapshot can saturate the volume for seconds, competing with Raft log fsyncs.
• Restore time on rejoin. A follower that restarts must restore the latest snapshot before it can accept AppendEntries. Restore is single-threaded and CPU-bound. On a large snapshot this can take minutes, during which the follower is a non-participant in consensus.

The dangerous failure mode is when snapshot creation or restore takes long enough to interfere with Raft timing. Snapshot save holds disk I/O that Raft needs for log writes. Snapshot restore keeps a follower silent long enough that the leader’s trailing logs (raft_trailing_logs, default 10000 since Consul 1.5.3) are exhausted, forcing another full snapshot install in a loop.

flowchart TD
    A[Catalog + KV grow slowly] --> B[Snapshots grow proportionally]
    B --> C[Snapshot save spikes memory and disk IO]
    B --> D[Snapshot restore takes minutes on rejoin]
    C --> E[raft.commitTime rises during save]
    D --> F[Follower falls behind trailing log window]
    E --> G[Leader election during snapshot save]
    F --> H[InstallSnapshot loop - follower never catches up]
    G --> I[Write outage]
    H --> I

Common causes

Quick checks

These are safe, read-only commands. Run them on a server. Adjust the data directory path to match your data_dir config; /opt/consul/data/ is a common default but not universal.

# Snapshot size on disk
ls -lh /opt/consul/data/raft/snapshots/
du -sh /opt/consul/data/raft/

# Per-type breakdown of the latest snapshot (KVS, Register, Index, sessions, etc.)
consul snapshot inspect /opt/consul/data/raft/snapshots/<latest-snapshot-file>
# TODO: verify exact flag name for KV prefix-level breakdown - some versions support -kvdetails

# Catalog and KV counts (rough proxy for FSM size)
curl -s http://127.0.0.1:8500/v1/catalog/services | jq 'length'
curl -s 'http://127.0.0.1:8500/v1/kv/?keys' | jq 'length'

# Raft apply rate and commit time (leader only reports commitTime).
# Note: /v1/agent/metrics returns JSON; grep is a quick filter, not a parsed read.
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -E 'raft.commitTime|raft.apply|raft.fsm.apply'

# Disk write latency during a snapshot window
iostat -x 1 5

# Memory during snapshot save - watch alloc_bytes while a save runs
watch -n 1 "curl -s http://127.0.0.1:8500/v1/agent/metrics | grep runtime.alloc_bytes"

How to diagnose it

1. Establish the slope. Snapshot size alone is not the signal; the slope is. Collect snapshot file sizes weekly and plot them. If snapshot size is growing faster than your legitimate service count, you have bloat, not growth.

2. Break the snapshot down by type. Use consul snapshot inspect to see which section of the FSM dominates. A healthy cluster typically has catalog and KV as the largest sections. If Register history or ACL tokens dominate, that is the leak source.

3. Correlate snapshot size with live state counts. Snapshot size should track roughly with consul.catalog.services, service instance count, KV key count, and session count. If snapshot size grows while these counts stay flat, something is accumulating inside the FSM that is not visible in the live API.

4. Measure restore time on a test follower. In a staging cluster or during a planned restart, time how long a fresh follower takes from join to voter. If this number has grown from seconds to minutes, you are inside the danger zone for the InstallSnapshot loop. The threshold for concern is any restore that approaches the time it takes the leader to write raft_trailing_logs entries.

5. Watch commit time during a snapshot save. Snapshots run on the leader. Pull consul.raft.commitTime alongside consul.runtime.alloc_bytes during a snapshot interval. If commit time spikes upward during snapshot save and the snapshot is large, disk contention or memory pressure is the mechanism.

6. Verify disk latency on the Raft volume. Slow disk is the multiplier that turns a large snapshot into an outage. iostat -x 1 should show await well under 10ms; under 1ms is the HashiCorp guidance for server volumes. Anything higher turns snapshot save into a sustained Raft stall.

Metrics and signals to monitor

The single most important signal is the one most teams do not collect: snapshot size over time. Catalog and KV state grow a few entries per week and nothing looks expensive in isolation, but over months the snapshot balloons, memory grows, restore times creep up, and one day a snapshot save takes long enough to cause a leader election during creation. Track snapshot size as a PLAN-level weekly trend, not a page.

Fixes

Catalog and KV bloat: remove the source

The only durable fix for snapshot growth is removing the state that should not be there. Identify the dominant section of the snapshot via consul snapshot inspect, then go after it:

• KV used as a database. Move high-throughput or large-value workloads out of Consul KV. Consul KV is for configuration and coordination, not primary application state. Each KV write is a Raft commit replicated to every server and serialized into every snapshot.
• Zombie service registrations. If ephemeral workloads (CI runners, batch jobs, short-lived pods) register services without deregistering, the catalog accumulates dead instances. Add deregister_critical_service_after to health checks so the catalog self-cleans.
• Unbounded check count. Each registered check is a row in the FSM. Services that register dozens of per-instance checks (one per dependency, one per endpoint) multiply catalog size. Consolidate checks where possible.

Removing bloat does not shrink existing snapshots immediately. The next snapshot after cleanup will be smaller, but the on-disk snapshot files from before remain until they are rotated out by newer snapshots. Plan cleanup with enough lead time before the next rolling restart.

Raft tuning: break the InstallSnapshot loop

If followers are already stuck in a restore loop because the snapshot is large and write rate is high, the immediate lever is raft_trailing_logs. Increasing it from the default 10000 to a higher value (50000 is a common operational choice ) gives the leader more log entries to send via incremental AppendEntries before falling back to a full snapshot install.

Since Consul 1.10, raft_snapshot_threshold, raft_snapshot_interval, and raft_trailing_logs are reloadable at runtime via consul reload (which sends SIGHUP), with no rolling restart required:

# Apply raft tuning changes from config without a restart
consul reload

This does not shrink the snapshot. It widens the window in which a follower can catch up. The real fix is still reducing snapshot size or improving disk I/O.

Disk I/O: remove the multiplier

A large snapshot on slow disk is an outage waiting to happen. The operational guidance is consistent across the Consul community: dedicated SSD for the Raft data directory, nothing else on the volume. If you are on EBS gp2 with exhausted burst credits, the fix is either io1/io2 with provisioned IOPS or a larger gp3 baseline. The signal that matters is write latency (await), not throughput.

Memory headroom for snapshot save

Snapshot save temporarily inflates the working set. Size server memory so that steady-state heap leaves headroom for the snapshot spike and GC overhead. If steady-state heap is already high, a snapshot save can push you into GC thrashing, which then pushes Raft commit times past the heartbeat timeout. A useful rule: plan for snapshot creation to roughly double the effective memory pressure for the duration of the save.

Prevention

• Track snapshot size weekly. This is a PLAN-level signal. Plot it next to catalog service count, KV key count, and total KV bytes. The slope tells you whether you have legitimate growth or bloat.
• Track restore duration on every follower restart. Each rolling restart is a free restore-time sample. Log how long each server takes from join to voter and alert on the trend.
• Set deregister_critical_service_after on ephemeral service checks. This is the single highest-leverage prevention for catalog bloat in dynamic environments.
• Enforce a KV value size budget. Alert on any KV value above a threshold (a few KB is a reasonable ceiling for configuration data). Large values in KV are almost always a misuse.
• Do not colocate the Raft data directory with anything else. No logs, no other databases, no shared volumes. The Raft volume is a single-tenant resource.
• Run periodic consul snapshot inspect on the latest snapshot and record the per-type breakdown. A sudden change in the dominant section (for example, Register overtaking KVS) is an early signal of a new bloat source.
• Capacity-plan against snapshot size, not just service count. Server memory and disk headroom should be sized for the projected snapshot size, not the current one.

How Netdata helps

• Per-second metric resolution exposes the commit-time spike during snapshot save that minute-granular monitoring misses. A snapshot save that pushes consul.raft.commitTime from 20ms to 400ms for 8 seconds is invisible at 1-minute aggregation.
• Correlated dashboards let you put consul.raft.commitTime, consul.runtime.alloc_bytes, disk await, and consul.raft.leader.lastContact on one view. The shape of a snapshot-induced stall is distinctive: memory and disk latency rise together, commit time follows, last-contact spikes on followers.
• ML anomaly detection flags the slow trend in snapshot size and catalog counts before they cross a static threshold. Snapshot growth is exactly the kind of slow monotonic drift that threshold-based alerting handles poorly.
• Disk I/O per device at per-second resolution catches the EBS burst-credit exhaustion pattern that turns a large snapshot into a Raft stall. The disk metric is the multiplier; without it you will diagnose the snapshot as the cause when the disk is the actual bottleneck.
• Composite alerts on the catalog-bloat pattern (snapshot size rising while service count is flat) catch the silent-catastrophe signature before it surfaces as a restore loop or an election during snapshot save.

Related guides

• How Consul actually works in production: a mental model for operators
• Consul raft commitTime high: the write pipeline is slowing down
• Consul raft lastContact rising: followers drifting toward an election
• Consul leader election storm: repeated elections and rolling write outages
• Consul “No cluster leader”: every write is failing
• Consul Raft log divergence: catching a corrupt follower before it wins an election
• Consul monitoring checklist: the signals every production cluster needs
• Consul monitoring maturity model: from survival to expert
• Consul gossip storm after mass recovery: rejoin floods and anti-entropy spikes
• Consul gossip flapping: nodes oscillating between alive, suspect, and failed