Consul Raft data directory full: the server that can no longer write

The server’s data_dir volume hits 100%. Raft cannot append to raft.db and cannot fsync snapshot files. The server either refuses writes or crashes. If the affected server is the leader, the commit index stalls and followers drift toward an election.

The most common cause is not disk hardware failure. It is a snapshot that keeps failing. Raft truncates its log only after a successful snapshot, so each failed attempt leaves raft.db larger than it should be. Eventually the log fills the volume.

What this means

A Consul server’s data_dir holds three Raft artifacts:

• peers.info: informational file describing the voter set.
• raft.db: the log store. On older Consul this is a single BoltDB file; on Consul 1.21+ the default LogStore is WAL, which uses rotating segment files in the same directory.
• snapshots/: the two most recent FSM snapshots, named TERM-INDEX-TIMESTAMP, each containing meta.json and state.bin.

When the volume fills, two things break in quick succession:

• Raft cannot append new log entries. The error in server logs is typically [ERROR] agent.server.raft: failed to append to logs: error="unable to store logs within log store, err: "write <data_dir>/raft/raft.db: no space left on device"".
• Snapshot creation cannot complete. Snapshots are staged in a temp directory (default Go os.TempDir, usually /tmp) and atomically renamed into snapshots/. If either /tmp or the data_dir volume fills mid-write, the snapshot fails and the .tmp directory is left behind as an orphan.

The compound effect is what makes this incident sticky: failed snapshots prevent log truncation, which keeps raft.db growing, which keeps the disk full, which keeps snapshots failing. Restarting the server does not break the loop. You have to free space first.

flowchart TD
    A[Snapshot creation fails] --> B[Raft cannot truncate log]
    B --> C[raft.db grows between snapshots]
    C --> D[Disk fills]
    D --> E[Append to raft.db fails]
    D --> F[Snapshot fsync fails]
    E --> G[Leader: commit index stalls]
    F --> G
    E --> H[Follower: read-only or crash]
    G --> I[Election triggered]

Common causes

Quick checks

All read-only. Run on the affected server.

# Disk usage on the volume backing data_dir
df -h <data_dir>

# Total size of the Raft directory
du -sh <data_dir>/raft/

# Size of the log store (BoltDB file or WAL segments)
ls -lh <data_dir>/raft/raft.db
ls -lh <data_dir>/raft/

# Snapshot directory contents, including orphaned .tmp dirs
ls -lah <data_dir>/raft/snapshots/

# Look specifically for orphaned snapshot temp directories
find <data_dir>/raft/snapshots -maxdepth 1 -name '*.tmp' -type d -printf '%p\n'

# Size of the snapshot temp dir (default /tmp). Consul stages snapshot
# files here before renaming into snapshots/. If /tmp is small and
# the snapshot is large, every snapshot attempt fails here.
df -h /tmp

# Recent Raft and snapshot errors in Consul logs
journalctl -u consul --since '1 hour ago' | grep -iE 'raft|snapshot|no space left'

# Is this server currently the leader?
curl -s http://127.0.0.1:8500/v1/status/leader

# Full Raft peer view from this server
consul operator raft list-peers

If you do not know the data_dir path, read it from config:

grep -r 'data_dir' /etc/consul* /opt/consul* 2>/dev/null

How to diagnose it

1. Confirm the volume is actually full and identify the offender. Run df -h <data_dir> and du -sh <data_dir>/raft/*. If raft.db plus snapshots/ do not add up to the used space, a colocated workload is the cause. Treat that as a separate incident: stop the colocated writer first, then reassess Consul.

2. Inspect the snapshot directory for orphaned .tmp entries. Consul does not reliably clean up .tmp directories when snapshot creation fails . Each orphan persists indefinitely and ensures the next snapshot attempt also runs out of room. This is known behavior, not a transient blip.

3. Check consul.raft.leader.oldestLogAge (gauge, in milliseconds). This shows the age of the oldest log entry still in the leader’s log store. If it is climbing monotonically over hours or days, snapshot compaction is not happening on the cadence you expect.

4. Check whether the affected server is the leader. If yes, the impact is broader: every write to the cluster is failing or stalling. Plan remediation as a leader-impacting event, not a single-node cleanup. If quorum is intact, transfer leadership to a healthy peer before doing destructive cleanup.

5. Identify the LogStore backend. consul info shows the raft configuration. If you are on BoltDB, freelist bloat after truncation can compound disk pressure. If you are on WAL (default on recent Consul), look at segment file counts in data_dir/raft/.

6. Check the snapshot temp directory separately. Even when data_dir has room, a snapshot can fail because /tmp is full. The error message references /tmp/snapshotXXXX, not the data directory. This is easy to miss.

7. Estimate runway. If raft.db is growing because snapshots keep failing, you have until the disk fills before the server stops writing. The growth rate is roughly the Raft apply rate multiplied by average entry size, since no truncation is happening.

Metrics and signals to monitor

Fixes

Free space without losing Raft state

The only artifact in snapshots/ that is safe to delete wholesale is an orphaned .tmp directory. Real snapshots (named TERM-INDEX-TIMESTAMP, no .tmp suffix) are needed for recovery. raft.db and peers.info must not be touched.

# DESTRUCTIVE: stop Consul on this server first to ensure no snapshot
# is currently being written.
systemctl stop consul

# List orphaned .tmp directories before removing anything.
find <data_dir>/raft/snapshots -maxdepth 1 -name '*.tmp' -type d

# Remove ONLY orphaned .tmp snapshot directories.
find <data_dir>/raft/snapshots -maxdepth 1 -name '*.tmp' -type d -exec rm -rf {} +

# Verify the freed space.
df -h <data_dir>

# Restart and watch the snapshot succeed in the logs.
systemctl start consul
journalctl -u consul -f | grep -i snapshot

If the .tmp directories do not account for the bulk of the usage, the problem is raft.db itself (the snapshot failure loop has run long enough that the log is genuinely oversized) or a colocated workload. Do not delete raft.db as a space-recovery measure on a running leader.

Break the snapshot failure loop

If .tmp cleanup frees space but the next snapshot still fails, the underlying snapshot mechanism is broken. Two common fixable causes:

1. /tmp is too small. Set TMPDIR for the Consul process to a directory on the same volume as data_dir (or any volume with enough headroom for one full snapshot). For a systemd unit, add Environment=TMPDIR=<data_dir>/tmp to the [Service] section, create the directory, and restart.

2. Snapshot cadence is too slow for the write rate. raft_snapshot_threshold, raft_snapshot_interval, and raft_trailing_logs have been reloadable via consul reload or SIGHUP since Consul 1.10.0. Lowering raft_snapshot_interval to 5s and raft_snapshot_threshold to 8192 will compact the log faster, at the cost of more frequent disk-heavy snapshot writes. Verify the disk can sustain the higher snapshot rate before committing to it.

After either fix, watch consul.raft.leader.oldestLogAge drop back to a stable low value. If it does not, snapshots are still failing and you have not fixed the root cause.

Migrate off colocated workloads

If a colocated workload (application logs, another database, container images, a Prometheus data directory, anything write-heavy) filled the volume, the cleanup above is temporary. The durable fix is a dedicated volume for data_dir. HashiCorp guidance has been consistent on this: Raft data lives on a dedicated SSD, with nothing else on the volume. No amount of monitoring recovers from a neighbor that decides to write 50GB of logs.

Recover when raft.db itself is oversized

If you have cleaned .tmp directories, confirmed snapshots now succeed, but raft.db is still consuming the bulk of the volume, the existing log file will not shrink on its own. On BoltDB this is freelist bloat. On WAL it is segment accumulation.

The recovery path for a non-leader:

# DESTRUCTIVE: only do this on a follower, never on the leader.
# Confirm this server is not the leader first.
curl -s http://127.0.0.1:8500/v1/status/leader

systemctl stop consul
mv <data_dir>/raft <data_dir>/raft.bak.$(date +%s)
systemctl start consul

# The server will rejoin gossip, request a snapshot from the leader,
# and rebuild its raft directory. Monitor:
journalctl -u consul -f | grep -iE 'raft|snapshot|install'

This is the same recovery path as a corrupted data directory. The server will be a non-voter until it has caught up. Do not run this on more than one server at a time. Do not run this on the leader.

For BoltDB specifically, upgrading to a Consul version that defaults to WAL will avoid the freelist bloat pattern on future restarts.

Prevention

• Dedicated volume for data_dir. Nothing else writes to it. This is the single highest-leverage prevention step.
• Set TMPDIR explicitly. Do not rely on /tmp being large enough for a snapshot.
• Monitor consul.raft.leader.oldestLogAge. Any sustained growth means compaction is not happening on schedule. This metric gives you hours or days of warning before the disk fills.
• Monitor snapshot directory size trend. Growing snapshots indicate catalog or KV growth that will eventually make snapshot creation slow or fail.
• Monitor disk usage and disk write await on the data_dir device. Treat these as PAGE-level signals, not capacity planning signals. The cliff edge between “slow disk” and “leader election storm” is narrow.
• Audit what else writes to the volume. A quarterly check for accidental colocated writers (logrotate misconfiguration, a deploy that put containerd storage on the same LV) is cheaper than a 3am incident.
• Do not use Consul KV as a database. Large or frequently-written KV values inflate every snapshot and every Raft log entry.

How Netdata helps

• Per-second disk space and disk write await metrics on the data_dir volume catch filling before the cliff edge. A 1-minute poll interval is too coarse when the runway is measured in minutes.
• ML anomaly detection on consul.raft.leader.oldestLogAge flags the snapshot failure loop hours before the disk fills, while the metric is still well below any static threshold.
• Correlating disk write await on the data_dir device against consul.raft.commitTime and consul.raft.leader.lastContact distinguishes a disk problem from a network or GC problem in seconds.
• Disk usage per mount point, with per-process attribution where available, identifies the colocated workload that filled the volume without manual lsof work.
• Tracking consul.raft.state.candidate alongside disk saturation surfaces the moment the disk problem turns into a leader election, which is when the incident escalates from “one degraded server” to “cluster-wide write impact”.

Related guides

• Consul gossip flapping: nodes oscillating between alive, suspect, and failed
• Consul serf queue backlog: an agent falling behind on gossip
• Consul gossip storm after mass recovery: rejoin floods and anti-entropy spikes
• How Consul actually works in production: a mental model for operators
• Consul leader election storm: repeated elections and rolling write outages
• Consul monitoring checklist: the signals every production cluster needs
• Consul monitoring maturity model: from survival to expert
• Consul “No cluster leader”: every write is failing
• Consul raft commitTime high: the write pipeline is slowing down
• Consul raft lastContact rising: followers drifting toward an election
• Consul Raft log divergence: catching a corrupt follower before it wins an election
• Consul lost quorum: Raft peers below the majority needed to elect a leader