Consul Raft log divergence: catching a corrupt follower before it wins an election

A divergent Consul follower can gossip normally, answer Serf probes, and appear in consul members as alive while its persisted Raft state is corrupt or inconsistent. The leader replicates to the healthy majority and clients see correct data. The divergence becomes catastrophic only when that follower wins an election and serves state the cluster never agreed on.

Two failure modes look identical from gossip. Replication lag: a follower falls behind on commit index due to slow disk or degraded network. Raft’s Leader Completeness Property prevents a lagging follower from winning, so this is self-correcting once the bottleneck clears. Silent corruption: the follower’s log index and term match the leader’s, but the entries it applied are wrong. The up-to-date check passes, the follower can win, and the divergence surfaces as incorrect registrations, stale KV, or broken ACL state.

The signal that catches both is consul.raft.last_index compared across every server. In a healthy cluster, all servers report nearly identical values. Large divergence means a follower is behind or inconsistent. This metric is rarely watched in steady state, which is exactly why it matters during election failures.

What this means

flowchart TD
    A[Follower Raft log diverges] --> B{Divergence type}
    B -->|Replication lag| C[Lagging index, consistent data]
    B -->|Silent corruption| D[Matching index, wrong data]
    C --> E[Up-to-date check fails]
    E --> F[Cannot win election]
    D --> G[Up-to-date check passes]
    G --> H[Corrupt follower wins election]
    A --> I[Gossip stays healthy]
    I --> J[No alarm fires]
    J --> H
    H --> K[Clients read divergent state]
    F --> L[Recovers once bottleneck clears]

Raft replication assumes every server’s FSM converges on the same state. Divergence breaks that assumption. A divergent follower may be applying entries in the wrong order, replaying stale-term entries, or holding corrupted log data it will reapply on restart. It does not crash, does not leave gossip, and does not report unhealthy. Serf membership and Raft state are independent subsystems; a server can be fully alive in one and corrupt in the other.

The risk materializes during leader election. Raft restricts candidates to those whose log is at least as up-to-date as a majority. A lagging follower fails this check and cannot win. A follower with silent corruption at the same index and term passes. If it collects a majority of votes, it becomes leader and the corruption is now authoritative for the cluster.

Common causes

Quick checks

Read-only and safe to run during an incident.

# List all Raft peers with voter status and leader
consul operator raft list-peers

# Per-server Raft internals (run on each server)
consul info | grep -A 15 "^raft"

# Raft metrics from the local agent's metrics API
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -E 'last_index|applied_index'

# Raft data directory size on a server
du -sh /opt/consul/data/raft/
ls -lh /opt/consul/data/raft/raft.db

# Check for divergence warning patterns in follower logs
journalctl -u consul --since "1 hour ago" | grep -iE "skipping application|old log|appendEntries.*fail"

How to diagnose it

Step through these in order. The goal is to distinguish lag, which is recoverable in place, from corruption, which requires removing the follower.

1. Snapshot the peer list. Run consul operator raft list-peers and record every voter, the leader, the commit index, and the Trails Leader By gap. Note any follower whose index trails by more than a few hundred entries.

2. Compare last_log_index on every server. SSH to each server and run consul info | grep last_log_index. In a healthy cluster the values match within a handful of entries. A follower far below the leader is lagging. A follower matching the leader but returning different query results is corrupt.

3. Check the per-follower trailing gap. The list-peers output shows how many commits each follower trails the leader by. A follower that never converges to zero despite a healthy leader has a stuck replication pipeline.

4. Compare raft.db size. A corrupt follower often has a raft.db file dramatically larger or smaller than peers. Documented upstream cases show a corrupt follower ballooning to tens of gigabytes while healthy servers sit at a few hundred megabytes, accompanied by constant disk writes and “Skipping application of old log” warnings. Run ls -lh on each server’s raft directory.

5. Check for apply warnings. Grep the follower’s logs for “Skipping application of old log” or AppendEntries pipeline failures. These indicate the follower is processing entries it should not be, or the leader is struggling to replicate to that specific follower.

6. Verify the leader’s per-follower view. On the leader, check consul.raft.leader.lastContact. If one follower shows consistently elevated lastContact while others stay normal, the problem is network or disk on that follower, not cluster-wide.

7. Distinguish lag from corruption. If the follower’s commit index is actively growing toward the leader, it is replication lag. If the index matches the leader but state queries return different results, it is corruption.

Metrics and signals to monitor

The Go/JSON names above may render differently in Prometheus, StatsD, or DogStatsD; confirm rendered names in your sink, but preserve the underlying verifier counter identities.

Fixes

Replication lag: follower is behind but consistent

Address the underlying bottleneck. The follower is not corrupt, it is slow.

• Slow disk: Check iostat -x 1 on the follower. If write latency (await) is elevated, the follower cannot acknowledge entries fast enough. SSD is baseline for Consul servers; rotational disks or shared network storage cause exactly this pattern.
• Network degradation: Verify pairwise latency between the leader and the trailing follower. Asymmetric partitions are common and only surface when you test the specific server pair.
• Stuck snapshot install: If the leader has truncated its log past the point the follower needs, the follower must receive a full snapshot. At high write rates, a follower whose snapshot restore takes longer than the leader’s log retention window will never catch up unless write volume drops.

Once the bottleneck clears, the follower converges on its own. Do not remove it from the peer set for lag alone.

Silent corruption: follower’s state diverges at the same index

The follower cannot be trusted. The safe path is to remove it from Raft, wipe its data directory, and let it rejoin from a fresh snapshot.

1. Remove the follower from the peer set. Run consul operator raft remove-peer -address=<server-address> to drop it from the voting set while preserving quorum on the remaining servers.
2. Stop the corrupt server. Halt the Consul process so it cannot win an election during cleanup.
3. Back up, then wipe the data directory. Copy the contents of the raft directory aside for post-incident analysis, then remove the raft files so the server starts clean.
4. Rejoin with clean state. Restart the server. It will receive a full snapshot from the leader and rebuild its log from scratch.

Warning: removing a peer changes quorum math. In a 3-server cluster, removing one peer leaves 2 voters and zero failure margin. Do not remove a second server until the first has rejoined and caught up.

Suspected corruption without confirmation

If you see divergence symptoms but cannot confirm corruption, treat the follower as untrusted. Prefer removing and rebuilding it over leaving it in the voting set, especially before any planned leader election or rolling restart. The cost of a clean rebuild is minutes of reduced redundancy. The cost of a corrupt leader serving divergent state is an incident.

Prevention

• Enable Raft log verification. Consul 1.15.0 added raft_logstore; configure raft_logstore.verification.enabled = true and a non-zero raft_logstore.verification.interval (HashiCorp recommends 30s–5m). The leader must have verification enabled to write checkpoints; enable it on followers so they verify those checkpoints. It works with any LogStore backend. Failures surface as consul.raft.logstore.verifier.read_checksum_failures (storage corruption) and write_checksum_failures (in-flight corruption).

• Monitor last-index divergence continuously. Alert when any server’s consul.raft.last_index differs from the cluster maximum by more than a small threshold. This is the single most effective early warning for both lag and corruption.

• Track raft.db size per server. Set a baseline and alert on outliers. A server whose raft.db is growing while peers stay stable is accumulating log entries it should have compacted away.

• Test failover before you need it. Trigger a leader election during a maintenance window and verify the new leader serves consistent state.

• Verify backend consistency after upgrades. The WAL LogStore backend shipped in Consul 1.15 and became the default in Consul 1.20. If you have crossed those releases, confirm every server uses the Raft LogStore backend you expect; mixed backends complicate storage-corruption diagnosis even though Raft replication semantics remain cluster-wide.

How Netdata helps

• Per-server lastLogIndex correlation. Netdata collects consul.raft.lastLogIndex from every server at per-second resolution. Overlaying these on one chart makes divergence visible the moment one server’s line separates from the pack, before an election forces the issue.
• commitTime and lastContact side by side. Correlating consul.raft.commitTime on the leader with consul.raft.leader.lastContact per follower separates disk-driven lag from network-driven lag from outright corruption.
• Anomaly detection on index divergence. Netdata’s ML flags the moment a follower’s log index stops tracking the leader, even when the absolute gap is small. Healthy divergence is near-zero, making manual thresholds unreliable.
• Filesystem metrics for the raft directory. Disk usage and I/O latency on the Raft data directory surface the ballooning-raft-db pattern before it becomes a corruption incident.
• Verifier failure surfacing. If you enable raft log verification, Netdata surfaces checksum-failure counters as they increment, turning an opt-in safety feature into an actionable alert.

Related guides

• How Consul actually works in production: a mental model for operators
• Consul leader election storm: repeated elections and rolling write outages
• Consul monitoring checklist: the signals every production cluster needs
• Consul monitoring maturity model: from survival to expert
• Consul “No cluster leader”: every write is failing
• Consul raft commitTime high: the write pipeline is slowing down
• Consul raft lastContact rising: followers drifting toward an election
• Consul lost quorum: Raft peers below the majority needed to elect a leader