Consul raft lastContact rising: followers drifting toward an election

consul.raft.leader.lastContact is the predictive signal before a Raft election fires. It measures the elapsed time since the leader last successfully contacted each follower server. When this value rises, a follower is drifting toward the election timeout. Cross that threshold and the follower starts a new election: writes stall, clients see “no cluster leader” errors, and downstream consumers retry in a thundering herd.

Most Consul metrics tell you something already broke. lastContact tells you something is about to break. A healthy cluster holds this value in the low tens of milliseconds. Trending above 200ms warrants investigation. Sustaining above 500ms means an election is imminent. The election timeout sits at approximately 1000ms with raft_multiplier set to 1 (the production recommendation). Higher multipliers raise the timeout proportionally.

Default raft_multiplier is 5 (since Consul 0.7); setting it to 1 gives highest-performance mode equivalent to pre-0.7 timing

The diagnostic pattern matters more than any single reading. All followers trending upward means the leader cannot send heartbeats on time. One follower drifting while others stay flat means the network path to that specific follower is degrading. These two scenarios have different root causes and different fixes.

What this means

consul.raft.leader.lastContact is a timer reported in milliseconds. It captures the gap between the current moment and the last successful heartbeat or AppendEntries acknowledgment from each follower. The leader sends heartbeats from the same goroutine loop that processes Raft log writes, so anything blocking that goroutine (disk fsync, GC pause, CPU starvation) delays heartbeats and inflates lastContact.

flowchart TD
    A["Healthy: lastContact under 50ms"] --> B["Drifting: 200-500ms, trending up"]
    B --> C["Imminent: over 500ms sustained"]
    C --> D{"Reaches election timeout?"}
    D -->|Yes| E["Follower starts election"]
    D -->|No, heartbeats resume| B
    E --> F{"Stable new leader?"}
    F -->|Yes| A
    F -->|No| G["Leader thrashing: repeated elections"]
    G --> C

Key properties:

• Moves between servers on leadership change. In Prometheus, the time series may briefly vanish during the transition as the old leader stops emitting and the new leader starts.
• Zero values are the leader itself. Filter them in Prometheus with consul_raft_leader_lastContact != 0.
• Per-follower granularity is the whole point. A single follower with high lastContact is a network story. All followers with high lastContact is a leader story.

The relationship between lastContact and the election timeout is what makes this metric predictive. When lastContact on any follower approaches the election timeout, that follower will request a new election. With raft_multiplier set to 1, the election timeout is approximately 1000ms. The 500ms page threshold gives you roughly half the timeout window to react before an election fires.

Autopilot adds a second threshold. Its LastContactThreshold defaults to 200ms. When a server exceeds this, Autopilot marks it unhealthy, and CleanupDeadServers may remove it from the Raft configuration. A lastContact spike can therefore cascade from “slow follower” to “removed peer” to “reduced quorum margin” in a single cleanup cycle.

Confirmed: hashicorp/raft#524 is open (no library fix yet); affects Consul >= 1.13.0; root cause is a check introduced in Raft v1.3.7; workaround is to reset the returning node

Common causes

Quick checks

All safe, read-only operations. Run on the current leader unless noted. Adjust scheme and port if your API listener uses TLS or a non-default port.

# Identify the current leader
curl -s http://127.0.0.1:8500/v1/status/leader

# Check lastContact values
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep lastContact

# Check commit time (correlates with leader pressure)
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep commitTime

# List Raft peers and voter status
consul operator raft list-peers

# Check disk I/O latency on the leader's data volume
iostat -x 1 5

# Check GC pause duration on the leader
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep gc_pause

# Check Autopilot configuration (LastContactThreshold, CleanupDeadServers)
curl -s http://127.0.0.1:8500/v1/operator/autopilot/configuration

# Check gossip membership (gossip and Raft can disagree)
consul members

# Pairwise network check to a specific follower
ping -c 10 <follower-address>

How to diagnose it

Step 1: Determine scope. Pull lastContact from the current leader. Are all followers trending upward, or is it isolated to one?

• All followers high: The leader cannot send heartbeats fast enough. The problem is on the leader.
• One follower high: The network path between the leader and that follower is degraded, or the follower itself cannot process incoming RPCs.

Step 2: If all followers are high, check leader-side causes.

Run iostat -x 1 5 on the leader. Look at await (write latency) and %util on the volume hosting the Raft data directory. Sustained await above 10ms is a red flag. The Raft goroutine that sends heartbeats also handles log writes. When fsync blocks, heartbeats are delayed.

Check consul.raft.commitTime on the leader. If commitTime is also elevated, the Raft write pipeline is saturated. Disk I/O is the most common cause. If commitTime is normal but lastContact is high, suspect GC pauses or CPU starvation.

Check consul.runtime.gc_pause_ns. Stop-the-world GC pauses block the Raft goroutine directly. Even tens of milliseconds of GC pause can push lastContact past the 200ms Autopilot threshold if they coincide with heartbeat intervals.

Step 3: If one follower is high, check the network path.

Run pairwise latency checks between the leader and the affected follower. Use ping, mtr, or tcpdump on port 8300 (the Raft RPC port). Look for packet loss, asymmetric routing, or latency spikes that do not appear on paths to other followers.

Check the follower’s own resource usage. A follower that is CPU-starved or disk-saturated may not process AppendEntries RPCs fast enough, even if the network is fine.

Step 4: Check for Autopilot interaction.

Review the Autopilot configuration. If CleanupDeadServers is enabled and LastContactThreshold is at the default 200ms, a lastContact spike can trigger peer removal. Check consul operator raft list-peers to verify all expected voters are present. A missing peer after a lastContact spike suggests Autopilot cleanup.

Step 5: Check for version-specific issues.

If the metric seems absent rather than elevated, verify your Consul version. Earlier releases had telemetry library bugs that could suppress metric emission under certain conditions.

Confirmed: hashicorp/consul#15231 describes elections taking 3–15 minutes after a node eviction starting from 1.13.x; resolution status unclear

Metrics and signals to monitor

Fixes

Leader disk I/O saturation

This is the single most common cause of lastContact spikes. The Raft goroutine sends heartbeats from the same loop that handles log writes. When fsync on the data directory blocks, heartbeats stop until the write completes.

Immediate actions:

• Identify the source of write churn. Health check flapping and KV write storms generate excessive Raft log entries. Check consul.catalog.register and consul.raft.apply rates for abnormal spikes.
• If running on AWS EBS gp2, check burst credit balance. Exhausted credits cause sudden latency cliffs with no warning in Consul metrics.

Structural fixes:

• Move the Raft data directory to a dedicated SSD volume. Never colocate with application logs, other databases, or noisy neighbors.
• Use provisioned IOPS rather than burst-dependent volumes. The cost difference is minor compared to the cost of a leader election storm.

Network path degradation to one follower

When only one follower shows high lastContact, the leader is fine. The problem is between the leader and that specific follower.

• Check pairwise latency between all server pairs, not just leader to follower. Asymmetric partitions (A reaches B, but B cannot reach A) are common and easy to miss if you only test one direction.
• Verify firewall rules on port 8300 (Raft RPC). A security group change can block Raft traffic while leaving gossip (port 8301) untouched, creating a split view where the node looks alive in gossip but is unreachable for consensus.
• Use mtr to identify where packets are lost or delayed in the path.
• Check the follower’s own CPU and disk. A follower that cannot process AppendEntries fast enough will show high lastContact even with a healthy network.

Leader GC pauses

Large Go heaps with high allocation rates produce stop-the-world pauses that block the Raft goroutine. Each pause inflates lastContact on all followers simultaneously.

• Check consul.runtime.alloc_bytes and consul.runtime.heap_objects trends. Growing heaps with no corresponding service growth indicate a leak.
• Correlate goroutine count with heap growth. Blocking query leaks and watch handler accumulation are common drivers of heap pressure.
• Consider tuning GOGC. The default of 100 may cause frequent pauses on large heaps. Increasing it reduces pause frequency at the cost of higher baseline memory usage.

Autopilot cleanup cascade

When LastContactThreshold (default 200ms) is exceeded, Autopilot may remove the server from the Raft peer set. This reduces quorum margin and, in Consul 1.13+, has been associated with election livelock when the removed server reconnects and requests votes with higher terms.

• Review whether CleanupDeadServers is appropriate for your environment. In networks with periodic latency spikes, aggressive cleanup can destabilize the cluster.
• Consider raising LastContactThreshold if your baseline network latency between servers is naturally elevated (cross-AZ, cross-region).
• Monitor consul.raft.peers count. An unexpected drop after a lastContact spike confirms Autopilot removal rather than a crash.

Leader CPU starvation

In containerized deployments, insufficient CPU limits cause scheduling delays that propagate to the Raft goroutine.

• Check container CPU limits against actual usage. Consul servers need burst capacity for Raft processing, TLS handshakes, and gossip encryption.
• CPU pinned at 100% causes gossip probe timeouts and heartbeat delays simultaneously. lastContact and gossip member health degrade together.
• Separate the server workload from other containers on the same host. Noisy neighbors steal CPU cycles at the worst possible moment.

Prevention

• Monitor lastContact at per-second resolution. Sub-minute drift that coarser aggregation smooths over is exactly the drift that precedes an election.
• Alert on trend, not just threshold. A static threshold of 500ms catches the moment before disaster. A trend alert at 200ms catches the drift that leads there. Use both, but weight the trend.
• Track disk I/O write latency as a first-class signal. The await metric on the Raft data volume is the leading indicator for disk-driven elections. Alert on sustained values above 10ms.
• Keep Raft data on dedicated SSDs. This is the single most impactful infrastructure decision for Consul stability.
• Monitor GC pause distributions, not just heap size. Go’s GC can keep heap stable while spending significant time in stop-the-world pauses. Track consul.runtime.gc_pause_ns percentiles.
• Verify Autopilot thresholds match your network. The default 200ms LastContactThreshold assumes low-latency LAN. Cross-AZ or cross-region deployments may need a higher threshold.
• Pair lastContact with election count monitoring. If consul.raft.state.leader increments alongside lastContact spikes, you are already in a thrashing pattern.

How Netdata helps

• Per-second resolution catches the sub-minute drift that precedes an election. The trend from 50ms to 200ms can develop in seconds; coarser aggregation smooths it over.
• Correlation across metrics distinguishes disk saturation from GC pressure from CPU starvation. When all followers drift, lastContact alongside commitTime, GC pauses, and disk await in a single view narrows the cause without switching tools.
• Anomaly detection flags gradual creep (for example, 30ms to 150ms over an hour) that static thresholds miss but that still precedes an election.
• Multi-node correlation handles the metric’s migration during leadership changes. When leadership moves, the metric follows the new leader.
• Disk I/O on the same node completes the root cause picture. The await on the leader’s data volume is one correlation away.

Related guides

• How Consul actually works in production: a mental model for operators
• Consul monitoring checklist: the signals every production cluster needs
• Consul monitoring maturity model: from survival to expert
• Consul “No cluster leader”: every write is failing