Consul Go GC pauses: stop-the-world stalls that disturb Raft timing

Consul servers are Go binaries, and Go’s garbage collector still has stop-the-world phases despite doing most of its work concurrently. On a server with a large live heap and a high allocation rate, those pauses can stretch into hundreds of milliseconds. The Raft leader sends heartbeats from a goroutine in the same process. When a pause runs long enough, the leader stops heartbeating; followers see consul.raft.leader.lastContact climbing toward the election timeout and start a new election. Writes block for the duration.

This failure mode is subtle because the usual suspects, disk I/O and network, look clean. The operator instinct on leader thrashing is to check iostat and packet loss. Both come back normal. GC pause monitoring is usually added reactively, after an incident where disk and network were ruled out. The signal that pinpoints the cause is consul.runtime.gc_pause_ns correlated against lastContact and the leader election counter.

The interaction with raft_multiplier is what makes this dangerous. HashiCorp’s production recommendation is raft_multiplier=1, which sets the Raft HeartbeatTimeout to 1000ms; the default multiplier 5 sets it to 5000ms. A GC pause of a few hundred milliseconds consumes a large fraction of that window. With the default raft_multiplier=5, the same pause would be absorbed harmlessly. Clusters tuned for fast failover are the most exposed to GC-induced elections.

What this means

The cascade is a feedback loop. Heap size and allocation rate drive GC frequency and pause duration. Each long pause risks an election. Each election causes client retries and watch reconnections, which raise allocation pressure, which drives more GC. Breaking the loop means attacking at least one of: heap size, allocation rate, pause cost, or the Raft timing margin.

flowchart TD
    A["Large live heap
(multi-GB)"] --> B["High allocation rate
(catalog/KV churn)"]
    B --> C["Frequent GC cycles"]
    C --> D["Stop-the-world pause
(multi-ms)"]
    D --> E["Leader stops sending
Raft heartbeats"]
    E --> F["Followers: lastContact
climbs toward timeout"]
    F --> G{"Pause exceeds
heartbeat window?"}
    G -->|Yes| H["Follower starts election"]
    H --> I["Writes block
during election"]
    I --> J["Client retries
more alloc pressure"]
    J --> B
    G -->|No| K["Pause absorbed
no election"]

A few specifics matter for diagnosis:

• Go’s GC is concurrent but not pause-free. The stop-the-world phases (sweep termination, mark termination) are brief on small heaps but grow with live object count and, per the Go runtime’s design, with GOMAXPROCS contention. Pause cost is not purely a function of heap size.
• consul.runtime.gc_pause_ns is the per-cycle pause timer. consul.runtime.total_gc_pause_ns is the cumulative nanoseconds spent in stop-the-world pauses since process start. The cumulative counter must be differenced (a rate or non_negative_difference) to get a per-interval value. In Prometheus scrape format these appear as consul_runtime_gc_pause_ns and consul_runtime_total_gc_pause_ns.
• HashiCorp’s telemetry guidance warns when the cumulative total_gc_pause_ns exceeds 2 seconds per minute and marks critical above 5 seconds per minute.
• The leader is the critical node. A GC pause on a follower delays its own apply but does not trigger an election. A pause on the leader delays heartbeats to all followers simultaneously, which is what produces the symmetric lastContact spike across the cluster.

Common causes

Quick checks

Run these on the leader first. They are read-only except where noted.

# Identify the current leader
curl -s http://127.0.0.1:8500/v1/status/leader

# GC pause and heap metrics on this server
curl -s http://127.0.0.1:8500/v1/agent/metrics \
  | grep -E "runtime.*(gc_pause|alloc_bytes|heap_objects|num_goroutines)"

# Raft timing and election counters
curl -s http://127.0.0.1:8500/v1/agent/metrics \
  | grep -E "raft.*(lastContact|state\.leader|commitTime)"

# Build info, including the Go version Consul was compiled with
curl -s http://127.0.0.1:8500/v1/agent/self | grep -E "version|go_version"

# Runtime environment the process was launched with (GOGC, GOMEMLIMIT, GOMAXPROCS)
cat /proc/$(pgrep -x consul)/environ | tr '\0' '\n' | grep -E "^GO(GC|MAXPROCS|MEMLIMIT)="

# CPU usage and any cgroup quota (cgroup v2 then v1 fallback)
top -bn1 -p $(pgrep -x consul) | tail -1
cat /sys/fs/cgroup/cpu.max 2>/dev/null || cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us 2>/dev/null

# Disk latency on the Raft volume to rule out the #1 cause
iostat -x 1 3

For a heap or goroutine profile, Consul exposes Go pprof endpoints under /debug/pprof/ on the HTTP API port when enable_debug = true (the current default). A heap snapshot is read-only but briefly affects allocation timing, so capture it during an incident, not on a whim.

How to diagnose it

1. Confirm the symptom is leader elections, not just slow writes. Look at the election counter rate. More than 2 transitions in 10 minutes outside a maintenance window is the thrashing signature. A single transition with clean handoff is a different problem.

2. Look at lastContact across all followers. If lastContact spikes symmetrically on every follower immediately before each election, the leader is failing to send heartbeats. If only one follower spikes, the problem is that follower or its network path, not leader-side GC.

3. Rule out disk I/O on the leader. GC pauses and disk latency produce nearly identical lastContact symptoms but different root signals. If iostat await is clean and consul.raft.commitTime is not the leading indicator, disk is unlikely. Disk is by far the more common cause of leader thrashing, so do not skip this step.

4. Pull the GC pause series on the leader. Look for periodic spikes in consul.runtime.gc_pause_ns that line up in time with the lastContact spikes and the election events. The temporal alignment is the diagnosis. Aggregate the cumulative counter as a per-minute rate and compare against the 2s/min and 5s/min guidance.

5. Pull the heap series. Confirm the heap is large (multi-GB) and that allocation rate is high. A small heap with frequent GC suggests a leak or churn driving the cycle. A large, stable heap with long pauses suggests the per-cycle scan cost itself is the problem.

6. Check GOMAXPROCS and CPU headroom. Stop-the-world pause cost scales with GOMAXPROCS contention. A many-core server under CPU pressure can produce longer pauses than heap size alone would predict. Confirm GOMAXPROCS matches the intended core allocation; container runtimes sometimes expose a host-sized GOMAXPROCS inside a CPU-limited cgroup.

7. Capture profiles during an incident. A heap profile shows what holds the live memory. A goroutine profile shows whether blocking queries or watches are inflating allocation pressure. Compare a profile taken during a pause burst against one taken during steady state.

Metrics and signals to monitor

Fixes

Tune GOGC

GOGC controls the GC trigger ratio. The default of 100 means GC triggers when the heap doubles relative to the live set after the previous cycle. On a multi-GB heap, that means GC runs relatively infrequently but each cycle scans a large live set, producing longer pauses. Raising GOGC (for example to 200) lets the heap grow larger before triggering, reducing GC frequency at the cost of higher peak memory. Lowering it runs GC more often on a smaller heap, which can reduce per-cycle pause time but increases total CPU spent in GC.

There is no HashiCorp-published recommendation for GOGC on Consul servers, and the Consul source does not set a non-default value. The tradeoff is workload-specific: benchmark your catalog size and write rate before and after any change. GOGC is read at process start, so changes require a restart.

Consider GOMEMLIMIT

Go 1.19 introduced GOMEMLIMIT, a soft memory limit that lets the runtime run GC harder as it approaches the limit instead of waiting for the heap-ratio trigger. For Consul servers built with Go 1.19 or later, GOMEMLIMIT can cap heap growth and keep the process away from the OOM cliff. Set it below the container or host memory limit to leave headroom for snapshot creation, which temporarily increases working memory. A limit set too low can cause premature GC cycles and increase pause frequency, so size it with headroom.

Reduce heap pressure

The most durable fix is to shrink the working set and the allocation rate. Common drivers: large KV values being used as a datastore, unbounded health check result accumulation without deregister_critical_service_after, leaking blocking queries or watches inflating goroutine count, and catalog churn from flapping registrations. Each of these raises allocation rate or live object count, which feeds the GC cycle. Reducing churn often lowers both the pause frequency and the steady-state heap.

Loosen raft_multiplier

If GC pauses are modest in absolute terms (low hundreds of ms) but elections still fire, the Raft timing margin may simply be too tight for the runtime. Moving from raft_multiplier=1 to an intermediate value like 2 or 3 widens the heartbeat window without abandoning the production-tuning direction. Some operators at very large scale run an intermediate multiplier as a compromise between failover speed and tolerance to runtime and disk transients; HashiCorp documents the general lastContact-based tradeoff but does not publish a specific intermediate value. The cost is slower leader failover; the benefit is that a GC or disk transient no longer tips straight into an election.

Add CPU headroom

Because stop-the-world pause cost scales with GOMAXPROCS contention, CPU starvation directly amplifies GC pauses. If the Consul server is CPU-limited by a cgroup quota or a noisy neighbor, raising the limit or moving the server to a less contended host can reduce pause duration more than any GC tuning. Pair this with a check that GOMAXPROCS reflects the actual core allocation.

Prevention

• Instrument GC before the first incident, not after. Add consul.runtime.gc_pause_ns and the rate of total_gc_pause_ns to the server monitoring profile. This is a Level 3 maturity signal in the Consul monitoring model, and it is the one most teams add only after a leader-thrashing incident traced to GC.
• Alert on cumulative pause rate. Use HashiCorp’s 2s/min warning and 5s/min critical guidance as a starting point, and alert on individual pause spikes aligned with lastContact.
• Track heap and heap-object count as trends. A steadily growing heap with stable catalog size is a leading indicator that GC pauses will worsen over time.
• Treat GC and CPU headroom as part of the Raft stability budget when running raft_multiplier=1. Tight Raft timing assumes a well-behaved runtime. If you cannot guarantee that, widen the multiplier.
• Account for snapshot creation in memory sizing. Snapshot creation loads state into memory; a heap already near the limit will GC aggressively during snapshots, which is exactly when Raft timing is already stressed.
• Track the Go runtime version across Consul upgrades. GC behavior and GOMEMLIMIT support change between Go releases.

How Netdata helps

• Per-second resolution on consul.runtime.gc_pause_ns, heap metrics, and Raft timing makes the time alignment between a pause and an election visible. Aggregated minute windows often hide the sub-second GC transients that actually trigger elections.
• Correlating GC pause spikes with consul.raft.leader.lastContact and the leader election counter on one timeline confirms GC, rather than disk or network, as the root cause.
• ML anomaly detection on the GC pause series flags a shift in the pause distribution before it crosses a fixed threshold. The failure here is about pause duration trends, not just peaks.
• Heap and heap-object charts alongside catalog register/deregister rates show whether allocation pressure comes from churn or steady-state growth.
• Per-core CPU charts show whether pauses coincide with CPU pinning, which points at cgroup limits rather than heap size.

Related guides

• Consul catalog bloat: too many services and checks slowing everything down
• Consul registration storm: catalog churn overwhelming Raft
• Consul anti-entropy not syncing: local agent state and the catalog drifting apart
• Consul client rpc failed: agents alive but the catalog is going stale
• Consul DeregisterCriticalServiceAfter: instances vanishing from the catalog
• Consul DNS latency high: slow lookups stalling connections and failovers
• Consul DNS SERVFAIL: service discovery is broken for your applications
• Consul stale DNS queries: the agent is answering from cache
• Consul on EBS: burst-credit exhaustion and the sudden latency cliff
• Consul gossip encryption key mismatch: a botched keyring rotation splits the pool
• Consul gossip flapping: nodes oscillating between alive, suspect, and failed
• Consul serf queue backlog: an agent falling behind on gossip