Consul server memory climbing toward OOM: heap, catalog, and snapshots

A Consul server’s RSS is climbing and OOMKill is close. The cause could be legitimate catalog growth, a goroutine leak, snapshot amplification, or a runtime pathology. Memory growth in Consul usually mixes real state growth, Go runtime behavior, and at least one subsystem leaking.

Go’s garbage collector does not return memory to the OS immediately. RSS of roughly 2x the live heap is normal and is not a leak. What you should investigate is monotonic RSS growth that survives a GC, or live heap (consul.runtime.alloc_bytes) that does not return to a baseline after a transient event.

Consul server memory scales with five things: catalog size (services x instances x checks), KV store size, active blocking queries, goroutine count, and gossip membership. The leader also pays a premium during Raft snapshot creation, because the snapshot path loads full FSM state into RAM before persisting. If steady-state heap is near your memory ceiling, a snapshot will push you over.

What it means

The failure mode is not just OOMKill. As the heap grows, the Go runtime spends more time in GC. GC pauses are stop-the-world. Once pauses approach the Raft heartbeat or election timeout, you get leader churn, which cascades into client retries, watch re-establishment, and even more goroutines. Memory pressure converts into availability pressure well before the kernel kills the process.

Three patterns to distinguish:

1. Heap-driven growth. Live heap is expanding because the working set (catalog, KV) is growing or because goroutines are accumulating references the runtime cannot collect.
2. Catalog-driven growth. Legitimate state growth that exceeds capacity. The fix is capacity, not bug hunting.
3. Snapshot-driven spikes. Acute spikes during Raft snapshot creation that push an otherwise healthy server over its limit. Steady state is fine, but the periodic spike kills you.

Separating these requires correlating consul.runtime.alloc_bytes, consul.runtime.num_goroutines, catalog and KV counts, snapshot events, and OS RSS over the same time window. Single-metric analysis will mislead you.

Common causes

Quick checks

Run these on the affected server. All are read-only.

# Go runtime memory gauges
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -E "consul_runtime_alloc_bytes|consul_runtime_sys_bytes|consul_runtime_heap_objects"

# OS-level RSS for comparison
grep VmRSS /proc/$(pgrep -x consul)/status

# Goroutine count
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep consul_runtime_num_goroutines

# GC pause distribution
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep consul_runtime_gc_pause

# Catalog size (number of services)
curl -s http://127.0.0.1:8500/v1/catalog/services | jq 'length'

# KV key count (CAUTION: response can be large on big KV stores; avoid on a server near OOM)
curl -s http://127.0.0.1:8500/v1/kv/?keys | jq 'length'

# Snapshot files and sizes (adjust path to match your data_dir)
ls -lh /opt/consul/data/raft/snapshots/

# Heap profile (safe to capture during an incident)
curl -s http://127.0.0.1:8500/debug/pprof/heap > /tmp/consul-heap-$(date +%s).pb.gz

# Goroutine profile, top stack sites
curl -s "http://127.0.0.1:8500/debug/pprof/goroutine?debug=1" | head -60

# Confirm leader so you interpret leader-only metrics correctly
curl -s http://127.0.0.1:8500/v1/status/leader

Capture the heap and goroutine profiles early and again ten minutes later. The delta shows what is growing.

How to diagnose it

Work through the decision tree in order. The goal is to classify the growth before you change anything.

flowchart TD
  A["RSS or alloc_bytes climbing"] --> B{"Goroutines also rising?"}
  B -- yes --> C["Leak: blocking queries, watches, gRPC streams"]
  B -- no --> D{"Catalog or KV growing?"}
  D -- yes --> E["Legitimate state growth: resize"]
  D -- no --> F{"Spikes during snapshots?"}
  F -- yes --> G["Snapshot memory amplification"]
  F -- no --> H{"Sustained GC pauses?"}
  H -- yes --> I["Heap pressure: GC tuning"]
  H -- no --> J["Version-specific bug or CVE"]

1. Confirm it is real growth, not GC lag. Compare consul.runtime.alloc_bytes (live heap) to OS RSS. RSS of roughly 2x live heap is expected. If both are climbing together, or if alloc_bytes itself is monotonically rising, you have real growth. If only RSS is high while alloc_bytes oscillates, you are seeing Go’s retention of released pages, not a leak.

2. Check goroutine count trend. If consul.runtime.num_goroutines is climbing, capture /debug/pprof/goroutine?debug=1 twice, ten minutes apart. The top stack sites tell you the leak source. Common offenders are blocking queries from clients that never closed the watch, gRPC streams from Connect sidecars, and retry loops after a leader change. WAN federation and peering have known goroutine leak patterns in some versions.

3. Quantify catalog and KV growth. Compare total services, total checks, and total KV keys against your known inventory. If registrations are growing without corresponding deployments, you have a churn problem (see Consul catalog bloat) or a deregistration gap. If KV keys are accumulating, identify the application treating Consul KV as a database.

4. Correlate snapshot events with memory spikes. Snapshot creation loads the full FSM into RAM. If periodic spikes line up with snapshot creation, snapshot amplification is the trigger. The steady state may be fine but the spike kills the process. Measure snapshot size on disk as a proxy for in-memory cost.

5. Check GC pause impact on Raft. Pull consul.runtime.gc_pause_ns and consul.raft.commitTime over the same window. If GC pause spikes line up with commit time spikes or leader transitions, heap pressure has become an availability problem, not just a memory problem.

6. Verify the binary architecture. A 32-bit Consul binary (arch=386) is constrained to a ~4GB address space and will OOM well before exhausting physical RAM. Check consul version and the binary’s architecture. The fix is the 64-bit (amd64) binary.

7. Check version against known issues. CVE-2025-11374 (HCSEC-2025-29) allows a PUT to /v1/kv/ without a Content-Length header to bypass KVMaxValueSize and OOM the server. Reported fix versions: 1.22.0, 1.21.6, 1.20.8, and 1.18.12. WAN federation and peering have caused unbounded goroutine growth in some 1.20.x and 1.13.x releases for some 1.20.x and 1.13.x releases; consult the issue tracker. If your growth pattern matches a known issue, an upgrade is the remediation.

Metrics and signals to monitor

Leader-only metrics (commitTime, fsm.apply, catalog.register) move between servers on leadership change. Your monitoring must follow the leader or you will see gaps.

Fixes

Goroutine leaks

Capture /debug/pprof/goroutine?debug=1 and identify the dominant stack. Most leaks trace to blocking queries from clients that disconnect without closing, consul-template instances multiplied beyond expectation, or gRPC streams from Envoy sidecars not being reaped. The immediate mitigation is to restart the offending clients. The long-term fix is to ensure clients close watches on shutdown and to bound blocking query concurrency. For WAN federation or peering goroutine leaks tied to a specific Consul version, the fix is an upgrade.

Catalog and KV growth

If growth is legitimate (more services, larger KV), the fix is capacity planning. The sizing rule is 2 to 4 times the working set, measured from consul.runtime.alloc_bytes on the leader. If KV is the driver, move high-throughput or large-value workloads off Consul KV. For churn-driven growth, see Consul registration storm.

Snapshot memory amplification

Confirmed: both are reloadable since Consul 1.10.0

Two levers:

• Raise raft_snapshot_interval (default 30s) to reduce snapshot frequency and give the server more time to recover between spikes.
• Lower raft_snapshot_threshold (confirmed default: 16384 entries since Consul 1.1.0; previously 8192) to make each snapshot smaller, which reduces the in-memory state loaded during creation.

Lowering the threshold triggers more frequent snapshots (more disk IO) with smaller state per snapshot

The more important fix is to keep steady-state heap below ~50% of available memory so the snapshot spike fits. If steady-state heap is already near the limit, no snapshot tuning will save you. You need more RAM or less state.

GC pressure

If consul.runtime.gc_pause_ns is the problem rather than raw heap, tune the Go runtime. GOGC (default 100) controls the GC trigger relative to live heap. Raising it reduces GC frequency at the cost of higher peak heap. GOMEMLIMIT sets a soft ceiling the runtime respects. In containerized environments, set GOMEMLIMIT to roughly 90% of the cgroup memory limit to trigger GC before the kernel does. Leave headroom for non-Go allocations and cgroup overhead; setting GOMEMLIMIT too close to the hard limit can itself cause OOMKill.

No evidence that Consul sets GOMEMLIMIT automatically; it must be configured by the operator

GODEBUG=madvdontneed=1 forces the runtime to return memory to the OS more aggressively after GC, which can reduce RSS at the cost of page faults. Most useful in containers where RSS triggers OOMKill before live heap does.

GODEBUG=madvdontneed=1 is no longer needed since Go 1.16 (MADV_DONTNEED is the default)

Version-specific issues

If the growth pattern matches CVE-2025-11374 (unbounded KV body via missing Content-Length), upgrade to a fixed version. For WAN federation or peering goroutine leaks in 1.13.x or 1.20.x, consult the issue tracker for the specific fix version.

Prevention

• Track steady-state heap against 50% of memory. The 50% threshold leaves room for snapshot spikes. Alert when sustained above 70%, page when approaching the cgroup or system limit.
• Trend goroutine count. It should be flat in steady state. Any monotonic climb is a leak, regardless of absolute number.
• Trend snapshot size. Growing snapshot size means growing state means growing memory.
• Set GOMEMLIMIT in containers. Without it, the Go runtime has no awareness of the cgroup limit and will allocate until OOMKill. No evidence that Consul sets GOMEMLIMIT automatically; it must be configured by the operator
• Run the 64-bit binary. The 32-bit binary’s 4GB ceiling is a silent cap.
• Follow the leader in monitoring. Leader-only metrics disappear and reappear on different servers during transitions.

How Netdata helps

• Per-second resolution on consul.runtime.alloc_bytes, sys_bytes, and heap_objects distinguishes monotonic growth from normal GC oscillation. A 10-second aggregation window hides the pattern.
• ML anomaly detection on goroutine count catches slow leaks before they reach the scheduling-overhead threshold. A 1% per day drift is invisible to static thresholds.
• Correlation of GC pause with Raft commit time on the same dashboard shows when memory pressure becomes availability pressure.
• OS RSS alongside Go runtime gauges answers the “real growth or GC retention?” question in seconds.
• Snapshot event correlation with memory spikes confirms snapshot amplification as the cause.

Related guides

• Consul catalog bloat: too many services and checks slowing everything down
• Consul registration storm: catalog churn overwhelming Raft
• Consul anti-entropy not syncing: local agent state and the catalog drifting apart
• Consul client rpc failed: agents alive but the catalog is going stale
• Consul DeregisterCriticalServiceAfter: instances vanishing from the catalog
• Consul DNS latency high: slow lookups stalling connections and failovers
• Consul DNS SERVFAIL: service discovery is broken for your applications
• Consul stale DNS queries: the agent is answering from cache
• Consul on EBS: burst-credit exhaustion and the sudden latency cliff
• Consul gossip encryption key mismatch: a botched keyring rotation splits the pool
• Consul gossip flapping: nodes oscillating between alive, suspect, and failed
• Consul serf queue backlog: an agent falling behind on gossip