Consul goroutine count climbing: the leak behind slow resource exhaustion

consul.runtime.num_goroutines is a gauge of how many concurrent operations the process is juggling: every blocking query, every gRPC or xDS stream, every health check handler runs inside a goroutine. In a healthy cluster the number is baseline-dependent but flat. When it creeps upward hour over hour, something is spawning goroutines and not cleaning them up.

The absolute number is a distraction. A medium cluster idles between a few hundred and 20,000 goroutines; a large Connect deployment legitimately runs higher. The trend is the signal. Monotonic growth without a matching increase in services, watchers, or sidecar count is a leak, regardless of where the number sits.

Each leaked goroutine holds a stack that starts at 2 KB and grows, plus references to any memory it touches. The Go runtime must scan every goroutine stack during stop-the-world GC phases, so a rising count lengthens GC pauses even when heap growth is modest. Sustained GC pause growth eats into Raft heartbeat budgets and can trigger leadership changes, turning a quiet resource leak into a cluster availability incident.

Frequent restarts mask the problem. If you restart agents weekly for deploys, the count resets each time and you never see the curve. The leak is still there. Plot the trend over the longest window your agents stay up.

What this means

A climbing goroutine count means the Consul process is creating goroutines faster than it is reaping them. The Go runtime does not garbage-collect goroutines; they exit only when the code running inside them returns or is unblocked. A leak is a code path that blocks forever, or a caller that spawns without tracking the lifecycle.

The leak almost always falls into one of three buckets.

Blocking queries that never terminate cleanly. A client opens an HTTP long-poll watch and then disconnects without sending the signal that ends the watch. The server goroutine parked on that watch waits for the full wait timeout (default five minutes) Confirmed by official API docs before it reaps, and a steady stream of misbehaving clients keeps the count growing.

xDS or gRPC streams that are not torn down. When Consul Connect is enabled, each Envoy sidecar holds a gRPC stream to a server. If the stream is abandoned rather than closed, the server goroutine serving it stays parked. At thousands of sidecars this adds up.

Health checks for services that were deregistered. The check loop is supposed to stop when the service is deregistered, but registration bookkeeping bugs or anti-entropy races can leave a check running against a service ID that no longer exists.

Each has a distinct signature when you correlate goroutine count with secondary signals.

flowchart TD
  A["num_goroutines climbing"] --> B{"Memory climbing too?"}
  B -- yes --> C["Classic leak: goroutines hold state"]
  B -- no --> D{"RPC connections climbing?"}
  D -- yes --> E["External client leak"]
  D -- no --> F["Lightweight leak: check handlers, watchers"]
  C --> G["Capture pprof goroutine profile"]
  E --> G
  F --> G
  G --> H["Group stack traces by call site"]

Common causes

Quick checks

All read-only. Run on the suspect agent.

# Current goroutine count
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep num_goroutines

# Heap allocation, for the classic-leak correlation
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -E "alloc_bytes|heap_objects"

# GC pause trend
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -i gc

# File descriptor usage and limit
ls /proc/$(pgrep -x consul)/fd | wc -l
grep "Max open files" /proc/$(pgrep -x consul)/limits

# Client RPC failure rate
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -E "client.rpc"

# Consul version, to rule out a known fixed leak
consul version

# Goroutine stack dump (safe, read-only; output can be large)
curl -s http://127.0.0.1:8500/debug/pprof/goroutine?debug=1 | head -40

How to diagnose it

1. Confirm the trend, not the absolute number. Pull consul.runtime.num_goroutines over the longest window the agent has been up. If the curve is flat or oscillates with load, this is not a leak. If it is monotonically rising, proceed.

2. Correlate with memory. Pull consul.runtime.alloc_bytes over the same window. Rising goroutines with rising memory is the classic leak signature: parked goroutines hold references to result sets, watch state, or stream buffers. Rising goroutines with flat memory means the goroutines are lightweight (parked on a channel, not holding data). The leak is still real but the failure mode is slower.

3. Correlate with RPC connections. Rising goroutines with rising RPC or HTTP active connections points at an external client leak: clients opening watches or streams and not closing them. Flat connection count points at an internal leak (health check handlers, watch handlers local to the agent).

4. Pull a goroutine profile during the climb. The single most useful artifact is /debug/pprof/goroutine?debug=1. It dumps every goroutine with its stack. Review the output for repeated stack patterns parked in blockingQuery, Watch, xDS serve handlers, or health check loops. The call sites with the highest repetition are your leak source.

   # Capture full profile to a file for offline analysis
   curl -s http://127.0.0.1:8500/debug/pprof/goroutine?debug=1 > /tmp/consul-goroutines.txt
   wc -l /tmp/consul-goroutines.txt
   

5. Identify the caller. For blocking query leaks, check which clients are issuing long-poll requests and whether they close them properly. Load balancers without session affinity can break index tracking and force clients to re-issue watches. For xDS leaks, compare stream count against the number of registered Envoy sidecars. For health check leaks, look for services deregistered while their checks were still running.

6. Check the version against known fixed leaks. Several goroutine leaks have been fixed in specific Consul releases. The go-memdb blocking-query leak (large services, default wait timeouts, goroutine counts spiking sharply) was addressed in Consul 1.14.0 backported to active branches. xDS subsystem goroutine leaks were also addressed in 1.14.0, and a further xDS stream handling leak was fixed in 1.18.1. If you are running an older release, the fix may be an upgrade.

Metrics and signals to monitor

Fixes

Blocking query leaks

The fastest mitigation is to stop the source. Identify misbehaving clients from access logs or by correlating connection counts with goroutine growth. Common culprits are consul-template fleets with too many templates, custom polling loops that do not respect X-Consul-Index, and load balancers that spread blocking queries across servers (which breaks index affinity).

If the source is large-service health queries, enable the streaming backend. use_streaming_backend replaces HTTP long-polling with server-pushed streaming RPC for endpoints that support it, eliminating the parked-goroutine pattern for those queries. It defaults to on in current Consul releases Confirmed: use_streaming_backend defaults to true in current releases; confirm in your agent config.

If you cannot stop the source, lower the blocking query wait timeout so parked goroutines are reaped faster. This trades freshness for headroom.

xDS stream leaks

Compare active xDS stream count against the number of registered Envoy sidecars. If streams significantly exceed proxies, sidecars are reconnecting without cleanly closing old streams.

curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -i xds

Short-term mitigation is a rolling restart of the affected sidecars or the Consul server. Long-term, verify you are on a release with the xDS leak fixes (1.14.0 for the subsystem-wide leak, 1.18.1 for stream handling). If you have a very large mesh, review whether enable_xds_load_balancing is appropriate; the stream limiter it introduces has its own failure modes ; verify behavior on your version before enabling.

Health check leaks

This is harder to mitigate without a code or version fix because the goroutine is internal to the agent. Identify the pattern by correlating the goroutine rise with service deregistration events. Services that register and deregister frequently with script checks are the prime suspects.

The durable fix is to ensure you are on a release where check handlers are correctly torn down on deregistration, and to set deregister_critical_service_after on services with transient registrations so that critical services are cleaned up rather than lingering with running checks.

Restarting as a last resort

A restart reaps every goroutine and resets the count to baseline. It also resets every cache, invalidates every watch, and triggers a reconnect storm from clients. Use it to buy time during an active incident, not as a substitute for finding the leak. If you are restarting agents on a schedule to keep goroutine count manageable, you have a leak that needs a code or config fix.

Prevention

• Plot goroutine count on the longest window agents stay up. A weekly restart schedule will hide a slow leak. If your agents restart weekly, capture the trend inside that week and watch the slope.
• Alert on the slope, not the absolute number. A useful starting point is sustained growth of more than 1000 goroutines per minute, or any monotonic climb over 24 hours. Adjust to your baseline.
• Correlate goroutine count with memory, FD usage, and connection count. Each correlation narrows the cause.
• Track Consul version against known leak fixes. Several significant leaks were fixed in the 1.14.x and 1.18.x lines. Running an older release is the most common reason a leak persists.
• Audit blocking query consumers. Every consul-template instance, every custom watcher, every long-poll client is a potential leak source. Track their count against your goroutine baseline.

How Netdata helps

• The Consul collector captures consul.runtime.num_goroutines, alloc_bytes, GC pause, and consul.raft.commitTime at per-second resolution, so you can correlate goroutine growth with memory, GC pressure, and write latency on a single timeline without manual instrumentation.
• Anomaly detection flags the slope change on goroutine count regardless of where the absolute number sits, which is the failure mode that hides leaks behind restart schedules.
• Per-node dashboards let you isolate a single misbehaving agent from a cluster-wide pattern.

Related guides

• Consul catalog bloat: too many services and checks slowing everything down
• Consul registration storm: catalog churn overwhelming Raft
• Consul anti-entropy not syncing: local agent state and the catalog drifting apart
• Consul client rpc failed: agents alive but the catalog is going stale
• Consul DeregisterCriticalServiceAfter: instances vanishing from the catalog
• Consul DNS latency high: slow lookups stalling connections and failovers
• Consul DNS SERVFAIL: service discovery is broken for your applications
• Consul stale DNS queries: the agent is answering from cache
• Consul on EBS: burst-credit exhaustion and the sudden latency cliff
• Consul gossip encryption key mismatch: a botched keyring rotation splits the pool
• Consul gossip flapping: nodes oscillating between alive, suspect, and failed
• Consul serf queue backlog: an agent falling behind on gossip