$ guides / consul / consul-blocking-query-accumulation ▌

Operations Guides

Consul blocking query accumulation: leaked watches that pile up goroutines

Goroutine count on your Consul servers is creeping upward. File descriptor usage follows the same trajectory. Neither reverses during quiet periods. Eventually the server hits its FD limit or goroutine scheduling overhead degrades everything: new connections are refused, health checks stop reaching the catalog, anti-entropy sync stalls, and the cluster cascades.

The culprit is leaked blocking queries. Consul’s watch mechanism and any client using the HTTP long-polling API (?index= and ?wait= parameters) holds a goroutine and an FD on the server for up to the wait timeout (5 minutes default, 10 minutes maximum). A fleet of consul-template instances, application-level watchers, or service mesh control planes can open thousands of concurrent blocking queries. This is fine when queries are properly opened and closed. The leak starts when they are not: a client crashes without cleanly closing its connection, a consul-template bug prevents goroutine cleanup, a watch configuration exceeds the server’s tracking capacity, or a go-memdb WatchSet bug causes goroutine proliferation.

The accumulation is slow enough that it is visible in telemetry for days or weeks before it becomes an incident, but nothing alerts on it because the absolute count has not crossed a hard threshold.

How blocking queries work

A blocking query is Consul’s alternative to polling. The client sends a request with ?index= (from the previous response’s X-Consul-Index header) and ?wait= (how long the server holds the connection open). The server parks the request in a goroutine, watching the state store for changes. If the data changes, the server responds immediately. If nothing changes within the wait period, the server responds with the same data and the client re-issues with the updated index.

Each blocking query consumes:

One goroutine for the duration of the wait (up to 5 minutes default, 10 minutes maximum)
One file descriptor for the HTTP connection
Memory for the goroutine stack and any cached result set

A consul-template instance watching 10 service endpoints opens 10 concurrent blocking queries. A fleet of 100 opens 1000. The leak occurs when queries are opened but never torn down.

flowchart TD
    A["consul-template fleet
app watchers"] -->|"?index= + ?wait= query"| B["Server parks goroutine
+ FD per query"]
    B -->|"client crash or
misconfigured watch"| C["Query leaks: goroutine
+ FD not released"]
    C -->|"accumulates over
hours to weeks"| D["Goroutine count climbs
FD count climbs"]
    D -->|"hits FD or goroutine
limit"| E["New connections refused"]
    E --> F["Health checks fail
anti-entropy stalls"]
    F --> G["Catalog goes stale
discovery breaks"]

Several distinct mechanisms produce the same symptom: rising goroutine count and FD usage.

Client disconnect without cleanup. When a client abruptly terminates a blocking query (process crash, network timeout, load balancer killing the connection), the server-side goroutine and FD may remain until the full wait timeout expires. A misbehaving client that repeatedly opens and aborts queries piles up goroutines faster than they expire.

consul-template goroutine leaks. consul-template creates and destroys watches on configuration reload. Lifecycle bugs can leak goroutines. consul-template v0.37.4 fixed a leak where dependencies could be added after a runner stops. Earlier versions had similar issues on config reload, including a Vault token watcher leak fixed in v0.29.4.

go-memdb WatchSet goroutine leak. A bug in WatchSet.watchMany() caused goroutine proliferation during blocking queries on large service catalogs. With a service of 10,000 instances, goroutines could spike from 40,000 to over 1 million. Fixed in Consul 1.12.7, 1.13.4, and 1.14.0 via a go-memdb update (HashiCorp merged a 1.11 backport, but published no later 1.11 patch release).

Watch limit fallback busy-loop. Consul caps the number of watch channels per blocking query (originally 2048, increased to 8192 in Consul 1.7.0). In the unoptimized health-query path, each instance can add three watches (node, node-level checks, service-specific checks); modern Consul optimizes eligible services to a service-level watch to avoid this. When a service still exceeds this limit, the WatchSet falls back to coarse-grained root nodes, watching all services, checks, and nodes. The blocking query returns early on any unrelated change, re-queries the tree, and repeats. This amplifies CPU load without leaking goroutines per se.

Streaming backend gRPC leak. When the streaming backend is enabled, gRPC connections can accumulate on client agents during router rebalancing. The cache entry is evicted after its 20-minute idle TTL. This produces a different leak surface than HTTP long-polling but the same class of resource exhaustion on client agents.

Common causes

Cause	What it looks like	First thing to check
consul-template goroutine leak	Goroutine count grows after config reloads and never fully recovers	consul-template version
Client crash without closing query	Steady goroutine growth correlating with deploys or restarts; FD count tracks goroutine count	Access logs for source IPs of blocking queries
Watch limit fallback busy-loop	High RPC query rate with stable Raft commit index; elevated cache misses; CPU spike on servers	Service instance count vs watch limit (up to three watches per instance in the fallback path)
go-memdb WatchSet leak	Massive goroutine spike (100K+) correlating with high-churn service or large catalog	Consul version (patched in 1.12.7, 1.13.4, 1.14.0)
Streaming backend gRPC leak	gRPC connection count grows on client agents and does not reverse	`consul.grpc.client.connections` on client agents

Quick checks

Run these read-only commands on the affected Consul server. None are disruptive.

# Goroutine count
curl -s http://localhost:8500/v1/agent/metrics | grep consul_runtime_num_goroutines

# Count open FDs directly from the OS (assumes single consul process)
ls /proc/$(pgrep -x consul | head -1)/fd/ | wc -l

# OS-level FD limit
cat /proc/$(pgrep -x consul | head -1)/limits | grep "Max open files"

# Goroutine profile: stack traces grouped by call site
curl -s http://localhost:8500/debug/pprof/goroutine?debug=1 | head -100

# Watch-related metrics
curl -s http://localhost:8500/v1/agent/metrics | grep consul_watch

# HTTP request latency (blocking queries inflate aggregate latency)
curl -s http://localhost:8500/v1/agent/metrics | grep consul_http | grep -E "Mean|Count"

# Verify streaming backend for health queries (this is itself a 2-second blocking query)
curl -s -o /dev/null -D - "http://localhost:8500/v1/health/service/consul?index=0&wait=2s" | grep -i x-consul-query-backend

The pprof goroutine dump is the single most useful check. It shows stack traces grouped by call site, so you can immediately see whether leaked goroutines cluster in blocking query handling, watch processing, or something else entirely.

Metric names above use Prometheus-style underscore format. If your Consul telemetry uses the default dot format, replace underscores with dots (for example, consul.runtime.num_goroutines).

How to diagnose it

Confirm goroutine growth is abnormal. Compare against baseline. For medium clusters, 10,000 to 20,000 goroutines is typical. Monotonic increase over hours or days without load increase is the signal. A spike to 100K or more is an active leak.
Capture a goroutine profile during the leak. Run curl -s http://localhost:8500/debug/pprof/goroutine?debug=1 > goroutine-dump.txt. Look for stacks containing blockingQuery, WatchSet, WatchCtx, or watchMany. Count goroutines sharing the same stack. The call site with the most goroutines is your leak source.
Identify the source clients. Check Consul HTTP access logs or network-level connection tracking for clients with many concurrent connections. Each blocking query holds a separate HTTP connection. A single consul-template instance or application pod with hundreds of connections to a server is suspicious. Look for patterns: is growth from a specific service, deployment, or namespace?
Verify consul-template version. If consul-template is in the stack, versions before v0.37.4 have known goroutine leaks on config reload. If you cannot upgrade immediately, reducing config reload frequency slows the leak.
Check for watch limit fallback. For health queries, use up to three watches per instance in the unoptimized path. If this exceeds 8192 (or 2048 on Consul before 1.7.0), the query falls back to root nodes and busy-loops. The symptom is high consul_rpc_query rate with a stable consul_raft_commitIndex and elevated cache miss rate despite stable data.
Check Consul version for known bugs. The go-memdb WatchSet leak was fixed in 1.12.7, 1.13.4, and 1.14.0. If your servers are on 1.11.x or an earlier patch in 1.12.x/1.13.x with services that have thousands of instances, this may be the root cause.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`consul_runtime_num_goroutines`	Direct proxy for concurrent blocking queries	Monotonic growth without load increase; above 50K warrants investigation, above 100K is urgent
FD usage (read `/proc/<consul-pid>/fd` and `/proc/<consul-pid>/limits`)	Blocking queries hold FDs; exhaustion cascades to refused connections	Above 70% of soft limit, or sustained growth without corresponding connection count
`consul_watch` metrics	Watch handler count and execution rate	Count growing without corresponding service growth
HTTP API latency (`consul_http_*`)	Blocking queries with 5-minute waits inflate latency	Elevated mean latency that does not match user-perceived slowness
`consul_rpc_query` rate	Forwarded queries indicate cache misses or stale reads	High rate with stable `consul_raft_commitIndex` suggests read amplification, not write load
`consul_cache` miss rate	Cache thrashing from watch limit fallback or improper index handling	Elevated miss rate despite stable catalog data
`consul.grpc.client.connections` (if streaming enabled)	Streaming backend can leak gRPC connections during rebalance	Growing connection count on client agents that does not reverse

Note on HTTP latency: blocking queries with a 5-minute wait that return after 3 minutes report 3 minutes of latency. This is normal for that query type but pollutes aggregate latency metrics. Filter out requests carrying ?wait= or ?index= when analyzing API performance.

Fixes

Upgrade consul-template

If the goroutine profile shows leaked goroutines in consul-template’s watch lifecycle and the version is below v0.37.4, upgrade. The fix prevents dependencies from being added after a runner stops.

Upgrade Consul

If goroutine profiles show proliferation in WatchSet.watchMany() and the Consul version predates the 1.12.7, 1.13.4, or 1.14.0 patch line, upgrade. This matters most for clusters with large service catalogs where individual services have thousands of instances.

Enable the streaming backend

Consul 1.9+ introduced a streaming backend for service health queries that replaces HTTP long-polling with gRPC subscriptions. When active, API responses include X-Consul-Query-Backend: streaming. The streaming backend eliminates the per-query goroutine model for health queries, reducing server goroutine pressure.

Caveat: the streaming backend has its own known issue where gRPC connections accumulate on client agents during router rebalancing. Old subscriptions hold connections open until cache eviction. If you enable streaming to fix a goroutine leak, monitor consul.grpc.client.connections on client agents for this new failure mode.

Rate-limit or shed abusive clients

If a specific client or service is opening excessive blocking queries, rate-limit the blocking query loop on the client side. Use a token bucket with a small burst (for example, burst of 2) rather than a fixed sleep. A fixed sleep delays updates in the happy case and worsens busy-loop behavior during high churn.

If the abusive client cannot be identified or fixed quickly, consider temporarily blocking its connections at the network level while the root cause is addressed. This is disruptive to that client; use it as a stopgap, not a permanent fix.

Reduce watch scope for large services

For health queries on services with enough instances to exceed the watch limit, the watch falls back to root nodes and busy-loops. This produces server-side pressure similar to a goroutine leak (high CPU, elevated RPC rate) even though the mechanism is different. Options:

Split the large service into smaller logical groups that each stay under the watch limit
Use the streaming backend, which does not have the same per-instance channel limit
Watch the version you deploy: the limit is the internal watchLimit constant, not a supported user-facing setting

Increase FD limits as a stopgap

If the server is approaching FD exhaustion and the root cause cannot be fixed immediately, increasing the FD limit buys time. Check the systemd unit’s LimitNOFILE and the process-level ulimit -n. Consul documentation recommends a minimum of 65536 for servers. This does not fix the leak; it only delays the cliff.

Do not restart Consul servers as a primary fix. A restart clears goroutines and FDs temporarily but the accumulation resumes as soon as clients reconnect and re-establish their watches.

Prevention

Monitor goroutine count as a trend, not just a threshold. Alert on rate of change: goroutine count increasing by more than 1000 per minute sustained, or any monotonic increase over 24 hours without corresponding load increase. A threshold alert at 100K fires too late.

Track expected blocking query load. Know your normal: consul-template instances times templates per instance, plus application watchers, plus Connect xDS streams. If goroutine count diverges from expected blocking query load, investigate.

Verify consul-template and Consul versions during upgrades. Both have had multiple goroutine leak fixes across versions. Track versions in your fleet and prioritize upgrades when leak fixes land.

Watch for the watch limit fallback pattern. If your largest services approach the instance count where health queries exceed the watch limit (up to three watches per instance in the fallback path), plan for the streaming backend or service decomposition before the busy-loop starts.

Size FD limits with headroom. FD usage should stay below 60% of the limit in steady state. Each new consul-template instance, Connect proxy, or blocking query consumer adds to the baseline. Size the FD limit at roughly 3x current usage.

How Netdata helps

Per-second goroutine count metrics expose the leak trend early. A slow goroutine leak produces a visible upward slope that per-second resolution makes obvious within minutes rather than hours.
Correlate goroutine count with FD usage, HTTP API latency, and RPC request rate in a single view to distinguish a blocking query leak from other goroutine consumers (health check execution, gRPC streams, internal processing).
Anomaly detection on goroutine count flags deviations from baseline before a hard threshold fires, which matters for slow leaks that develop over days.
File descriptor utilization alerts (configurable at 70% and 90%) provide early warning before refused connections cascade.
HTTP API latency segmentation helps isolate blocking query pollution in aggregate latency metrics, so you do not chase a false latency problem when the real issue is goroutine accumulation.