$ guides / consul / consul-catalog-bloat ▌

Operations Guides

Consul catalog bloat: too many services and checks slowing everything down

DNS latency is creeping up. API calls to /v1/catalog/services take longer than they used to. Server restarts that used to take 20 seconds now take minutes. The Raft snapshot on disk keeps growing, and so does server RSS. No single failure point, no stack trace, no page. Just a slow, compounding drag on everything Consul does.

This is catalog bloat. Registered service instances and health checks grow the in-memory state store, the on-disk Raft snapshot, anti-entropy reconciliation work, and catalog-scan latency for every DNS and API query. The cost compounds week over week until a snapshot creation tips a leader election, a restart takes long enough to miss a deploy window, or DNS p99 crosses the threshold where applications time out on service discovery.

The hard part is not slowing the catalog. It is distinguishing legitimate fleet growth from leaked entries. Consul treats the agent’s local state as authoritative for agent-initiated registrations. If an agent registers a service and then dies permanently without a graceful leave, the service and its checks stay in the catalog until something explicitly removes them. Ephemeral environments (Kubernetes pods, autoscaling VMs, ECS tasks) are the classic source of slow, invisible accumulation.

Where the cost lands

Every catalog entry carries recurring cost across every server:

State store memory, proportional to services times instances times checks.
Snapshot bytes on disk, plus a memory spike during snapshot creation because the FSM serializes the full catalog state.
Anti-entropy work on every sync cycle: each agent reconciles its full local state against the server catalog.
Catalog-scan time on every DNS and API query that walks the catalog.
Raft log and FSM apply work whenever entries churn.

The relationship between catalog size and per-operation performance is roughly linear for individual reads and writes. Snapshot-related overhead is proportional to total state size. That asymmetry is what makes bloat dangerous: restart time, snapshot creation time, and the memory spike during snapshot creation all scale with the whole catalog, not the active working set. A cluster can handle its current query load fine and still fall over the next time a server restarts and has to rehydrate a 20 GB snapshot.

HashiCorp’s published guidance is roughly 5,000 client agents per datacenter for a basic server cluster, with 10,000 or more requiring careful tuning. Those are agent-count heuristics. The number that actually drives bloat is total service instances plus total health checks. A 2,000-agent cluster where each agent registers 15 services with 3 checks each is carrying 90,000 service instances and 270,000 checks. That is the sizing problem, not the agent count.

flowchart TD
  A[More service instances and checks registered] --> B[Larger in-memory state store on every server]
  A --> C[More anti-entropy work per sync cycle]
  B --> D[Larger Raft snapshots on disk]
  D --> E[Slower snapshot creation: memory spike and disk I/O]
  E --> F[Longer server restart and recovery]
  A --> G[Slower catalog scans for DNS and API]
  G --> H[Higher raft.commitTime and fsm.apply latency]
  E --> I[Risk of leader election during snapshot]

Common causes

Cause	What it looks like	First thing to check
Leaked ephemeral registrations	Instance count for a service grows monotonically; node list includes hosts that no longer exist; critical-check count drifts up	`GET /v1/catalog/service/<name>` and look for nodes absent from `consul members`
Missing `deregister_critical_service_after`	Checks sit critical indefinitely; services never disappear after their host dies	Inspect check definitions for the timeout field
Health check proliferation	Total check count is a large multiple of service count; each service carries script, HTTP, and TTL checks	Count checks per service and look for redundant checks
Legitimate fleet growth	Instance counts track real infrastructure inventory; deploy and autoscale events explain the curve	Compare instance count against infrastructure inventory
KV store used as a database	Snapshot grows but service counts are stable	Count keys and inspect value sizes
Sync-controller churn bugs	Registration rate is high and steady with no deploy; leader CPU elevated	Check the catalog sync controller version (consul-ecs, consul-k8s) for known re-registration bugs

Quick checks

All read-only.

# Total services registered
curl -s http://127.0.0.1:8500/v1/catalog/services | jq 'length'

# Instance count per service, sorted descending.
# Warning: issues one API call per service. Run once, off-peak.
curl -s http://127.0.0.1:8500/v1/catalog/services | jq -r 'keys[]' | \
  while read svc; do \
    n=$(curl -s "http://127.0.0.1:8500/v1/catalog/service/$svc" | jq 'length'); \
    echo "$n $svc"; \
  done | sort -rn | head -20

# Check counts by state
for state in passing warning critical; do
  count=$(curl -s "http://127.0.0.1:8500/v1/health/state/$state" | jq 'length')
  echo "$state: $count"
done

# Snapshot and raft directory size (path depends on data_dir)
du -sh /opt/consul/data/raft/
ls -lh /opt/consul/data/raft/snapshots/ 2>/dev/null

# Registration churn rate from telemetry
curl -s http://127.0.0.1:8500/v1/agent/metrics | \
  jq '.Counters[] | select(.Name | test("catalog.register|catalog.deregister")) | {Name, Count}'

# Leader-side write pipeline latency
curl -s http://127.0.0.1:8500/v1/agent/metrics | \
  jq '.Samples[] | select(.Name | test("raft.commitTime|raft.fsm.apply")) | {Name, Mean, Count}'

How to diagnose it

Measure the whole catalog, not just service count. Total services is a weak signal. Total instances and total checks are what drive cost. Run the per-service instance count above and record the top contributors.
Separate steady state from churn. Pull consul.catalog.register and consul.catalog.deregister counters and compute a rate over five minutes. A stable catalog with growth of a few instances per week is legitimate fleet growth. A high, steady registration rate with no corresponding deploy is churn: usually flapping checks, a buggy sync controller, or anti-entropy fighting itself.
Identify leaked entries. Cross-reference the node list in a service’s instances against consul members. Any instance whose node is not alive in gossip (or not present at all) is a leak. Leaked entries are the highest-value cleanup target because they cost full catalog overhead while serving no consumer.
Check for missing deregistration timeouts. Pull check definitions and look for deregister_critical_service_after. Services without it that belong to ephemeral workloads accumulate forever once their host dies.
Confirm bloat is the bottleneck, not disk or network. Correlate catalog size growth against consul.raft.commitTime, consul.raft.fsm.apply, DNS query latency, and snapshot size. If all of them trend up together over weeks while topology is stable, bloat is the common cause. If only commit time is up and DNS is flat, look at disk I/O first.
Project runway. Trend total instances and total checks weekly. If growth is super-linear, extrapolate when you will cross the operating size where snapshot creation or restart time becomes a reliability risk. Test with a synthetic catalog at that size before you hit it.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Total service instances	Primary driver of state store memory and snapshot size	Steady growth without corresponding infrastructure growth
Total health checks	Checks multiply the per-instance cost; often the hidden bloat source	Check-to-instance ratio climbing over time
`consul.catalog.register` / `deregister` rate	Distinguishes steady growth from pathological churn	High steady rate with no deploy or autoscale event
Raft snapshot size	Proportional to total catalog state; drives restart and creation cost	Monotonic growth over weeks
`consul.runtime.alloc_bytes`	Server heap tracks catalog size; snapshot creation spikes it	Growth correlated with instance count, not load
`consul.raft.commitTime`	Write pipeline latency; bloat increases apply cost per entry	Slow upward trend independent of disk I/O
`consul.raft.fsm.apply`	Per-entry apply latency; large catalogs make each apply more expensive	p99 climbing while write rate is stable
`consul.dns.domain_query` latency	User-visible cost of catalog scan	p99 creeping up over weeks
`consul.cache` hit ratio	Churn invalidates caches; low hit ratio amplifies catalog scan cost	Drop correlated with registration churn
`consul.runtime.num_goroutines`	Blocking-query and watch accumulation tracks catalog churn	Slow monotonic growth

Fixes

Clean up leaked ephemeral registrations

Highest-value, lowest-risk fix. Find services with instances on dead nodes and remove them.

# Deregister a specific service instance on a specific node.
# This is a Raft write. Batch cleanups to avoid pressuring the write pipeline.
curl -X PUT "http://127.0.0.1:8500/v1/catalog/deregister" -d '{
  "Node": "<dead-node-name>",
  "ServiceID": "<service-id>"
}'

To remove an entire dead node and all its registrations at once:

# Destructive: removes the node and everything registered to it from the catalog.
# Only use on nodes confirmed permanently gone.
consul force-leave -prune <dead-node-name>

Then prevent recurrence by setting deregister_critical_service_after on checks for any ephemeral workload. When a check stays critical longer than the timeout, Consul deregisters the service automatically. Pick a timeout longer than your worst-case transient failure to avoid mass deregistration during a brief dependency outage.

Reduce check cardinality

Every check is a catalog entry, an anti-entropy sync item, and a potential churn source. A service with a script check, an HTTP check, and a TTL check triples its per-instance cost. Audit for redundant checks and combine where the signal allows.

Enable the streaming backend for health queries

Consul 1.9+ can stream health query updates over gRPC instead of long-polling, sending diffs rather than full result sets. Enable use_streaming_backend = true on agents that consume /v1/health/service/:name heavily.

Streaming historically covered only the /v1/health/service/:name endpoint. Verify current scope against your Consul version before assuming it covers other blocking queries.

Upgrade past known blocking-query cliffs

Before Consul 1.4.4, each instance of a health watch added multiple internal watches against a hardcoded limit. Services above roughly 682 instances forced the server to fall back to coarse-grained watching, triggering full tree queries on every catalog change. This was fixed in 1.4.4 and the internal watch limit was raised in 1.7.0. If your catalog has large services and you are below those versions, the fix is an upgrade, not tuning.

Add capacity or split the cluster

When the catalog is legitimately large and growing, tuning runs out. Options, in increasing cost and disruption:

Size servers to the catalog. HashiCorp’s production server guidance calls for multi-core CPU, substantial RAM, and fast SSD. Bigger catalogs need the upper end of whatever range is current.
Add servers. More servers spread read and RPC load, though Raft write latency does not improve and the FSM is fully replicated on each server.
Split the datacenter. Partition services into separate Consul clusters when a single catalog crosses the comfortable operating range. This is a planned architecture change, not an incident response.

Stop using the KV store as a database

If the snapshot is growing but service and check counts are stable, inspect the KV store. Consul KV is for configuration and coordination, not high-throughput application state. Large or high-churn KV values inflate every snapshot and every Raft commit. Move that workload to a real data store.

Prevention

Track total service instances and total checks weekly as a trending signal, not a threshold alert. The goal is to see the curve, not to page on it.
Track which services contribute most to the catalog. A few services often account for most of the growth.
Require deregister_critical_service_after on any check attached to an ephemeral workload. Enforce it in your registration pipeline.
Monitor registration churn rate. A high steady rate with no deploy is the earliest sign of a sync bug or flapping checks.
Watch snapshot size and server RSS alongside instance counts. When they diverge from the instance curve, investigate (usually KV bloat or a memory leak).
Load-test at projected catalog sizes before you reach them. Restart time and snapshot creation time are failure modes you cannot afford to discover in production.

How Netdata helps

Per-second consul.catalog.register and consul.catalog.deregister collection makes the steady-growth-vs-churn distinction a visual exercise. Chart registration rate alongside deploy or autoscaling markers.
In a single dashboard, chart consul.raft.commitTime and consul.raft.fsm.apply against total service instances. When all three trend up together while topology is stable, bloat is confirmed as the root cause rather than disk I/O or network saturation.
Tracking Raft snapshot size and server RSS week over week alongside instance counts gives runway projection before a snapshot-induced leader election.
DNS query latency percentiles surface the user-visible cost of catalog scan growth, the symptom that usually triggers investigation.
Anomaly detection on goroutine count and cache hit ratio catches the secondary effects of churn (cache invalidation storms, watch accumulation) before they become resource exhaustion.