Consul registration storm: catalog churn overwhelming Raft

The complaint comes in as “Consul is slow.” DNS lookups lag, service discovery returns stale results, HTTP API writes take longer than usual. The cluster has a leader, gossip is healthy, no node is down. The Raft commit time metric is climbing. The root cause is often hiding in the catalog registration rate.

Every service registration and deregistration is a Raft write. The leader appends the entry to its log, persists it (fsync), replicates to followers (who also persist), waits for quorum, and applies the entry to the in-memory state store. This is the same pipeline that handles KV writes, health-check state updates, session creation, and ACL operations. When the registration rate is high enough, it saturates the pipeline and every consumer of the consensus layer pays the cost.

The downstream effects compound. Each catalog update invalidates caches on servers and clients. Consumers that watch the catalog (consul-template, Envoy xDS streams, blocking queries) must reprocess the update. Anti-entropy generates additional registrations as it reconciles agent state with the catalog.

The operation timers to watch are consul.catalog.register and consul.catalog.deregister. These are leader-only metrics, so your monitoring must track the leader. In a healthy cluster, spikes correlate with deployments and other intentional events. The problem is sustained high churn with no corresponding deployment, or churn that tracks with a pathological source: flapping health checks, crash-restart loops in sidecars, or anti-entropy fighting a drift condition that never resolves.

flowchart TD
  SRC2[Flapping health checks] --> SRC[Churn source]
  SRC3[Deploy or rollback loop] --> SRC
  SRC4[Sidecar crash-restart] --> SRC
  SRC5[Anti-entropy drift] --> SRC
  SRC -->|catalog.register / catalog.deregister| RAFT[Raft write pipeline]
  RAFT -->|fsync + apply| COMMIT[consul.raft.commitTime rising]
  COMMIT --> LAST[lastContact rising on followers]
  LAST --> ELEC[Election timeout approaching]
  ELEC --> OUTAGE[Write outages during elections]
  RAFT -->|cache invalidation| CONSUMERS[consul-template, Envoy xDS, LB integrations reprocess]

Common causes

Quick checks

These are read-only and safe to run on any server, preferably the leader.

# Identify the current leader
curl -s http://127.0.0.1:8500/v1/status/leader

# Pull catalog registration and deregistration timer samples
curl -s http://127.0.0.1:8500/v1/agent/metrics | python3 -c "
import sys,json
d=json.load(sys.stdin)
for s in d.get('Samples',[]):
    if 'catalog.register' in s['Name'] or 'catalog.deregister' in s['Name']:
        print(s['Name'], 'Count:', s.get('Count'), 'Mean:', s.get('Mean'))
"

# Check Raft commit time (leader only)
curl -s http://127.0.0.1:8500/v1/agent/metrics | python3 -c "
import sys,json
d=json.load(sys.stdin)
for s in d.get('Samples',[]):
    if 'raft.commitTime' in s['Name']:
        print(s['Name'], 'Mean:', s.get('Mean'), 'Count:', s.get('Count'))
"

# Check Raft apply rate (leader only)
curl -s http://127.0.0.1:8500/v1/agent/metrics | python3 -c "
import sys,json
d=json.load(sys.stdin)
for s in d.get('Counters',[]):
    if 'raft.apply' in s['Name']:
        print(s['Name'], 'Count:', s.get('Count'))
"

# Check last contact time on followers
curl -s http://127.0.0.1:8500/v1/agent/metrics | python3 -c "
import sys,json
d=json.load(sys.stdin)
for s in d.get('Samples',[]):
    if 'raft.leader.lastContact' in s['Name']:
        print(s['Name'], 'Mean:', s.get('Mean'))
"

# Count critical health checks
curl -s http://127.0.0.1:8500/v1/health/state/critical | python3 -c "import sys,json; print('Critical checks:', len(json.load(sys.stdin)))"

# Check disk I/O on the Raft data volume
iostat -x 1 3

How to diagnose it

1. Confirm churn is the problem. Compare consul.catalog.register and consul.catalog.deregister rates against consul.raft.commitTime. If all three rise together, churn is driving Raft load. If commit time is elevated but registration rates are normal, the bottleneck is elsewhere: disk I/O, large KV writes, or FSM apply latency.

2. Identify the churn source. The registration rate does not tell you who is registering. Cross-reference with deployment timestamps, health-check transition logs, and sidecar process logs. In Kubernetes, check pod scheduling events against the rate. A mass recovery (rolling restart, AZ recovery) produces a sharp spike that usually resolves within minutes. Sustained churn without a known event is the signal to chase.

3. Check for health-check flapping. Query /v1/health/state/critical for checks oscillating between passing and critical. Each transition triggers catalog updates. Look for checks with short intervals bouncing because the upstream dependency is genuinely unhealthy or the check timeout is too tight.

4. Check for anti-entropy drift. Anti-entropy reconciles local agent state with the server catalog periodically. If an agent’s state perpetually disagrees with the catalog (misconfigured check, registration that keeps being rejected, client reconnecting with different metadata), anti-entropy generates registrations on every sync cycle. Look for repeated sync events for the same node or service in agent logs.

5. Assess Raft headroom. The critical relationship is between consul.raft.commitTime and the election timeout. If commit time approaches the election timeout, the cluster is at risk of leader elections. A conservative target: commit time below one-tenth of the election timeout. Above 50 percent of the election timeout, elections are imminent.

6. Check downstream consumer load. High catalog churn forces consumers to process constant updates. Check goroutine counts and connection counts on servers. Consul-template instances, Envoy xDS streams, and blocking queries all hold server-side resources that scale with the update rate. Cache hit ratios collapse when invalidation outpaces warming.

Metrics and signals to monitor

Fixes

Stop the source of churn

The most effective fix is to stop whatever is generating the registrations. Everything else is a workaround.

Health-check flapping. If a check oscillates between passing and critical, widen the check interval, increase the timeout, or fix the underlying dependency. Consul is correctly reporting what it sees. Do not disable the check unless you understand the routing consequences.

Deploy or rollback loops. A deployment system that repeatedly deploys and rolls back generates a registration burst on each cycle. Identify and stop the loop in the pipeline. The rate drops once the loop is broken.

Sidecar crash-restart loops. If Connect sidecars crash and restart, each restart re-registers the service. Check Envoy sidecar logs for the crash cause: resource exhaustion, certificate problems, or configuration errors.

Anti-entropy drift. If anti-entropy fights a perpetual disagreement between agent and catalog state, find the source. Common causes: invalid check configurations the server rejects, or clients reconnecting with different service metadata after a network blip. Agent logs show sync rejection messages or repeated reconciliation for the same service.

Reduce Raft pressure temporarily

If the churn source cannot be stopped immediately, reduce leader load to keep commit time below the election timeout.

Switch reads to stale mode. Consistent reads go through the leader and add to its load. Stale reads serve from any server. Clients add ?stale to their queries. Tradeoff: stale reads may return slightly outdated data. In a crisis, this is usually acceptable.

Disable non-critical health checks. Deregister individual checks with PUT /v1/agent/check/deregister/:checkId. This is a targeted, reversible action, but services with disabled checks will not be removed from the catalog even if they fail, so traffic may route to dead instances. Use only to prevent a cluster-wide outage.

Reduce periodic check-state writes. Consul sizes periodic anti-entropy runs by cluster size and does not expose a sync-interval key in current releases. For volatile steady-state output, use check_update_interval (default 5m) to defer output synchronization and consider discard_check_output to prevent noisy output from becoming Raft writes; state changes still synchronize immediately.

Address disk I/O if it is the amplifier

Slow disk does not cause catalog churn, but it reduces the pipeline’s ability to absorb it. If consul.raft.commitTime is high and disk write latency (await) is above 10 milliseconds, the disk is compounding the problem. Check for EBS burst credit exhaustion, noisy neighbors, or rotational media on the Raft data volume. Dedicated SSDs with adequate IOPS are the baseline for Consul servers. Fsync latency on Raft log writes is the dominant component of commit time; when commit time approaches the election timeout, the leader cannot maintain its lease.

Prevention

• Baseline the registration rate. Track consul.catalog.register and consul.catalog.deregister during normal operations. Alert on sustained rates more than 5x baseline that do not correlate with deployments.
• Set deregister_critical_service_after on services. Prevents dead services from accumulating in the catalog, which reduces catalog size and snapshot cost over time.
• Monitor anti-entropy sync success. Failing syncs indicate a pipeline problem that will manifest as catalog staleness or churn.
• Track commit time trend. A gradual upward trend in consul.raft.commitTime over days or weeks means the cluster is approaching write capacity. Investigate before it becomes an election problem.
• Watch catalog size growth. Total service instances and health checks drive snapshot size, memory, and apply cost. Track weekly.
• In Kubernetes, correlate pod churn with registration rate. If pod churn saturates Raft, the fix may be at the scheduling layer (deployment strategy, pod disruption budgets, graceful shutdown) rather than in Consul.

How Netdata helps

• The catalog registration rate (consul.catalog.register, consul.catalog.deregister) and Raft commit time (consul.raft.commitTime) are collected per second. When a registration storm begins, the correlation between rising churn and rising commit time is visible within seconds.
• ML anomaly detection flags unusual registration rate patterns even when no explicit threshold is set, which is useful where the baseline is dynamic (Kubernetes pod churn, autoscaling).
• Raft apply rate, last contact time, and FSM apply latency are collected alongside registration metrics, making it possible to confirm in a single view that churn is the Raft bottleneck rather than disk I/O or KV writes.
• Disk I/O metrics on the Raft data volume are collected at the system level, so you can immediately distinguish “churn overwhelming a healthy pipeline” from “churn plus slow disk amplifying the problem.”

Related guides

• Consul on EBS: burst-credit exhaustion and the sudden latency cliff
• Consul gossip encryption key mismatch: a botched keyring rotation splits the pool
• Consul gossip flapping: nodes oscillating between alive, suspect, and failed
• Consul serf queue backlog: an agent falling behind on gossip
• Consul gossip storm after mass recovery: rejoin floods and anti-entropy spikes
• How Consul actually works in production: a mental model for operators
• Consul leader election storm: repeated elections and rolling write outages
• Consul monitoring checklist: the signals every production cluster needs
• Consul monitoring maturity model: from survival to expert
• Consul “No cluster leader”: every write is failing
• Consul raft commitTime high: the write pipeline is slowing down
• Consul Raft data directory full: the server that can no longer write