Consul monitoring maturity model: from survival to expert

Consul runs three independent subsystems that must all be healthy for the cluster to function: Raft consensus, Serf gossip, and the catalog state machine with its anti-entropy sync pipeline. Most teams monitor one or two well and discover the third during an incident.

The model is cumulative. Each level adds signals to the previous one; you cannot skip to Mature without the Operational baseline in place. A team with composite-pattern detection at Expert but no client-agent RPC monitoring at Operational is blind to the most common silent failure mode: the catalog drifting from reality while every server-side metric looks healthy.

Use this as both an audit of current coverage and a prioritized roadmap. Signals within each level are ordered roughly by incident frequency. Treat the boundaries as maturity gates: a team can be Mature on servers and Survival on clients, and the weakest tier wins.

flowchart TD
  L1["Survival
leader, gossip, process, disk"]
  L2["Operational
commitTime, lastContact, RPC failures, peer count"]
  L3["Mature
goroutines, gossip queues, catalog churn, cert expiry, GC pauses"]
  L4["Expert
log divergence, anti-entropy timing, blocking queries, cross-DC RPC"]
  L1 --> L2 --> L3 --> L4

Level 1: Survival

If any of these fail, the cluster is either down or about to be. A team with only these signals will detect total outages but miss every silent degradation and most partial failures.

Signals to cover:

Leader exists. Query /v1/status/leader and assert a non-empty response. Without a leader, all writes fail: service registrations, KV writes, session creation, and health check state updates.
All server nodes alive in gossip. Query /v1/agent/members, filter for Tags.role == consul, and assert Status == 1 for each. A failed server in a 3-server cluster is one failure from quorum loss.
Agent process up on every node. A dead agent means no health checks execute locally, and the catalog shows last-known state until gossip marks the node failed.
HTTP API responsive. A 200 from /v1/status/leader within a tight timeout distinguishes “Consul is slow” from “Consul is gone.”
Server data directory not full. Raft cannot persist log entries when the volume fills; the server will crash or fall behind and lose quorum.

Gotchas:

/v1/status/leader can return a stale result from a server that has not yet processed the leader change. Cross-check with multiple servers during incidents.
A server can be alive in gossip while being non-functional for Raft. Gossip and Raft are independent protocols; one being healthy does not imply the other.
Brief leaderless periods of 1 to 5 seconds during rolling restarts are normal. Use a sustained-absence window of 15 to 30 seconds for paging thresholds, or you will alert on every upgrade.

Level 2: Operational

This is where a competent team running Consul in production should live. The signals here catch performance degradation, partial failures, and the conditions that lead to leader elections.

Signals to add:

Raft commit time (consul.raft.commitTime). End-to-end write latency and the single best indicator of Raft health. Captures disk write latency, network replication latency, and FSM apply time in one number. Leader-only.
Raft last contact (consul.raft.leader.lastContact). Time since the leader last heard from each follower. Growing values indicate replication lag and approaching election timeouts. Leader-only.
Leader election count (consul.raft.state.leader). This is a counter incremented each time this server becomes leader; alert on increments outside maintenance. More than 2 transitions per 10 minutes outside a maintenance window indicates a systemic problem.
DNS query latency. User-facing latency of service discovery for any consumer using Consul DNS on port 8600.
Health check distribution. Passing, warning, and critical counts across the catalog. A sudden spike in critical checks usually indicates a shared downstream dependency failure.
Server memory RSS. Tracks resource consumption and growth trends. Correlate with catalog size and goroutine count.
Client agent RPC failure rate (consul.client.rpc.failed). Elevated rates mean agents cannot push state to the catalog. This is the single most under-monitored signal in Consul deployments.
File descriptor usage. Approaching the limit causes connection failures, gossip instability, and Raft timeouts. Default OS ulimit of 1024 is too low; servers should run with at least 65536.
Raft peer count (consul.raft.peers or /v1/operator/raft/configuration). Must match expected cluster size. A drop below quorum is a page.

Gotchas:

Many Raft metrics are leader-only: commitTime, kvs.apply, catalog.register. When leadership changes, your monitoring must follow the leader or you will have gaps in the time series.
Blocking queries inflate HTTP latency metrics. Filter requests with ?wait= or ?index= parameters when analyzing API performance, or your p99 numbers will be meaningless.
consul.client.rpc.failed is a counter. Alert on the rate of change, not the absolute value.
Autopilot’s dead server cleanup can remove a failed server automatically, causing the peer count to drop even when a replacement is being provisioned. This is by design but confusing if unexpected.

Level 3: Mature

Full coverage for a professional SRE team. These signals catch slow leaks, capacity trends, and the secondary effects that compound primary failures into outages.

Signals to add:

Goroutine count (consul.runtime.num_goroutines). Monotonic growth indicates a leak. Each blocking query, each gRPC stream, and each watch holds a goroutine for its lifetime.
Gossip queue depth (consul.serf.queue_Event, consul.serf.queue_Intent, consul.serf.queue_Query). Sustained non-zero values mean the agent is receiving gossip faster than it can process it, causing delayed failure detection and false positives.
Catalog registration and deregistration rate (consul.catalog.register, consul.catalog.deregister). High churn drives Raft load, anti-entropy overhead, and downstream consumer updates.
KV operation latency (consul.kvs.apply). Write path for KV specifically. If KV latency is high but general Raft metrics are normal, the issue is KV-specific: large values, deep key trees, or transaction contention.
HTTP API latency per endpoint. Catches single-subsystem bottlenecks masked by aggregate averages.
xDS stream count (consul.xds.server.streams). Connect control-plane health; should match your Envoy proxy count. Each stream holds a goroutine and a file descriptor on the server.
Certificate expiration time for both leaf and CA root. Cliff-edge failure: everything works until the exact moment it does not.
ACL resolution latency (consul.acl.ResolveToken). Every authenticated API request pays this cost. High latency with a high cache-miss ratio means the token cache is undersized.
consul members -wan membership state per DC. A remote DC disappearing from the WAN pool breaks prepared-query failover and cross-DC service lookups.
Raft snapshot size. Tracks state growth. Large snapshots slow recovery, compete for disk I/O during creation, and can temporarily double memory usage.
Disk I/O latency on server volumes. Track await and %util, not just throughput. This is the leading indicator for Raft issues and the most common source of leader-instability incidents.
Go GC pause duration (consul.runtime.gc_pause_ns, a nanosecond summary). Stop-the-world pauses affect Raft timing. Pauses approaching the election timeout will trigger elections.
Session operation timing and invalidation (consul.session.apply and consul.session_ttl.invalidate are timers; track application-side lock loss for rate). Spikes mean distributed locks releasing across the cluster, which can trigger application-level leader elections and cache flushes.

Gotchas:

Disk I/O latency is the most common root cause of leader instability. Teams provision acceptable CPU and memory, then put the Raft data directory on EBS gp2 with exhaustible burst credits, shared NFS, or spinning disks. The first visible symptom is leader elections, by which point the cluster is already degraded. Page on disk write latency, not on downstream Raft symptoms.
Go’s garbage collector does not immediately return memory to the OS. runtime.alloc_bytes (live heap) and VmRSS (OS-level resident set) can diverge significantly. Investigate monotonic RSS growth, not absolute ratios.
Certificate expiration monitoring without renewal success tracking is a countdown clock with no early warning. Track both. A Vault CA backend outage will not surface as expiry-time degradation until hours later, when certificates start failing to rotate.
Gossip and Raft are independent. A server can be alive in gossip while having corrupt Raft state, applying entries incorrectly and diverging from the leader. This only manifests when that server wins an election.
Composite patterns matter more than individual thresholds at this level. Leader churn, health-check thundering herds, blocking-query amplification, and gossip/Raft divergence each have distinctive multi-signal signatures. Build alerts that fire on the combination, not on any single metric.

Level 4: Expert

Signals that experienced operators add after their third or fourth major incident. They catch subtle internal state and provide the 30-minute warning that would have averted the last outage.

Signals to add:

Raft log index divergence across servers (consul.raft.last_index and consul.raft.applied_index gauges on all servers). Indices should be nearly identical across the peer set. Large divergence means a follower is falling behind or has inconsistent state, and is a candidate for serving corrupt data if it wins an election.
Anti-entropy sync timing per agent. Detects agents that cannot sync even though they are alive in gossip. The catalog drifts silently while every server metric looks healthy.
Blocking-query count and distribution. Identifies watch accumulation before it becomes a goroutine leak. Each consul-template instance and each watch holds a goroutine and a connection; leaks accumulate for weeks before cascading.
Prepared-query failover event rate. Detects when consumers are silently being served from remote DCs. Receiving results does not mean local health; it may mean prepared-query failover is masking a complete DC service failure.
Cross-DC RPC latency. WAN gossip being healthy does not guarantee that cross-DC RPC is fast enough for application timeouts.
Certificate renewal success rate. Not just expiry time. Track CSR acceptance, signing latency, and Envoy acknowledgment separately.
Catalog size by service. Which services contribute most to catalog bloat, driving snapshot size, memory, and query latency.
Raft compaction timing and success. If snapshot creation fails, the log grows indefinitely.
Agent event handler execution time. Custom handlers blocking the agent main loop.
Network partition detection via cross-referenced member lists. Query consul members from multiple servers and diff. Asymmetric partitions, where server A sees B but B does not see A, are common and invisible to leader-centric metrics.

Gotchas:

Pairwise Raft network latency matters, not just leader-to-follower. Measure between all server pairs. Asymmetric partitions are the failure mode most likely to cause confusing split-brain symptoms in a 5-server cluster.
These signals rarely fire in steady state. Their value is during incidents and capacity planning. Do not expect them to populate dashboards with activity; expect them to provide critical context when everything else is on fire.
Consul’s telemetry uses time-window aggregation, and rapid transients under a second may not appear in metrics sampled per 10 seconds. Per-second collection matters here more than at any other level.

Where most teams get stuck

Disk I/O ignored until it causes elections. The leading indicator is write latency (await), not throughput or utilization. This should be page-level on server volumes. If you only learn about disk pressure from consul.raft.commitTime spikes, you are reacting too late.
Client-to-server RPC health unmonitored. Server-side monitoring misses the pipeline from client agents to the catalog. The catalog silently goes stale. Monitor consul.client.rpc.failed on every client agent and alert on sustained non-zero rates.
Blocking query accumulation treated as normal load. Each watch and each consul-template instance holds a goroutine and a connection. Leaks accumulate for weeks before cascading into FD exhaustion or goroutine saturation. Track goroutine count trends and correlate with expected blocking-query load.
Composite pattern detection absent. Individual thresholds miss multi-signal failure modes. Teams with 100 metrics but no composite alerts miss these until they escalate.
Version-specific behavior untracked. Consul 1.19 removed state-store metrics with the doubled consul.consul prefix; use the single-prefix names such as consul.state.config_entries. The telemetry.disable_compat_1.9 option was removed in Consul 1.13 after the 1.9-style consul.http metrics had been deprecated. Adjust monitoring assumptions on every upgrade, and treat the version-specific upgrade notes as required reading for whoever owns the dashboards.

How Netdata helps

Netdata’s Consul collector scrapes the agent telemetry endpoint at per-second resolution, which matters because the most expensive Consul failures (Raft timing violations, brief gossip partitions) produce transients that 10- or 15-second pollers miss.

Per-second Raft commit time correlated with disk I/O latency shows the causal chain from disk await degradation to commit-time spikes to leader elections in a single view, instead of triangulating across three dashboards after the fact.
Goroutine count and file descriptor usage trended together surfaces slow blocking-query leaks before they become outages. Anomaly detection flags monotonic growth that static thresholds miss.
Client-agent RPC failure rate collected on every node, not just servers closes the most common blind spot. Each client agent reports its own consul.client.rpc.failed without depending on a central scraper that only talks to servers.
Leader-only metrics follow the leader automatically. Each node reports its own telemetry, so consul.raft.commitTime and consul.kvs.apply appear on whichever server is currently leader, with no gaps during transitions.
ML anomaly scoring across Raft, gossip, and RPC signals catches the multi-signal signatures of leader churn, thundering herds, and gossip/Raft divergence that single-metric thresholds cannot.

Consul monitoring maturity model: from survival to expert

Level 1: Survival

Level 2: Operational

Level 3: Mature

Level 4: Expert

Where most teams get stuck

How Netdata helps

Related guides