Consul monitoring checklist: the signals every production cluster needs

Consul runs three subsystems concurrently: Raft consensus, Serf gossip, and a catalog state machine backed by BoltDB. Each fails independently. A monitoring setup that checks only “is there a leader” and “are all members alive” will miss the most common production incidents: slow disk causing leader churn, client agents losing RPC connectivity while gossip stays green, blocking query leaks, and Connect certificate rotation failures.

This checklist organizes signals into four maturity levels. Each builds on the previous one and maps to specific failure patterns: leader thrashing, split brain after partition, gossip storms during mass recovery, silent catalog staleness, and KV store saturation. Use it as an audit tool. Identify which level your current monitoring reaches, then close gaps at that level before chasing the next tier.

Do not skip levels. Expert signals like Raft log index comparison and certificate renewal success rate add no value if your survival checks do not page on a missing leader or a failed server node.

flowchart TD
    L4["Level 4 - Expert
Asymmetric partition detection, GC pause tails,
certificate renewal success, blocking query leaks"]
    L3["Level 3 - Mature
GC pauses, gossip queues, catalog churn,
certificate expiry, xDS stream health"]
    L2["Level 2 - Operational
Raft commit time, last contact, elections,
DNS latency, RPC failures, FD usage"]
    L1["Level 1 - Survival
Leader exists, servers alive, process up,
API responsive, disk space"]
    L1 --> L2 --> L3 --> L4

Level 1: survival

If any of these fail, the cluster is down or about to be. Page on all of them.

Alerting guidance. Page on no leader for more than 15 seconds outside maintenance windows. Page on any server in failed state. Page on voter count below quorum (fewer than 2 for a 3-server cluster, fewer than 3 for a 5-server cluster). Brief leaderless periods during rolling restarts are expected.

Level 2: operational

These signals tell you whether Consul is performing well, not just whether it is alive. Most teams stop here, which is adequate for small clusters but leaves blind spots as scale grows.

Alerting guidance. Page on commit time sustained above 500ms. Page on RPC failure rate above 1% on any client agent. Page on disk write latency sustained above 10ms on the Raft volume. Ticket on election count above 1 per hour outside maintenance.

Why disk latency is critical. The Raft log is persisted to disk with fsync on every write. When disk write latency approaches the heartbeat timeout, the leader cannot send heartbeats fast enough, followers start elections, and the cluster enters a thrashing cycle. This is the most common Consul incident pattern. Rotational disks, NFS, and EBS gp2 with exhausted burst credits are the usual culprits. SSD is not optional for server data directories.

Level 3: mature

Full coverage for a professional SRE team. These signals tell you what is trending toward a problem and where the bottlenecks are.

Alerting guidance. Ticket on goroutine count above 2x baseline with monotonic growth. Ticket on gossip queue depth sustained above 0 for more than 1 minute. Ticket on certificate expiry within 30 days for root CA, 7 days for intermediate. Page on xDS stream drain rate spiking, since that means Envoy proxies are losing configuration updates.

Level 4: expert

Signals that experienced operators add after their third or fourth major incident. These give you 30 minutes of warning before the next one.

Alerting guidance. Most Level 4 signals are investigative rather than threshold-based. Use them during incident triage and for weekly trend review. The exception is certificate renewal success rate: page on any sustained renewal failure, because the countdown to expiry has started.

Common blind spots

These are gaps most teams discover during incidents, not during planning.

Disk I/O latency. The most common Consul incident pattern. Teams provision servers with adequate CPU and memory, then put the Raft data directory on general-purpose storage. Everything works until write volume increases or burst credits deplete. The first visible symptom is leader elections, by which point the cluster is already degraded. Monitor disk write latency (w_await from iostat), not just throughput or utilization. This should be a page-level alert.

Client-to-server RPC health. Server-side monitoring gets all the attention. But client agents can be alive, running health checks, and participating in gossip while being unable to push state to the server catalog. The catalog silently goes stale. Monitor consul.client.rpc.failed on every client agent, not just servers.

Gossip and Raft divergence. Gossip is the most visible protocol because it powers consul members. Teams assume “all members alive” means the cluster is healthy. Gossip and Raft are independent subsystems. A server can be alive in gossip while having corrupt Raft state. Cross-reference consul operator raft list-peers against consul members regularly.

Blocking query leaks. Every consul-template instance, every watch, and every application using blocking queries holds a goroutine and a connection on the server. When clients crash without cleanly ending queries, these accumulate. Goroutine count creeps up week over week. It becomes a problem only when servers exhaust goroutines or file descriptors. Track goroutine count trends and correlate with expected blocking query load.

Connect certificate lifecycle. Teams enable Connect, verify it works during setup, and never monitor the certificate pipeline again. When the Vault backend has a brief outage, nobody notices until certificates start expiring. Monitor both certificate expiration time and renewal success rate as separate signals.

Catalog size growth. The catalog grows slowly: a few new services per week, a few more checks per deployment. Over months, the catalog can grow 10x. Snapshot sizes grow proportionally. Server memory grows. Raft commit times creep up. Track total service instances, total checks, and snapshot size as trend signals with weekly review.

Cross-DC federation. Multi-datacenter Consul is configured once, verified once, and then assumed to work. WAN gossip encryption keys are rotated independently. Network changes between DCs break WAN gossip silently. Prepared query failover stops working without anyone noticing because the primary DC is still serving traffic.

How Netdata helps

Netdata collects Consul signals at per-second resolution, which catches transients invisible to 10-second or 1-minute aggregation windows.

• Raft pipeline correlation. Per-second consul.raft.commitTime, consul.raft.leader.lastContact, and consul.raft.state.leader on the same timeline let you see disk latency translating into commit time spikes translating into elections, before the cluster thrashes.
• OS-level signals alongside Consul metrics. Disk write latency on the Raft volume, file descriptor usage, memory RSS, and goroutine count are collected at the host level and displayed next to Consul telemetry. The connection between slow disk and leader churn becomes visible without switching tools.
• Client agent coverage. consul.client.rpc.failed is collected from every agent, not just servers. Silent catalog staleness from client-to-server RPC failures shows up before consumers notice stale DNS responses.
• ML-based anomaly detection. Goroutine count, gossip queue depth, and catalog registration rate benefit from trend-based anomaly detection. Gradual leaks that stay within static thresholds but deviate from the learned baseline are flagged without manual threshold tuning.
• Health check distribution. Passing, warning, and critical check counts are tracked over time, making mass-failure events and recovery visible as a coordinated shift rather than individual check alerts.
• DNS latency. Per-second DNS query latency distribution separates agent-side issues from server-side issues, helping you determine whether the bottleneck is catalog scans, RPC backing up, or agent-level parsing.

Related guides

• How Consul actually works in production: a mental model for operators
• Consul monitoring maturity model: from survival to expert
• Consul “No cluster leader”: every write is failing
• Consul leader election storm: repeated elections and rolling write outages
• Consul lost quorum: Raft peers below the majority needed to elect a leader
• Consul raft commitTime high: the write pipeline is slowing down
• Consul raft lastContact rising: followers drifting toward an election
• Consul leader stable but commits stalled: writes silently failing
• Consul stale Raft peer: removing a failed server from the configuration
• Consul Raft log divergence: catching a corrupt follower before it wins an election
• Consul server in failed state: reading consul members during an incident
• Consul gossip flapping: nodes oscillating between alive, suspect, and failed