Consul critical health checks spiking: real outage or broken checks?

The alert fires when /v1/health/state/critical jumps from single digits to dozens or hundreds of checks in minutes. The page says “services are unhealthy,” but that is ambiguous: either real services are down, or the checks themselves are broken. Pick the wrong fork and you burn an hour on the wrong dependency.

The first decision is not “what failed.” It is “is this a service-level event, a node-level event, or a monitoring-level event?” The cheapest signal is the check Output text field, which carries the raw error from check execution. It holds strings like connection refused, i/o timeout, certificate has expired. That field usually resolves the diagnosis in seconds, and most teams ignore it until their second or third incident.

The second fork is gossip. A spike that correlates with gossip member failures is an infrastructure event. A spike against healthy gossip is service-level. Mixing the two produces the wrong runbook.

What this means

/v1/health/state/critical returns every check in critical status. The response includes CheckID, Name, Status, Output, ServiceID, ServiceName, Node, and Namespace. The CheckID prefix tells you which subset is spiking.

Three classes of spike:

• serfHealth checks. The node-level check maintained by the gossip protocol. A spike here is a node or gossip event, not a service-level event. When serfHealth goes critical, Consul excludes all services on that node from DNS and service discovery. The Output is Agent not live or unreachable.
• service: prefixed checks. Service-level. The service itself is failing its configured check.
• Maintenance checks (_service_maintenance:, _node_maintenance). Intentional. These should correlate with change tickets.

The next fork is whether the spike is real (services actually down) or whether the checks themselves are broken (misconfigured, timing out against a shared dependency, ACL-blocked, anti-entropy stalled, or racing a TTL). The Output field is the fastest path to that answer.

flowchart TD
    A["critical count spike"] --> B{"Dominant CheckID?"}
    B -->|"serfHealth"| C["node/gossip event"]
    B -->|"service: prefix"| D["service-level event"]
    B -->|"maintenance"| E["intentional - check change ticket"]
    C --> F{"gossip members failed?"}
    F -->|"yes"| G["infra or AZ event"]
    F -->|"no, Raft peers missing"| H["partition or quorum loss"]
    D --> I{"Output populated?"}
    I -->|"yes, shared string"| J["downstream dependency"]
    I -->|"empty"| K["query /v1/agent/checks locally"]
    D --> L{"local agent checks match catalog?"}
    L -->|"no, local passing"| M["anti-entropy stall - check RPC"]

Common causes

Quick checks

# Total critical checks right now
curl -s http://127.0.0.1:8500/v1/health/state/critical | jq length

# Bucket by CheckID prefix (serfHealth vs service-level vs maintenance)
curl -s http://127.0.0.1:8500/v1/health/state/critical | \
  jq -r '.[].CheckID | split(":")[0]' | sort | uniq -c | sort -rn

# Sample the Output field - the cheapest root-cause signal
curl -s http://127.0.0.1:8500/v1/health/state/critical | \
  jq -r '.[] | "\(.Node) \(.CheckID) => \(.Output[:120])"' | head -30

# Cluster common Output strings to find the shared dependency
curl -s http://127.0.0.1:8500/v1/health/state/critical | \
  jq -r '.[].Output' | sort | uniq -c | sort -rn | head -10

# Gossip view of node state
curl -s http://127.0.0.1:8500/v1/agent/members | \
  jq -r '.[].Status' | sort | uniq -c | sort -rn

# Check state from the local agent (real-time, bypasses anti-entropy delay)
curl -s http://127.0.0.1:8500/v1/agent/checks | jq -r '.[] | "\(.CheckID) [\(.Status)] \(.Output[:100])"'

# Client agent RPC failures (catalog staleness indicator)
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -E 'client.rpc|consul.client.rpc'

# Raft pipeline health if the spike is large
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -E 'raft.commitTime|raft.apply|raft.state.leader'

All read-only. None mutate catalog state.

How to diagnose it

1. Bucket the spike by CheckID prefix. If the bulk of criticals are serfHealth, you have a node or gossip event. Go to step 2a. If they are service: checks, go to step 2b.

2a. Node-level event. Correlate with consul members and the Serf member status gauge. If failed members cluster in one AZ or segment, suspect a network event. Cross-reference with the Raft peer list at /v1/operator/raft/configuration to see whether quorum is intact.

2b. Service-level event. Go to the Output field. Group criticals by Output text. A single repeated string (dial tcp ...: connect: connection refused, i/o timeout, certificate has expired) usually points at a shared dependency.

3. Check for anti-entropy stall. If /v1/agent/checks on the local agent shows the check passing but /v1/health/state/critical on the server shows it critical, the agent-to-server RPC pipeline is broken. Confirm with consul.client.rpc.failed rate and verify port 8300 reachability.

4. Rule out the known Output delay. If Output is empty in the catalog but the checks are recent, steady-state check output can wait up to check_update_interval before synchronizing (default 5m; a state change syncs immediately). Query /v1/agent/checks on the local agent for the real-time value. The local agent Output text is the workaround for this lag.

5. Check the Raft pipeline if the scope is wide. A massive spike in health updates generates a Raft apply storm. Watch consul.raft.commitTime and the consul.raft.apply rate. If commit time approaches the election timeout, the cluster can degrade into leader churn, which itself produces spurious criticals with empty Output during election windows.

6. Verify ACLs and check semantics if you recently upgraded. Consul 1.16.0 changed /v1/health/connect/ and /v1/health/ingress/ to return 403 “Permission Denied” for tokens lacking service:read; earlier behavior was an empty 200. Script checks became opt-in with enable_script_checks in Consul 0.9.0 (default false). Since Consul 1.0.0, invalid check definitions fail agent startup rather than being silently skipped.

Metrics and signals to monitor

Fixes

Real downstream dependency outage

The catalog is reporting truth. Focus on the dependency, not Consul. Do not manually deregister checks; let deregister_critical_service_after handle cleanup when configured to a positive duration. The reaper scans every check_reap_interval (default 30s), so removal can occur shortly after the configured critical duration expires; current source/docs do not impose a one-minute minimum. If the storm of state updates is pushing Raft toward saturation, you can temporarily lengthen deregister_critical_service_after to slow mass deregistration, or disable non-critical checks to shed write load. Tell service owners that Consul is the symptom, not the cause.

Node-level or gossip event

Verify network connectivity between all server pairs, not just to and from the leader. Check retry_join configuration and gossip encryption key consistency with consul keyring -list. For asymmetric partitions, member lists from each server will disagree. Do not remove Raft peers unless you are certain they are permanently gone.

Anti-entropy stall / client RPC failing

Identify why agents cannot reach servers: port 8300 blocked by a firewall change, server file-descriptor exhaustion, TLS certificate mismatch after a renewal. The catalog stays stale until the pipeline is restored. See Consul client rpc failed and Consul anti-entropy not syncing.

TTL check flapping

If TTL checks flap critical at near-identical times across hosts, suspect clock skew between agent and server, or an application renewal path that is racing the TTL. Issue #4742 reports TTL heartbeats not being seen until after the TTL expires; it is closed without a named fix, and the reporter pointed to broader RPC-latency issue #3738, still open at review. Verify NTP, then review the renewal interval relative to the configured TTL.

Check misconfiguration after a deploy

Validate check definitions at agent startup: since Consul 1.0.0, Consul rejects invalid checks and fails to start instead of silently skipping them. Confirm intervals and timeouts match observed response latency. For script checks, confirm enable_script_checks is set in agent config (it is opt-in and defaults to false).

ACL changes

Verify the token used by checks has service:read for the relevant endpoints. After upgrading to Consul 1.16.0 or later, expect /v1/health/connect/ and /v1/health/ingress/ to return 403 instead of an empty 200 for underprivileged tokens.

Prevention

• Track Output text, not just status. The single biggest gap in Consul health monitoring is ignoring the Output field. Capture Output strings and alert on patterns (connection refused, certificate expired, i/o timeout). These strings often precede or explain status transitions.
• Bucket criticals by CheckID prefix in dashboards. A combined “critical count” graph hides the difference between node-level and service-level events.
• Alert on the ratio, not the absolute count. Linear growth in criticals alongside linear service growth is normal. A step change in the critical-to-total ratio is not.
• Track transition rate alongside state. Critical ratio alone misses flapping checks.
• Monitor client agent RPC health separately from server health. Stale catalog data looks identical to healthy data if you only watch server metrics.
• Watch the Raft pipeline during health storms. Apply-rate spikes from health updates can cascade into leader churn, which itself creates spurious criticals.

How Netdata helps

• Per-second granularity on critical ratio and transition rate surfaces step changes as they happen.
• serfHealth criticals, gossip member status, and Raft peer count on the same dashboard distinguish a node event from a service event without cross-tool pivoting.
• Anomaly detection on the critical-count baseline separates seasonal and deploy-driven variation from genuine spikes.
• Raft commit time, apply rate, and leader election count correlate to expose when a health-check storm is stressing consensus.
• Per-agent client RPC failure visibility surfaces silent catalog staleness that masquerades as service failure.

Related guides

• Consul catalog bloat: too many services and checks slowing everything down
• Consul registration storm: catalog churn overwhelming Raft
• Consul anti-entropy not syncing: local agent state and the catalog drifting apart
• Consul client rpc failed: agents alive but the catalog is going stale
• Consul on EBS: burst-credit exhaustion and the sudden latency cliff
• Consul gossip encryption key mismatch: a botched keyring rotation splits the pool
• Consul gossip flapping: nodes oscillating between alive, suspect, and failed
• Consul serf queue backlog: an agent falling behind on gossip
• Consul gossip storm after mass recovery: rejoin floods and anti-entropy spikes
• How Consul actually works in production: a mental model for operators
• Consul leader election storm: repeated elections and rolling write outages
• Consul monitoring checklist: the signals every production cluster needs