$ guides / consul / consul-service-critical-health-checks ▌

Operations Guides

Consul critical health checks spiking: real outage or broken checks?

The alert fires when /v1/health/state/critical jumps from single digits to dozens or hundreds of checks in minutes. The page says “services are unhealthy,” but that is ambiguous: either real services are down, or the checks themselves are broken. Pick the wrong fork and you burn an hour on the wrong dependency.

The first decision is not “what failed.” It is “is this a service-level event, a node-level event, or a monitoring-level event?” The cheapest signal is the check Output text field, which carries the raw error from check execution. It holds strings like connection refused, i/o timeout, certificate has expired. That field usually resolves the diagnosis in seconds, and most teams ignore it until their second or third incident.

The second fork is gossip. A spike that correlates with gossip member failures is an infrastructure event. A spike against healthy gossip is service-level. Mixing the two produces the wrong runbook.

What this means

/v1/health/state/critical returns every check in critical status. The response includes CheckID, Name, Status, Output, ServiceID, ServiceName, Node, and Namespace. The CheckID prefix tells you which subset is spiking.

Three classes of spike:

serfHealth checks. The node-level check maintained by the gossip protocol. A spike here is a node or gossip event, not a service-level event. When serfHealth goes critical, Consul excludes all services on that node from DNS and service discovery. The Output is typically Agent not live or unreachable .
service: prefixed checks. Service-level. The service itself is failing its configured check.
Maintenance checks (_service_maintenance:, _node_maintenance). Intentional. These should correlate with change tickets.

The next fork is whether the spike is real (services actually down) or whether the checks themselves are broken (misconfigured, timing out against a shared dependency, ACL-blocked, anti-entropy stalled, or racing a TTL). The Output field is the fastest path to that answer.

flowchart TD
    A["critical count spike"] --> B{"Dominant CheckID?"}
    B -->|"serfHealth"| C["node/gossip event"]
    B -->|"service: prefix"| D["service-level event"]
    B -->|"maintenance"| E["intentional - check change ticket"]
    C --> F{"gossip members failed?"}
    F -->|"yes"| G["infra or AZ event"]
    F -->|"no, Raft peers missing"| H["partition or quorum loss"]
    D --> I{"Output populated?"}
    I -->|"yes, shared string"| J["downstream dependency"]
    I -->|"empty"| K["query /v1/agent/checks locally"]
    D --> L{"local agent checks match catalog?"}
    L -->|"no, local passing"| M["anti-entropy stall - check RPC"]

Common causes

Cause	What it looks like	First thing to check
Real downstream dependency outage	Many independent services critical simultaneously with similar Output text	Output field grouped across checks
Network partition or AZ event	`serfHealth` criticals concentrated on one segment	`consul members` vs Raft peer list
Anti-entropy stall / client RPC failing	Services look healthy locally, catalog shows stale critical	`consul.client.rpc.failed` on client agents
Check misconfiguration after deploy	Spike coincides with deploy window; Output shows timeout or refused	Check intervals and timeouts vs observed response time
TTL check expiry or clock skew	TTL checks flap critical at near-identical times across hosts	NTP skew; application renewal path
ACL change blocking health endpoints	403 in Output, or empty check results after upgrade	Token policy on the check; `service:read` coverage
Leader re-election	Criticals appear with empty Output during election window	`consul.raft.state.leader` election count
Output propagation delay	Catalog shows critical with empty Output, local agent has Output	Query `/v1/agent/checks` on the agent

Quick checks

# Total critical checks right now
curl -s http://127.0.0.1:8500/v1/health/state/critical | jq length

# Bucket by CheckID prefix (serfHealth vs service-level vs maintenance)
curl -s http://127.0.0.1:8500/v1/health/state/critical | \
  jq -r '.[].CheckID | split(":")[0]' | sort | uniq -c | sort -rn

# Sample the Output field - the cheapest root-cause signal
curl -s http://127.0.0.1:8500/v1/health/state/critical | \
  jq -r '.[] | "\(.Node) \(.CheckID) => \(.Output[:120])"' | head -30

# Cluster common Output strings to find the shared dependency
curl -s http://127.0.0.1:8500/v1/health/state/critical | \
  jq -r '.[].Output' | sort | uniq -c | sort -rn | head -10

# Gossip view of node state
curl -s http://127.0.0.1:8500/v1/agent/members | \
  jq -r '.[].Status' | sort | uniq -c | sort -rn

# Check state from the local agent (real-time, bypasses anti-entropy delay)
curl -s http://127.0.0.1:8500/v1/agent/checks | jq -r '.[] | "\(.CheckID) [\(.Status)] \(.Output[:100])"'

# Client agent RPC failures (catalog staleness indicator)
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -E 'client.rpc|consul.client.rpc'

# Raft pipeline health if the spike is large
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -E 'raft.commitTime|raft.apply|raft.state.leader'

All read-only. None mutate catalog state.

How to diagnose it

Bucket the spike by CheckID prefix. If the bulk of criticals are serfHealth, you have a node or gossip event. Go to step 2a. If they are service: checks, go to step 2b.

2a. Node-level event. Correlate with consul members and the Serf member status gauge. If failed members cluster in one AZ or segment, suspect a network event. Cross-reference with the Raft peer list at /v1/operator/raft/configuration to see whether quorum is intact.

2b. Service-level event. Go to the Output field. Group criticals by Output text. A single repeated string (dial tcp ...: connect: connection refused, i/o timeout, certificate has expired) usually points at a shared dependency.

Check for anti-entropy stall. If /v1/agent/checks on the local agent shows the check passing but /v1/health/state/critical on the server shows it critical, the agent-to-server RPC pipeline is broken. Confirm with consul.client.rpc.failed rate and verify port 8300 reachability.
Rule out the known Output delay. If Output is empty in the catalog but the checks are recent, the Output field may not yet have propagated . Query /v1/agent/checks on the local agent for the real-time value. The local agent Output text is the workaround for this lag.
Check the Raft pipeline if the scope is wide. A massive spike in health updates generates a Raft apply storm. Watch consul.raft.commitTime and the consul.raft.apply rate. If commit time approaches the election timeout, the cluster can degrade into leader churn, which itself produces spurious criticals with empty Output during election windows.
Verify ACLs if you recently upgraded. Consul 1.16.0+ returns 403 “Permission denied” on /v1/health/connect/ and /v1/health/ingress/ for tokens lacking service:read; applications expecting an empty 200 will mishandle this. Script checks have been opt-in via enable_script_checks , so upgrades that drop this flag silently fail script-based checks. Invalid check definitions also fail agent startup rather than being silently skipped .

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`consul_health_service_status` critical ratio	Core spike signal	Sudden 10x increase; more than 50% of business-critical service checks critical
Health check transition rate	Distinguishes sustained failure from flapping	More than 2 transitions per check per 5 minutes
Check `Output` text via API	Most underused diagnostic	Empty Output during an incident means fall back to `/v1/agent/checks`
`serfHealth` critical count	Node-level vs service-level fork	Any non-zero means a node is unreachable in gossip
`consul.serf.lan.members` alive vs failed	Gossip-level node state	Failed members indicate infra event
`consul.client.rpc.failed`	Agent-to-server pipeline	Sustained non-zero means the catalog is going stale
`consul.raft.commitTime` and apply rate	Write pipeline saturation from update storm	Commit time approaching election timeout
`consul.raft.state.leader` election count	Spurious criticals during elections	Elections coinciding with the spike
Anti-entropy sync interval vs configured	Sync stall detection	Interval more than 2x the configured `sync_interval`
ACL 403 rate, `consul.acl.resolveToken` latency	Authorization path health	Spike shortly after a policy change

Fixes

Real downstream dependency outage

The catalog is reporting truth. Focus on the dependency, not Consul. Do not manually deregister checks; let deregister_critical_service_after handle cleanup (minimum one minute; reaper interval is implementation-dependent ). If the storm of state updates is pushing Raft toward saturation, you can temporarily lengthen deregister_critical_service_after to slow mass deregistration, or disable non-critical checks to shed write load. Tell service owners that Consul is the symptom, not the cause.

Node-level or gossip event

Verify network connectivity between all server pairs, not just to and from the leader. Check retry_join configuration and gossip encryption key consistency with consul keyring -list. For asymmetric partitions, member lists from each server will disagree. Do not remove Raft peers unless you are certain they are permanently gone.

Anti-entropy stall / client RPC failing

Identify why agents cannot reach servers: port 8300 blocked by a firewall change, server file-descriptor exhaustion, TLS certificate mismatch after a renewal. The catalog stays stale until the pipeline is restored. See Consul client rpc failed and Consul anti-entropy not syncing.

TTL check flapping

If TTL checks flap critical at near-identical times across hosts, suspect clock skew between agent and server, or an application renewal path that is racing the TTL. Issue #4742 documents TTL heartbeats not being seen until after the TTL expires. Verify NTP, then review the renewal interval relative to the configured TTL.

Check misconfiguration after a deploy

Validate check definitions at agent startup: Consul rejects invalid checks and fails to start instead of silently skipping them . Confirm intervals and timeouts match observed response latency. For script checks, confirm enable_script_checks is set in agent config (it is opt-in and defaults to false).

ACL changes

Verify the token used by checks has service:read for the relevant endpoints. After upgrading past 1.16.0 , expect /v1/health/connect/ and /v1/health/ingress/ to return 403 instead of an empty 200 for underprivileged tokens.

Prevention

Track Output text, not just status. The single biggest gap in Consul health monitoring is ignoring the Output field. Capture Output strings and alert on patterns (connection refused, certificate expired, i/o timeout). These strings often precede or explain status transitions.
Bucket criticals by CheckID prefix in dashboards. A combined “critical count” graph hides the difference between node-level and service-level events.
Alert on the ratio, not the absolute count. Linear growth in criticals alongside linear service growth is normal. A step change in the critical-to-total ratio is not.
Track transition rate alongside state. Critical ratio alone misses flapping checks.
Monitor client agent RPC health separately from server health. Stale catalog data looks identical to healthy data if you only watch server metrics.
Watch the Raft pipeline during health storms. Apply-rate spikes from health updates can cascade into leader churn, which itself creates spurious criticals.

How Netdata helps

Per-second granularity on critical ratio and transition rate surfaces step changes as they happen.
serfHealth criticals, gossip member status, and Raft peer count on the same dashboard distinguish a node event from a service event without cross-tool pivoting.
Anomaly detection on the critical-count baseline separates seasonal and deploy-driven variation from genuine spikes.
Raft commit time, apply rate, and leader election count correlate to expose when a health-check storm is stressing consensus.
Per-agent client RPC failure visibility surfaces silent catalog staleness that masquerades as service failure.