$ guides / consul / consul-health-check-flapping ▌

Operations Guides

Consul health check flapping: the passing/critical oscillation that churns the catalog

Every health check state transition is a catalog write. The client agent runs the check locally, detects a status change, and pushes the update to the leader. The leader commits a Raft log entry, the FSM applies it, per-service caches invalidate, blocking queries watching that service return, and every downstream consumer (load balancer, Envoy control plane, consul-template, DNS resolver) re-evaluates. One flapping check is an annoyance. A dozen flapping in unison saturates the Raft write pipeline and looks indistinguishable from a registration storm.

The hard part is separating two problems with the same symptom. A misconfigured check (interval too aggressive, timeout too short, no dampening) flaps against a healthy service. A genuinely marginal service (intermittent timeouts, resource starvation, dependency instability) flaps against a correctly configured check. The fix for the first is check tuning. The fix for the second is fixing the service. Applying the wrong fix gives you either a silent outage (you damped away real failures) or continued thrash (you tuned a check that was already correct).

This is the diagnostic runbook for that distinction: the write cascade, the signals that separate flapping from real degradation, the tuning levers Consul provides, and the gap that catches teams later. A check that always returns passing is not evidence it would detect a real failure.

What this means

Health checks run on client agents, not servers. The agent executes the check (HTTP, TCP, script, or TTL heartbeat), compares the result to the last known state, and only on a state transition pushes an update to the server cluster. That update becomes a Raft commit. The leader replicates it to quorum, applies it to the FSM (the catalog), and the new state is visible cluster-wide.

Continuous oscillation produces a sustained stream of Raft commits, each invalidating caches and firing watches. At scale, this consumes write pipeline capacity that should serve real registration and deregistration events.

flowchart TD
    A[Agent executes health check] --> B{Result differs from last state?}
    B -->|No| A
    B -->|Yes: state transition| C[Agent pushes update to leader via RPC]
    C --> D[Leader commits Raft log entry]
    D --> E[FSM applies: catalog mutated]
    E --> F[Per-service cache invalidated]
    E --> G[Blocking queries return to watchers]
    F --> H[Downstream consumers re-evaluate]
    G --> H
    H --> A

The downstream impact is what makes this painful. Load balancers that poll Consul’s health API pull endpoints in and out of rotation on every transition. Service mesh control planes push new xDS configuration to sidecars on every catalog change. consul-template instances regenerate configuration files. DNS caches expire and re-query. A check flapping at 5-second intervals against a service with 50 instances can generate hundreds of downstream reactions per minute, none reflecting a real change in health.

Common causes

Cause	What it looks like	First thing to check
Interval/timeout ratio too tight	Check flips at regular intervals, correlating with latency spikes	Compare `timeout` to `interval`; if timeout approaches or exceeds interval, probes overlap
No dampening thresholds	Every failed probe immediately transitions to critical, then back	Check whether `success_before_passing` and `failures_before_critical` are set (defaults are 0)
Check testing the wrong thing	Check passes or fails independently of actual service health	Read the check Output field; verify the logic matches the failure mode you care about
Marginally loaded service	Check fails under load but recovers quickly; pattern correlates with CPU, memory, or dependency latency	Correlate check transitions with host and dependency metrics
TTL heartbeat race	TTL checks flap even though the service sends heartbeats on time	Check if TTL is too close to the heartbeat interval; agent-side processing latency can cause the heartbeat PUT to miss the window
deregister window too aggressive	Service disappears from catalog during brief blips, then reappears with a re-registration write	Check `deregister_critical_service_after`; minimum effective timeout is 1 minute, reaper runs every 30 seconds

Quick checks

# Count checks currently in critical state
curl -s http://localhost:8500/v1/health/state/critical | jq length

# List critical checks with their output (the diagnostic text field)
curl -s http://localhost:8500/v1/health/state/critical | jq '.[] | {CheckID, Name, Output}'

# Health check status (sample repeatedly; Consul has no direct transition-rate metric)
curl -s http://localhost:8500/v1/health/state/critical | jq length

# Raft transaction rate (is the write pipeline seeing churn?)
curl -s http://localhost:8500/v1/agent/metrics | grep -E 'raft.apply'

# Cache hit ratio (are check transitions invalidating caches?)
curl -s http://localhost:8500/v1/agent/metrics | grep consul_cache

# Catalog registration and deregistration rate (is churn reaching the catalog?)
curl -s http://localhost:8500/v1/agent/metrics | grep -E 'catalog.(register|deregister)'

# Anti-entropy disagreement: compare local and catalog service state
curl -s http://localhost:8500/v1/agent/services | jq 'keys'
curl -s http://localhost:8500/v1/catalog/service/<service-name> | jq '[.[].ServiceID]'

# Client RPC failure rate (can the agent reach servers at all?)
curl -s http://localhost:8500/v1/agent/metrics | grep 'client.rpc.failed'

# Inspect a specific flapping check's configuration and recent output
curl -s "http://localhost:8500/v1/health/checks/<service-name>" | jq '.[] | {CheckID, Status, Output}'

How to diagnose it

Identify which checks are flapping, not just which are critical. A check sitting steadily in critical is a different problem from one oscillating. Query the health state repeatedly and look for checks that appear and disappear from the critical list across samples. The transition rate matters more than the absolute count.
Read the check Output field. The Output field contains the diagnostic text: “connection refused,” “i/o timeout,” “HTTP 503,” “certificate has expired.” This is the fastest path to root cause. Query it via the API during the incident rather than guessing from the status code.
Correlate transitions with Raft apply rate. If consul.raft.apply spikes coincide with health check transitions, flapping is consuming write pipeline capacity. If the apply rate is flat but checks still flip locally, the agent may be churning without successfully pushing all updates. Check client.rpc failure metrics for agent-to-server connectivity issues.
Check the interval/timeout relationship. If timeout is close to or equal to interval, probes overlap. A slow probe from the previous interval interferes with the next. Default HTTP check timeout is 10 seconds. If your interval is also 10 seconds, you have zero margin.
Check for dampening configuration. The defaults for success_before_passing and failures_before_critical are both 0, meaning a single probe result transitions the state immediately. If unset, every transient blip becomes a full state transition with a full catalog write.
Correlate with host and dependency metrics. If the check is an HTTP check hitting a service that is genuinely slow under load, the flapping is real. Look at the service’s own latency, CPU, memory, and downstream dependency metrics. If those are stable and the check still flaps, the check configuration is the problem.
Distinguish from anti-entropy disagreement. If the agent and catalog states perpetually disagree, anti-entropy sync generates registration events on every cycle even without real changes. Check anti-entropy sync success and failure rates.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Health check transition rate	Direct measure of flapping; each transition is a catalog write	More than 2 transitions per check per 5 minutes
`consul.raft.apply` rate (leader)	Shows whether check churn is consuming write pipeline capacity	Spike correlating with health check status changes
Cache hit ratio (`consul_cache`)	Check transitions invalidate per-service caches	Sudden 20%+ drop in hit ratio for service caches
Catalog register/deregister rate	Each transition may trigger registration events via anti-entropy	Rate more than 5x baseline without deployments
Client RPC failure rate	Agent cannot push check updates to servers; catalog goes stale	Any sustained non-zero rate
Check Output content	Contains the actual diagnostic text (timeout, refused, expired)	Query via API during incidents
Anti-entropy sync success rate	Failed sync causes local agent state and catalog to drift	Any non-zero failure rate
Per-service healthy instance count	Downstream impact of flapping on traffic routing	Instance count oscillating in sync with check transitions

Fixes

Interval and timeout tuning

The most common cause of flapping is a check interval and timeout that are too close together. If timeout approaches or equals interval, a slow probe from one cycle bleeds into the next. Practical rules:

Interval should be significantly longer than timeout. If your check takes up to 2 seconds under load, set timeout to 3 seconds and interval to at least 10 seconds.
Do not set interval below 5 seconds for HTTP checks. Sub-5-second intervals leave no room for transient network latency or agent scheduling delays.
For script checks, account for script startup overhead. A script that takes 25 seconds under load with a 30-second interval will flap.

If the check must detect failures faster than 5 seconds, use TTL checks with heartbeats sent by the application itself, and set the TTL to at least 2x the heartbeat interval. This absorbs agent-side request processing latency, which can delay heartbeat PUTs by seconds under load and cause false TTL expirations.

Dampening thresholds

Consul provides two fields to dampen state transitions (available since Consul 1.7.0):

success_before_passing (default 0): Number of consecutive successful probes required before a check transitions from critical or warning to passing. Setting this to 2 or 3 prevents a single successful probe from flipping the state back immediately after a failure.
failures_before_critical (default 0): Number of consecutive failed probes required before a check transitions to critical. Setting this to 2 or 3 prevents a single timeout from marking the service critical.

Both default to 0, so every single probe result can transition the state and trigger a catalog write. For most production checks, setting both to 2 or 3 eliminates noise from transient blips without meaningfully delaying detection of real failures.

Tradeoff: Before Consul 1.11.0, setting failures_before_critical above 1 meant the check appeared passing while probes were actually failing. This hid early failures from monitoring during rolling deployments, making the scope of impact invisible. The failures_before_warning field (introduced in 1.11.0, defaults to the same value as failures_before_critical) addresses this by letting the check enter warning state before reaching the critical threshold. If you are on a version older than 1.11.0, dampening hides intermediate failures from dashboards.

deregister_critical_service_after

This field controls how long a check can remain critical before the service is automatically removed from the catalog. The minimum effective timeout is 1 minute because the reaper runs every 30 seconds. If set too aggressively (for example, 30 seconds), a brief blip deregisters the service entirely, and when the check passes again the service must re-register, generating additional catalog writes on top of the health state transitions.

For services that occasionally blip, set this to several minutes. For services that should be removed quickly when truly down, keep it short, but understand the re-registration cost on recovery.

Fixing the check logic

If the check is testing the wrong thing, no amount of interval tuning helps. Common patterns:

Health endpoint that returns 200 regardless of downstream dependency health. The check passes even when the service cannot serve requests. Verify the health endpoint actually exercises critical dependencies (database connectivity, cache availability).
TCP check on a port that accepts connections but serves errors. A successful TCP dial does not mean the service is functional. Use an HTTP check that validates response status or content.
Check timeout longer than the consumer’s timeout. If your load balancer gives up after 2 seconds but your health check waits 10 seconds, the LB routes traffic to a service that Consul still considers healthy.

This is the meta-gap that catches teams: a check that always passes is not proof of health. Periodically verify detection by deliberately failing a service instance and measuring how long until Consul marks it critical and downstream consumers stop routing to it. The detection latency matters as much as the detection itself.

Fixing the service

If the check is correctly configured and the service is genuinely marginal, the fix is in the service, not Consul. Common root causes: CPU throttling, memory pressure, downstream dependency instability (database connection exhaustion, cache eviction storms), or connection pool exhaustion under burst load. The check is doing its job. Damping it away masks a real problem that will eventually become a full outage.

Prevention

Set dampening thresholds on all production checks. success_before_passing and failures_before_critical at 2 or 3 should be the default, not the exception.
Audit interval/timeout ratios during deployment. Timeout should be no more than half the interval. Document the ratio in the service registration template.
Track health check transition rate as a monitoring signal. More than 2 transitions per check per 5 minutes indicates instability. Alert on it before it becomes a storm.
Monitor check Output content, not just status. The Output field contains the diagnostic text. Surface it during incidents instead of requiring a manual API query at 3 a.m.
Periodically chaos-test check detection. Deliberately fail a service instance and measure detection latency end to end. A check that never fails may not be testing anything useful.
Set deregister_critical_service_after deliberately. Understand the re-registration cost and the reaper interval before accepting a short window.

How Netdata helps

Per-second metric resolution captures health check transition patterns that minute-aggregated telemetry smooths over. A check flapping at 5-second intervals may show only as an elevated average critical ratio in 1-minute windows, but per-second data shows the oscillation directly.
Correlating consul_raft_fsm_apply rate with health check status changes in a single view makes it immediately visible whether check churn is consuming write pipeline capacity, without cross-referencing separate dashboards.
Cache hit ratio displayed alongside health metrics reveals whether transitions are driving cache invalidation storms, a downstream effect invisible if you only watch check status counts.
Client agent RPC failure rate on the same dashboard as health check status distinguishes “the check is flapping” from “the agent cannot push updates and the catalog is going stale,” which require different fixes.
ML anomaly detection on transition rate surfaces flapping checks before they cross static thresholds, useful for services whose baseline check behavior shifts gradually as load grows.