$ guides / consul / consul-serfhealth-check-failing ▌

Operations Guides

Consul serfHealth check failing: the node-level check behind mass deregistration

Every Consul node carries an automatic health check called serfHealth. It is not user-defined, it cannot be removed, and it reflects the node’s membership in the Serf LAN gossip pool. When this check goes critical, Consul treats the entire node and every service registered on it as unhealthy, regardless of what individual service checks report.

This is the most commonly misdiagnosed failure in Consul. Operators see a service “go down” across all instances on a host, investigate the service checks, and find them still passing. The services are fine. The node’s gossip membership broke, and serfHealth cascaded that failure into every service on the node.

This guide covers why this happens, how to distinguish a serfHealth failure from a genuine service-level outage, and how DeregisterCriticalServiceAfter interacts with it.

What this means

Consul computes effective health as the node’s serfHealth status ANDed with each service-level check status. If serfHealth is critical, the effective health of every service on that node is critical, no matter what the service checks report.

When gossip detects that a node is unreachable, the cluster sets that node’s serfHealth check to critical automatically. The check output reads Agent not live or unreachable. The individual service checks on that node freeze in their last known state, which is usually passing. This is by design: the dead agent cannot update its own checks, so only the cluster-maintained serfHealth check moves.

The visible result looks like mass deregistration. DNS queries for the affected services return fewer or zero healthy instances. The HTTP API excludes the node’s services from ?passing=true results. Load balancers and sidecars consuming Consul health data drain or stop sending traffic. From the outside, every service on the node appears to have failed simultaneously.

flowchart TD
    A[UDP 8301 probe fails] --> B[Indirect probes through peers]
    B --> C[All probes fail]
    C --> D[Node marked suspect then failed]
    D --> E[serfHealth set to critical by cluster]
    E --> F[Node health ANDed with service checks]
    F --> G[All services treated as unhealthy]
    G --> H[DNS and API drop node services]

Common causes

Cause	What it looks like	First thing to check
UDP 8301 blocked	TCP works but gossip flaps; node oscillates between alive and failed	Firewall and security group rules for UDP 8301 in both directions
Network partition	Multiple nodes in the same segment go critical at once; Raft may also be affected	Cross-reference Raft peer status and gossip member lists from multiple servers
Container IP routing	Consul client in a container flaps because the server cannot route to the container IP	Whether the server can reach the container gossip IP on port 8301
Agent resource starvation	Agent process alive but too slow to respond to gossip probes within timeout	CPU, memory, and file descriptor usage on the agent host
Asymmetric firewall rules	One node sees another as alive, but the reverse fails	Run gossip connectivity tests from both directions

Quick checks

Run these read-only commands during the incident. They are safe and produce the data needed to separate a gossip failure from a service-level problem.

# List all critical checks; look for serfHealth entries
curl -s http://localhost:8500/v1/health/state/critical | jq '.[] | {Node, CheckID, Status, Output}'

# Gossip membership from this agent's perspective
curl -s http://localhost:8500/v1/agent/members | jq '.[] | {Name, Addr, Status, Tags: .Tags.role}'

# Local agent serf state
consul info | grep -A 20 'serf_lan'

# Per-node health, including service checks
curl -s "http://localhost:8500/v1/health/node/<node-name>" | jq '.[] | {CheckID, Status, Output}'

# TCP 8301 reachability from a server to the affected node (does not test UDP)
nc -z -w2 <node-ip> 8301 && echo "TCP 8301 open" || echo "TCP 8301 closed"

# Leader and Raft configuration, to rule out a cluster-wide issue
curl -s http://localhost:8500/v1/status/leader
curl -s http://localhost:8500/v1/operator/raft/configuration | jq '.[] | {Index, Servers: [.Servers[] | {Node, Address, Voter, Leader}]}'

The diagnostic pattern is straightforward. If the critical checks are all serfHealth and the individual service checks on those nodes show passing, you have a gossip-level failure, not a service-level outage.

How to diagnose it

Identify which nodes have critical serfHealth checks. Query /v1/health/state/critical and filter for CheckID: "serfHealth". Each result names the affected node and includes the output string.
Check gossip membership from multiple servers. Run consul members or query /v1/agent/members from at least two servers. If one server sees a node as alive and another sees it as failed, you have an asymmetric partition. If all servers agree the node is failed, the node is genuinely unreachable on gossip.
Verify the affected node’s service checks individually. Query /v1/health/node/<node-name>. Service checks showing passing while serfHealth shows critical means the services themselves are fine and the problem is gossip connectivity.
Test bidirectional gossip connectivity. Gossip uses both UDP and TCP on port 8301 between all agents, not just client to server. From the affected node, test connectivity to a server on 8301. From a server, test connectivity to the affected node on 8301. TCP is easy to verify with nc. UDP is harder to test directly with simple tools; if TCP works but gossip still fails, suspect UDP blocking.
Check for container or overlay network issues. If the affected agent runs in a Docker container or Kubernetes pod, verify that its gossip IP is routable from the servers. A common failure is the agent advertising a container bridge IP that servers cannot reach.
Check agent resource usage. An agent that is CPU-starved, memory-pressured, or out of file descriptors may be too slow to respond to gossip probes within the timeout window. Check consul info for goroutine count and runtime stats, and check the OS for CPU and memory pressure.
Review gossip protocol parameters. If flapping is chronic, the probe interval, probe timeout, or suspicion multiplier may be too aggressive for your network latency. These are configured in the serf_lan stanza.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`consul members` LAN states	Tracks gossip health independently of Raft	Drop in `alive` count without corresponding leave events
`consul_health_service_status` critical count	Mass critical events may be serfHealth-driven, not service-driven	Spike in critical count correlated with gossip member failures
`consul.client.rpc.failed`	Agent-to-server RPC failures indicate connectivity problems	Sustained non-zero rate on client agents
Health check `Output` field	The `serfHealth` output reveals the failure mode	Output reads `Agent not live or unreachable`
Gossip queue depth	Agent falling behind on message processing	Sustained non-zero queue depth
Raft last contact on followers	Rules out a cluster-wide issue versus node-level gossip	Rising values suggest a network problem affecting the whole cluster

Fixes

UDP 8301 blocked

Gossip uses both UDP and TCP on port 8301. If TCP is open but UDP is blocked, the failure detector trips continuously while TCP-based anti-entropy sync recovers, producing chronic flapping. This is the most common cause of serfHealth oscillation.

Open UDP 8301 in both directions between all Consul agents. This includes client-to-client traffic, not just client-to-server. All nodes must be able to reach each other over port 8301.

Network partition

If the partition is infrastructure-level (switch failure, security group change), fix the underlying network. Consul reconciles automatically once connectivity is restored. Do not remove nodes from the Raft configuration unless you are certain they are permanently gone.

If the partition has healed but nodes remain in failed state, it may take several gossip intervals for membership to converge. Monitor the alive count and wait. Forcing re-joins is rarely necessary.

Container IP routing

If a Consul client runs inside a container, gossip probes target the container’s IP. If servers cannot route to that IP, serfHealth flaps. Configure the agent’s bind_addr or advertise_addr to use an IP reachable by all cluster members. On Docker bridge networking, advertise the host IP or use host networking. On Kubernetes, verify the pod IP is routable from server nodes.

Agent resource starvation

If the agent process is alive but too slow to respond to gossip probes, raise resource allocation or reduce load. Common causes include CPU limits in containers that are too low (causing throttling during gossip processing), file descriptor exhaustion preventing the agent from accepting probe responses, and GC pauses on a large Go heap exceeding the gossip probe timeout.

Compare the gossip timeout against the gossip_lan configuration. If your network latency exceeds the default, increase gossip_lan.probe_timeout or reduce gossip_lan.probe_interval; the advanced gossip docs warn that improper tuning can cause unexpected failures.

DeregisterCriticalServiceAfter interactions

DeregisterCriticalServiceAfter is a per-check field on user-defined service checks. It specifies how long a check can remain critical before Consul deregisters the associated service. It does not apply to the serfHealth check itself.

When a node fails and serfHealth goes critical, the node itself is not immediately deregistered. Services on that node are treated as unhealthy by the AND logic, but the catalog entries remain. The node is eventually reaped by Serf’s reconnect timeout. reconnect_timeout controls how long a failed node takes to be completely removed; it defaults to 72 hours, is configurable, and must be at least 8 hours. reconnect_timeout_wan is the WAN server equivalent.

If services on the failed node have DeregisterCriticalServiceAfter configured, those services may be deregistered after their critical timeout elapses. This is independent of the serfHealth mechanism and depends on per-service check configuration.

Prevention

Monitor gossip membership independently of Raft. Alert on any agent in failed or suspect state for more than 30 seconds. The consul members CLI and /v1/agent/members API are the primary signals.
Test gossip connectivity proactively. Include UDP 8301 in your network health checks. Teams that test only TCP ports miss UDP failures until serfHealth starts flapping.
Validate container networking before registering services. Ensure the advertised gossip address is routable from all cluster members before starting the agent.
Set file descriptor limits appropriately. Consul recommends a minimum ulimit -n of 65536 for servers. Insufficient file descriptors cause gossip probe failures under load.
Correlate serfHealth failures with service-level alerts. When serfHealth failure is the root cause, service-level alerts fire in bulk. If all services on a single node go critical simultaneously, suspect serfHealth first.
Tune gossip parameters for your network. If your latency exceeds the defaults, increase gossip_lan.probe_timeout or reduce gossip_lan.probe_interval to reduce false positives; treat this as advanced tuning and verify the change carefully.

How Netdata helps

Per-second gossip membership metrics let you see the exact moment a node transitions from alive to suspect to failed, and whether it recovers or stays down.
Correlation between consul.serf.lan.members and consul_health_service_status makes it obvious when a mass critical event is gossip-driven rather than service-driven. If the critical spike coincides with a gossip membership change, the root cause is the node, not the services.
Agent-level RPC failure metrics (consul.client.rpc.failed) distinguish gossip failures from agent-to-server connectivity problems, which have different root causes and fixes.
ML anomaly detection on gossip queue depth catches agents that are falling behind on message processing before peers mark them as failed.
Composite dashboards showing gossip health, Raft health, and service health side by side let you quickly determine whether a failure is node-level, cluster-level, or service-level.