Consul server in failed state: reading consul members during an incident

You ran consul members during an incident and one of your servers shows failed. Before fixing anything, understand what the status means. Gossip failure and Raft failure are separate subsystems. A server in failed state is not answering gossip probes, but it may still be a voting Raft peer, or its agent process may still be running but starved of resources.

The distinction that matters during an incident: gossip health is not Raft health and is not agent health. A node can be alive in gossip yet useless for consensus. A node can show failed in gossip while its process is still running. These distinctions are the difference between a two-minute diagnosis and a misdiagnosis that makes the incident worse.

What the status field means

The /v1/agent/members endpoint and the consul members CLI report each member with a status field. The API returns numeric codes:

The CLI renders the numeric codes through the same Serf status strings, so it can display alive, leaving, left, or failed.

A server showing failed (status 4) has stopped answering gossip probes. The Serf gossip protocol, a SWIM variant, probes each member periodically over UDP (LAN port 8301, WAN port 8302). If a node does not acknowledge a direct probe within the probe timeout, the protocol tries indirect probes through other members. Only after both direct and indirect probes fail does the node move through suspect and eventually to failed.

Detection takes time. Serf failed status may lag the actual failure by 10 to 60 seconds depending on gossip interval and suspicion multiplier settings. This lag is useful operationally: gossip failure detection often fires before Raft detects leader loss, giving you an early warning window. But it also means the Raft cluster may still have quorum when you first see the gossip alert. A single server in failed in a 3-server cluster means you are one failure away from losing quorum. In a 5-server cluster, a single failed server reduces headroom but does not threaten quorum.

flowchart TD
    A["Server shows failed in gossip"] --> B{"Raft quorum intact?"}
    B -- "Yes, leader exists" --> C["Reduced redundancy, no write outage"]
    B -- "No leader or below quorum" --> D["PAGE: cluster cannot commit writes"]
    A --> E{"Agent process running?"}
    E -- "No" --> F["Crash, OOM kill, or manual stop"]
    E -- "Yes but unresponsive" --> G["CPU starvation, disk saturation, or partition"]

Common causes

Quick checks

All commands below are read-only and safe to run during an incident.

# List failed members from this server's gossip view
consul members -status=failed

# API view with numeric status codes, servers only
curl -s http://127.0.0.1:8500/v1/agent/members | \
  python3 -c "import sys,json; [print(m['Name'], m['Status'], m['Addr']) for m in json.load(sys.stdin) if m.get('Tags',{}).get('role')=='consul']"

# Compare gossip views across servers (run on each server)
consul members

# Check Raft peer configuration and voter status
consul operator raft list-peers

# Check if a leader exists (empty string = no leader)
curl -s http://127.0.0.1:8500/v1/status/leader

# Check this server's own Serf LAN state
consul info | grep -A 20 serf_lan

# Check telemetry counters for member state transitions
curl -s http://127.0.0.1:8500/v1/agent/metrics | grep -E "consul.serf.member"

# Verify the failed server's process state
ssh <failed-server> "pgrep -x consul || echo 'consul process not found'"

How to diagnose it

1. Confirm the failure is real and not a stale view. The /v1/agent/members endpoint is eventually consistent. Results may differ between agents. Run consul members from at least two other servers. If they agree the node is failed, proceed. If they disagree, you may be looking at an asymmetric partition where one side can see the other but not vice versa.

2. Check whether Raft still has quorum. Run consul operator raft list-peers. Count the voters. For a 3-server cluster you need 2 reachable voters. For a 5-server cluster you need 3. A server can be failed in gossip but still listed as a voter in the Raft configuration, because gossip failure detection and Raft peer removal are independent mechanisms. The distinguishing signal of imminent quorum loss: Serf shows failed members while Raft still shows them as voters.

3. Determine if the failed server’s process is running. SSH to the host. If the process is gone (crash, OOM kill, systemd stop), gossip is accurately reporting a dead server. If the process is running but the host is starved (CPU pegged, disk saturated), the server is alive but cannot respond to probes within the timeout window.

4. Check pairwise network connectivity on the gossip port. Gossip uses UDP on port 8301 for LAN and 8302 for WAN. A firewall rule, MTU problem, or VXLAN overlay issue can drop UDP probes while leaving TCP traffic (Raft on 8300, RPC) unaffected. This creates the confusing situation where a server is unreachable for gossip but partially reachable for other protocols. Use tcpdump to confirm probes are actually reaching the host:

# On the failed server, capture incoming gossip traffic (read-only, safe)
sudo tcpdump -i any -n udp port 8301 -c 20

If you see no incoming probes, the problem is network path or firewall, not the Consul process.

5. Correlate with Raft-specific signals. Check consul.raft.commitTime on the leader and consul.raft.leader.lastContact on followers. If both are healthy, the Raft cluster is functioning despite the gossip failure. You have reduced redundancy but no active write outage. If lastContact is climbing toward the election timeout, the gossip failure may be a symptom of a broader network problem that is about to take down Raft as well.

Metrics and signals to monitor

Severity guidance:

• PAGE: any server in failed state in a 3-server cluster. You are one failure from quorum loss.
• PAGE: 2 or more servers in failed in a 5-server cluster.
• TICKET: any single server in failed in a 5-server cluster. Quorum is maintained but redundancy headroom is reduced.

Fixes

Process crash or OOM kill

If the process is gone, investigate the cause before restarting. Check dmesg for OOM killer messages. Check disk space on the Raft data directory. A server that ran out of disk cannot persist its Raft log and will crash or stall repeatedly. Restart the process once you understand why it died, otherwise it will fail again under the same conditions.

Network partition

If the partition is real, focus on restoring connectivity. Do not remove the failed peer from the Raft configuration unless you are certain it is permanently gone. Premature removal risks split-brain if the node returns with divergent state. When the partition heals, gossip will return the node to alive within several gossip intervals. This may take longer than expected because the returning node must be re-confirmed by multiple peers.

Server overloaded

If the server process is running but the host is CPU-starved or disk-saturated, gossip probes are timing out because the agent cannot process them fast enough. This is often the same root cause that drives Raft instability. Address the resource pressure: increase CPU limits in containerized deployments, move the Raft data directory to faster storage (dedicated SSD, not shared or network-attached), or shed write load. Slow disk I/O is a common root cause of cascading Consul failures that begin as gossip instability.

Stale or stuck member entries

Failed nodes remain in the member list because Consul attempts reconnection for reconnect_timeout (LAN and WAN default 72 hours). If a node is permanently gone and the stale entry is causing operational noise, use consul force-leave <node> to transition it to left state. The -prune flag (added in Consul 1.6.2) removes the node from the member list entirely.

Two important caveats. First, force-leave changes gossip state but does not remove the node from the Raft voter list. For server nodes, you may also need consul operator raft remove-peer -address=<host>:8300 or ... -id=<server-id> to clean up the consensus configuration. Second, applying -prune to many nodes simultaneously in large clusters can trigger gossip storms. Process one node at a time.

Prevention

• Alert on consul.serf.member.failed increments for server nodes. This is the earliest gossip-layer signal that a server has stopped responding.
• Track alive member count in the LAN pool. Alert on sudden drops that do not correlate with known scale-down events.
• Compare gossip membership against Raft peer configuration regularly. A node alive in gossip but missing from Raft, or vice versa, indicates partition or configuration drift that will cause confusion during the next incident.
• Ensure file descriptor limits are adequate. Consul documentation recommends at least 65536 for servers. FD exhaustion causes gossip probe failures and false failed detections.
• Run Consul servers on dedicated SSD storage. Slow disk I/O degrades Raft commit times and cascades into gossip instability when the agent cannot keep up with probe processing.
• Do not alert on left (status 3) nodes. Graceful departures are retained for the reconnect window. Only failed (status 4) indicates an unexpected problem.

How Netdata helps

• Per-second collection of consul.serf.member.failed and related counters shows the exact moment a member transitions to failed, not a minute later when a longer polling interval catches up.
• Correlate the gossip failure counter with consul.raft.leader.lastContact and consul.raft.commitTime in the same dashboard to determine immediately whether the gossip failure has spread to Raft or is isolated to the membership layer.
• The alive member count gauge provides a baseline so sudden drops are obvious without manual threshold tuning.
• When the Netdata agent runs on each server, comparing gossip views across nodes reveals asymmetric partitions that a single-server perspective cannot detect.

Related guides

• How Consul actually works in production: a mental model for operators
• Consul leader election storm: repeated elections and rolling write outages
• Consul monitoring checklist: the signals every production cluster needs
• Consul monitoring maturity model: from survival to expert
• Consul “No cluster leader”: every write is failing
• Consul raft commitTime high: the write pipeline is slowing down
• Consul raft lastContact rising: followers drifting toward an election
• Consul lost quorum: Raft peers below the majority needed to elect a leader