RabbitMQ node down: telling a dead broker apart from a partitioned one

A node has disappeared from your cluster view. The first question is not “how do I bring it back” but “what actually happened to it.” A dead broker (VM crashed, process killed, host gone) and a partitioned broker (Erlang node still running but cut off from its peers) look almost identical from one vantage point and completely different from another. The recovery steps are opposite: you restart a dead node, but restarting a partitioned node mid-partition can make split-brain worse.

The trap is that running=false from the management API is ambiguous. It can mean the Erlang VM is gone, or it can mean the node you are querying from cannot reach the peer. Similarly, running=true does not mean the node is part of the cluster: a partitioned node reports itself as running and will happily answer probes while serving a divergent view of the world.

This guide walks through the checks that separate the two cases, in the order you should run them during an incident.

What this means

RabbitMQ nodes communicate over the Erlang distribution protocol (port 25672 by default). When that link breaks, each node keeps running independently. From the majority side, the missing node looks down. From the missing node’s perspective, the rest of the cluster is down. Both are “running” locally.

Three facts drive the diagnosis:

• The running field in GET /api/nodes reflects what the querying node believes, not ground truth on the target.
• Each node maintains a partitions array listing nodes it considers partitioned. This is the authoritative split-brain signal.
• A node can be alive but non-service-impacting to lose, or alive and quorum-critical. Losing a quorum-critical node takes quorum queues and streams with it.

flowchart TD
  A[Node alert fires] --> B{rabbitmq-diagnostics ping on the node itself}
  B -->|fails| C{beam.smp process alive?}
  C -->|no| D[Dead broker - VM crashed or host down]
  C -->|yes| E[rabbit application stopped - VM alive, broker down]
  B -->|succeeds| F{partitions array empty on all peers?}
  F -->|no| G[Partitioned - running but isolated]
  F -->|yes| H[Transient probe failure or management API issue]
  G --> I{check_if_node_is_quorum_critical}
  I -->|critical| J[Service-impacting - quorum queues at risk]
  I -->|not critical| K[Redundant node loss]

Common causes

Quick checks

Run these from the affected node itself first, then from a healthy peer. All are read-only.

# 1. Is the Erlang node alive at all? Run on the affected node.
rabbitmq-diagnostics -q ping

# 2. Is the OS process there? Distinguishes VM crash from app stop.
pgrep -f beam.smp

# 3. Is the rabbit application running inside the VM?
rabbitmq-diagnostics check_running

# 4. What do the peers think? Run from a healthy node.
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, running, partitions}'

# 5. Cluster view from CLI on a healthy node.
rabbitmq-diagnostics cluster_status

# 6. How long was the node up before it vanished?
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, uptime_seconds: (.uptime / 1000)}'

# 7. Is losing this node service-impacting?
rabbitmq-queues check_if_node_is_quorum_critical
# or via API from a peer:
curl -s -u guest:guest http://localhost:15672/api/health/checks/node-is-quorum-critical

# 8. Is the AMQP data plane still alive even if management is dead?
# (TCP connect to 5672 from a client host; management on 15672 can be down independently)

A note on the tools: rabbitmq-diagnostics ping only proves the node runtime is up and CLI authentication works. It says nothing about whether the rabbit application is running, which is what check_running verifies. And the management API on port 15672 can be down while AMQP on 5672 keeps serving clients, so never treat a management API failure alone as proof of a dead broker. Conversely, /api/health/checks/is-in-service is a readiness/drain signal, not a liveness check: it intentionally returns 503 on a node paused for maintenance. Do not wire it to a “node dead” alert.

How to diagnose it

1. Rule out the alert artifact. If uptime was under 300 seconds before the node vanished, you are probably looking at a rolling restart or a crash-restart loop, not a steady-state failure. Check deployment and maintenance windows before touching anything.

2. Probe the node directly. Run rabbitmq-diagnostics -q ping on the affected host. If it fails and pgrep -f beam.smp returns nothing, the VM is dead: check dmesg for the OOM killer and the RabbitMQ log for crash reports. If beam.smp is alive but ping or check_running fails, the VM is up but the rabbit application is stopped or hung. A single failed probe proves nothing; the paging condition is running == false or unreachable for more than 2 consecutive collection intervals, precisely because brief management API timeouts happen during heavy GC.

3. Ask the cluster, not the node. From a healthy peer, pull GET /api/nodes and inspect partitions on every node. A non-empty array anywhere means split-brain: the “down” node is actually a live, isolated partition side. Cross-check with rabbitmq-diagnostics cluster_status, which shows the querying node’s view of running peers.

4. Confirm with cluster link traffic. On a healthy peer, inspect the cluster_links field in GET /api/nodes. Traffic (recv_bytes, send_bytes) dropping to zero between previously communicating peers, combined with a non-empty partitions array, confirms network isolation. Zero traffic alone is not enough: idle clusters legitimately show near-zero inter-node bytes between intervals.

5. Classify the blast radius. Run rabbitmq-queues check_if_node_is_quorum_critical (or the equivalent API health check). A non-zero exit means taking this node down leaves at least one quorum queue or stream without an online majority. That is the difference between a page-the-team service outage and a redundant-node loss you can fix during business hours. Also list queue states: quorum queues on the minority side will show state: minority (read-only at best), classic queues mastered on the lost node show down.

6. If it is a partition, check whether it is real or self-inflicted. GC pauses exceeding net_ticktime (default 60s) cause transient false partition detection that self-heals. Correlate with the Erlang run queue: a sustained run queue above the core count delays heartbeat processing and manufactures phantom partitions. A partition that appears and clears within one collection interval is usually this, not a network fault.

Metrics and signals to monitor

Fixes

If the broker is genuinely dead

Find out why before restarting. A VM that crashed from memory exhaustion will crash again on restart as queues repopulate, and restarts lose in-memory messages. Check the RabbitMQ log and dmesg for OOM kills, disk exhaustion on the data partition, or Erlang process/atom limit errors. Once the cause is understood and the host is healthy, start the node and let it rejoin. Expect a warm-up window: memory will spike as queues reload, and quorum followers need time to resync. Suppress resource alerts for that period rather than paging on warm-up noise.

If the node is partitioned

Do not restart it while the partition is active. Each side may be accepting writes independently, and premature intervention deepens divergence. First fix the connectivity: verify the path between nodes, including the Erlang distribution port 25672 (firewall rules, NAT timeouts, switch state). Then check the partition handling strategy in rabbitmq.conf (cluster_partition_handling) so you know what recovery will do: pause_minority freezes the minority side until it can see a majority; autoheal restarts the losing side when the network heals (non-replicated messages on those nodes are lost); ignore does nothing and leaves the cluster split until you act manually. After recovery, verify the partitions array is empty on every node and that quorum queues have left the minority state.

If it was a false partition from GC or scheduler saturation

Treat the run queue as the incident, not the partition. Sustained scheduler saturation delays heartbeats and triggers net_ticktime breaches. Reduce load, revisit TLS termination costs, or add capacity. Tuning net_ticktime upward to mask this trades slower real-partition detection for fewer false ones; fix the saturation first.

Prevention

• Alert with gating conditions. Page only when running == false or unreachable is sustained over more than 2 collection intervals AND prior uptime exceeded 300 seconds. This one change eliminates most cold-start and rolling-restart pages.
• Alert on partitions separately and require persistence. GC pauses past net_ticktime self-heal; only a partition persisting across intervals deserves a page.
• Probe both planes. Monitor management API reachability and actively probe the AMQP listener (TCP connect to 5672). They fail independently.
• Check quorum-criticality before any planned stop. Run check_if_node_is_quorum_critical before every rolling restart or decommission. It is the difference between a boring maintenance window and a self-inflicted outage.
• Monitor the run queue as a partition precursor. Scheduler saturation is upstream of heartbeat timeouts; catch it before it manufactures a false split.
• Keep classic mirrored queues out of the picture. They are removed in current versions; on quorum queues a partition degrades to minority rather than divergent writes.

How Netdata helps

• Node availability with uptime context: Netdata tracks the running state and uptime per node, so a down alert carries the warm-up context needed to distinguish a restart from a failure.
• Partition detection per node: the partitions array from /api/nodes is surfaced per node, letting you see at a glance which side of a split each node believes it is on.
• Cluster link traffic rates: per-peer send/receive byte rates make the “zero traffic plus partition detected” correlation visible on one screen instead of two curl commands.
• Queue state tracking: down and minority queue states are charted alongside node events, so you can see whether a node loss took quorum queues with it.
• Run queue correlation: the Erlang run queue is collected next to partition events, which is how you spot GC-driven false partitions without pulling logs.

Related guides

• RabbitMQ channel leak and churn: the channel-per-message anti-pattern
• RabbitMQ connection blocked / blocking: publishers frozen by a resource alarm
• RabbitMQ connection in flow state: credit-based backpressure explained
• RabbitMQ connection storm: reconnect loops, FD pressure, and CPU spent on handshakes
• RabbitMQ consumer timeout: delivery acknowledgement timed out and the channel is closed
• RabbitMQ consumer_utilisation low: consumers attached but not keeping up
• RabbitMQ consumers connected but not acknowledging: the consumer black hole
• RabbitMQ disk free limit alarm: free disk space insufficient and publishing halted
• RabbitMQ disk_free_limit at the default 50MB: the production footgun
• RabbitMQ file descriptor exhaustion: fd_used near fd_total and refused connections
• RabbitMQ flow control vs resource alarms: the two throttling mechanisms operators confuse
• RabbitMQ head message age: the queue latency that depth alone cannot show