RabbitMQ quorum queue in minority: lost majority and read-only queues

You are looking at a queue with state: minority in the management API, or publishers are timing out on a queue that was healthy an hour ago. A quorum queue enters minority when fewer than a majority of its Raft members are online. Without a majority, the queue cannot elect a leader and cannot commit new writes.

This is not a queue-level bug. It is the direct, by-design consequence of a network partition or the loss of multiple cluster nodes. The queue is protecting your data by refusing writes it cannot replicate safely.

For a queue with active traffic (messages > 0 or consumers > 0), treat this as urgent. Publishers to that queue are stuck, and depending on client behavior they will block, time out, or fail over. This guide walks through confirming the scope, deciding between member recovery and shrinking the member set, and preventing recurrence.

What this means

A quorum queue is replicated across N members using Raft consensus (typically 3 or 5 members). Every write must be committed by a majority: 2 of 3 members, 3 of 5. When the online member count drops below that majority, the queue cannot elect a leader and cannot commit anything new.

In the management API (GET /api/queues), the queue’s state field flips from running to minority. A queue in minority cannot accept writes but may still serve reads of previously committed messages. Treat it as effectively unavailable for new work: publishers will stall, and any client expecting confirms will hang or fail.

The two triggers, in order of likelihood:

1. Network partition. The cluster split and this queue’s majority ended up on the other side. Other quorum queues with different member placement may still be running, which makes the failure look arbitrary.
2. Multi-node loss. Two or more nodes hosting this queue’s members are down (crash, hardware failure, botched rolling restart, eviction).

stateDiagram-v2
  running --> minority: majority of members offline (partition or node loss)
  minority --> running: members recover, quorum restored
  minority --> running: dead members removed, member set shrunk
  minority --> unavailable: members permanently lost, quorum unrecoverable
  unavailable --> running: force delete and recreate queue (data loss)

Common causes

Quick checks

All of these are read-only and safe to run during an incident. Substitute your actual credentials for guest:guest in the API examples.

# Find every queue currently in minority state
curl -s -u guest:guest http://localhost:15672/api/queues | \
  jq '.[] | select(.state=="minority") | {name, vhost, node, state, type, messages: .messages_ready, consumers}'

# Check cluster-wide partition status
rabbitmq-diagnostics cluster_status
# or via API
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, running, partitions}'

# Per-queue Raft detail: leader, followers, online members
rabbitmq-queues quorum_status <queue_name>

# Member list and which are online
rabbitmqctl list_queues name type online members

# Is each expected node actually alive?
rabbitmq-diagnostics -q ping

# Before any further node shutdown: would it break quorum anywhere else?
rabbitmq-diagnostics check_if_node_is_quorum_critical

Two things to note in the output. First, the activity fields: a queue in minority with messages_ready > 0 or consumers > 0 is actively impacting traffic and should be treated as a page. An idle queue in minority is still a correctness problem but rarely urgent. Second, which specific members are offline. That answer drives the entire recovery decision.

How to diagnose it

1. Confirm the scope. Run the minority-state query above. One queue in minority points to unlucky member placement. Many queues in minority points to a cluster-level event (partition or multiple node loss). Also check GET /api/vhosts for partially available vhosts if several queues are affected.

2. Check for a partition first. A non-empty partitions array, corroborated by cluster link traffic dropping to zero between previously communicating peers, confirms a split brain. Verify with independent network checks between nodes (Erlang distribution runs on port 25672 by default). Do not confuse this with a false partition: GC pauses or a saturated Erlang run queue exceeding net_ticktime (default 60s) can trigger transient partition detection that self-heals. Require the condition to persist before acting.

3. Check node availability. For each node that hosts a member of the affected queue, confirm running status. A node can be running: true but partitioned, so node health and partition health are separate checks.

4. Classify the offline members: transient or permanent. A node that crashed and will restart cleanly is transient. A node with a dead disk or a decommissioned VM is permanent. This distinction decides the fix: recovery versus shrinking the member set.

5. Assess downstream impact. Publish rates to the affected queue will be zero or near zero. Look for publisher-side symptoms: connections piling up, timeouts, or retry storms. Consumers may still drain previously committed messages, but plan for the queue to be effectively out of service for new work.

6. Freeze further node changes. Until quorum is restored, do not stop, restart, or decommission any node that hosts a member of an affected queue. Run rabbitmq-diagnostics check_if_node_is_quorum_critical before touching anything. One more member offline can turn a recoverable minority into permanent loss.

Metrics and signals to monitor

Fixes

Recover the failed members (preferred)

If the offline members are transiently down (crashed node, fixed network partition, restarted VM), bring them back and do nothing to the queue itself. Raft restores quorum automatically once a majority is online again.

After recovery, followers must resynchronize with the leader. On deep queues this resync can take minutes to hours, during which the queue serves traffic while replication catches up. Expect elevated inter-node traffic and disk I/O during this window; it is normal recovery behavior, not a second incident.

If the cause was a partition, resolve the network issue first and let the cluster heal before restarting anything. Premature restarts during an unstable network extend the outage.

Shrink the member set (members permanently lost)

If one or more members are never coming back, the remaining recovery path is to reduce the member set so the survivors form a new majority. For example, a 5-member queue with 3 members permanently lost can be reduced so the 2 survivors constitute a majority.

This is a deliberate, queue-by-queue operation. Do not attempt it while a partition is still unresolved; you can end up with two sides that each believe they are the queue.

Force delete and recreate (last resort)

If a majority of members is permanently and unrecoverably lost (for example, 2 of 3 nodes destroyed with no backups), the queue cannot be recovered. RabbitMQ’s own documentation states the queue is permanently unavailable and must be force deleted and recreated.

This loses all messages in the queue. Treat it as a data-loss event: confirm with the owning team that the messages are expendable or recoverable upstream, then delete the queue, redeclare it, and let clients reconnect. Normal deletion may fail on a queue with no online replicas; check your version’s documentation for the force-delete procedure.

What not to do

• Do not restart healthy nodes hosting surviving members. You are one member away from permanent loss.
• Do not redeploy the queue while a partition is unresolved.
• Do not wait passively if the queue is active. Minority on an idle queue can wait for business hours; minority on a hot queue is an active outage.

Prevention

• Use odd member counts and spread them. Three members tolerates 1 failure; five tolerates 2. Place members across failure domains (racks, availability zones) so a single event cannot take a majority.
• Gate all node maintenance on quorum safety. Run rabbitmq-diagnostics check_if_node_is_quorum_critical (or GET /api/health/checks/node-is-quorum-critical) before every stop, restart, or decommission. During rolling restarts, wait for each node to return and for followers to resync before proceeding.
• Alert on state: minority with an activity condition. Page when the queue has messages or consumers; ticket otherwise. A bare minority alert on every idle test queue trains people to ignore the real one.
• Alert on partitions with a sustained condition. Transient false partitions from GC pauses self-heal; real ones do not. Corroborate with cluster link traffic before paging.
• Rebalance leaders after restarts. Rolling restarts leave leaders concentrated on surviving nodes. rabbitmq-queues rebalance all redistributes them; avoid running it at peak traffic since leader transfer causes brief per-queue unavailability.
• Keep the run queue healthy. Sustained scheduler saturation causes heartbeat timeouts, which cause false partitions, which cause minority states that were entirely avoidable.

How Netdata helps

• Per-queue state tracking surfaces the transition from running to minority the moment it happens, with the queue’s message and consumer counts attached so you can judge urgency without a manual query.
• Partition and node availability signals together let you separate a true split brain (partitions array plus zero cluster link traffic) from a dead node (running: false) and from a false positive (elevated run queue, self-healing).
• Publish rate collapse alongside stable connection count confirms the queue is refusing writes while clients stay attached, which distinguishes minority from a publisher-side failure.
• Cluster link traffic rates corroborate isolation between specific peers, telling you which side of the partition holds the majority.
• Erlang run queue monitoring catches the CPU/GC saturation that causes heartbeat timeouts and false partition detection before it cascades into a real minority event.

Correlating queue state, partition status, and node availability on one timeline turns “some queue is broken” into “nodes 2 and 3 partitioned at 03:14, this queue’s majority was on the other side” in one look.

Related guides

• RabbitMQ connection blocked / blocking: publishers frozen by a resource alarm
• RabbitMQ flow control vs resource alarms: the two throttling mechanisms operators confuse
• RabbitMQ connection in flow state: credit-based backpressure explained
• RabbitMQ connection storm: reconnect loops, FD pressure, and CPU spent on handshakes
• RabbitMQ consumer timeout: delivery acknowledgement timed out and the channel is closed
• RabbitMQ consumer_utilisation low: consumers attached but not keeping up
• RabbitMQ consumers connected but not acknowledging: the consumer black hole
• RabbitMQ disk free limit alarm: free disk space insufficient and publishing halted
• RabbitMQ disk_free_limit at the default 50MB: the production footgun
• RabbitMQ file descriptor exhaustion: fd_used near fd_total and refused connections
• RabbitMQ channel leak and churn: the channel-per-message anti-pattern
• RabbitMQ head message age: the queue latency that depth alone cannot show