RabbitMQ quorum queue uncommitted Raft entries growing: followers falling behind

You are watching rabbitmq_raft_log_uncommitted_entries climb on a quorum queue and it is not coming back down. In a healthy quorum queue this metric hovers near zero with brief spikes during publish bursts. Sustained growth means the leader is appending entries to its Raft log faster than a quorum of followers can replicate and acknowledge them.

This is a durability problem before it is an availability problem. Every uncommitted entry is a message that exists only on the leader. If the leader fails now, those messages are lost even though publishers may already have received confirms for some of them, depending on client behavior. A sustained count above roughly 1000 entries is a replication bottleneck that needs diagnosis, not observation.

The causes are almost always one of three things: a slow follower disk, inter-node network latency, or saturation of the Erlang distribution port. All three are measurable.

What this means

Quorum queues replicate every write via Raft. The leader appends an entry to its write-ahead log, ships it to followers, and commits it once a majority acknowledges. The uncommitted entries gauge is the gap between what the leader has written and what a quorum has confirmed.

A small, oscillating value is normal: publish bursts fill the gap, followers catch up, the gauge returns to zero. The failure mode is a persistent gap, where the leader’s append rate exceeds the replication path’s throughput. The backpressure also propagates upstream: when Raft replication falls behind, credit-based flow control throttles the publishing connections, so you will often see connections in flow state alongside the growing gauge.

There is a second, related failure: quorum queue WAL growth. A follower that cannot keep up also cannot snapshot and truncate its log, so disk usage grows on followers as well. Left alone, the replication lag incident becomes a disk space incident, and the disk alarm, when it fires, blocks publishers cluster-wide.

flowchart LR
  P[Publishers] -->|publish| L[Quorum queue leader]
  L -->|replicate| F1[Follower A]
  L -->|replicate| F2[Follower B]
  F1 -->|ack| L
  F2 -->|slow ack| L
  L --> UC[Uncommitted entries grow]
  F2 -.root cause.-> D[Slow disk / network lag / dist port saturation]
  UC -->|leader fails| LOSS[Uncommitted messages lost]
  UC -->|no snapshot| WAL[WAL grows on followers]
  WAL --> DISK[Disk alarm risk]

Common causes

Quick checks

# 1. Confirm which queues are affected and who is online
rabbitmqctl list_queues name type leader online members state messages

# 2. Check whether any node is quorum-critical before you touch anything
rabbitmq-diagnostics check_if_node_is_quorum_critical

# 3. Look at the Erlang distribution port send queue on the leader node
#    A persistently non-empty Send-Q means the inter-node pipe is saturated
#    (needs root or sudo for -p)
sudo ss -tnp | grep 25672

# 4. Check cluster link traffic per peer (cumulative counters; sample twice and diff)
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, cluster_links}'

# 5. Check for partitions: a partitioned follower cannot replicate at all
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, partitions}'

# 6. Check disk space and the disk alarm on every member node
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, disk_free, disk_free_limit, disk_free_alarm}'

# 7. Look for WAL accumulation on followers (path differs if the data dir is customized)
du -sh /var/lib/rabbitmq/mnesia/*/quorum/ 2>/dev/null

# 8. Check whether publishers are already being throttled
curl -s -u guest:guest http://localhost:15672/api/connections | jq 'group_by(.state) | map({state: .[0].state, count: length})'

The guest:guest credentials only work from localhost; substitute real monitoring credentials for remote checks. All of these are read-only. Do not restart any node as a first response: a restart triggers leader elections for queues mastered on that node and forces follower resync, which makes replication lag worse before it gets better.

How to diagnose it

1. Scope the problem. Is the lag on one queue, all queues on one leader node, or all quorum queues cluster-wide? One queue points to a hot queue or a specific slow follower in that queue’s membership. Everything at once points to shared infrastructure: the network or the distribution port.

2. Identify the laggard. Each quorum queue has a leader and members. Compare state across nodes for the same queue. If one member’s online list is short, or the queue shows reduced membership, that follower is your suspect. Note that per-queue Raft metrics are reported by the node where the queue leader runs; followers do not always expose per-queue Raft detail.

3. Rule out the obvious failure modes first. Check the partitions array on every node: a partitioned follower cannot replicate. Check node uptime: a follower that restarted recently is resyncing, and transient lag during resync is expected behavior, not an incident.

4. Measure the follower disk. On the suspected follower, watch write latency on the RabbitMQ data partition. Quorum queue WAL writes are fsync-heavy; high-latency storage (network-attached disks, throttled cloud volumes) is the single most common root cause of followers falling behind. If the WAL directory for affected queues is growing on the follower while the leader’s stays bounded, the disk path is the bottleneck and snapshotting is not keeping up either.

5. Measure the inter-node path. Check RTT and retransmits between nodes, then check the Erlang distribution port send queue with ss -tnp | grep 25672 on the leader. The distribution port is a single TCP connection per node pair that carries all inter-node traffic, including Raft replication for every quorum queue. A persistently growing send queue means the pipe is saturated and replication traffic is queueing behind Mnesia, stats, and other distribution traffic.

6. Check the demand side. Compare the publish rate on affected queues against what the replication path can sustain. If publish rate legitimately exceeds follower disk or network throughput, no amount of tuning fixes it; you need faster disks, more capacity, or load spread across more queues and leaders.

7. Check knock-on effects. Look for connections in flow state (backpressure reaching publishers), disk_free trending down on followers (WAL accumulation), and whether the queue has lost redundancy (a 3-member queue with 1 member down is still committing but has no remaining fault tolerance).

Metrics and signals to monitor

Version note: in RabbitMQ 4.2 the Raft metric names changed. The older rabbitmq_raft_log_* names were replaced with names such as rabbitmq_raft_commit_index, rabbitmq_raft_last_applied, and rabbitmq_raft_snapshot_index. On 4.2 and later, follower lag is typically computed by comparing rabbitmq_raft_last_applied across nodes for the same queue rather than reading a single uncommitted gauge. Check which naming your version exposes before writing alerts.

Fixes

Slow follower disk

Move the RabbitMQ data directory to local SSD if it is on network-attached or heavily contended storage. Quorum queue WAL writes are fsync-heavy and amplify any storage latency directly into replication lag. If the follower is sharing its disk with other workloads, isolate it. If WAL segments are accumulating because snapshotting cannot keep up, reducing the segment size via raft.segment_max_entries from its default of 32768 to something smaller (for example 1024) makes truncation more granular; this is a tuning tradeoff, not a free win, so change it on one node and observe.

Network latency or distribution port saturation

If the send queue on port 25672 is persistently non-zero, reduce what else travels over the distribution link. Concentrating queue leaders on fewer node pairs, reducing management stats collection frequency, or splitting very large clusters reduces distribution traffic. Verify MTU, retransmits, and any cloud provider bandwidth caps between the nodes. If nodes span availability zones or regions, confirm the latency budget is realistic for synchronous Raft replication; quorum queues are sensitive to inter-node RTT by design.

Follower down or decommissioned

If a member is permanently gone, remove it from the queue’s membership so the queue stops trying to replicate to it and can compact its log:

# Remove a dead member, then optionally add a healthy replacement
rabbitmq-queues delete_member <queue_name> <node>
rabbitmq-queues add_member <queue_name> <node>

Until the replacement finishes syncing, the queue runs with reduced redundancy. Adding a member triggers a full sync from the leader snapshot, so expect a burst of inter-node traffic and follower disk I/O while it catches up.

Leader imbalance concentrating replication load

If one node leads most quorum queues, its network and disk carry most of the replication load. Rebalance leaders so the write work spreads across the cluster:

# Redistribute quorum queue leaders across nodes (3.10+)
rabbitmq-queues rebalance all

Rebalancing causes a brief unavailability per queue during leader transfer. Run it off-peak.

Publish rate exceeding replication capacity

If demand genuinely exceeds what the replication path can sustain, spread the load: shard across more quorum queues so more Raft groups replicate in parallel, distribute publishers across nodes, or reduce per-message size. This is a capacity problem; flow control will keep throttling publishers until you change the shape of the load.

Prevention

• Alert on the lag, not just the alarm. Uncommitted entries sustained above a few hundred, or any monotonic growth over 10 minutes, should page before the disk alarm does.
• Track redundancy separately. A 3-member quorum queue running with 2 members is committing fine but has zero remaining fault tolerance. Alert on online member count below expected.
• Monitor the distribution port. Send queue depth on port 25672 is the earliest signal of inter-node saturation; it is not exposed via the management API, so collect it at the OS level.
• Provision disks for WAL headroom. Followers under lag grow WAL. Size data partitions with room for the WAL of a follower catching up from a snapshot, and set disk_free_limit well above the 50MB default so a growing WAL does not trigger a cluster-wide publisher halt prematurely.
• Run check_if_node_is_quorum_critical before any restart. Rolling maintenance is the most common way to turn tolerable lag into quorum loss.
• Test failover. Know, from a drill rather than an incident, how long a new follower takes to sync and what the lag looks like while it does.

How Netdata helps

• Netdata’s RabbitMQ collector tracks publish, deliver, and ack rates per node and per queue, so you can see publish demand outrunning replication throughput on the specific queues whose uncommitted entries are growing.
• Connection state monitoring surfaces the downstream effect: connections flipping into flow as Raft backpressure propagates to publishers.
• Per-node disk metrics on the RabbitMQ data partition show the secondary failure mode in the same view: follower WAL growth heading toward disk_free_limit while the lag persists.
• Network interface and TCP metrics between cluster nodes let you correlate replication lag with inter-node saturation instead of guessing whether the network is the bottleneck.
• Node-level signals such as file descriptors, memory ratio, and Erlang run queue on follower nodes help rule out a generally overloaded follower masquerading as a replication problem.
• Because all of these series share a timeline, you can line up the moment uncommitted entries started growing with the follower restart, network event, or publish spike that caused it.

Related guides

• RabbitMQ connection blocked / blocking: publishers frozen by a resource alarm
• RabbitMQ connection in flow state: credit-based backpressure explained
• RabbitMQ flow control vs resource alarms: the two throttling mechanisms operators confuse
• RabbitMQ disk free limit alarm: free disk space insufficient and publishing halted
• RabbitMQ disk_free_limit at the default 50MB: the production footgun
• RabbitMQ consumer_utilisation low: consumers attached but not keeping up
• RabbitMQ head message age: the queue latency that depth alone cannot show