RabbitMQ queue backlog growing: messages_ready climbing and how to drain it

A queue’s messages_ready counter has been climbing for 20 minutes and it is not coming back down. Publishers are still publishing; upstream is fine. You need two answers fast: why consumers are not keeping up, and whether the backlog triggers a memory or disk alarm before you fix it.

Rising messages_ready means exactly one thing: messages arrive faster than they are delivered. The cause is one of three: consumers are absent, consumers are present but too slow, or consumers are present but stuck. The diagnostic work is separating those cases, because the fix differs for each and the wrong fix (restart the broker, purge the queue) can turn a slow drain into data loss.

What this means

messages_ready is the count of messages sitting in the queue waiting for delivery. It grows whenever publish rate exceeds delivery rate, which is normal in bursts. It becomes an incident when growth is sustained, because a backlog has three second-order effects:

  1. Memory pressure. For classic queues, messages in RAM count against the node. Past the paging threshold (default: 50% of the memory watermark), RabbitMQ pages queue contents to disk, which slows queue processes further and can start a feedback loop. Unacknowledged messages are worse: they are pinned in memory and cannot be paged, because the broker must be able to redeliver them instantly.
  2. Cliff-edge alarms. If the backlog pushes any node’s memory past vm_memory_high_watermark or disk below disk_free_limit, publishers are blocked cluster-wide. There is no gradual degradation before that crossing.
  3. Latency. Depth alone does not tell you latency; the head message age does. A queue with 10,000 messages and a 5-second-old head is fine. Ten messages with a 3-hour-old head is a stalled pipeline.

Lazy queues and quorum queues decouple depth from memory: messages live mostly on disk, so a deep backlog does not directly threaten the memory alarm. It still threatens latency, and on quorum queues it consumes disk via the Raft log.

flowchart TD
    A[messages_ready climbing] --> B{consumers on queue?}
    B -->|0 consumers| C[Absent: consumer app down or never deployed]
    B -->|> 0 consumers| D{ack rate > 0?}
    D -->|ack near 0| E[Stuck: deadlock, failed downstream, zombie connection]
    D -->|ack > 0| F{publish > deliver_get?}
    F -->|yes| G[Slow: consumers cannot keep up with ingest]
    F -->|redeliver high, depth stable| H[Poison message looping at queue head]

Common causes

CauseWhat it looks likeFirst thing to check
Consumer application down or undeployedconsumers = 0 on the queue, depth growing unboundedrabbitmqctl list_queues name consumers messages_ready
Consumers too slow for ingestconsumers > 0, ack rate > 0, publish > deliver_get sustainedCompare publish_details.rate vs deliver_get_details.rate on the queue
Consumer stuck on a downstream dependencydeliver rate near 0 or ack rate near 0 while consumers > 0Consumer application logs; downstream DB/API health
Zombie connectionsconsumers > 0, ack rate = 0, connections open but client process deadConsumer-side process health, connection connected_at age
Poison message loopdepth stable, head age growing, redeliver rate elevatedredeliver rate on the queue; consumer error logs
Prefetch misconfigurationunacked pinned at exactly prefetch x consumers, or one consumer hoardingmessages_unacknowledged vs consumer count
Deployment windowbrief consumer count drop, depth grows then recoversDeployment timeline; does depth recover within the window?

Quick checks

All read-only and safe to run during an incident. The examples use the guest user against localhost, which is the default; substitute real credentials for a remote broker.

# The single most useful triage view: depth, unacked, consumers per queue
rabbitmqctl list_queues name messages_ready messages_unacknowledged consumers state

# Global totals, for blast radius
curl -s -u guest:guest http://localhost:15672/api/overview | jq '.queue_totals'

# Per-queue rates: is publish outpacing delivery? Is ack happening at all?
# Rates are null if the queue has had no message activity since stats reset.
curl -s -u guest:guest 'http://localhost:15672/api/queues/%2f/my_queue' | \
  jq '{messages_ready, messages_unacknowledged, consumers, consumer_utilisation,
       publish_rate: .message_stats.publish_details.rate,
       deliver_get_rate: .message_stats.deliver_get_details.rate,
       ack_rate: .message_stats.ack_details.rate,
       redeliver_rate: .message_stats.redeliver_details.rate}'

# Head message age: how old is the oldest waiting message?
# Note: only present if publishers set the AMQP timestamp property.
curl -s -u guest:guest 'http://localhost:15672/api/queues/%2f/my_queue' | \
  jq '{head_message_timestamp, messages_ready}'

# Are alarms already firing or close?
curl -s -u guest:guest http://localhost:15672/api/nodes | \
  jq '.[] | {name, mem_alarm, disk_free_alarm, mem_used, mem_limit, disk_free, disk_free_limit}'

# Which queues are heaviest on memory (sort via API)
curl -s -u guest:guest 'http://localhost:15672/api/queues?sort=memory&sort_reverse=true' | \
  jq '.[] | {name, memory, messages_ready, messages_unacknowledged}' | head -40

How to diagnose it

Work the decision tree from the diagram, in this order.

  1. Check consumer count first. If consumers = 0, nothing is subscribed. Confirm the consumer application is deployed and connected. With zero consumers, messages_ready grows unbounded until a resource alarm fires and blocks publishers cluster-wide. This is the most common root cause of backlog incidents. Skip to fixes.
  2. Check the ack rate. Consumers > 0 with ack rate at or near zero is the “consumer black hole”: connected but not processing. If deliver rate is also zero, consumers are not even pulling messages (fully stalled or prefetch exhausted). If deliver is non-zero but ack is zero, they receive messages and never finish. Check messages_unacknowledged: if it sits at exactly prefetch x consumer count, consumers are holding a full prefetch buffer and not completing work.
  3. Check for zombie connections. A client process can die while its TCP connection stays open, and the broker still shows the consumer as subscribed. Correlate broker-side consumer presence with the consumer application’s process health and logs. If the app is dead but the connection persists, close the connection via the management API to force requeue and let a healthy instance pick up the work.
  4. Check the poison message case. Depth roughly stable, consumers > 0, redeliver rate elevated, head message age climbing: one message at the head is being delivered, failing, requeued, and redelivered in a loop, and everything behind it is stuck. Consumer logs show the same failure repeating.
  5. Check for genuine slowness. Consumers healthy, ack rate > 0, but publish rate exceeds deliver_get rate sustained. Consumers are doing their best and losing. Check consumer_utilisation: values well below 1.0 on a queue with a backlog mean consumers cannot accept messages as fast as the queue can deliver (slow processing, low prefetch, or network latency between broker and consumer).
  6. Estimate your runway before choosing a fix. Backlog growth consumes memory (classic queues) and possibly disk (persistent messages, quorum queue WAL). Estimate time to alarm: (mem_limit - mem_used) / memory_growth_rate, where the growth rate tracks (publish_rate - consume_rate) x average_message_size. On lazy or quorum queues, depth is mostly on disk and memory runway is longer, but check disk_free against disk_free_limit instead.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
messages_ready rate of changeThe backlog itself. Alert on sustained positive trend, not absolute depth (depth is workload-dependent)Positive growth for 10+ minutes
messages_unacknowledgedPinned in memory, cannot be paged. The top precursor to memory alarmsGrowing, or pinned at prefetch x consumers with no acks
Consumer count per queueZero consumers on a queue with messages = unbounded growthDrop to 0, or below expected minimum
Ack rate vs deliver rateDivergence means consumers receive but never finishAck < 50% of deliver for 5+ minutes
Publish rate vs deliver_get rateThe delta is the backlog growth rateSustained publish » deliver_get
consumer_utilisationWhether attached consumers can actually keep up< 0.5 with messages_ready > 0
Head message age (head_message_timestamp)Depth without age is incomplete. Age is the latency signalAge exceeding the queue’s SLA
redeliver rate per queuePoison message loops and consumer failure cyclesElevated on one queue while depth stays flat
mem_used / mem_limit and messages_paged_outRunway to the memory alarm; paging is the leading indicatorRatio > 0.7, or paged_out appearing under traffic

Fixes

Grouped by cause. Restarting the broker is not on this list; it risks losing in-memory messages and makes recovery slower, not faster.

Consumers absent (count = 0)

Redeploy or restart the consumer application. This is the whole fix. Once consumers attach, the queue drains at whatever rate they can sustain. If the backlog is huge and the messages are expendable (metrics, events with TTL elsewhere), purging the queue is an acceptable shortcut. Purge is destructive and irreversible: confirm the data is disposable first.

Consumers stuck (ack rate = 0)

  • Fix the downstream dependency the consumer is blocked on (database, API). The backlog drains on its own once processing resumes.
  • If consumers are zombie connections, close them from the management API. Their unacked messages requeue immediately, in one burst, back to the front of the queue. Plan for that burst if the unacked count is large.
  • If the consumer code is in an exception loop (fails, nacks with requeue, fails again), deploy a fix that nacks with requeue=false and routes to a dead-letter exchange.

Poison message

Move the offending message out of the head: dead-letter it manually, or consume it with a throwaway consumer that acks and discards. Then add a delivery limit (x-delivery-limit on quorum queues) and a DLX so the next poison message ejects itself after N attempts instead of looping forever. Inspect the message payload before discarding if you need it for the bug report.

Consumers too slow

  • Scale consumers out. More consumer instances on the same queue parallelizes delivery. Check consumer_utilisation first: if it is already near 1.0 and the backlog still grows, per-message processing time is the bottleneck, not consumer count, and scaling helps only if processing parallelizes cleanly.
  • Tune prefetch deliberately. Too low and consumers idle waiting for the next batch after each ack; too high and messages move from messages_ready into messages_unacknowledged, where they sit pinned in consumer buffers (and broker memory) and hidden from the ready count. Watch both metrics while tuning so the backlog does not just move.
  • Reduce per-message work. If consumers call a slow downstream per message, batch or cache. The broker cannot fix a 200 ms database call per message.

Buying time when an alarm is close

If mem_used / mem_limit is climbing toward 1.0 and you cannot restore consumption fast enough, stop or throttle the publishers if you control them. Unlike a broker-side memory alarm, a publisher-side pause is surgical and reversible. Purging a large non-critical queue is emergency relief that frees memory immediately, at the cost of the data. See RabbitMQ memory resource limit alarm if the alarm has already fired.

Prevention

  • Alert on rate of change, not depth. Absolute depth thresholds are workload-dependent and either noisy or blind. Alert on sustained positive growth of messages_ready and on consumer count dropping to zero on queues that should always have consumers.
  • Pair depth with age. Track head_message_timestamp alongside depth. Require publishers to set the AMQP timestamp property, or the field will be absent or zero.
  • Watch unacked separately. messages_unacknowledged growth is the memory alarm’s advance warning. It also catches stuck consumers before the backlog itself does.
  • Set sane prefetch defaults in client libraries, and dead-letter exchanges with delivery limits on queues where poison messages are plausible.
  • Know your queue types. Lazy and quorum queues keep backlogs on disk; classic queues keep them in RAM. Your runway math and your alarm risk differ accordingly.

How Netdata helps

  • Netdata charts messages_ready and messages_unacknowledged per queue separately, so you can see a backlog shifting into consumer buffers instead of draining.
  • Publish, deliver, and ack rates per queue are collected together, making the “publish > deliver_get” imbalance and the “deliver but no ack” stall visible without querying the API by hand.
  • Consumer count per queue is charted next to depth, so the “zero consumers, unbounded growth” case is obvious at a glance.
  • Node memory usage and memory/disk alarm states are correlated on the same dashboard as queue depth, so you can see how much runway the backlog has left.
  • Anomaly detection on queue depth rate of change catches slow-building backlogs that absolute thresholds miss.