RabbitMQ publish outpacing deliver: reading the rate imbalance before the backlog

When publish rate runs ahead of deliver rate on a RabbitMQ broker, nothing is broken yet. That is exactly why this signal matters. Queue depth is a lagging indicator: by the time messages_ready looks alarming, the imbalance has usually been running for minutes. The rate comparison tells you the same thing earlier, and it tells you which side of the pipe changed.

The comparison has one trap that produces both false alarms and missed incidents: the correct counter to compare against publish is deliver_get, not deliver. Pull-mode consumers using basic.get only show up in get, and deliver_get is the sum of deliver plus get. Compare publish against deliver alone on a workload with pull consumers and you will see a permanent fake imbalance.

This article covers how to read the publish versus deliver_get rates correctly, what the different imbalance shapes mean, and which signals to correlate before you act. For the broader broker mental model and failure pattern catalogue, see How RabbitMQ actually works in production.

What this means

RabbitMQ exposes message rates as cumulative counters: publish, deliver, get, deliver_get, ack, redeliver, and others, available cluster-wide via GET /api/overview, per vhost via GET /api/vhosts, and per queue via GET /api/queues/{vhost}/{name} in each object’s message_stats block. Your monitoring system differentiates the counters into rates; the management API also ships pre-computed *_details.rate fields from a sliding window.

Three rate relationships carry almost all of the diagnostic value:

  • publish vs deliver_get: the queue growth equation. If publish exceeds deliver_get sustained, depth will grow. This is the leading read; queue depth is the confirmation.
  • deliver vs ack: the consumer health equation. If deliver is positive but ack is zero, consumers are receiving messages and not completing them. That is a different failure than consumers being absent.
  • publish dropping to zero: publisher-side failure or publisher-side blocking. A rate collapse is not a capacity problem; it is publishers dead, blocked by a resource alarm, or throttled by flow control.

Batch publishers are spiky by design. A job that pushes 50,000 messages in ten seconds and then goes quiet will show publish far above deliver_get on any short sample. Use 1-minute averages for the comparison, not instantaneous samples, and judge imbalance over 5 to 10 minutes, not seconds.

flowchart LR
  P[publish rate] --> Q[queue process]
  Q --> D[deliver_get rate]
  D --> A[ack rate]
  P -- "publish > deliver_get" --> G[depth grows]
  G --> M[memory rises]
  M --> PG[paging to disk]
  PG --> F[flow control on publishers]
  M --> AL[memory alarm: all publishers blocked]
  D -- "deliver > 0, ack = 0" --> U[unacked builds: RAM pinned]

Common causes

CauseWhat it looks likeFirst thing to check
Consumers absent or crashedpublish > 0, deliver_get near 0, consumers = 0 on the queuePer-queue consumer count
Consumers present but too slowpublish > deliver_get sustained, deliver_get > 0, depth growingconsumer_utilisation and ack rate on the queue
Consumers connected but not ackingdeliver > 0, ack = 0, messages_unacknowledged pinned at prefetchUnacked count vs prefetch times consumer count
Pull-consumer miscountpublish looks above deliver, but deliver_get matches publishCompare against deliver_get, not deliver
Traffic spike beyond capacitypublish 2-3x baseline, deliver_get at its ceiling, both healthyRolling baseline comparison, 1-minute averages
Publishers blocked or failedpublish drops to zero or near zeromem_alarm, disk_free_alarm, connection states
Silent routing losspublish > 0, deliver_get = 0, depth flat and emptyExchange bindings and return_unroutable
Fanout multiplierdeliver_get higher than publishExpected: one publish routed to N queues counts N deliveries

Quick checks

All read-only. Substitute your management user and vhost (the default vhost / is URL-encoded as %2f).

# Cluster-wide message counters and rates
curl -s -u guest:guest http://localhost:15672/api/overview | jq '.message_stats'

# Global queue totals: ready vs unacknowledged
curl -s -u guest:guest http://localhost:15672/api/overview | jq '.queue_totals'

# Per-queue depth and consumer state in one pass
rabbitmqctl list_queues name messages_ready messages_unacknowledged consumers consumer_utilisation

# Per-queue rates: message_stats for a specific queue
curl -s -u guest:guest 'http://localhost:15672/api/queues/%2f/my_queue' | \
  jq '.message_stats | {publish: .publish_details.rate, deliver_get: .deliver_get_details.rate, ack: .ack_details.rate}'

# Consumer effectiveness on the same queue
curl -s -u guest:guest 'http://localhost:15672/api/queues/%2f/my_queue' | \
  jq '{consumer_utilisation, consumers, head_message_timestamp}'

# If publish collapsed: check resource alarms first
curl -s -u guest:guest http://localhost:15672/api/nodes | \
  jq '.[] | {name, mem_alarm, disk_free_alarm}'

# If publish collapsed: count connection states
curl -s -u guest:guest http://localhost:15672/api/connections | \
  jq 'group_by(.state) | map({state: .[0].state, count: length})'

Two notes on collection cost. Iterating /api/connections to count states is expensive on deployments with many connections; sample it, do not poll it tightly. And the management API itself has overhead: polling more frequently than every 5-10 seconds can measurably load the broker.

How to diagnose it

  1. Confirm the imbalance is real. Pull message_stats from /api/overview twice, 60 seconds apart, and compare 1-minute average rates. A single spiky sample from a batch publisher is not an imbalance. Sustained publish above deliver_get for 5+ minutes is.

  2. Localize it. Cluster-wide rates can hide a single hot queue. List per-queue depths and consumers with rabbitmqctl list_queues, then pull message_stats per queue from the HTTP API and sort by publish rate. Per-queue rates can be zero while global rates look healthy because load is concentrated elsewhere.

  3. Check which side moved. If publish jumped above baseline, you have an ingress event: a flood, a retry storm, a new producer. If publish is at baseline and deliver_get fell, you have a consumer-side problem. The fix paths are completely different.

  4. Characterize the consumer side. On the affected queue, read consumers, consumer_utilisation, and the deliver versus ack relationship. Zero consumers means nobody is subscribed. Utilisation below 0.5 with a growing backlog means consumers are attached but not effective. Deliver positive with ack zero means the consumer black hole: connected, receiving, never completing. See consumers connected but not acknowledging.

  5. Check depth as confirmation, not discovery. messages_ready should now be growing at roughly (publish - deliver_get) per second. If publish exceeds deliver_get but depth is flat and empty, suspect routing loss: check exchange bindings and return_unroutable. Messages published without mandatory=true to an exchange with no matching bindings are silently discarded and appear in no counter.

  6. Project the runway. For classic queues, the imbalance rate times average message size approximates memory growth while messages sit in RAM. Compare against mem_used / mem_limit: paging begins at the paging ratio (default 0.5 of the watermark) and the cluster-wide publisher block fires at ratio 1.0. That gives you a time-to-alarm estimate instead of a surprise.

  7. If publish dropped to zero, switch playbooks. Zero publish with connections still open means publishers are blocked, not slow. Check mem_alarm and disk_free_alarm first (memory resource limit alarm, disk free limit alarm), then connection states. Connections in blocked or blocking mean a resource alarm; connections in flow mean per-connection credit throttling, which is surgical and transient rather than cluster-wide. The distinction matters: flow control vs resource alarms.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
publish rate (1-min avg)Ingress ground truthDrop >80% from rolling baseline for 5+ min: publisher failure or block
deliver_get rate (1-min avg)Total egress, push plus pullSustained gap vs publish: depth will grow
publish / deliver_get ratioThe imbalance itself>2x sustained for 10+ min
deliver vs ack rateConsumer completiondeliver > 0 with ack = 0: stuck consumers
messages_ready trendConfirmation of the rate mathPositive growth for 10+ min
messages_unacknowledgedRAM pinned by in-flight messagesPinned at prefetch x consumers, no acks
consumer_utilisationConsumer effectiveness, not presence< 0.5 with messages_ready > 0
redeliver rateConsumer failure loopsElevated: nacks, timeouts, poison messages
head_message_timestampLatency depth cannot showOldest message aging past queue SLA
return_unroutableRouting loss (mandatory=true only)Any sustained rate > 0

Fixes

Consumers absent or crashed. Restore the consumer fleet first. Depth will drain on its own once deliver_get exceeds publish. Do not restart the broker: it does not fix a consumer problem and it forces queue recovery on top of an active backlog.

Consumers too slow. If consumer_utilisation is low, the bottleneck is consumer-side processing or prefetch tuning, not consumer count; adding consumers that all wait on the same slow downstream changes nothing. If utilisation is high and depth still grows, you genuinely need more consumer capacity. See consumer_utilisation low.

Consumers not acking. This is the black hole pattern. Check consumer application health and downstream dependencies, and consider the consumer timeout behavior introduced in 3.12, which closes channels whose deliveries go unacknowledged past the timeout (default 30 minutes). Unacked messages are pinned in RAM and cannot be paged to disk, so this pattern reaches the memory alarm faster than ordinary backlog.

Publish-side flood. If the ingress spike is legitimate (batch backfill, replay), the options are consumer scale-up, or letting the backlog drain if your message-age SLA tolerates it. Quorum and lazy-queue depth decouples backlog from RAM more than classic queues do, which buys time but not infinite time.

Publish rate zero. Do not tune consumers. Find the block: memory alarm, disk alarm, or widespread flow state. Rate imbalance analysis resumes after publishers can publish.

Silent routing loss. Fix the binding, and set mandatory=true on publishers so unroutable messages surface in return_unroutable instead of vanishing. This is the one imbalance shape where depth never confirms the problem, because the messages never arrive.

Prevention

  • Alert on the ratio, not just depth. Sustained publish / deliver_get above 2x for 10+ minutes catches capacity mismatches while depth is still boring. Depth alerts alone fire late.
  • Alert on publish collapse. A drop of more than 80% from rolling baseline for 5+ minutes catches publisher failure and alarm-driven blocks before users report missing work.
  • Baseline per queue. Per-queue rates can be zero while global rates are healthy. The queues that matter need their own publish and deliver_get baselines.
  • Pair depth with age. Depth tells you volume; head message age tells you latency. The rate imbalance predicts both.
  • Watch unacked separately. messages_unacknowledged is the fast path to the memory wall and the clearest stuck-consumer signal.
  • Respect the batch pattern. If your publishers are batch jobs, alert on 1-minute averages and longer sustained windows, or you will page yourself every time a job runs.

How Netdata helps

  • Netdata collects RabbitMQ message_stats counters and differentiates them into publish, deliver_get, and ack rates, so the imbalance is visible as a rate comparison rather than something you compute by hand.
  • Per-queue charts for ready, unacknowledged, and message rates let you localize a cluster-wide imbalance to the one queue actually driving it.
  • Correlating the rate gap against memory usage and mem_used / mem_limit on the same dashboard turns “depth will grow” into a concrete time-to-alarm estimate.
  • Unacked messages and consumer counts alongside ack rate make the black hole pattern (deliveries without acknowledgements) obvious in one view.
  • Alarm flags and connection states on the same timeline explain publish-rate collapses immediately: blocked connections plus an active alarm is a resource problem, not a publisher bug.