RabbitMQ memory resource limit alarm: publishers blocked across the whole cluster

Your RabbitMQ cluster just stopped accepting messages. Publishers are connected but nothing flows. The broker log shows:

memory resource limit alarm set on node rabbit@node1.
*** Publishers will be blocked until this alarm clears ***

One node crossed its memory high watermark, and RabbitMQ responded by blocking every publisher on every node in the cluster. There is no gradual degradation: the transition from “fine” to “all ingestion halted” is a single threshold crossing.

The important operational facts: the broker is alive, consumers still work, and the alarm is a deliberate circuit breaker, not a crash. Do not restart the broker. Restarting throws away the diagnostic state (unacked messages, connection states) that tells you why memory grew, and it can make things worse when queues repopulate from disk into an already tight memory budget.

What this means

RabbitMQ tracks the Erlang VM’s memory usage against a high watermark (vm_memory_high_watermark). The default is 0.4 (40% of detected RAM) on RabbitMQ 3.x and 0.6 (60%) on RabbitMQ 4.x. When mem_used reaches mem_limit on any node, that node raises mem_alarm, and the alarm propagates cluster-wide: all publishing connections on all nodes are blocked.

Connection states tell you exactly what is happening:

  • running: normal
  • blocking: a resource alarm is active, but this connection has not tried to publish yet
  • blocked: a resource alarm is active and this connection tried to publish; it is frozen
  • flow: credit-based flow control, a different mechanism entirely (per-connection, transient, surgical)

Do not confuse flow with blocked. Flow control throttles one publisher outpacing a downstream queue. A resource alarm is cluster-wide: everything stops.

Two more properties matter for triage:

  • Consumer-only connections are not blocked. Deliveries continue, so consumers can drain queues and relieve the pressure. This is your recovery path.
  • There is no hysteresis. The alarm clears at the same threshold where it fired. If memory hovers at the watermark, the alarm can flap, blocking and unblocking publishers in cycles. The alarm also does not cap memory growth; it only stops ingestion. The node can still OOM if consumers do not drain.
flowchart TD
  A[Consumers slow, stuck, or dead] --> B[Ready and unacked messages grow]
  B --> C[mem_used rises toward mem_limit]
  C --> D[Paging to disk on classic queues]
  D --> C
  C --> E[mem_used crosses the watermark]
  E --> F[mem_alarm = true on one node]
  F --> G[Alarm propagates cluster-wide]
  G --> H[All publishers blocked on all nodes]
  H --> I[Publish rate drops to zero]

Common causes

CauseWhat it looks likeFirst thing to check
Unacked message accumulationmessages_unacknowledged grows while messages_ready looks normal; ack rate near zero with non-zero deliver rateGlobal unacked count in /api/overview, then per-queue unacked
Dead or stuck consumersConsumer count > 0 but ack rate = 0; unacked pinned at prefetch sizeConsumer application logs and health
Consumers scaled to zeromessages_ready growing on queues with 0 consumersPer-queue consumers field
Message flood from upstreamPublish rate spiked before the alarm; ready depth climbingPublish vs deliver_get rates
Watermark too low for workloadAlarm fires at memory levels the workload routinely needs; no backlog growthmem_limit vs normal operating memory
Large message bodiesMemory climbs with modest message counts; binary heap dominatesrabbitmq-diagnostics memory_breakdown
Very high prefetchA few consumers hold thousands of unacked messages; other consumers idleUnacked vs prefetch x consumer count

The most common case is unacked accumulation. Queue depth dashboards look fine because operators watch messages_ready, while delivered-but-unacknowledged messages pin memory invisibly. Unacked messages also cannot be paged to disk in classic queues, because the broker must be able to redeliver them.

Quick checks

All read-only and safe to run during the incident.

# 1. Confirm the alarm and see which node raised it
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, mem_alarm, disk_free_alarm}'

# 2. Check how far over the limit each node is
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, mem_used, mem_limit, ratio: (.mem_used / .mem_limit)}'

# 3. Alternative: alarm state from the CLI
rabbitmqctl eval 'rabbit_alarm:get_alarms().'

# 4. Global queue totals: is unacked the driver?
curl -s -u guest:guest http://localhost:15672/api/overview | jq '.queue_totals'

# 5. Find the heaviest queues by memory
curl -s -u guest:guest 'http://localhost:15672/api/queues?sort=memory&sort_reverse=true' | jq '.[] | {name, messages_ready, messages_unacknowledged, consumers, memory}'

# 6. Count blocked and blocking connections
curl -s -u guest:guest http://localhost:15672/api/connections | jq 'group_by(.state) | map({state: .[0].state, count: length})'

# 7. See what is consuming memory on the node
rabbitmq-diagnostics memory_breakdown

Iterating /api/connections is expensive on deployments with many connections, and there is no aggregate blocked-connection metric. Run check 6 once for diagnosis, not in a loop.

How to diagnose it

  1. Confirm the scope. Check mem_alarm on every node (check 1). Any node with mem_alarm: true blocks the whole cluster. Also check disk_free_alarm, because a disk alarm produces the same publisher-blocked symptom with a different fix.

  2. Identify the memory driver. Look at global queue_totals (check 4). If messages_unacknowledged is large and growing, consumers are the bottleneck. If messages_ready dominates, consumers are absent or too slow. If both are modest but memory is high, run rabbitmq-diagnostics memory_breakdown to see whether connections, queue processes, or the binary heap are responsible.

  3. Find the offending queues. Sort queues by memory (check 5). Usually one or two queues hold most of the pressure. Note their consumers count.

  4. Classify the consumer problem.

    • consumers: 0: nobody is subscribed. The queue grows until you restore consumers or purge it.
    • consumers > 0, unacked high, ack rate zero: consumers are connected but stuck. Check consumer application logs, downstream dependencies (databases, APIs), and prefetch settings.
    • Unacked pinned at exactly prefetch size times consumer count: consumers fetched everything and stopped processing. Classic stuck-consumer signature.

  5. Check for zombie connections. A consumer process can crash while its TCP socket stays open; the broker keeps the channel and all its unacked messages. Look for connections from hosts whose application you know is down. Closing these connections requeues their unacked messages, which is what you want.

  6. Check whether the alarm is flapping. If mem_used / mem_limit oscillates around 1.0, the alarm is firing and clearing repeatedly. Pressure is marginal: small relief may clear it, but it returns until the root cause is fixed.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
mem_alarm per nodeThe alarm itself; cluster-wide impact from any nodetrue sustained > 60s with publish traffic present
mem_used / mem_limitLeading indicator; the alarm fires at 1.0Ratio > 0.7 sustained, or steady upward trend
messages_unacknowledged (global and per-queue)The most common alarm precursor; pins memory, cannot be pagedPositive growth trend for 10+ minutes
messages_ready with consumer countBacklog building without drainGrowth with 0 consumers, or growth with consumers present
Connection states (blocked, blocking)Confirms real traffic impact of the alarmAny blocked publishers during an alarm
Publish vs deliver_get vs ack ratesShows the imbalance driving memory growthPublish » deliver_get sustained; ack = 0 with deliver > 0
messages_paged_outBroker is spilling to disk under pressureAny significant paging during active traffic
Publish rate during the alarmConfirms the blockDrops to zero while connection count stays stable

Alert text matters. The alarm is cluster-wide, so “node X memory high” undersells it. The page should say “cluster publishing halted.”

Fixes

Stuck or dead consumers (unacked accumulation)

Fix or restart the consumer applications, not the broker. If consumer processes crashed but their connections are still open, close the zombie connections via the management API. Tradeoff: closing a connection requeues all its unacked messages in one burst back to the head of the queue, which briefly spikes messages_ready. That is correct behavior; the messages were never processed.

If the root cause is a downstream dependency (the consumer’s database or API is down), fixing that dependency unblocks the consumers, acks resume, memory drains, and the alarm clears on its own.

No consumers at all

Redeploy or scale up the consumer fleet. As emergency relief for non-critical queues, purging the queue frees memory immediately. Purging is destructive: every message in the queue is discarded. Confirm the queue is expendable before you purge, and never purge as a first response on queues with data you cannot replay.

Watermark too low for the workload

If the alarm fires repeatedly at memory levels your workload legitimately needs, raise vm_memory_high_watermark (or set an absolute limit). Tradeoff: the watermark is also your guardrail against the OOM killer. Raising it narrows the margin between “publishers blocked” and “node killed by the kernel,” and the alarm does not prevent memory growth past the watermark. Size the limit so the broker, plus OS and anything else on the host, fits in RAM with headroom. In containers, verify RabbitMQ sees the container memory limit rather than the host’s RAM; an absolute limit is safer there.

A related technique: rabbitmqctl set_vm_memory_high_watermark 0 sets the limit to zero, which raises the alarm and blocks all publishing. Operators sometimes use this as an emergency brake during migrations or upstream floods. Treat it as a controlled outage of ingestion.

Message flood or oversized messages

Throttle the upstream producer, and check whether average message size grew; a deployment that started sending larger payloads can move memory dramatically at the same message rate. Long term, large bodies belong in a payload-offload pattern, not in the broker.

Prevention

  • Track mem_used / mem_limit with a warning at 0.7. The alarm is a cliff edge; the only warning you get is the ratio trending up. Paging activity is the secondary tell on classic queues.
  • Monitor unacked separately from ready. Unacked growth is the number-one precursor to memory alarms and is invisible in a total-depth chart.
  • Alert on ack rate divergence. Deliver > 0 with ack = 0 means consumers are connected but dead in practice. Catch this hours before the alarm.
  • Set sane prefetch counts. A small number of consumers with huge prefetch can pin enormous memory in unacked messages.
  • Size the watermark deliberately. Know your steady-state memory, size the watermark above it with real headroom, and use an absolute value in containers.
  • Suppress alarm pages during warm-up. After a restart, memory spikes as queues repopulate. Gate the page on uptime > 10 minutes so cold starts do not page you.

How Netdata helps

  • Netdata’s RabbitMQ collector pulls the Management API and charts mem_used against mem_limit per node, so you see the ratio approaching 1.0 before the alarm fires instead of discovering it at the cliff.
  • messages_ready and messages_unacknowledged are charted as separate dimensions, making the unacked-accumulation pattern visible without custom queries.
  • Publish, deliver, and ack rates on one dashboard let you spot the imbalance driving memory growth, and the publish rate dropping to zero confirms the block in the same view.
  • Alarm state and per-queue depth correlation shortens the path from “publishing stopped” to “this specific queue with zero acks is the cause,” which is the whole diagnosis.
  • Historical retention around the incident lets you reconstruct whether this was unacked buildup, a flood, or a slow leak, which determines the fix and the prevention work.