$ guides / rabbitmq ▌

RABBITMQ · OPERATIONS PLAYBOOK

RabbitMQ's two circuit breakers: a memory wall, a disk halt, and a cluster that can split in two

An Erlang message broker where every connection, channel, and queue is its own process, publishers are throttled by two completely different mechanisms, and a single node crossing its memory or disk watermark blocks writes everywhere. We trace how that design behaves under load, where it turns from throttling into a cluster-wide stop, and what to do when it does.

> Start with the monitoring checklist → # Jump to the full guide list

RabbitMQ's defaults get you to production quickly, then hand you a set of cliff-edges that most teams only discover during an incident.

The defaults work. Until consumers stall, messages_unacknowledged piles up in memory that cannot be paged, and one node crosses vm_memory_high_watermark — at which point mem_alarm fires and every publisher on every node is blocked at once. Until disk_free_limit (still 50MB out of the box) is crossed by a log rotation and publishing halts cluster-wide. Until a consumer holds a delivery past consumer_timeout and the broker closes its channel. Until a redeploy redeclares a queue with changed arguments and every publish throws PRECONDITION_FAILED - inequivalent arg. Until a GC pause longer than net_ticktime looks like a dead node and the cluster declares a partition.

These guides are written for engineers who already run RabbitMQ, not for people learning what a queue is. The goal is the mental model of how the broker actually behaves under load, the failure patterns that keep recurring, the monitoring story that catches them before they page anyone, and the runbooks you wish someone had handed you before your last incident.

How RabbitMQ actually runs in production

RabbitMQ is not just a queue. It is an Erlang runtime where every connection, channel, and queue is a separate process, messages flow through exchanges into per-queue processes, and two independent throttles — per-connection flow control and cluster-wide resource alarms — sit on the write path. Most production failures live between these layers, not inside any one of them.

publishers / clients

Each AMQP connection is one Erlang process; each channel within it is another. Publishers, consumers, and management clients each cost a file descriptor and a socket. Credit-based flow control can throttle a single fast publisher (<code>state: flow</code>) without touching anyone else.

CLIENT

exchanges + routing

Exchanges are routing tables, not storage. Direct and fanout are cheap; topic exchanges match routing keys against binding patterns and get expensive with thousands of bindings. A publish with no matching binding is returned (<code>mandatory=true</code>) or silently dropped (the default).

ROUTE

queues (processes)

Every queue is a single Erlang process holding ready and unacknowledged messages, delivery state, and consumer assignments. A hot queue is single-threaded and cannot scale past one core. Classic, quorum (Raft), and stream queues have different storage and failure modes.

QUEUE

message store + index + WAL

Classic queues share an on-disk message store and per-queue index; quorum queues write a per-queue Raft write-ahead log. Under memory pressure, queues page ready messages to disk — but unacknowledged messages stay pinned in RAM.

STORE

consumers + acks

Consumers pull work via <code>basic.consume</code> with a prefetch window. A message is only done when acknowledged; delivered-but-unacked messages hold memory and, past <code>consumer_timeout</code>, get the channel closed. Rejections and requeues drive redelivery and dead-lettering.

CONSUME

memory + disk alarms

The node-wide circuit breakers. When <code>mem_used</code> crosses the watermark or <code>disk_free</code> drops below <code>disk_free_limit</code>, an alarm fires and blocks all publishers on all nodes. This is a cliff-edge, not a slowdown — and one node's alarm is the whole cluster's problem.

ALARM

Erlang VM (BEAM)

Schedulers (one per core) run every process cooperatively. <code>run_queue</code> is the direct CPU-saturation signal; <code>proc_used</code> counts processes toward a hard limit; the binary heap holds reference-counted message bodies. A saturated scheduler delays heartbeats and can trigger false partitions.

BEAM

clustering + Erlang distribution

Mnesia replicates metadata; quorum queues replicate messages via Raft. All of it crosses a single TCP link per node pair (port 25672). Saturate that link and you get Mnesia timeouts, quorum lag, and false partition detection — many symptoms, one cause.

CLUSTER

Why this matters: 'RabbitMQ is slow' or 'publishers are stuck' can come from one connection under flow control, a cluster-wide resource alarm, a hot single-threaded queue, unacked messages pinned in memory, a saturated Erlang scheduler, a congested distribution port, or a partition. The symptom rhymes but each layer has a different signal — and a different fix.

The failures you'll actually see

Most RabbitMQ incidents fall into a small set of recurring patterns. Recognise the shape, and triage gets dramatically faster.

CRITICAL

The memory wall

Consumers fall behind, messages_unacknowledged climbs (and cannot be paged to disk), and one node crosses vm_memory_high_watermark. mem_alarm goes true and every publisher on every node is blocked instantly — a cliff-edge, not a slowdown. Publish rate drops to zero while connections stay open in blocked state. The alarm is cluster-wide even though only one node hit its limit.

mem_alarm = true on any node (publishing halted cluster-wide)
mem_used / mem_limit at or near 1.0
messages_unacknowledged growing while ack rate sits near zero
Connections in blocked / blocking state, publish rate at zero

Investigate →

CRITICAL

The disk halt

Free disk on the data partition drops below disk_free_limit — often because it was left at the 50MB default and a log rotation or crash dump crossed it, or because a quorum queue's WAL grew faster than it compacted. disk_free_alarm fires and all publishers are blocked cluster-wide. Unlike the memory wall, consumers keep draining, which can free space if the messages are persistent.

disk_free_alarm = true (publishers blocked, consumers still draining)
disk_free approaching disk_free_limit on the data partition
Quorum queue WAL directory growing under sustained throughput
Erlang crash dump or unrotated logs filling the partition

Investigate →

CRITICAL

The split-brain partition

Cluster nodes lose contact — a real network fault, or a GC pause longer than net_ticktime that only looks like one. The partitions array goes non-empty and behaviour depends on cluster_partition_handling: ignore lets both sides diverge, pause_minority freezes the minority, autoheal restarts the losing side. Quorum queues in the minority go read-only.

Network Partitions listed in rabbitmqctl cluster_status
Non-empty partitions array on /api/nodes, sustained > 2 intervals
Cluster link traffic between peers dropping to zero
Quorum queues reporting minority state

Investigate →

ACTIVE

The consumer black hole

Consumers are connected and channels are open, but nothing is being acknowledged — a deadlock, a failed downstream dependency, or a zombie connection whose process died but whose socket is still open. messages_ready climbs, ack rate sits at zero, and any delivery held past consumer_timeout gets its channel force-closed. Open connections are not working consumers.

Consumers > 0 but ack rate near zero on the queue
delivery acknowledgement on channel N timed out in the logs
messages_ready climbing, consumer_utilisation low
Unacked count stuck at exactly prefetch x consumer count

Investigate →

ACTIVE

Publishing into the void

Publishers report success but messages never arrive. Either the target exchange or queue does not exist and the channel is closed with NOT_FOUND, or the exchange exists with no matching binding and — with mandatory=false, the default — every message is silently discarded. No alarm, no error, no queue growth. Common after a deploy that renames or reorders declarations.

NOT_FOUND - no exchange / no queue closing channels
Non-zero publish rate with zero deliver_get and flat queue depth
return_unroutable > 0 (only when mandatory=true)
Downstream systems reporting missing data hours later

Investigate →

IMMINENT

The redeploy topology clash

A deploy declares a queue or exchange with arguments that differ from how it already exists — a changed TTL, a new x-queue-type, durable versus transient — and the broker refuses with PRECONDITION_FAILED - inequivalent arg, closing the channel. Every instance retries and fails identically, so consumers cannot attach and publishers cannot publish until the topology is reconciled.

PRECONDITION_FAILED - inequivalent arg on queue.declare
Channel exceptions spiking in step with a deployment
Clients in a declare / fail / reconnect loop
Consumers unable to attach to the affected queue

Investigate →

RabbitMQ monitoring maturity levels

RabbitMQ observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your broker matters. Most production clusters should land at the second level.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor. With these signals you can answer one question: is the broker alive and is it accepting messages? You will not learn what broke, but you will learn that something broke before users do. Survival is enough for dev brokers and non-critical pipelines.

Node running / aliveness Is the rabbit application up and answering, not just beam.smp alive?
Memory alarm (mem_alarm) If true, every publisher on every node is blocked.
Disk free alarm (disk_free_alarm) If true, publishing is halted cluster-wide.
Total messages ready + unacked Are queues accumulating unboundedly across the cluster?
Connection count Can clients actually connect right now?

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production clusters should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident on its own: stalled consumers, backlogs, resource pressure, flow control, partitions.

Per-queue ready and unacked Unacked pinned in RAM is the number-one alarm precursor.
Consumer count per queue Zero consumers on a live queue is an immediate investigation.
Publish / deliver_get / ack rates The imbalance predicts queue growth before depth does.
Memory usage ratio (mem_used/mem_limit) The gauge that leads the memory alarm; watch past 0.7.
File descriptor and socket ratio Cliff-edge: at the limit, new connections are refused.
Flow / blocked connection counts flow is normal backpressure; blocked means an alarm.
Cluster membership and partitions All expected nodes running, partitions array empty.
Disk free space trend Runway to the disk alarm, not just the current value.

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone up. Messages paging to disk, head-of-line latency creeping, connections churning, redelivery climbing, a quorum queue drifting toward minority. None of these page you on day one. They become page-out incidents on day thirty.

Memory breakdown by category binary vs queue-proc vs connection vs Mnesia — where is it going?
Consumer utilisation per queue Consumers attached but below 0.5 means they cannot keep up.
Head message age Latency the depth number alone cannot show.
Redeliver and return-unroutable rate Retry storms and routing misconfiguration.
Connection / channel / queue churn Stable counts can hide a leak churning underneath.
Erlang run queue Direct CPU-saturation signal; sustained > cores is trouble.
Quorum queue leader distribution Leaders skewed onto one node after a restart.
Messages paged out Queues spilling to disk is the early memory warning.

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. Distribution-port saturation, quorum WAL growth, binary-heap bloat, Mnesia latency, publisher confirm latency. Most teams never need every signal here. Add the ones your incident history says you do.

Erlang distribution port send queue The single link carrying all cluster traffic (ss on 25672).
Quorum uncommitted Raft entries Followers behind; leader loss would drop those messages.
Quorum queue WAL segment size on disk Can fill the disk faster than classic queue growth.
Binary heap vs RSS discrepancy Reference-counted bodies awaiting lazy GC (force_gc reveals it).
Mnesia table sizes and tx latency 100k+ queues slow declarations and node rejoins.
Publisher confirm latency (client-side) A 5-10 minute warning before flow control or an alarm.
Per-queue redeliver count Localises the poison-message loop.
TLS certificate expiry Expired certs are a silent, total outage; not a broker metric.

Operating mistakes worth avoiding

The traps RabbitMQ teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

Leaving disk_free_limit at the 50MB default

On a modern server the 50MB default is dangerously small — a single log rotation, crash dump, or burst of persistent messages crosses it in seconds and halts publishing across the entire cluster. Set <code>disk_free_limit.relative</code> to 1.0-2.0 (1-2x RAM) or an absolute 2GB+, and alert when free space drops below the larger of 3x the limit or 1GB.

Watching messages_ready but not messages_unacknowledged

Everyone watches total queue depth. The silent killer is <code>messages_unacknowledged</code>: in classic queues it cannot be paged to disk, so it stays pinned in RAM and is the number-one precursor to a memory alarm. A consumer holding thousands of unacked messages with a flat ack rate is the shape to catch.

Confusing connection health with consumer health

An open connection with open channels does not mean a working consumer — it can be deadlocked, waiting on a dead downstream service, or a zombie whose process died with the socket still open. Verify ack rate and <code>consumer_utilisation</code>, not connection counts. Deliveries held past <code>consumer_timeout</code> then get the channel force-closed.

Treating the memory alarm as a per-node alert

The memory alarm is cluster-wide: one node crossing its watermark blocks publishers on every node. Teams that alert per-node with the text 'node X memory high' miss that message ingestion has stopped everywhere. The alert should read 'cluster publishing halted', because that is what happened.

Alerting on flow control like it were an alarm

A connection in <code>state: flow</code> is credit-based backpressure throttling one publisher — surgical, transient, and healthy. A connection in <code>blocked</code> is a cluster-wide resource alarm. Alerting on flow the way you alert on blocked generates constant false positives and buries the distinction that actually matters.

Under-provisioning file descriptors

Each connection costs a file descriptor and a socket; classic queues consume FDs for index and store files; quorum queues use FDs for Raft WAL. The default ulimit of 1024 is exhausted fast, and at the limit the node refuses connections and cannot open segment files — often misdiagnosed as a disk problem. Raise both the OS ulimit and the systemd limit.

Judging queues by depth without message age

A queue with 1,000 messages is not inherently alarming; a queue with 1,000 messages where the oldest is three hours old is. Depth is volume, <code>head_message_timestamp</code> is latency. Without age, teams react to normal batch spikes and miss the latency degradation that actually breaks an SLA.

Not detecting silent routing loss

With <code>mandatory=false</code> (the default) and no publisher confirms, messages published to an exchange with no matching binding are silently discarded — no alarm, no return, no queue growth. Teams discover it when downstream systems report missing data days later. Use publisher confirms, set <code>mandatory=true</code>, and audit bindings proactively.

RabbitMQ runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

Memory and disk alarms

▸

Flow control and backpressure

▸

Queue backlogs and latency

▸

Consumers, acks, and prefetch

▸

Throughput, routing, and lost messages

▸

Connections, channels, and protocol errors

▸

Clustering, partitions, and node failure

▸

Quorum queues and replication

▸

Erlang runtime and resources

▸

Publisher confirms and durability

▸

Authentication, permissions, and TLS

WHERE TO GO NEXT

Setting up RabbitMQ monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.

> Start with the checklist > Back to Operations Guides