RabbitMQ Operations Guides on Netdata

How RabbitMQ actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How RabbitMQ actually works in production: a mental model for operators

Most RabbitMQ incidents look mysterious only because the operator is reasoning about the wrong abstraction. RabbitMQ is not a daemon with threads and shared memory. It is an Erlang/OTP application running on the BEAM VM, and almost everything you interact with (a connection, a channel, a queue) is an independent Erlang process with its own mailbox, scheduled alongside thousands of others. Once you internalize that, the weird symptoms become predictable: why one hot queue can stall an entire connection, why a single node crossing a memory threshold freezes publishers cluster-wide, and why a “healthy” broker can appear to drop messages.

RabbitMQ ACCESS_REFUSED - Login was refused: authentication failures and credential rotation

Mon, 01 Jan 0001 00:00:00 +0000

Your application logs are filling with ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN, and the broker log shows matching PLAIN login refused lines. No messages flow from the affected clients, and depending on the client library’s retry behavior, the broker may also be spending CPU on a tight connect-authenticate-reject loop.

This error means the broker rejected the username/password pair (or the backend that validates them) during AMQP connection establishment. It happens before any vhost, exchange, or queue is touched. The connection is closed immediately, which is why a single misconfigured deployment can turn into connection churn that looks like a capacity problem.

RabbitMQ binary heap bloat: reference-counted message bodies and force_gc

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ binary heap bloat: reference-counted message bodies and force_gc

A consumer fleet finishes draining a queue. Queue depth drops to zero. And the RabbitMQ node’s memory graph stays high. It sits at two or three times what the message volume should account for, sometimes for hours, while dashboards keep telling you the broker is nearly out of memory. The instinct is to call it a leak. In most cases it is not one.

RabbitMQ channel leak and churn: the channel-per-message anti-pattern

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ channel leak and churn: the channel-per-message anti-pattern

Your connection count is flat and queue depths look normal, but broker memory is creeping up and the Erlang process count keeps climbing. That is the classic shape of a RabbitMQ channel leak: a slow resource drain that hides behind a stable connection count until the node runs out of processes or memory.

Channels are where all AMQP work happens: publishing, consuming, acknowledging. Each channel is a separate Erlang process on the broker. They are lightweight but not free. When a client opens a new channel per message instead of reusing long-lived channels, or recreates a channel on every exception, the broker pays for setup and teardown on every operation. If channels open faster than they close, the leak accumulates Erlang processes and memory indefinitely.

RabbitMQ classic mirrored queues removed: migrating to quorum queues

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ classic mirrored queues removed: migrating to quorum queues

If you upgraded RabbitMQ and your ha-mode policies silently stopped doing anything, that is not a bug. Classic queue mirroring was deprecated in 3.9 and removed in 4.0. On 4.x the mirroring-related keys in your policies (ha-mode, ha-params, ha-sync-mode, ha-promote-on-shutdown, ha-promote-on-failure, ha-sync-batch-size) have no effect. The classic queues themselves keep working, but each one now runs as a single, non-replicated queue on one node.

RabbitMQ connection blocked / blocking: publishers frozen by a resource alarm

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ connection blocked / blocking: publishers frozen by a resource alarm

Your publishers are hanging. No errors, no exceptions, no disconnections. Publish calls simply never return, and queue depth stops growing. When you check the management UI or rabbitmqctl list_connections, you see connections in state blocking or blocked instead of running.

This is not a network problem and not a client bug. RabbitMQ has raised a resource alarm (memory or disk) and has deliberately stopped reading from publisher connections across the entire cluster. It is a circuit breaker doing its job. The danger is that many client libraries hide this completely: from the application’s perspective, the publish call just hangs with no error, which sends operators chasing ghosts in the wrong place.

RabbitMQ connection in flow state: credit-based backpressure explained

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ connection in flow state: credit-based backpressure explained

You opened the management UI or ran rabbitmqctl list_connections state and saw one or more connections sitting in state: flow. No alarm fired. Nothing is blocked cluster-wide. Is it a problem? Usually no, and treating it like one is a common source of alert noise.

Connection state flow is RabbitMQ’s internal, per-connection, credit-based backpressure doing what it was designed to do. A specific publishing connection is outrunning something downstream: a queue process, the message store, disk I/O, or an inter-node link. The broker throttles just that connection, transiently, toggling between blocked and unblocked many times per second. It is surgical. It is not a resource alarm.

RabbitMQ connection storm: reconnect loops, FD pressure, and CPU spent on handshakes

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ connection storm: reconnect loops, FD pressure, and CPU spent on handshakes

A connection storm happens when many clients open connections to RabbitMQ at the same time, or when a smaller set of clients reconnect in a tight loop. The broker tolerates thousands of idle connections far better than it tolerates creating them. Connection setup is expensive: TCP handling, TLS handshake, AMQP negotiation, authentication, a new Erlang process per connection, plus one process per channel the client opens.

RabbitMQ consumer timeout: delivery acknowledgement timed out and the channel is closed

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ consumer timeout: delivery acknowledgement timed out and the channel is closed

You found this in the broker log or your client logs:

delivery acknowledgement on channel 1 timed out

In 3.13+ the fuller broker-side message names the consumer, queue, vhost, delivery tag, and the timeout value used, and the channel is closed with a PRECONDITION_FAILED (406) channel exception. The client library then typically reopens the channel and resubscribes, until it happens again.

RabbitMQ consumer_utilisation low: consumers attached but not keeping up

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ consumer_utilisation low: consumers attached but not keeping up

The queue has messages. It has consumers. And yet the backlog keeps growing. The management UI shows consumer_utilisation (renamed consumer_capacity in newer RabbitMQ versions) at 0.2 or 0.3, and your first instinct, “just add more consumers”, may or may not be the right move.

consumer_utilisation is a per-queue estimate from the broker: the fraction of time the queue was able to deliver a message immediately to a waiting consumer. A value of 1.0 means every time a message was ready, a consumer could take it. A value below 0.5 with a non-empty backlog means the queue had messages sitting ready while no consumer was in a position to accept them. The bottleneck is on the delivery path between the queue and the consumers, not on publishing, and not on routing.

RabbitMQ consumers connected but not acknowledging: the consumer black hole

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ consumers connected but not acknowledging: the consumer black hole

The queue has consumers. The management UI shows consumers: 3. Yet messages_ready keeps climbing, the ack rate is flat at zero, and nothing is draining. Everything looks connected and nothing is working. This is the consumer black hole: dangerous because every surface-level health check passes.

The trap is confusing connection health with consumer health. An open TCP connection and a registered consumer tag do not mean messages are being processed. A consumer can be connected, have channels open, and be completely stuck: deadlocked, blocked on a failed downstream dependency, or a zombie where the application process died but the OS never closed the socket. The broker will hold unacknowledged messages for it, pin them in RAM, and wait. Left alone, this escalates into the memory wall: unacked messages are not paged out, memory grows, the watermark is crossed, and every publisher in the cluster is blocked.

RabbitMQ dead-letter queue growing: TTL expiry, max-length overflow, and rejections

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ dead-letter queue growing: TTL expiry, max-length overflow, and rejections

Your dead-letter queue (DLQ) has 400,000 messages and no consumer. Nothing paged you, because RabbitMQ does not alarm on a DLQ by default. But every message in that queue is a record of something that already failed upstream: a consumer that rejected a delivery, a message that sat past its TTL, or a queue that overflowed its length limit and pushed the oldest message out.

RabbitMQ disk free limit alarm: free disk space insufficient and publishing halted

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ disk free limit alarm: free disk space insufficient and publishing halted

Your RabbitMQ cluster just stopped accepting messages. Publishers are connected but frozen. Consumers are still draining queues normally. The broker log shows a disk alarm, and the management UI shows disk_free_alarm: true on at least one node.

This is the RabbitMQ disk free limit alarm. It fires when free space on the node’s data partition drops below disk_free_limit, and it blocks every publisher on every node in the cluster, not just the affected node. It is a hard circuit breaker, not a gradual degradation: one threshold crossing and ingestion stops instantly.

RabbitMQ disk_free_limit at the default 50MB: the production footgun

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ disk_free_limit at the default 50MB: the production footgun

RabbitMQ ships with disk_free_limit set to 50MB. When free disk space on the node’s data partition drops below that limit, the broker raises a disk alarm and blocks every publisher on every node in the cluster. Consumers keep draining, but message ingestion stops until free space climbs back above the threshold.

The 50MB default exists so development installs on tiny machines do not silently eat a laptop’s disk. On a production server with terabytes of storage, it is not a safety margin; it is a tripwire one inch off the floor. A single log rotation, an Erlang crash dump, or a burst of persistent messages can consume 50MB in seconds, and the resulting incident looks exactly like a broker outage: publishers frozen, applications timing out, queue depth flat at zero growth while upstream systems back up.

RabbitMQ Erlang distribution port saturation: the single link that carries all cluster traffic

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ Erlang distribution port saturation: the single link that carries all cluster traffic

Your RabbitMQ cluster is acting haunted. Quorum queues lag behind their leaders. Metadata operations time out intermittently. Nodes occasionally report network partitions even though latency between nodes is sub-millisecond and nothing is actually down. You investigate each symptom separately and find nothing.

The likely common cause is Erlang distribution port saturation. In a RabbitMQ cluster, all inter-node traffic flows over a single TCP connection per node pair, by default on port 25672. Mnesia metadata replication, quorum queue Raft consensus traffic, management stats aggregation, and internal RPC all share that one connection. There is no separate channel, no QoS, no prioritization. When the volume of inter-node traffic exceeds what that one connection can carry, everything it carries degrades at once.

RabbitMQ Erlang process exhaustion: proc_used near the process limit

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ Erlang process exhaustion: proc_used near the process limit

Your monitoring shows proc_used climbing toward proc_total on a RabbitMQ node. The default limit is 1,048,576 Erlang processes, and when the count hits the ceiling, the BEAM VM terminates with a system_limit error. There is no graceful degradation: the node crashes, taking every connection, channel, and non-replicated queue on it down.

Process exhaustion is almost never the root cause. Every connection, channel, and queue in RabbitMQ is at least one Erlang process, so when the process table fills, something upstream is creating these objects faster than they are being destroyed. The usual suspects are channel-per-message publishers, leaked temporary queues (auto-delete or exclusive queues whose owning connections never cleanly close), and connection proliferation. Raising the +P limit buys time; it does not fix anything.

RabbitMQ Erlang run queue high: scheduler saturation and slow heartbeats

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ Erlang run queue high: scheduler saturation and slow heartbeats

The RabbitMQ node is alive, queues are accepting messages, but everything feels slow: publishes lag, acks lag, the management API responds sluggishly, and the cluster log starts showing “missed heartbeats” from peers or clients. The node’s run_queue metric sits persistently above the CPU core count. OS-level CPU may even look unremarkable.

This is Erlang scheduler saturation. The BEAM VM has more runnable processes than its scheduler threads can execute, so work queues up and every operation on the node, including heartbeat processing, waits its turn. The secondary effect is worse than the primary one: delayed heartbeats get misread as dead peers, and the cluster starts reporting network partitions that do not exist.

RabbitMQ file descriptor exhaustion: fd_used near fd_total and refused connections

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ file descriptor exhaustion: fd_used near fd_total and refused connections

Clients that were connected keep working, but every new connection attempt fails. Publishers in a reconnect loop after a deployment start timing out. The broker log shows too many open files, and the management API shows fd_used sitting at or near fd_total. This is RabbitMQ file descriptor exhaustion, and it is a cliff-edge failure: no graceful degradation, just a hard refusal.

RabbitMQ flow control vs resource alarms: the two throttling mechanisms operators confuse

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ flow control vs resource alarms: the two throttling mechanisms operators confuse

Your RabbitMQ dashboard shows connections that are not in running state, your alert fired, and you need to know: is the broker in trouble, or working as designed? The answer depends entirely on whether those connections show flow or blocked. The difference is not a matter of degree. They are two independent mechanisms with different scope, different causes, and different required responses.

RabbitMQ head message age: the queue latency that depth alone cannot show

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ head message age: the queue latency that depth alone cannot show

A queue with 10 messages whose oldest is 3 hours old is in worse shape than a queue with 10,000 messages whose oldest is 5 seconds old. Queue depth tells you volume. It says nothing about how long messages are actually waiting. Operators who alert on depth alone end up paged for healthy batch queues and blind to a real-time queue that has been stalled for an hour.

RabbitMQ memory resource limit alarm: publishers blocked across the whole cluster

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ memory resource limit alarm: publishers blocked across the whole cluster

Your RabbitMQ cluster just stopped accepting messages. Publishers are connected but nothing flows. The broker log shows:

memory resource limit alarm set on node rabbit@node1.
*** Publishers will be blocked until this alarm clears ***

One node crossed its memory high watermark, and RabbitMQ responded by blocking every publisher on every node in the cluster. There is no gradual degradation: the transition from “fine” to “all ingestion halted” is a single threshold crossing.

RabbitMQ memory usage climbing toward the watermark: mem_used vs mem_limit

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ memory usage climbing toward the watermark: mem_used vs mem_limit

The memory alarm has not fired yet. Publishers are still running. But mem_used has been creeping up for hours and mem_used / mem_limit just crossed 0.7. This is the last quiet window you get: RabbitMQ’s memory failure mode is a cliff edge. When mem_used reaches mem_limit, the alarm fires and every publisher on every node in the cluster is blocked at once.

RabbitMQ message TTL expiry: messages vanishing before consumers read them

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ message TTL expiry: messages vanishing before consumers read them

A queue shows a stable, modest depth. Publish rate is healthy. Consumers are connected. Then a downstream team reports missing data, and you discover the queue was never keeping up. Messages were expiring faster than consumers could read them, and RabbitMQ silently discarded every one.

This is the signature failure of message TTL without a dead-letter exchange. A queue-level x-message-ttl argument or a per-message expiration property tells RabbitMQ to discard messages that sit too long. Unless a dead-letter exchange (DLX) is configured, expired messages are dropped with no error, no return to the publisher, and no counter that obviously screams “data loss.” Expiry also works against your monitoring: queue depth stays flat because the backlog is being deleted, and head message age resets every time the head message expires.

RabbitMQ messages lost on restart: durable queues, persistent messages, and transient traps

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ messages lost on restart: durable queues, persistent messages, and transient traps

You restarted a RabbitMQ node, it came back healthy, and now downstream systems are reporting missing data. The queues are there. The exchanges and bindings are there. But messages that were sitting in queues before the restart are gone, or a queue itself vanished entirely.

This is almost never a RabbitMQ bug. It is a mismatch between what the operator thinks durability means and what RabbitMQ actually guarantees. The broker did exactly what it was configured to do; the configuration just guarantees less than you assumed.

RabbitMQ Mnesia growth: too many queues and bindings, slow declarations and rejoins

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ Mnesia growth: too many queues and bindings, slow declarations and rejoins

Your cluster still routes messages fine, but everything else has gone soft. queue.declare calls that used to return in milliseconds now take a second or more. The management UI takes ten seconds to render the queue list. A node restarted for a routine upgrade has been “starting” for five minutes and is serving no traffic. No resource alarm has fired, and yet the system is clearly sick.

RabbitMQ monitoring checklist: the signals every production broker needs

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ monitoring checklist: the signals every production broker needs

Most RabbitMQ incidents are not exotic. They are memory alarms nobody saw coming, disk alarms triggered by a 50MB default limit, unacked messages pinning RAM while the queue depth graph looked fine, and silent routing loss that only surfaced when a downstream team reported missing data. The broker exposed the signals the whole time. The gap is in what the team chose to collect and alert on.

RabbitMQ monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ monitoring maturity model: from survival to expert

Most RabbitMQ monitoring setups fail in one of two ways. Either they watch almost nothing and discover outages from user reports, or they chart everything and page on noise. The difference is rarely tooling. It is knowing which signals actually predict the failure modes RabbitMQ is prone to: the memory wall, the disk halt, consumer black holes, split-brain, and silent message loss.

RabbitMQ network partition detected: split-brain in cluster_status

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ network partition detected: split-brain in cluster_status

You ran rabbitmqctl cluster_status (or your monitoring scraped /api/nodes) and saw the “Network Partitions” section listing node names, or a non-empty partitions array on one or more nodes. The cluster has split: some nodes can no longer see each other over the Erlang distribution link, and each side now has its own view of the world.

This is a paging condition. What happens next depends on cluster_partition_handling: the two sides may be accepting writes independently and diverging (the ignore default), the minority side may have frozen and stopped serving clients (pause_minority), or nodes may be about to restart themselves (autoheal). Each outcome has a different blast radius, and the wrong response makes it worse.

RabbitMQ node down: telling a dead broker apart from a partitioned one

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ node down: telling a dead broker apart from a partitioned one

A node has disappeared from your cluster view. The first question is not “how do I bring it back” but “what actually happened to it.” A dead broker (VM crashed, process killed, host gone) and a partitioned broker (Erlang node still running but cut off from its peers) look almost identical from one vantage point and completely different from another. The recovery steps are opposite: you restart a dead node, but restarting a partitioned node mid-partition can make split-brain worse.

RabbitMQ node is quorum critical: checking before a rolling restart

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ node is quorum critical: checking before a rolling restart

You are about to restart a RabbitMQ node: a rolling upgrade, an OS patch, an instance resize. Before you stop it, one question decides whether this is routine maintenance or a queue outage: if this node goes down right now, does any quorum queue or stream lose its online majority?

RabbitMQ ships a purpose-built check for exactly this. rabbitmq-diagnostics check_if_node_is_quorum_critical returns unhealthy when stopping the target node would drop a quorum queue below the number of online members it needs to accept writes. The same logic is exposed over HTTP at GET /api/health/checks/node-is-quorum-critical, which makes it usable from automation, load balancer health gates, and CI-driven upgrade pipelines.

RabbitMQ NOT_FOUND - no queue or exchange: publishing to something that does not exist

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ NOT_FOUND - no queue or exchange: publishing to something that does not exist

Your publisher or consumer logs show NOT_FOUND - no exchange 'orders.events' in vhost '/', or the queue variant, NOT_FOUND - no queue 'billing.jobs' in vhost '/'. The RabbitMQ server log has the matching line: operation basic.publish caused a channel exception not_found: no exchange 'orders.events' in vhost '/'. The channel that raised it is now closed, and depending on the client library, your application may be spinning in a declare-fail-reconnect loop.

RabbitMQ paging messages to disk: the paging ratio as an early memory warning

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ paging messages to disk: the paging ratio as an early memory warning

RabbitMQ does not usually go from healthy to blocked in one step. Before the memory alarm blocks publishers cluster-wide, the broker can cross a quieter threshold: queues begin paging message contents out of RAM and onto disk.

That threshold is controlled by vm_memory_high_watermark_paging_ratio, which defaults to 0.5 of the effective memory limit. If vm_memory_high_watermark is 0.40 of RAM, paging starts near 20 percent of RAM. If the watermark is 0.60, paging starts near 30 percent. Check the configured values rather than inferring them from the RabbitMQ version.

RabbitMQ partition handling: pause_minority vs autoheal vs ignore

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ partition handling: pause_minority vs autoheal vs ignore

When a RabbitMQ cluster splits across a network fault, the broker cannot keep every node writable without risking divergent state, and it cannot keep every node consistent without taking some nodes out of service. The cluster_partition_handling setting in rabbitmq.conf is where you decide, in advance, which one gives: availability, or data.

Most teams set this once during cluster setup and never test what it does. Then a real partition happens at 3 a.m. and they discover their choice the hard way: autoheal restarts nodes and drops non-replicated messages, pause_minority freezes nodes that might be serving most of your clients, and ignore lets both sides accept writes and silently discards one side’s state when the network heals. None of these is a bug. They are the documented behavior you opted into.

RabbitMQ permission denied on vhost: configure, write, and read access errors

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ permission denied on vhost: configure, write, and read access errors

Your application connected to RabbitMQ, authenticated successfully, and then failed the moment it tried to do actual work. The client log shows something like ACCESS_REFUSED - access to exchange 'events' in vhost 'billing' refused for user 'svc-billing' or 403 ACCESS_REFUSED on a queue declare, publish, or consume call.

This is a distinct failure class from a login failure. The credentials are valid; the broker rejected a specific operation because the user’s permissions on that vhost or resource do not cover it. The fix is completely different: a login failure means wrong credentials or a broken auth backend, while ACCESS_REFUSED means the authorization layer is doing its job and the user, vhost, or permission set is wrong.

RabbitMQ poison message loop: a stuck queue head and endless redelivery

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ poison message loop: a stuck queue head and endless redelivery

A queue with active consumers is making no progress. Queue depth is flat, messages are being delivered, but almost nothing is being acknowledged. The oldest message in the queue keeps getting older. This is a poison message loop: one message at the head of the queue crashes or is rejected by every consumer that picks it up, gets requeued, and is delivered again, forever.

RabbitMQ PRECONDITION_FAILED - inequivalent arg: redeclaring a queue with different properties

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ PRECONDITION_FAILED - inequivalent arg: redeclaring a queue with different properties

Your application logs are filling with channel exceptions and consumers or publishers keep reconnecting:

PRECONDITION_FAILED - inequivalent arg 'x-queue-type' for queue 'email.queue'
in vhost 'Email': received none but current is the value 'quorum' of type 'longstr'

This is AMQP error 406, a channel-level exception. A client declared a queue (or exchange) that already exists, but one or more of the declared properties do not match how the entity was originally created. RabbitMQ refuses the declaration and closes the channel. The connection usually survives, but most client libraries treat a channel exception as fatal and tear down the whole connection, so a single mismatched declaration often looks like a reconnect storm.

RabbitMQ prefetch count: unacked hoarding versus idle consumers

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ prefetch count: unacked hoarding versus idle consumers

The basic.qos prefetch count is the most consequential consumer-side setting in RabbitMQ, and the one most teams never touch. The default is unlimited. That default is why a single slow consumer can pin thousands of messages in RAM while every other consumer on the same queue sits idle, and why “we added more consumers but nothing got faster” is such a common report.

RabbitMQ publish outpacing deliver: reading the rate imbalance before the backlog

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ publish outpacing deliver: reading the rate imbalance before the backlog

When publish rate runs ahead of deliver rate on a RabbitMQ broker, nothing is broken yet. That is exactly why this signal matters. Queue depth is a lagging indicator: by the time messages_ready looks alarming, the imbalance has usually been running for minutes. The rate comparison tells you the same thing earlier, and it tells you which side of the pipe changed.

RabbitMQ publisher confirms: proving the broker actually accepted your message

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ publisher confirms: proving the broker actually accepted your message

By default, AMQP publishing is fire-and-forget. Your client writes a message to the socket, the write succeeds, and the application moves on. That TCP write proves exactly one thing: the bytes left your process. It says nothing about whether the broker routed the message, persisted it, or even finished reading it before the connection dropped.

Publisher confirms close this gap. When enabled, the broker sends an explicit acknowledgement for every published message, after it has taken responsibility for it. Combined with the mandatory flag, confirms are the only way to get end-to-end proof that a message was both accepted and routed. Without them, entire classes of message loss are invisible: unroutable messages are silently discarded, in-flight messages vanish on connection failure, and the first symptom is a downstream consumer reporting missing data hours later.

RabbitMQ queue backlog growing: messages_ready climbing and how to drain it

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ queue backlog growing: messages_ready climbing and how to drain it

A queue’s messages_ready counter has been climbing for 20 minutes and it is not coming back down. Publishers are still publishing; upstream is fine. You need two answers fast: why consumers are not keeping up, and whether the backlog triggers a memory or disk alarm before you fix it.

Rising messages_ready means exactly one thing: messages arrive faster than they are delivered. The cause is one of three: consumers are absent, consumers are present but too slow, or consumers are present but stuck. The diagnostic work is separating those cases, because the fix differs for each and the wrong fix (restart the broker, purge the queue) can turn a slow drain into data loss.

RabbitMQ queue with zero consumers: zombie queues that grow until they OOM the node

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ queue with zero consumers: zombie queues that grow until they OOM the node

A queue with zero consumers and a non-zero publish rate is a memory leak with a routing key. Every published message lands in the queue, nothing drains it, and the backlog grows until something stops it: a TTL, a queue length limit, a manual purge, or the node’s memory alarm. If none of the first three happen, you find out when publishers across the whole cluster get blocked.

RabbitMQ quorum queue in minority: lost majority and read-only queues

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ quorum queue in minority: lost majority and read-only queues

You are looking at a queue with state: minority in the management API, or publishers are timing out on a queue that was healthy an hour ago. A quorum queue enters minority when fewer than a majority of its Raft members are online. Without a majority, the queue cannot elect a leader and cannot commit new writes.

This is not a queue-level bug. It is the direct, by-design consequence of a network partition or the loss of multiple cluster nodes. The queue is protecting your data by refusing writes it cannot replicate safely.

RabbitMQ quorum queue uncommitted Raft entries growing: followers falling behind

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ quorum queue uncommitted Raft entries growing: followers falling behind

You are watching rabbitmq_raft_log_uncommitted_entries climb on a quorum queue and it is not coming back down. In a healthy quorum queue this metric hovers near zero with brief spikes during publish bursts. Sustained growth means the leader is appending entries to its Raft log faster than a quorum of followers can replicate and acknowledge them.

This is a durability problem before it is an availability problem. Every uncommitted entry is a message that exists only on the leader. If the leader fails now, those messages are lost even though publishers may already have received confirms for some of them, depending on client behavior. A sustained count above roughly 1000 entries is a replication bottleneck that needs diagnosis, not observation.

RabbitMQ quorum queue WAL explosion: Raft log outpacing snapshots

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ quorum queue WAL explosion: Raft log outpacing snapshots

Every operation on a quorum queue is appended to a Raft write-ahead log on disk. That log is supposed to be bounded: the queue periodically snapshots its state and discards the log segments the snapshot made redundant. When appends outpace snapshots, the WAL grows without bound, the data partition fills, and the node trips the disk alarm.

The disk alarm is not a quorum-queue-local event. When free disk on any node drops below disk_free_limit, RabbitMQ blocks all publishers on all nodes in the cluster. Consumers keep draining, but nothing new comes in. A single hot quorum queue can halt message ingestion for the entire cluster, including classic queues and streams that share the node.

RabbitMQ redelivered messages climbing: nacks, requeues, and retry storms

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ redelivered messages climbing: nacks, requeues, and retry storms

The redeliver counter on your broker is climbing and it will not stop. Deliveries look healthy on the dashboard, throughput appears fine, but ack rate is flat or falling, and the same messages keep cycling through consumers. This is not message loss. Every one of those messages is being delivered, rejected or abandoned, requeued, and delivered again. You are paying full processing cost for zero net progress.

RabbitMQ RESOURCE_LOCKED - cannot obtain exclusive access to queue

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ RESOURCE_LOCKED - cannot obtain exclusive access to queue

Your application logs show a channel-level exception, the channel closes, and the consumer never comes back:

reply-code=405, reply-text=RESOURCE_LOCKED - cannot obtain exclusive access to locked queue 'reply.xyz' in vhost '/'

This is AMQP error 405, a channel exception raised when a connection tries to declare, consume from, purge, or delete an exclusive queue that is still owned by a different connection. Exclusive queues are bound to the lifecycle of the connection that declared them: only that connection may use them, and they are deleted when that connection closes or is lost. The error is not a capacity problem, not an alarm, and not a permissions failure. It is an ownership conflict.

RabbitMQ silent message loss: unroutable messages dropped when mandatory is false

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ silent message loss: unroutable messages dropped when mandatory is false

Your publisher is healthy. The broker is healthy. Publish rate is non-zero, connections are up, no alarms, no errors in the log. And yet downstream systems are missing data. Queue depth is flat. Deliver rate is zero. Nothing looks broken, because nothing is broken in the usual sense: RabbitMQ is doing exactly what it was told to do.

When a message is published to an exchange and no queue binding matches its routing key, the broker has nowhere to route it. If the publisher did not set mandatory=true (and most client libraries default it to false), the broker silently discards the message. No return, no log line, no queue growth, no error surfaced to the publisher.

RabbitMQ TLS certificate expiry: expired certs and handshake failures that lock everyone out

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ TLS certificate expiry: expired certs and handshake failures that lock everyone out

Every client that connects to your RabbitMQ cluster over TLS fails at the same time. No memory alarm, no disk alarm, no queue buildup, no crash. The broker process is running, the management API may even respond, but every AMQP connection attempt dies during the TLS handshake. If inter-node (Erlang distribution) TLS is also enabled, cluster nodes cannot talk to each other and CLI tools may fail to connect as well.

RabbitMQ unacknowledged messages growing: the pinned-memory leak behind most alarms

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ unacknowledged messages growing: the pinned-memory leak behind most alarms

The queue depth chart looks fine. messages_ready is flat or low. Then the memory alarm fires and every publisher in the cluster freezes. When you dig in, the culprit is almost always the same: messages_unacknowledged had been climbing for hours, quietly pinning RAM that RabbitMQ cannot page out.

Watching total queue depth while ignoring unacknowledged messages as a separate signal is the most common RabbitMQ memory monitoring mistake. A consumer holding 10,000 unacked messages with no acks for 5 minutes is the top precursor to memory alarms, and it is invisible if you only chart messages_ready.

RabbitMQ unroutable messages returned: mandatory publishes with no matching binding

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ unroutable messages returned: mandatory publishes with no matching binding

Your broker is accepting publishes, no queues are growing, and yet the return_unroutable counter is climbing. Messages are being published with mandatory=true to an exchange that cannot route them to any queue, and RabbitMQ is handing them back to the publisher instead of delivering them.

This is a configuration fault, not an infrastructure fault. The broker is healthy. The routing topology is not. Something changed: a binding was deleted, a queue was removed, an exchange was redeployed without its bindings, or a publisher is using a routing key that matches nothing. Because the publishers set mandatory=true, RabbitMQ is telling you exactly that messages have nowhere to go. The worse case is the default, mandatory=false, where unroutable messages are silently dropped and you see nothing at all.

RabbitMQ vhost stopped or partially available: queues unreachable on some nodes

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ vhost stopped or partially available: queues unreachable on some nodes

You are looking at GET /api/vhosts and one vhost does not report running on every node. Its cluster_state map shows stopped on one or more nodes, or nodedown, while the rest of the cluster looks fine. Applications that connect to the affected node and target that vhost fail to open connections, or their channels error out immediately.

This is a per-node condition, which is what makes it confusing. A vhost is not a single process with one state; it is a supervision tree that runs independently on each cluster node. When it shows stopped on one node, that node’s message store, vhost supervisor, and queue processes for that vhost are not running. Clients pinned to that node by a load balancer or connection string see failures, while clients on other nodes keep working.

RabbitMQ vm_memory_high_watermark: setting the memory alarm threshold correctly

Mon, 01 Jan 0001 00:00:00 +0000

RabbitMQ vm_memory_high_watermark: setting the memory alarm threshold correctly

Your publishers are blocked cluster-wide because one node crossed its memory watermark. Or the alarm is firing and clearing every few seconds, and every flap sends a wave of connection.blocked / connection.unblocked notifications through your clients. Both investigations land on the same setting: vm_memory_high_watermark.

This threshold is easy to get wrong in both directions. Set it too low and normal peak traffic trips a cluster-wide publishing halt. Set it too high (or inherit the new 0.6 default on a small node) and the OS OOM killer gets to the broker before the alarm ever fires, which is how you end up with Mnesia corruption instead of a clean circuit breaker.