RabbitMQ monitoring checklist: the signals every production broker needs

Most RabbitMQ incidents are not exotic. They are memory alarms nobody saw coming, disk alarms triggered by a 50MB default limit, unacked messages pinning RAM while the queue depth graph looked fine, and silent routing loss that only surfaced when a downstream team reported missing data. The broker exposed the signals the whole time. The gap is in what the team chose to collect and alert on.

This checklist organizes RabbitMQ monitoring into four maturity levels: survival, operational, mature, and expert. Each level assumes the previous one is in place. Scope: RabbitMQ 3.13+ with quorum queues as the recommended queue type. Classic mirrored queues are deprecated and not covered.

Use it two ways. As an audit: walk your current alerting against each level and note the gaps. As an on-call contract: the severity guidance per signal tells you what pages versus what tickets.

How the levels fit together

flowchart TD
  L1["Level 1 - Survival
node up, alarms, global depth, connections"]
  L2["Level 2 - Operational
rates, FD and socket ratios, per-queue state"]
  L3["Level 3 - Mature
run queue, churn, partitions, paging, redeliver"]
  L4["Level 4 - Expert
head age, consumer utilisation, GC, binary memory"]
  L1 --> L2 --> L3 --> L4

Two collection notes apply to every level. The Management API (port 15672) exposes most of what follows, but polling it faster than every 5 to 10 seconds adds measurable load to the broker. Message rates are cumulative counters, so your collector must compute derivatives; point-in-time samples hide brief spikes. Some signals (Raft commit latency, for example) are only available via the Prometheus endpoint, and socket usage is only available via the Management API. Plan on using both.

Level 1: survival

The minimum to know the broker is alive and not on fire. If any of these page, message ingestion is stopped or about to be.

  • Node running. Whether the Erlang node is up and responding. Check GET /api/nodes field running, or rabbitmq-diagnostics -q ping. The management plane can be down while AMQP on port 5672 still works, so probe the AMQP listener separately with a TCP connect. Page on sustained failure (>2 minutes), not a single sample, and ignore nodes with uptime under 300 seconds to exclude rolling restarts. Do not use /api/health/checks/is-in-service as a liveness check; it returns 503 intentionally during drains.
  • Memory alarm (mem_alarm). Boolean per node. When true on any node, all publishers on all nodes are blocked. This is a cluster-wide stop, not a node-local event. Page when sustained >60 seconds with evidence of traffic impact (publish rate was non-zero, or connections show blocked/blocking).
  • Disk free alarm (disk_free_alarm). Same semantics as the memory alarm: any node triggers it, all publishers halt. Consumers keep draining, which is your recovery path. Page with the same sustained-duration conditions.
  • Global queue depth (messages_ready + messages_unacknowledged). From GET /api/overview field queue_totals. Track both separately. Ready messages wait for delivery; unacknowledged messages are delivered but not acked, are pinned in RAM, and cannot be paged to disk. Ticket on sustained positive growth over 10+ minutes, not on absolute values.
  • Connection count. From object_totals.connections. A sudden drop is a client disconnect storm; a sudden spike is a deployment or a leak. Ticket on >50% deviation from rolling baseline in 5 minutes.

Level 2: operational

Level 1 plus the signals that tell you whether message flow is healthy and whether the node is approaching resource ceilings.

  • Message rates: publish, deliver_get, ack. From message_stats on /api/overview. Use deliver_get (deliver plus get) for comparisons against publish, because pull consumers only show up in get. Publish greater than deliver_get, sustained, means queues are growing. Deliver non-zero with ack at zero means consumers are receiving but not processing. These are counters; alert on the computed rate, using 1-minute averages because batch publishers create normal spikes.
  • File descriptor ratio (fd_used / fd_total). Each connection costs at least one FD, and queues with persistence cost more. Exhaustion is a cliff: new connections refused, paging can fail. Ticket at ratio >0.8 sustained. Both the OS limit and the Erlang limit must be raised together; raising one alone does nothing.
  • Socket ratio (sockets_used / sockets_total). Sockets are a subset of FDs with their own limit, typically about 90% of the FD total. You can exhaust sockets before FDs. Ticket at >0.8. Management API only.
  • Vhost status. GET /api/vhosts field cluster_state. Any node showing a vhost as stopped or down outside a maintenance window needs investigation.
  • Per-queue state. GET /api/queues field state. running and idle are normal. Page on crashed or terminated persisting across collection intervals on a queue with recent activity. Ticket on down (follows a node failure) and on minority for quorum queues, upgrading to page if the queue is known-critical.
  • Per-queue consumer count. Zero consumers on a queue with messages growing is a guaranteed backlog. This is the cheapest check that catches the most common incident class.
  • Unacked trend per queue. A consumer holding unacked messages at exactly its prefetch size, with ack rate zero, is stuck or deadlocked. This is the number one precursor to memory alarms.

Level 3: mature

Leading indicators and cluster-wide signals. This is where you stop reacting to alarms and start seeing them 10 to 60 minutes early.

  • Memory ratio (mem_used / mem_limit). The alarm fires at ratio 1.0, but paging starts at the paging ratio (default 0.5 of the watermark), which degrades throughput before anything blocks. Ticket at >0.7 sustained; treat consistent upward drift over hours as a capacity-planning item even below that. Do not multiply mem_limit by the watermark again; the API value already includes it. Version note: the default watermark changed from 0.4 of RAM in 3.x to 0.6 in 4.x.
  • Erlang run queue. The count of processes waiting for a scheduler, and the only direct measure of broker CPU saturation. Sustained run queue above the core count means every operation, including heartbeats, gets slower. High run queue during a GC pause is the mechanism behind false partition detection.
  • Connection, channel, and queue churn. From churn_rates. A stable connection count can hide massive open/close churn, and each connection setup costs a TLS handshake, AMQP negotiation, and an Erlang process. Ticket when creation rate exceeds 3x rolling baseline for 2+ minutes outside deploy windows. Queue creation persistently exceeding deletion over 30+ minutes is a queue leak.
  • Network partition status. GET /api/nodes field partitions. Any non-empty array sustained across two collection intervals, in a cluster, pages. The sustained condition matters: GC pauses exceeding net_ticktime (default 60s) cause transient false positives that self-heal. Corroborate with cluster link traffic dropping to zero and quorum queues entering minority.
  • Cluster link traffic. cluster_links recv/send bytes per peer. Zero traffic between previously communicating peers, corroborated by partition status, confirms isolation. Never alert on zero traffic alone; idle clusters legitimately have near-zero inter-node bytes.
  • Per-queue messages_paged_out. RabbitMQ moving queue contents to disk under memory pressure. This is the leading indicator before the memory alarm that most teams do not watch.
  • Redeliver rate. High redeliver with high deliver means consumers are receiving, failing, and receiving again. Net progress is zero; usually a poison message or a broken downstream dependency.
  • Return unroutable rate. Any sustained rate above zero is a routing misconfiguration. Caveat: it only fires for messages published with mandatory=true. The complementary pattern for silent loss is publish rate non-zero, deliver_get zero, queue depth flat.
  • Node uptime. Use it to gate other alerts. Suppress resource alarms for a warm-up window (5 to 30 minutes depending on workload) after any restart; memory spikes during queue repopulation are normal.
  • Flow control state. Connection state: flow is per-connection backpressure, surgical and transient. state: blocked or blocking means a resource alarm. Never alert on flow the way you alert on alarms. Ticket only when >10% of connections sit in flow for >5 minutes, which indicates a systemic write-path bottleneck. There is no aggregate metric; you must iterate /api/connections, so sample rather than poll aggressively.

Level 4: expert

The signals teams add after their third or fourth major incident. Each one exists because a specific failure class is invisible at Level 3.

  • Erlang process ratio (proc_used / proc_total). Every connection, channel, and queue is a process. Exhaustion crashes the node. Ticket at >0.8. Growth here is always a symptom; find what is spawning processes.
  • Disk free, absolute and trend. The alarm fires at disk_free_limit, but the default limit is 50MB, which is dangerously small on any real server. Set disk_free_limit.relative to 1.0-2.0 (1-2x RAM) or a floor of 2GB+, and alert when disk_free drops below max(3 * disk_free_limit, 1GB). Compute runway from the consumption rate, not the absolute value.
  • Head message age. head_message_timestamp per queue. Depth tells you volume; age tells you latency. A queue with 1,000 messages one second old is fine; 1,000 messages two hours old is an incident. Caveats: it depends on publishers setting the timestamp property, and it is absent on empty queues.
  • Consumer utilisation. consumer_utilisation per queue, 0.0 to 1.0. Consumer count tells you consumers exist; utilisation tells you they are effective. Below 0.5 sustained on a queue with a backlog means the consumers are the bottleneck. Only meaningful when messages_ready > 0.
  • Publisher confirm latency. A 5 to 10 minute warning before flow control and alarms. Not available broker-side; instrument it in the publisher client.
  • Erlang GC activity and binary memory. Binary heap can stay inflated after queues drain because of reference counting. rabbitmqctl eval 'erlang:memory(binary).' shows it; reported mem_used may not shrink immediately after a drain for this reason.
  • Per-queue memory contribution. Sort queues by memory to find the heaviest: GET /api/queues?sort=memory&sort_reverse=true. This is the first query to run during a memory alarm.
  • Raft health for quorum queues. Commit latency and leadership stability via the Prometheus endpoint, plus rabbitmq-diagnostics check_if_node_is_quorum_critical before any rolling restart.
  • The silent-loss composite. return_unroutable plus the pattern of zero depth, zero deliver_get, non-zero publish. Together they catch messages vanishing into exchanges with no bindings, the failure mode with the longest time-to-detection in most organizations.

Signals this checklist cannot give you

Some operationally critical signals are not in the Management API or standard endpoints and need external collection: AMQP listener health (active TCP probing of 5672/5671), TLS certificate expiry and handshake failures (certificate inspection and logs), protocol errors like PRECONDITION_FAILED and ACCESS_REFUSED (logs only), and authentication failure rates (log parsing, patterns like “authentication failed” and “Login failed” in /var/log/rabbitmq/[email protected]). There is also no broker-side CPU field; use OS-level CPU and the Erlang run queue as the proxy.

How Netdata helps

The hard part of this checklist is not collecting any single signal; it is correlating them fast enough to beat the cliff-edge transitions RabbitMQ is known for.

  • Netdata collects the Level 1 and Level 2 signals per node by default: node status, mem_alarm, disk_free_alarm, global messages_ready and messages_unacknowledged, connection count, publish/deliver_get/ack rates, and FD usage, so the survival and operational tiers work without custom instrumentation.
  • Because memory ratio, queue depth, and paging are on the same per-second timeline, you can see the actual cascade (depth rises, paging starts, ratio approaches 1.0) instead of discovering it when publishers block.
  • Per-queue collection surfaces per-queue depth, unacked counts, and redeliver rates, which is where Level 3 consumer-failure patterns live.
  • Alerting on ratios and trends rather than absolutes maps directly to the thresholds in this checklist: mem_used/mem_limit > 0.7, fd_used/fd_total > 0.8, sustained positive depth growth.
  • Uptime tracking lets you gate noisy resource alerts during post-restart warm-up, the most common source of false pages after rolling restarts.