RabbitMQ paging messages to disk: the paging ratio as an early memory warning
RabbitMQ does not usually go from healthy to blocked in one step. Before the memory alarm blocks publishers cluster-wide, the broker can cross a quieter threshold: queues begin paging message contents out of RAM and onto disk.
That threshold is controlled by vm_memory_high_watermark_paging_ratio, which defaults to 0.5 of the effective memory limit. If vm_memory_high_watermark is 0.40 of RAM, paging starts near 20 percent of RAM. If the watermark is 0.60, paging starts near 30 percent. Check the configured values rather than inferring them from the RabbitMQ version.
The trigger is total broker memory, not queue payload memory alone. Connections, channels, metadata, plugin state, and other Erlang allocations can push the node across the threshold and force queues to page. Paged messages therefore do not prove that queue contents caused the pressure.
This is the leading indicator many teams miss. The alarm is a cliff edge. Paging is the slope before it: disk I/O rises, queue operations become more expensive, and credit flow control becomes more likely while mem_alarm is still false.
What paging is and why it matters
Paging moves queue message contents from RAM to disk so the broker can keep accepting work without immediately crossing the memory watermark. It is a safety valve, not a free lunch. The broker trades memory pressure for disk writes, disk reads, index work, and slower queue processes.
Queue metadata still has to track what exists, in what order, for which consumer, and with what acknowledgement state. Message contents are what can be pushed out when memory pressure rises.
The meaning depends on queue type. For classic queues under memory pressure, paging is an emergency storage path. For lazy queues and other disk-first storage behavior, disk residency can be normal rather than a sign of an imminent alarm. Do not compare the same paging field across queue types without checking what it represents.
Paging can also make the system look calmer than it is. Memory growth may level off while queue depth continues to rise. Publish rate may remain acceptable. Underneath, the broker is doing more I/O and each queue operation is becoming more expensive.
How it works
The memory alarm threshold is vm_memory_high_watermark. The paging threshold is below it:
paging threshold = vm_memory_high_watermark x vm_memory_high_watermark_paging_ratio
In the Management API, mem_limit already incorporates the watermark calculation. Do not multiply mem_limit by the watermark again. Read mem_used and mem_limit from the same node snapshot, then compare their ratio with the configured paging ratio.
These read-only checks establish the current alarm, memory, and connection state:
rabbitmq-diagnostics check_alarms
rabbitmq-diagnostics memory_breakdown
rabbitmqctl list_connections name state
The connection listing can return many rows on a large cluster.
A useful mental sequence is:
flowchart LR P[Publishers] --> Q[Queue processes] Q --> R[RAM for message contents] R -->|mem_used reaches paging ratio x mem_limit| D[Page contents to disk] D --> IO[Higher disk I/O and slower queues] IO --> F[More credit flow control] R -->|mem_used reaches mem_limit| A[Memory alarm] A --> B[Publishers blocked cluster-wide] C[Consumers drain and ack] --> R U[Unacked messages may remain resident for redelivery] -.-> R
Three consequences follow.
First, paging is pre-alarm behavior. Once mem_used / mem_limit reaches the configured paging ratio, the broker may already be changing its storage behavior.
Second, unacknowledged messages are not ordinary ready backlog. They must remain available for redelivery after channel or consumer failure, and depending on queue implementation they may remain resident. Rising unacked messages with a low ack rate deserve investigation even if ready messages are being paged successfully.
Third, paging is a performance event before it is an availability event. Consumers may still work, publishers may still connect, and dashboards may still be green while disk latency and queue process costs rise.
The paging ratio as an early warning
Treat the paging ratio as the first memory threshold that matters operationally. The alarm threshold is where damage becomes obvious. The paging threshold is where you still have options.
The following are working bands, not RabbitMQ-defined thresholds:
mem_used / mem_limit | Interpretation | Operational posture |
|---|---|---|
| Below the configured paging ratio | Headroom remains | Watch the trend and identify non-queue memory growth. |
| From the paging ratio to 0.7 | Paging is likely or underway | Find the queues, consumers, and non-queue allocations driving pressure. |
| Above 0.7 | Little headroom before the alarm | Check alarm state, connection state, drain rate, and recovery options immediately. |
mem_alarm = true | Memory alarm active | Publishers can be blocked cluster-wide. Treat it as an incident. |
The ratio alone is not enough. A node at the paging threshold with fast consumers and falling queue depth is different from a node at the same ratio with rising unacked messages, near-zero ack rate, and climbing disk latency. The ratio tells you where the node is on the memory curve. Queue and consumer signals tell you why it is there.
The alarm also has a cluster-wide effect. When any node crosses its memory alarm threshold, publishing can be blocked across the cluster. That makes the paging ratio on the weakest node a cluster-level early warning, not just a per-node tuning detail.
Where it shows up in production
The first pattern is stable memory with growing queue depth. Queue contents are being moved out of RAM, so memory no longer reflects the backlog. The broker has not fixed the publish-consume imbalance; it has moved the symptom from RAM to disk while mem_alarm remains false.
The second pattern is messages_paged_out greater than zero while traffic is active. On queue types where this metric represents memory-pressure paging, it shows that contents are no longer fully resident. Correlate it with messages_ready, messages_unacknowledged, publish and delivery rates, and disk I/O. Paged-out messages with falling depth and a healthy ack rate can indicate recovery. Paged-out messages with rising depth and weak acknowledgements indicate pressure is building.
The third pattern is disk I/O rising before any alarm. Operators often expect memory trouble to appear first as mem_alarm = true. Paging inverts that expectation. The earliest visible cost can be write latency, read latency, or slower queue processes, followed by credit flow control on publishing connections.
The fourth pattern is widespread flow control without a resource alarm. A connection in flow state is not the same as blocked: flow control is targeted and often transient. Sustained flow across many connections while paging is active means the write path is struggling. Do not wait for blocking or blocked to treat it as real.
What paging does not tell you
Paging is not a complete memory explanation. A node can remain below the paging ratio and still be unhealthy because memory is consumed by connections, channels, ETS tables, metadata stores, plugin state, or Erlang binary retention rather than pageable queue contents.
Paging also does not prove that consumers are healthy. Consumers can be connected and still make no progress: ack rate zero, redelivery rate high, consumer_utilisation low, or head message age rising. In those cases, paging can continue while the actual fault is downstream of the broker.
Do not assume messages_paged_out has identical semantics for every queue type and RabbitMQ version. Classic queues, lazy queues, quorum queues, and streams use different storage models. If your deployment uses classic queues v2, quorum queues, or streams, verify which paging fields are emitted and whether they represent emergency memory pressure or normal disk-first behavior.
Finally, do not use the paging ratio as a substitute for disk monitoring. Paging consumes disk. If disk_free_limit is too low relative to the partition, or if the partition itself is small, the attempt to avoid a memory alarm can leave too little runway before a disk alarm.
Signals to watch in production
| Signal | Why it matters | Warning sign |
|---|---|---|
mem_used / mem_limit | Shows position relative to the effective memory limit. | Sustained movement through the paging ratio and 0.7, especially with rising depth. |
mem_alarm | Indicates whether the memory alarm is active. | Any transition to true. |
messages_paged_out per queue | On supported queue types, exposes contents moved from RAM toward disk. | Greater than zero with active traffic and rising backlog. |
messages_ready | Backlog waiting for delivery. | Sustained growth while consumers are attached. |
messages_unacknowledged | Messages delivered but not acknowledged. | At or above prefetch times consumers, or growing with a low ack rate. |
Publish, deliver_get, and ack rates | Show whether the queue is draining or accumulating. | Publish exceeds delivery for minutes, or delivery is positive while acks are near zero. |
Connection state | Distinguishes credit flow control from resource-alarm blocking. | Many connections in flow for minutes, or movement toward blocking or blocked. |
disk_free versus disk_free_limit | Paging and persistence need disk runway. | Free space below the limit or falling faster than queues can drain. |
| Data-partition I/O latency and throughput | Paging turns memory pressure into storage work. | I/O rises as mem_used / mem_limit crosses the paging region. |
consumer_utilisation | Shows whether attached consumers can take work. | Low utilisation while messages_ready is above zero. |
head_message_timestamp, where available | Adds queue latency to queue depth. | Head age rising while depth is flat, indicating a blocked head message rather than simple backlog. |
The strongest composite is mem_used / mem_limit above the paging threshold, messages_paged_out present, disk I/O rising, delivery below publish, and connections increasingly in flow. Together, those signals say the broker is absorbing pressure now and will block later if consumers do not catch up.
A subtler composite is stable memory, growing depth, low ack rate, and no active alarm. Nothing is red yet, but pressure has moved somewhere less visible: disk, unacked state, or consumer-side processing.
How Netdata helps
- Plot
mem_used,mem_limit, alarm state, queue depth, and message rates on one timeline so the pre-alarm slope is visible before publishers stop. - Where per-queue paging fields are exposed, compare them with
messages_readyandmessages_unacknowledgedto separate normal disk-first behavior from memory-pressure paging. - Overlay publish, delivery, ack, and redelivery rates to determine whether paging is buying time for healthy consumers or hiding a consumer black hole.
- Watch disk free and data-partition I/O next to node memory, because paging can turn a memory incident into a disk runway problem.
- Use queue-level context selectively on large deployments. The heaviest queues usually justify the detail; collecting every queue at full detail can create unnecessary cardinality.