RabbitMQ vm_memory_high_watermark: setting the memory alarm threshold correctly

Your publishers are blocked cluster-wide because one node crossed its memory watermark. Or the alarm is firing and clearing every few seconds, and every flap sends a wave of connection.blocked / connection.unblocked notifications through your clients. Both investigations land on the same setting: vm_memory_high_watermark.

This threshold is easy to get wrong in both directions. Set it too low and normal peak traffic trips a cluster-wide publishing halt. Set it too high (or inherit the new 0.6 default on a small node) and the OS OOM killer gets to the broker before the alarm ever fires, which is how you end up with Mnesia corruption instead of a clean circuit breaker.

Three things make this setting more subtle than it looks: the default changed from 0.4 in RabbitMQ 3.x to 0.6 in 4.x, the value is computed from detected RAM (which is not always your container limit), and the alarm has no hysteresis, so it flaps when usage hovers near the line. This guide walks through checking the effective value, choosing relative vs absolute, and leaving the right headroom.

What this means

vm_memory_high_watermark defines when the memory resource alarm fires. It can be a relative fraction of total detected RAM (0.6 = 60%) or an absolute value with units. When the node’s memory usage (mem_used) crosses the resulting limit (mem_limit), the node raises the memory alarm, and that alarm propagates to every node in the cluster: all publishers everywhere are blocked. Consumers keep working and can drain queues. This is the emergency circuit breaker described in RabbitMQ memory resource limit alarm; this article is about positioning the breaker itself.

Two properties of the mechanism drive most of the operational confusion:

• No hysteresis. The alarm clears at exactly the same value where it triggered. If usage oscillates around the watermark, the alarm flaps, and each transition churns publisher connections. There is no built-in dampening in current versions.
• The alarm is not a cap. The watermark only blocks publishers. It does not stop memory growth: unacked messages, connection churn, and GC transients can push the node past the watermark and into OOM territory even while the alarm is active.

One API detail that trips people up: mem_limit in the Management API already incorporates the watermark. Do not multiply it by the fraction again. To recover the RAM RabbitMQ detected, divide mem_limit by your configured fraction.

flowchart TD
  A[mem_used rises] --> B[paging zone
classic queues page to disk
disk I/O rises]
  B --> C[watermark reached
mem_used = mem_limit]
  C --> D[mem_alarm = true
all publishers blocked cluster-wide]
  D --> E{consumers drain?}
  E -->|yes| F[mem_used drops below watermark
alarm clears immediately]
  F -.->|no hysteresis: usage hovers| C
  E -->|no| G[memory keeps growing
OOM risk: alarm does not cap usage]

On 3.x with classic queues v1, paging to disk starts at vm_memory_high_watermark_paging_ratio (default 0.5 of the watermark), which is your earliest warning. In 4.x that knob went away with CQv1, but the pattern holds: rising disk I/O and messages_paged_out precede the alarm. For the broader resource model, see How RabbitMQ actually works in production.

Common causes

Quick checks

All read-only and safe to run during an incident.

# Check effective limit, current usage, and alarm state
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, mem_used, mem_limit, mem_alarm, ratio: (.mem_used / .mem_limit)}'

# List currently active alarms on the node
rabbitmqctl eval 'rabbit_alarm:get_alarms().'

# See what is actually holding memory (binary heap, queue procs, connections, mnesia)
rabbitmq-diagnostics memory_breakdown

# Identify the heaviest queues
curl -s -u guest:guest 'http://localhost:15672/api/queues?sort=memory&sort_reverse=true' | jq '.[] | {name, vhost, memory, messages_ready, messages_unacknowledged}'

# Compare RabbitMQ's view with the container limit
# cgroup v2:
cat /sys/fs/cgroup/memory.max
# cgroup v1:
cat /sys/fs/cgroup/memory/memory.limit_in_bytes

# Check whether the OOM killer has already visited the beam process
dmesg | grep -i "out of memory\|killed process"

How to diagnose it

1. Establish the effective threshold. Read mem_limit from GET /api/nodes. This is the real line the alarm fires at, regardless of what you think the config says. Divide it by your configured fraction to get the RAM RabbitMQ detected (e.g. mem_limit of 4.8 GiB at 0.6 means it detected 8 GiB).
2. Check detection against reality. Compare detected RAM to the node or container limit. If RabbitMQ running in a container detected host RAM, the relative watermark is meaningless: the kernel will OOM-kill the broker long before usage reaches the computed limit. Older releases did not reliably detect cgroup limits; verify after any runtime or version change rather than assuming.
3. Look at the shape of the trend. A slow, days-long climb in mem_used / mem_limit is workload growth or a leak. A vertical spike is a flood or consumer outage. A sawtooth right at the watermark is flapping, and the fix is headroom, not the alarm.
4. Attribute the memory. Use rabbitmq-diagnostics memory_breakdown and the heaviest-queues query above. Rising messages_unacknowledged is the dangerous case: unacked messages are pinned in RAM and cannot be paged out, so they push you through the watermark fast.
5. Decide which problem you have. If memory is consumed by legitimate backlog and stuck consumers, the watermark is doing its job and the fix is on the workload side: see RabbitMQ memory resource limit alarm. If the workload is healthy and the line itself is wrong for this node, fix the threshold as below.

Metrics and signals to monitor

Fixes

Set the watermark explicitly, relative

Do not inherit the default silently. Pin it in rabbitmq.conf:

# rabbitmq.conf
vm_memory_high_watermark.relative = 0.6

The documented production range is 0.4 to 0.7. The tradeoffs:

• Lower end (0.4-0.5): safer on small nodes and shared hosts; you give up usable RAM. The old 3.x default of 0.4 was considered overly conservative on modern Erlang/OTP, which is why 4.x moved to 0.6.
• Upper end (0.6-0.7): more headroom for message backlog, but less room for the Erlang VM’s transients. Erlang GC can transiently use up to roughly double the reported memory during collection, and RSS normally exceeds mem_used because the VM allocator retains freed memory. Above 0.7 you need careful monitoring and at least 30% of RAM left for the OS.
• Small nodes: on nodes with about 4 GB of RAM or less, the 0.6 default leaves thin absolute headroom, and an OOM kill risks Mnesia corruption and a cluster rebuild. Consider staying at 0.4-0.5 there.

Use an absolute value in containers

In containers, an absolute watermark removes the dependency on RAM detection:

# rabbitmq.conf - absolute limit sized to the container, not the host
vm_memory_high_watermark.absolute = 4GB

Set it below the container’s cgroup limit with room for non-VM overhead, and revisit it whenever the container size changes. Absolute values do not track resizes.

If you must keep a relative value but detection is wrong, total_memory_available_override_value pins the total RAM figure RabbitMQ uses for the calculation.

Change it at runtime during an incident

# Emergency runtime change: takes effect immediately, does NOT survive restart
rabbitmqctl set_vm_memory_high_watermark 0.4
# or as an absolute value
rabbitmqctl set_vm_memory_high_watermark absolute "4GB"

Use this to buy time (for example, lowering the threshold on a node that keeps OOM-killing before alarming, or briefly raising it while consumers catch up). Two caveats: the change is lost on restart, so follow it with a config change; and the command re-queries total system RAM, so on systems with hot-swappable memory even re-applying the same fraction can move the absolute limit.

Fix the memory calculation strategy

# rabbitmq.conf
vm_memory_calculation_strategy = rss

rss (resident set size) is the default on Linux and the most accurate reflection of what the OS will act on. The legacy erlang strategy underreports and is deprecated; if you inherited it, your alarm fires later than the OS view of memory would suggest.

Move off cgroup v1 for memory-heavy nodes

On cgroup v1 kernels, the page cache can be counted toward the container’s memory usage, producing false memory pressure and OOM kills, particularly with streams. Current guidance is to run on cgroup v2 (recent Kubernetes, or a distribution that defaults to it). If you are stuck on cgroup v1, prefer the rss strategy and validate against the cgroup limit directly.

Prevention

• Explicit configuration. Set the watermark in rabbitmq.conf on every node rather than relying on the default. The default changed between 3.x and 4.x, so upgrades silently move your alarm line.
• Absolute limits in containers. Use an absolute watermark or an override so the threshold tracks the container limit, not whatever RAM the host happens to have. Detection behavior is exactly the kind of thing that changes across runtime upgrades.
• Headroom budget. Keep peak mem_used / mem_limit below 0.6 in normal operation. This leaves room for traffic spikes, GC transients, and the gap between RSS and mem_used before you reach the cliff.
• Ratio alerting before alarm alerting. Alert on the ratio sustained above 0.7, and treat paging activity as an early warning. By the time the boolean alarm fires, publishers are already stopped.
• Flap-aware alarm alerts. Page only when mem_alarm has been active for more than 60 seconds and the node has been up for more than 10 minutes. The no-hysteresis design guarantees flapping near the line, and restart warm-up spikes are normal.
• Post-upgrade verification. After any RabbitMQ or Erlang/OTP upgrade, check mem_limit against expectations on every node. Defaults and detection behavior both change across versions.

How Netdata helps

Netdata’s RabbitMQ collector surfaces the signals this setting interacts with, per node and per second:

• Memory used vs limit as a continuous ratio, so you see the approach to the watermark minutes before the alarm instead of after.
• Alarm state over time, which makes flapping visible as repeated transitions rather than a single boolean.
• Queue depth and unacked messages alongside memory, so you can tell whether the watermark is being approached by backlog, by stuck consumers, or by something else.
• Paged-out message counts on classic queues, the leading indicator that the broker is already shedding memory pressure.
• Node uptime on the same dashboards, which lets you discount warm-up spikes after restarts without disabling alerts.

Because the watermark incident is always a correlation problem (memory plus queues plus consumers plus alarm state), having all four on one timeline is what shortens the diagnosis.

Related guides

• How RabbitMQ actually works in production: a mental model for operators
• RabbitMQ memory resource limit alarm: publishers blocked across the whole cluster
• RabbitMQ monitoring checklist: the signals every production broker needs
• RabbitMQ monitoring maturity model: from survival to expert