RabbitMQ connection blocked / blocking: publishers frozen by a resource alarm

Your publishers are hanging. No errors, no exceptions, no disconnections. Publish calls simply never return, and queue depth stops growing. When you check the management UI or rabbitmqctl list_connections, you see connections in state blocking or blocked instead of running.

This is not a network problem and not a client bug. RabbitMQ has raised a resource alarm (memory or disk) and has deliberately stopped reading from publisher connections across the entire cluster. It is a circuit breaker doing its job. The danger is that many client libraries hide this completely: from the application’s perspective, the publish call just hangs with no error, which sends operators chasing ghosts in the wrong place.

The distinction between the two states is narrow but useful:

• blocking: a resource alarm is active, but this connection has not attempted to publish yet. It can still do everything except publish.
• blocked: the connection attempted to publish while the alarm was active and is now frozen until the alarm clears.

Both states have the same root cause: a memory or disk alarm is active somewhere in the cluster. This article covers how to confirm that, how to find which alarm fired and why, and how to get publishers flowing again. For the alarm-specific deep dives, see the guides on the memory resource limit alarm and the disk free limit alarm.

What this means

RabbitMQ has two independent throttling mechanisms, and confusing them is the most common diagnostic mistake here:

1. Credit-based flow control (per-connection): internal backpressure. A fast publisher is temporarily throttled when a downstream queue or the persistence layer cannot keep up. The connection shows state: flow. This is surgical, transient (it can toggle many times per second), and healthy. It is the broker saying “slow down a little” to one publisher.

2. Resource alarms (cluster-wide): the emergency stop. When memory use crosses the high watermark (default 0.4 of RAM) on any node, or free disk drops below disk_free_limit (default 50MB) on any node, every publishing connection on every node is blocked. The connection shows blocking or blocked. Consumers are unaffected and keep draining.

Two properties of resource alarms catch operators off guard:

• They are cluster-wide. One node firing a memory alarm freezes publishers on all nodes, including nodes with plenty of headroom. Alert text that says “node X memory high” undersells it: the actual event is “cluster publishing halted.”
• They are cliff-edge. There is no gradual degradation. One second publishing works; the next, every publisher in the cluster is frozen. The only warning is the paging phase (queues writing messages to disk at the paging ratio, default 0.5 of the watermark) and the mem_used / mem_limit ratio approaching 1.0.

When a connection is blocked, the broker stops reading from that socket. The client’s TCP buffers fill, writes block, and the publish call hangs. Client libraries that support the connection.blocked / connection.unblocked protocol extension receive a notification, but many applications never register a handler for it, so the notification is silently dropped and the only visible symptom is the hang.

stateDiagram-v2
  running: running
  flow: flow (credit backpressure)
  blocking: blocking (alarm active, no publish attempted)
  blocked: blocked (publish attempted, frozen)
  running --> flow: downstream queue/disk slow
  flow --> running: downstream caught up
  running --> blocking: memory or disk alarm fires
  blocking --> blocked: app calls publish
  blocked --> running: ALL alarms clear
  blocking --> running: ALL alarms clear

Common causes

The state on the connection is a symptom. The cause is always an active alarm, and the alarm always has an underlying driver.

Quick checks

All read-only. Run these against any node in the cluster; remember the alarm may originate on a different node. The examples use guest:guest, which the management API only accepts from localhost by default.

# 1. Count connections by state
curl -s -u guest:guest http://localhost:15672/api/connections \
  | jq 'group_by(.state) | map({state: .[0].state, count: length})'

# Same via CLI (-q strips the banner and table headers)
rabbitmqctl -q list_connections state | sort | uniq -c

# 2. Check both alarm flags on every node
curl -s -u guest:guest http://localhost:15672/api/nodes \
  | jq '.[] | {name, mem_alarm, disk_free_alarm}'

# 3. Ask the alarm subsystem directly
rabbitmqctl eval 'rabbit_alarm:get_alarms().'

# 4. Memory headroom per node
curl -s -u guest:guest http://localhost:15672/api/nodes \
  | jq '.[] | {name, mem_used, mem_limit, ratio: (.mem_used / .mem_limit)}'

# 5. Disk headroom per node
curl -s -u guest:guest http://localhost:15672/api/nodes \
  | jq '.[] | {name, disk_free, disk_free_limit}'

# 6. Confirm publishing actually stopped (rates, not counters)
curl -s -u guest:guest http://localhost:15672/api/overview \
  | jq '{publish_rate: .message_stats.publish_details.rate, deliver_rate: .message_stats.deliver_get_details.rate, ack_rate: .message_stats.ack_details.rate}'

# 7. Find the queues holding memory
curl -s -u guest:guest 'http://localhost:15672/api/queues?sort=memory&sort_reverse=true' \
  | jq '.[:10][] | {name, messages_ready, messages_unacknowledged, consumers}'

Reading the results:

• If check 1 shows blocked or blocking connections and check 2 shows an alarm flag true on any node, the diagnosis is confirmed. The connection state is a symptom; go after the alarm.
• If check 6 shows publish rate at zero while deliver and ack rates are non-zero, that is the classic signature: publishers frozen, consumers still draining.
• If check 1 shows flow instead, no alarm is active and you are looking at ordinary credit-based backpressure, a different and much less severe problem.

How to diagnose it

1. Confirm the state. Count connections by state (check 1). A large population of blocked or blocking connections with stable total connection count is the fingerprint: clients are connected but frozen.

2. Identify which alarm fired. Check mem_alarm and disk_free_alarm on every node (check 2), not just the node your clients connect to. The alarm propagates cluster-wide; the node that fired it may not be the node you are looking at.

3. Rule out flow control. If connections show flow and no alarm flags are set, you are dealing with per-connection backpressure: a slow queue, slow disk, or saturated write path. That is a throughput problem, not an emergency stop.

4. If memory alarm: find the driver. Look at global messages_ready and messages_unacknowledged. Rising unacked with consumers present and ack rate near zero means stuck consumers. Rising ready with zero consumers on deep queues means absent consumers. Use check 7 to find the queues contributing the most memory.

5. If disk alarm: verify against the OS. Run df -h on the RabbitMQ data partition (typically /var/lib/rabbitmq). If the OS shows plenty of free space but the alarm is active, check disk_free_limit: if it is still the default 50MB, routine events like log rotation or a burst of persistent messages can trip it. See the guide on disk_free_limit at the default 50MB.

6. Check the client side. If the application shows no error at all, verify whether the client library supports connection.blocked notifications and whether the application registered a handler. Without a handler, the freeze is invisible to the app. Also check for TCP-level hangs: a publisher blocked mid-write will sit in a socket write until the alarm clears.

Metrics and signals to monitor

Fixes

Get publishing flowing again (memory alarm)

The alarm clears only when memory drops below the watermark. Consumers are not blocked, so the fastest recovery path is to let them drain:

• Restore or fix consumers. If consumers are dead, redeploy them. If they are stuck on a downstream dependency, fix the dependency. Every acknowledged message frees memory.
• Purge non-critical queues as emergency relief if consumers cannot be restored quickly. This is destructive: messages in the purged queue are gone. Confirm the queue is expendable first.
• Do not restart the broker. A restart loses in-memory messages and the repopulation spike after restart often re-fires the alarm immediately, leaving you worse off.

Get publishing flowing again (disk alarm)

• Free space on the data partition: rotate or remove old logs, remove Erlang crash dumps, move unrelated data off the partition.
• Let consumers drain. Consumers keep working during a disk alarm, and draining persistent messages frees disk.
• If disk_free_limit is the default 50MB and the disk is large, raise it deliberately, not just to clear the incident: set disk_free_limit.relative to 1.0 to 2.0 (1-2x RAM) or an absolute value of 2GB or more. Then treat the default as a misconfiguration to fix permanently, because it will fire again.

If the alarm flaps

There is no hysteresis: the alarm clears at the same threshold it fires at. If usage hovers at the watermark, connections oscillate between blocked and running, which is often worse for publishers than a clean stop. Fix the headroom problem (lower usage or raise the threshold with intent) rather than waiting it out.

Handle the application side

A client that hangs silently on publish is an operational liability even after the broker recovers:

• Register connection.blocked / connection.unblocked handlers where the client library supports them, and surface them as application metrics or log events.
• Add a client-side safeguard for indefinite blocks, such as a publish timeout or a blocked-connection timeout, so a frozen publisher fails fast and alerts instead of hanging a worker pool forever.

Prevention

• Monitor the alarm inputs, not just the alarm. Alert on mem_used / mem_limit > 0.7 and on disk_free below 3x disk_free_limit (with a 1GB absolute floor). The alarm itself is the cliff edge; the ratios are the warning. See the monitoring checklist.
• Watch paging as the early warning. messages_paged_out appearing under load means the broker is preparing for memory trouble 5-10 minutes before the alarm fires. See the guide on paging messages to disk.
• Track unacked messages separately from ready messages. Unacked accumulation is the most common memory alarm driver and is invisible if you only watch total depth.
• Fix the 50MB default on any production node before it fires for the first time.
• Alert on the right text. A memory or disk alarm means “cluster publishing halted,” not “node X resource high.” Operators should know the blast radius from the alert alone.
• Distinguish flow from blocked in dashboards and alerts. Alerting on flow the same way as blocked produces noise and trains people to ignore the state field.

How Netdata helps

• Netdata collects the node-level alarm flags (mem_alarm, disk_free_alarm) alongside the ratios that predict them (mem_used vs mem_limit, disk_free vs disk_free_limit), so you can see the approach to the cliff, not just the fall.
• Correlating the publish-rate drop against deliver and ack rates on the same dashboard confirms the alarm-freeze signature in seconds: publish at zero, consumers still draining.
• Per-queue ready, unacknowledged, and paged-out counts sit next to the alarm state, which shortens the path from “publishers frozen” to “this queue’s consumers are stuck.”
• Alerts on the resource ratios fire before the alarm, which is the only useful place to catch a cliff-edge event.

Related guides

• RabbitMQ memory resource limit alarm: publishers blocked across the whole cluster
• RabbitMQ disk free limit alarm: free disk space insufficient and publishing halted
• RabbitMQ disk_free_limit at the default 50MB: the production footgun
• RabbitMQ paging messages to disk: the paging ratio as an early memory warning
• How RabbitMQ actually works in production: a mental model for operators
• RabbitMQ monitoring checklist: the signals every production broker needs
• RabbitMQ monitoring maturity model: from survival to expert