RabbitMQ consumers connected but not acknowledging: the consumer black hole

The queue has consumers. The management UI shows consumers: 3. Yet messages_ready keeps climbing, the ack rate is flat at zero, and nothing is draining. Everything looks connected and nothing is working. This is the consumer black hole: dangerous because every surface-level health check passes.

The trap is confusing connection health with consumer health. An open TCP connection and a registered consumer tag do not mean messages are being processed. A consumer can be connected, have channels open, and be completely stuck: deadlocked, blocked on a failed downstream dependency, or a zombie where the application process died but the OS never closed the socket. The broker will hold unacknowledged messages for it, pin them in RAM, and wait. Left alone, this escalates into the memory wall: unacked messages are not paged out, memory grows, the watermark is crossed, and every publisher in the cluster is blocked.

This article is about finding the black hole fast and telling the three root causes apart, because the fix for each is different.

What this means

RabbitMQ tracks consumer reality through three independent signals, and the black hole is defined by their divergence:

  • consumers > 0 on the queue: consumer registrations exist.
  • ack rate at or near zero: no processing is completing.
  • messages_ready rising: work is accumulating.

The deliver rate splits the pattern in two, and this is the single most useful first observation:

flowchart TD
  A[consumers > 0, ack rate ~0] --> B{deliver rate?}
  B -->|near zero| C[Consumers not pulling]
  B -->|non-zero| D[Consumers pulling but not acking]
  C --> C1[Flow control on connection]
  C --> C2[Prefetch already exhausted]
  C --> C3[Zombie: process dead, socket open]
  D --> D1[Stuck on downstream dependency]
  D --> D2[Deadlock / infinite loop]
  D --> D3[Nack-requeue loop, poison message]
  C3 --> E[Unacked pinned in RAM]
  D1 --> E
  D2 --> E
  E --> F[Memory alarm: all publishers blocked]

If deliver is also zero, consumers are not even pulling messages: suspect flow control on the connection, exhausted prefetch slots against a backlog of unacked messages, or dead processes behind live sockets. If deliver is non-zero but ack is zero, consumers receive messages but never finish: suspect a downstream outage, a deadlock, or a nack loop where the same messages are redelivered and rejected forever. Check the redeliver rate to separate the last case from the others.

Common causes

CauseWhat it looks likeFirst thing to check
Downstream dependency failure (database, API, filesystem)Consumer process alive, near-zero CPU, unacked count static at exactly prefetch size, deliver may continue until prefetch fillsConsumer application logs and downstream dependency health
Deadlock or infinite loop in consumer codeSame as above, but dependencies are healthy; thread dump shows no progressThread dump / profiler on the consumer process
Zombie connection (process dead, TCP open)Consumers still registered, deliver rate zero, no consumer-side process exists on the client hostWhether the client host and process actually still exist
Nack/requeue loop (poison message)High deliver rate, near-zero ack rate, elevated redeliver rate, queue depth roughly stablePer-queue redeliver rate and consumer error logs
Flow control throttling the consumer’s connectionConnection shows state: flow; deliver stalls though consumer is healthy; often the consumer also publishes on the same connectionGET /api/connections connection states
Silent channel closure swallowed by client libraryConsumer process running, but its channel is gone on the broker; consumer count lower than the app expectsBroker log for channel.close events vs what the app believes
Prefetch misconfigurationUnacked pinned at exactly prefetch x consumer count; a small number of stuck consumers hoard the entire backlograbbitmqctl list_consumers prefetch values

Quick checks

All read-only. Run against any node; queue state is cluster-global.

# 1. Per-queue view: consumers, ready, unacked, utilisation
curl -s -u guest:guest http://localhost:15672/api/queues | \
  jq '.[] | {name, consumers, messages_ready, messages_unacknowledged, consumer_utilisation}'

# 2. Global rates: is ack actually zero, is deliver moving, is redeliver elevated?
curl -s -u guest:guest http://localhost:15672/api/overview | \
  jq '.message_stats | {publish, deliver, deliver_get, ack, redeliver}'

# 3. Detailed consumer view: which connections/channels, what prefetch, ack mode
rabbitmqctl list_consumers

# 4. Connection states: flow, blocked, blocking, running
curl -s -u guest:guest http://localhost:15672/api/connections | \
  jq 'group_by(.state) | map({state: .[0].state, count: length})'

# 5. Is this already escalating? Memory usage vs limit
curl -s -u guest:guest http://localhost:15672/api/nodes | \
  jq '.[] | {name, mem_used, mem_limit, mem_alarm}'

# 6. Recent channel/connection errors in the broker log
grep -i "channel.close\|connection.close\|error" /var/log/rabbitmq/rabbit@$(hostname).log | tail -30

What to look for:

  • Unacked == prefetch x consumers: classic stuck-consumer signature. Every consumer is holding a full prefetch buffer and not processing.
  • Unacked low, ready growing, deliver near zero: consumers are not pulling at all. Zombie connections or flow control.
  • Redeliver high with stable depth: nack loop, not a stuck consumer. See the poison message loop guide.
  • consumer_utilisation at 0.0 with ready > 0: the queue always has to wait for consumers, confirming consumers are the bottleneck rather than the broker. Note the field was called consumer_capacity in older versions and is absent for queues with no consumers.

How to diagnose it

  1. Split the pattern by deliver rate. From check 2: deliver near zero means “not pulling”; deliver positive with ack zero means “not finishing.” Everything downstream branches on this.
  2. Check redeliver. If redeliver is elevated, you are in a poison-message loop, not a black hole. The consumer is alive and rejecting; the fix is a delivery limit and a dead-letter exchange, not a consumer restart.
  3. Verify the consumers actually exist. rabbitmqctl list_consumers gives you the connection and channel behind each consumer. Take the peer host and port, then confirm the process is alive on that host. If the host was terminated (spot instance reclaim, SIGKILL, hard crash), the TCP connection can linger: the broker only detects the dead peer after the heartbeat timeout expires (default 60 seconds, negotiated with the client, detected after two missed heartbeats). Until then, messages already delivered sit unacked against a corpse.
  4. If the consumer process is alive, find what it is waiting on. Near-zero CPU plus static unacked means blocked on I/O: a database that is down, an HTTP call with no timeout, a filesystem hang. A thread dump or the consumer’s own logs will name the dependency. This is the most common production cause.
  5. Check for flow control on the consumer’s connection. If the same connection both publishes and consumes, credit-based flow control triggered by the publish path throttles the whole connection, including delivery. Look for state: flow on that connection. See connection in flow state and flow control vs resource alarms.
  6. Compare what the app believes with what the broker sees. If the client library swallowed a channel error, the application thinks it is consuming while the broker closed the channel long ago. Broker-side channel.close log events, matched against the app’s consumer count, expose this.
  7. Measure the escalation runway. Unacked messages are not paged out; they hold RAM until acked or requeued, and rising ready messages add their own memory pressure. Watch mem_used / mem_limit: if the ratio keeps climbing with paging active, you are on the path to a cluster-wide publisher halt. Paging to disk is your early warning; see paging messages to disk.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Per-queue ack rateGround truth that processing completesZero while consumers > 0
Per-queue deliver (deliver_get) rateSplits “not pulling” from “not finishing”Zero with ready growing
Per-queue redeliver rateDistinguishes nack loops from stuck consumersElevated with stable depth
messages_unacknowledgedHolds RAM and is not paged out; number-one precursor to memory alarmsStuck at prefetch x consumers, or growing
consumer_utilisation (formerly consumer_capacity)Consumer effectiveness, not just presence0.0 with messages_ready > 0
Connection statesFlow control or alarm involvement on consumer connectionsConsumer connections in flow
head_message_timestampLatency that depth alone cannot showHead age climbing while depth looks flat; see head message age
mem_used / mem_limitThe escalation pathRising steadily during a black hole
Connection churnReconnect loops from crashing consumersChurn high while consumer count looks stable

Fixes

Stuck on a downstream dependency

Restore the dependency or unblock the consumer (add timeouts, circuit breakers). Once the consumer resumes, it will drain its unacked backlog naturally. If the dependency will be down for a while, stop the consumers cleanly: closing channels requeues their unacked messages back to ready, where other consumers or a later restart can pick them up. A clean stop is far better than leaving a full prefetch buffer pinned per consumer.

Zombie connections

If the client process is dead but the broker still shows the connection, close it via the management API (DELETE /api/connections/{name}) or the management UI. This is disruptive to that connection by design: closing it requeues its unacked messages immediately instead of waiting for heartbeat detection. Longer term, configure AMQP heartbeats on the client: without them, detecting a dead peer depends on TCP timeouts, which can take far longer than the 60-second heartbeat default. Also check any load balancer in front of the broker: if the LB idle timeout is shorter than the heartbeat interval, it can silently kill the pipe while both ends believe they are connected.

Deadlock or infinite loop in consumer code

Capture a thread dump first (you will want it for the postmortem), then restart the consumer process. Restarting closes channels and requeues unacked messages. Do not restart the broker: it does not fix a client-side deadlock and it costs you in-memory state and a long queue recovery.

Flow control on a shared connection

Split publishing and consuming onto separate connections. Flow control is per-connection; a consumer that also publishes can have its deliveries throttled by its own publish path. This is a design fix, not a broker fix.

Nack/requeue loop

Not a black hole, but it presents identically at the ack-rate level. Configure a delivery limit (x-delivery-limit on quorum queues) and a dead-letter exchange so the failing message is routed away after N attempts instead of cycling forever. Details in the poison message guide.

Consumer timeout as a safety net

RabbitMQ’s consumer acknowledgement timeout (default 30 minutes, evaluated periodically) requeues messages from consumers that hold them too long. It has existed since RabbitMQ 3.8, but its enforcement scope has changed across versions. When the timeout fires, the consumer’s messages are requeued, which also means duplicate work if the consumer was merely slow. Thirty minutes is a long time for a queue that normally processes in milliseconds; if your consumers are fast, a shorter per-queue timeout (x-consumer-timeout queue argument or policy) limits how long a stuck consumer can hoard messages. Treat this as a backstop, not a fix: a consumer that regularly trips the timeout is a bug to fix, and a timeout that fires during legitimately slow processing causes redelivery and duplicates.

Prevention

  • Alert on the composite, not the parts. consumers > 0 AND ack rate ~ 0 AND messages_ready rising for 5+ minutes is the black hole signature. None of the three alone is actionable; together they are unambiguous.
  • Monitor messages_unacknowledged separately from messages_ready. Unacked is the memory-alarm precursor. A consumer holding thousands of unacked messages with no acks for 5 minutes should page before the memory watermark does.
  • Track consumer_utilisation on queues with backlog. Presence of consumers means nothing; utilisation tells you whether they are effective.
  • Set heartbeats explicitly on every client and keep the interval below any load balancer idle timeout on the path.
  • Add timeouts to every downstream call in consumer code. A consumer that can block forever will eventually block forever.
  • Separate publish and consume connections in apps that do both.
  • Configure delivery limits and a DLX on every queue with consumers, so one bad message cannot impersonate a dead consumer fleet.
  • Include consumer-side metrics (processing latency, error rate) in your dashboards. RabbitMQ cannot see inside your consumers; the broker-side signals in this article tell you that consumers are stuck, not why.

How Netdata helps

Netdata’s RabbitMQ collector pulls the management API signals that define this pattern, so you can correlate them on one dashboard instead of querying five endpoints during an incident:

  • Per-queue messages_ready vs messages_unacknowledged side by side, so unacked accumulation is visible instead of hidden inside total depth.
  • Per-queue publish, deliver, ack, and redeliver rates, so the deliver-zero-vs-ack-zero split and nack loops are distinguishable at a glance.
  • Per-queue consumer count and consumer utilisation, so “consumers attached but ineffective” shows up as a metric, not a guess.
  • Node memory usage against mem_limit plus alarm state, so you can see the black hole walking toward the memory wall and how much runway is left.
  • Connection and channel churn, which catches the crash-reconnect variant where consumer count looks stable but consumers are cycling.
  • Anomaly detection on ack rate, which flags the drop to zero even when absolute queue depth is still unremarkable.