RabbitMQ redelivered messages climbing: nacks, requeues, and retry storms

The redeliver counter on your broker is climbing and it will not stop. Deliveries look healthy on the dashboard, throughput appears fine, but ack rate is flat or falling, and the same messages keep cycling through consumers. This is not message loss. Every one of those messages is being delivered, rejected or abandoned, requeued, and delivered again. You are paying full processing cost for zero net progress.

A rising redeliver rate always means the same thing: messages are coming back. The consumer nacked with requeue=true, the channel or connection died with unacked messages outstanding, or the consumer timeout fired and the broker reclaimed the deliveries. What varies is why, and whether the loop involves one poison message or the entire traffic of a queue.

This guide covers how to localize the failing queue, distinguish the three mechanisms, and break the loop without losing messages. For the broader signal model, see How RabbitMQ actually works in production.

What this means

When a delivery cannot be completed, one of three things happens:

  1. Explicit negative acknowledgement. The consumer calls basic.nack or basic.reject with requeue=true. The broker puts the message back (at or near its original position, close to the head) and redelivers it, possibly to the same consumer.
  2. Channel or connection death. The consumer process crashes, the socket drops, or the client library closes the channel. Every unacked message on that channel is requeued in one burst. With high prefetch, that burst can be hundreds or thousands of messages at once.
  3. Consumer timeout. If a consumer holds a delivery longer than consumer_timeout (introduced in 3.12, default 30 minutes), the broker closes the channel and requeues the unacked messages.

In all three cases the redeliver counter increments and the message gets the redelivered flag on its next delivery. The dangerous pattern is “high deliver with equally high redeliver”: the queue looks busy, consumers look busy, but net progress is zero. Nothing alarms on this by default.

Redelivery is not inherently bad. A consumer restart during a deploy produces a one-time requeue burst and that is normal. The signal you care about is a sustained redeliver rate, especially one concentrated on a single queue.

Common causes

CauseWhat it looks likeFirst thing to check
Poison messageDepth stable, redeliver elevated on one queue, head message age growing, unacked pinned at a small numberConsumer logs for a repeating exception on the same payload
Consumer crash loop or deploy churnRedeliver spikes in bursts matching connection/channel churn; requeues come in large batchesConnection churn rate and consumer count over time
Consumer timeout firingRedeliver steps up every N minutes; channel closed errors in broker log; slow downstream dependencyBroker log for consumer timeout channel closures; consumer processing latency
Downstream dependency failureAck rate collapses across many queues at once, redeliver rises everywhere, consumers throwing the same errorHealth of the database/API the consumers call
Prefetch too high with slow consumersLarge unacked counts; one consumer crash requeues thousands; repeated retry stormsmessages_unacknowledged vs prefetch x consumer count
Nack-with-requeue retry logic in the appRedeliver rate roughly equals deliver rate; no DLX configured; loop never terminatesConsumer code path for failure handling

Quick checks

All read-only.

# Global redeliver counter and related message stats
curl -s -u guest:guest http://localhost:15672/api/overview | jq '.message_stats | {publish, deliver_get, ack, redeliver}'

If redeliver is a significant fraction of deliver_get (say, more than a few percent sustained), you have an active loop. These are cumulative counters: sample twice and compute the rate.

# Localize: per-queue redeliver, depth, unacked, consumers
curl -s -u guest:guest http://localhost:15672/api/queues | \
  jq '.[] | {name, messages_ready, messages_unacknowledged, consumers,
      redeliver: .message_stats.redeliver, ack: .message_stats.ack}'

One queue with a high redeliver count and stable depth points to poison messages. Many queues with rising redeliver point to a shared cause (downstream dependency, mass consumer failure).

# Head message age on the suspect queue (poison message signature)
curl -s -u guest:guest http://localhost:15672/api/queues | \
  jq '.[] | {name, head_message_timestamp, messages_ready}'

Caveat: head_message_timestamp depends on publishers setting the timestamp property, and may be absent or null for empty queues or some queue types.

# Consumers actually attached and their prefetch
rabbitmqctl list_consumers

# Broker log: channel closures from consumer timeout or consumer errors
grep -i "closing AMQP connection\|consumer.*timeout\|channel.*error" /var/log/rabbitmq/rabbit@$(hostname).log | tail -50

How to diagnose it

  1. Confirm the loop is real. Sample message_stats.redeliver twice, 60 seconds apart, from /api/overview. A rate near zero means you are looking at a historical burst, not a live problem.

  2. Find the queue. Compare per-queue redeliver counters. Poison message cases concentrate almost all redelivery on one queue. Systemic cases spread it across queues that share a consumer codebase or downstream dependency.

  3. Check the shape of the queue. On the suspect queue, look at messages_ready, messages_unacknowledged, consumers, and head message age together:

    • Stable depth + growing head age + elevated redeliver = poison message at or near the head blocking progress.
    • Growing depth + unacked pinned at prefetch x consumers + redeliver bursts = consumers crashing or timing out with full prefetch buffers.
    • Depth draining normally but redeliver still nonzero = a fraction of messages failing; likely a data-dependent consumer bug, not a single blocker.

  4. Read the consumer logs. The broker only tells you that messages came back. The reason (deserialization failure, null pointer, downstream 500, lock timeout) is always on the consumer side. Match exception timestamps against redeliver rate steps.

  5. Check for consumer timeout. If your consumers do slow work (large batches, slow queries), compare worst-case processing time against consumer_timeout (default 30 minutes since 3.12). A timeout close requeues everything the consumer held and, on most client libraries, kills the channel, which the application then has to recover. See RabbitMQ consumer timeout.

  6. Correlate with churn. If redeliver spikes align with connection or channel churn, the requeues are a symptom of consumer instability, not bad messages. Fix the crash loop first; the redeliver rate will follow it down.

flowchart TD
  A[Redeliver rate sustained above baseline] --> B{Concentrated on one queue?}
  B -->|Yes| C{Depth stable, head age growing?}
  B -->|No, many queues| F[Check shared downstream dependency and consumer fleet health]
  C -->|Yes| D[Poison message: inspect head message, nack requeue=false to DLX]
  C -->|No, unacked pinned at prefetch| E[Consumer crash or timeout: check logs and consumer_timeout]
  D --> G[Fix consumer handling, set x-delivery-limit on quorum queues]
  E --> G
  F --> G

Metrics and signals to monitor

SignalWhy it mattersWarning sign
message_stats.redeliver (global and per-queue)The direct measure of the loopSustained rate above a few percent of deliver_get
Ack rate vs deliver rateDivergence means deliveries are not completingAck < 90% of deliver sustained
messages_unacknowledgedIn-flight messages that will requeue en masse if the consumer diesPinned at prefetch x consumers; sudden drops followed by redeliver bursts
Head message ageThe poison message signature that depth cannot showAge growing while depth is stable and redeliver is elevated
Consumer count per queueA drop forces mass requeue of unacked messagesCount falling below expected minimum
Connection/channel churn ratesCrash loops produce requeue burstsCreation rate > 3x rolling baseline
DLX queue depth and publish rateWhere failed messages should be going instead of loopingDLX empty while redeliver is high (nothing is dead-lettering)

Fixes

Poison message

The loop exists because the consumer nacks with requeue=true (or crashes) on a message it can never process. Break the cycle:

  • Dead-letter the message. Consumers should nack with requeue=false for unprocessable messages, with a dead-letter exchange configured on the queue so the message is preserved for inspection instead of discarded.
  • Set a delivery limit on quorum queues. x-delivery-limit caps how many times a message is redelivered before it is dropped or dead-lettered. This is the built-in poison message protection and should be on every quorum queue that does not have it. Classic queues have no equivalent; there is no delivery count tracking, so protection must live in the consumer or an external counter.
  • Move the blocker manually during an incident. If one message is blocking the head and you cannot patch the consumer immediately, consume or route that message out of the queue so the backlog behind it can drain. Inspect it before discarding.

Consumer crashes and timeout requeues

  • Fix the crash, not the symptom. Mass requeues from dying consumers stop when the consumer stops dying. Redeliver rate is your confirmation metric after the fix ships.
  • Raise consumer_timeout only if processing is genuinely long. The timeout exists to reclaim deliveries from stuck consumers. Raising it to accommodate slow processing defers the requeue but also defers detection of a truly stuck consumer. Prefer making processing faster or splitting the work.
  • Tune prefetch down. High prefetch plus a crash means thousands of messages requeue at once and slam the surviving consumers, which can cascade into another crash round. A prefetch in the low tens (or single digits for slow work) bounds the blast radius of any single consumer failure. See RabbitMQ consumer_utilisation low for the delivery-side effects of prefetch.

Retry storms from application-level nack loops

If the consumer nacks with requeue=true on transient errors (database hiccup, rate-limited API), a dependency outage converts into a redelivery storm that adds load exactly when the system is least able to absorb it. The fix is backoff and eventual dead-lettering: either retry with delay at the application level, or dead-letter after a bounded number of attempts and re-drive the DLQ when the dependency recovers. Immediate requeue with no delay is the worst option because the message returns to near the head of the queue and can be redelivered to the same consumer instantly.

Prevention

  • Every quorum queue gets x-delivery-limit and a DLX. This converts infinite loops into a bounded failure with evidence preserved. Watch the DLQ depth; a growing DLQ is the same incident, but observable and non-blocking.
  • Alert on the redeliver ratio, not the raw count. Sustained redeliver / deliver_get above a small threshold (start at 2-5%) on any production queue is worth a ticket. Absolute counts mean nothing at varying traffic levels.
  • Size prefetch to bound requeue bursts. prefetch x consumers is the maximum number of messages that can requeue in a single failure event. Keep that number small relative to what the fleet can reprocess.
  • Make consumers idempotent. Redelivery is a normal part of at-least-once delivery (deploys, failovers, timeouts). Consumers that deduplicate by message ID or business key turn redeliveries from duplicate work into no-ops.
  • Track head message age on latency-sensitive queues. It catches the poison-message case (stable depth, growing age) that depth alerts miss entirely. See RabbitMQ head message age.

How Netdata helps

  • Per-queue redeliver rate alongside deliver and ack rates, so the “high deliver, high redeliver, zero progress” pattern is visible as diverging lines on one chart instead of three separate counters.
  • Queue depth and unacked counts on the same node dashboard, letting you correlate a redeliver spike with unacked accumulation (prefetch storm) versus stable depth (poison message) without switching tools.
  • Head message age per queue for the queues where publishers set timestamps, surfacing the stable-depth, growing-age signature directly.
  • Connection and channel churn charts, so you can see whether redeliver bursts align with consumer restarts or crash loops in the same time window.
  • Alerts on rate ratios and trends rather than absolute counters, which is what a redelivery loop actually requires to avoid noise at varying traffic levels.