RabbitMQ dead-letter queue growing: TTL expiry, max-length overflow, and rejections

Your dead-letter queue (DLQ) has 400,000 messages and no consumer. Nothing paged you, because RabbitMQ does not alarm on a DLQ by default. But every message in that queue is a record of something that already failed upstream: a consumer that rejected a delivery, a message that sat past its TTL, or a queue that overflowed its length limit and pushed the oldest message out.

A growing DLQ is a ledger of application failures. Read it as one. The first diagnostic move is not “why is the DLQ big” but “what does the x-death header say.” Every dead-lettered message carries an x-death header recording the queue it died in and the reason: expired, maxlen, rejected, or (for quorum queues with a delivery limit) delivery_limit. That reason field splits the investigation into completely different paths.

Left unmonitored, the DLQ also becomes the incident itself. It consumes memory and disk like any other queue, and in the worst case it participates in a routing loop where dead-lettered messages cycle back to their origin forever.

What this means

Dead-lettering is RabbitMQ’s rerouting of messages that can no longer live in their original queue. When you configure x-dead-letter-exchange on a queue, any message that is rejected without requeue, expires via TTL, or is pushed out by a length limit is republished to the dead-letter exchange (DLX) instead of being discarded. The DLX routes it, usually to a dedicated DLQ.

flowchart LR
  P[Publisher] --> Q[Origin queue]
  Q -->|nack/reject requeue=false| X[Dead-letter exchange]
  Q -->|TTL expired| X
  Q -->|max-length drop-head| X
  Q -->|delivery_limit exceeded| X
  X --> DLQ[Dead-letter queue]
  DLQ -.->|misconfigured DLX on DLQ| Q

Three things matter operationally:

  • The DLQ depth and its publish rate tell you failure volume over time, per origin.
  • The x-death header’s reason and count fields tell you the failure mode and how many times the message has been dead-lettered.
  • A DLQ with no consumer, no TTL, and no length limit grows unboundedly and eventually contributes to memory or disk pressure like any other backlog.

Common causes

CauseWhat it looks likeFirst thing to check
TTL expiry (expired)DLQ grows steadily; origin queue depth stays flat because messages age outx-death reason = expired; compare origin queue TTL against actual consumer latency
Max-length overflow (maxlen)DLQ grows during traffic bursts; origin queue pinned at its limitx-death reason = maxlen; check rabbitmqctl list_policies for max-length / max-length-bytes
Consumer rejections (rejected)DLQ grows in lockstep with a consumer error rate spikex-death reason = rejected; read consumer logs for the nack/reject path
Delivery limit exceeded (delivery_limit)Quorum queue poison messages dead-letter after N redeliveriesx-death reason = delivery_limit; check x-delivery-limit on the origin queue
DLX-to-origin loopSame messages cycle; x-death count climbs; depth oscillates between queuesInspect x-death array for repeated {queue, reason} entries
No DLQ consumer at allDLQ depth only ever increases, regardless of reasonrabbitmqctl list_queues name consumers for the DLQ

Quick checks

All read-only and safe to run during an incident. The guest:guest credentials below only work from localhost; substitute a monitoring user for remote access.

# 1. Find DLQs and their depth, unacked, and consumer counts
rabbitmqctl list_queues name messages_ready messages_unacknowledged consumers | grep -i -E "dlx|dlq|dead"

# 2. Check the publish rate into the DLQ (failure arrival rate)
curl -s -u guest:guest 'http://localhost:15672/api/queues/%2f/my-dlq' | jq '{messages_ready, consumers, publish_rate: .message_stats.publish_details.rate}'

# 3. See which policies drive dead-lettering and length limits
rabbitmqctl list_policies

# 4. Sample messages from the DLQ and inspect x-death headers
# ack_requeue_true only peeks; ack_requeue_false would remove them
curl -s -u guest:guest -X POST 'http://localhost:15672/api/queues/%2f/my-dlq/get' \
  -H 'content-type: application/json' \
  -d '{"count":5,"ackmode":"ack_requeue_true","encoding":"auto"}' | jq '.[].properties.headers["x-death"]'

# 5. Check per-queue redeliver rates on origin queues (rejection loops upstream)
curl -s -u guest:guest http://localhost:15672/api/queues | jq '.[] | {name, redeliver: .message_stats.redeliver_details.rate, ack: .message_stats.ack_details.rate}'

The x-death header is an array compressed by {queue, reason} pair. Each entry carries the reason, the original queue, the routing keys, a timestamp, and a count of how many times the message died that way in that queue. The broker also stamps x-first-death-reason / x-first-death-queue (immutable, recorded at the first dead-lettering) and x-last-death-reason / x-last-death-queue (updated each time). First-death tells you where the trouble started; last-death tells you where it most recently surfaced.

One behavior note if you rely on count for retry bookkeeping: from RabbitMQ 3.13.0 the broker no longer increments the count when a message is dead-lettered repeatedly from the same queue for the same reason. Older versions incremented on every event. Check this before building alerting on count values.

How to diagnose it

  1. Confirm the growth rate. Check the DLQ’s publish rate, not just its depth. Depth is a stock; the publish rate is the flow of failures. A DLQ at 500,000 messages with a publish rate of zero is a backlog to drain; one with a publish rate of 200/s is an active fire upstream.

  2. Read the reason. Sample a handful of messages with the /get endpoint using ack_requeue_true and look at x-death and x-first-death-reason. This single field determines everything that follows.

  3. If expired: compare the origin queue’s TTL (policy or per-message expiration) against real consumer latency. If consumers routinely take longer than the TTL to reach a message, messages expire before processing and dead-letter. Either the TTL is too tight for the workload, or consumers are too slow. Note that when a message dead-letters due to TTL, the broker removes the original expiration and records it separately so it does not immediately re-expire in the DLQ.

  4. If maxlen: the origin queue hit max-length or max-length-bytes. The default overflow behavior is drop-head: the oldest ready message is pushed out (and dead-lettered, if a DLX is set) to make room. Length limits count only ready messages, not unacknowledged ones, so a queue at its limit can mean consumers are behind or absent. Check whether the overflow is a safety valve doing its job during a burst, or a constant leak meaning the limit is wrong for the traffic.

  5. If rejected: a consumer called basic.nack or basic.reject with requeue=false. This is an application decision. Correlate the dead-letter arrival time with consumer logs and error rates. A spike here usually means a schema change, a poison message class, or a downstream dependency failure that makes the consumer give up on messages.

  6. If delivery_limit: a quorum queue redelivered the message past its x-delivery-limit (default 20 in RabbitMQ 4.x; unlimited in 3.13 unless explicitly set). This is the poison-message path working as designed. The question is why the message keeps failing: check consumer exception logs for the payload in question.

  7. Rule out a loop. If the same messages appear to cycle, look for a DLX on the DLQ itself that routes back toward the origin. RabbitMQ detects cycles and drops a message only if no rejection occurred anywhere in the cycle; if any rejection is in the path, the cycle is allowed to continue. A loop with a rejection in it never terminates on its own.

  8. Check the DLQ’s own hygiene. Consumers? TTL? Length limit? A DLQ with none of these is a queue that can only grow.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
DLQ messages_ready depthThe accumulated failure ledgerAny sustained positive trend
DLQ publish rateThe live failure arrival rate per originNon-zero when the application baseline says zero
x-death reason distributionSplits the root cause: TTL vs overflow vs rejection vs delivery limitA shift in the dominant reason after a deploy
Origin queue redeliver rateConsumers receiving, failing, and re-receivingHigh redeliver with near-zero ack = poison loop upstream
Origin queue depth at its max-lengthExplains maxlen dead-letteringDepth pinned at limit while publish continues
mem_used / mem_limit and disk_freeAn unbounded DLQ eventually pressures broker resourcesDLQ depth climbing alongside memory ratio
DLQ consumer countWhether anyone is draining the ledgerZero consumers on a DLQ with a non-zero publish rate

Fixes

TTL expiry

Loosen the TTL if it was set defensively and is now tighter than real end-to-end latency, or speed up consumers if the latency is the actual problem. If some messages legitimately expire (time-bound offers, stale commands), the DLQ traffic is expected: give the DLQ a consumer or a retention policy so the ledger drains. Do not fix this by raising TTL blindly; an over-long TTL just converts expirations into queue depth and memory pressure.

Max-length overflow

Decide what the limit is for. If max-length exists to cap backlog during consumer outages, dead-lettering on overflow is the design working. If the DLQ grows during normal traffic, the limit is undersized for bursts, or drop-head is silently discarding data you cannot afford to lose. Alternatives: raise the limit, or switch overflow to reject-publish so publishers get a nack instead of the oldest message being pushed out. Be aware of a sharp edge: a quorum queue acting as a dead-letter target with overflow=reject-publish does not stop the internal dead-letter publisher from exceeding its max-length. If your DLQ is a quorum queue, prefer drop-head or use a classic queue as the dead-letter target.

Rejections

This is an application fix, not a broker fix. Find the consumer code path that nacks with requeue=false and the condition that triggers it. Common root causes: a schema version the consumer cannot deserialize, a missing dependency that makes processing impossible, or a catch-all error handler that dead-letters on any exception. Until the consumer is fixed, the DLQ keeps absorbing the failures, which is exactly what it is for. Alert on the rate so “the DLQ is doing its job” does not become “the DLQ hid an outage.”

Delivery limit (poison messages)

The mechanism worked. Fix the consumer for the failing message class. If you do not have a delivery limit on quorum queues and poison messages loop forever via requeue, set x-delivery-limit so they dead-letter after N attempts instead of consuming delivery capacity indefinitely.

DLX-to-origin loops

Break the cycle at the DLQ: remove the x-dead-letter-exchange policy on the DLQ itself, or give the DLQ’s DLX a routing key that cannot lead back to the origin queue. Then drain the in-flight looping messages manually.

Unbounded DLQ growth

Give every DLQ the same operational hygiene as a primary queue: an owner, a consumer or retention TTL, a length limit, and an alert on depth. Messages in a DLQ with no consumer have no path out except manual intervention or expiry.

Prevention

  • Alert on DLQ publish rate and depth, not just depth. Rate tells you an active failure; depth tells you an un-drained ledger.
  • Alert on the reason mix. Sample or export x-death reasons so a shift from expired to rejected after a deploy is visible immediately.
  • Set x-delivery-limit on quorum queues so poison messages dead-letter instead of looping on requeue.
  • Never dead-letter a DLQ back toward its origin. If the DLQ needs its own DLX, route it to a terminal queue.
  • Give the DLQ retention. A TTL or length limit on the DLQ bounds how much failure history you keep, and forces a decision about draining it.
  • Watch redeliver rates on origin queues. Rejections that dead-letter started as redeliveries; catching the loop upstream prevents the DLQ entry entirely.
  • Include the DLQ in capacity planning. DLQ messages consume memory and disk like any others; an ignored DLQ can contribute to the memory wall or disk halt that blocks all publishers cluster-wide.

How Netdata helps

  • Per-queue messages_ready and messages_unacknowledged for DLQs alongside origin queues, so failure inflow and the resulting backlog are on the same timeline.
  • Per-queue publish, deliver, ack, and redeliver rates, which let you see rejections upstream (high redeliver, low ack) before they arrive in the DLQ.
  • Queue depth trends over days, distinguishing a DLQ that drains from one that only accumulates.
  • Memory ratio (mem_used / mem_limit) and disk free correlation, catching the case where the DLQ itself becomes the capacity problem.
  • Alerts on DLQ publish rate or depth, so the failure ledger pages someone before it becomes the incident.