ActiveMQ poison message loop: redelivery storms and DLQ growth

A queue looks busy. Consumers are connected, dequeue counters are moving, and yet the business workflow has stalled: orders are not completing, messages are getting older, and somewhere in the broker a queue called ActiveMQ.DLQ is quietly filling up. This is the poison message loop, and it is one of the most misread failure modes in ActiveMQ Classic because the headline metrics look healthy while useful work has stopped.

The mechanism is simple. A message arrives that the consumer cannot process: a deserialization error, a schema mismatch after a producer deploy, a consumer bug, a downstream dependency returning something the consumer never handles. The consumer rolls back (or the session recovers), and the broker redelivers the message. The client-side RedeliveryPolicy allows this up to maximumRedeliveries, which defaults to 6, before the broker moves the message to the Dead Letter Queue. If the poisoned input is a class of messages rather than a single message, every one of them burns through its redelivery budget, and the broker spends its dispatch capacity retrying failures instead of doing work.

Two properties make this pattern dangerous. First, messages moved to the DLQ are counted as dequeued from the source queue, so dequeue rate can look healthy while messages are actually being discarded. Second, the DLQ has no TTL by default and every message in it pins KahaDB journal space, so a poison message incident that is never cleaned up becomes a store-exhaustion incident weeks later.

What this means

flowchart LR
  A[Message dispatched] --> B{Consumer processes?}
  B -->|ack| C[Dequeued, store entry freed]
  B -->|rollback / nack| D{Delivery count <= maximumRedeliveries?}
  D -->|yes| E[Redelivered after delay]
  E --> B
  D -->|no, default 6| F[Moved to DLQ]
  F --> G[Counted as dequeue from source queue]
  F --> H[Pins KahaDB journal space, no TTL by default]

The two indicators fire in a fixed order, and the order matters for alerting. Redelivery rate is the leading indicator: it rises the moment a poison class starts flowing, while messages are still cycling through retries. DLQ growth is the lagging indicator: it begins only after the first poisoned messages exhaust their redelivery budget. If you alert only on DLQ depth, you find out about the incident at least six delivery attempts and several minutes late. The composite signature is: redelivery rate rises on specific queues, DLQ growth begins, dequeue looks busy but useful completion stalls, and message age climbs.

The consequence most teams miss is the store angle. KahaDB journal files (default 32MB each) are only reclaimable when every message in the file has been consumed. DLQ messages are dequeued from the source queue’s perspective but still live in the store as pending messages on the DLQ. A DLQ that is never drained holds journal files indefinitely, which is how a poison message incident turns into the store exhaustion spiral weeks after the original bug was fixed.

Common causes

Cause	What it looks like	First thing to check
Deserialization or schema mismatch	Redelivery starts immediately after a producer or consumer deploy; DLQ messages share a type	Broker log for `ClassNotFoundException` / `InvalidClassException`; compare producer and consumer versions
Consumer bug on a specific message shape	One message class fails, others flow; exception in consumer logs on every attempt	Consumer application logs correlated with `JMSXDeliveryCount` climbing
Downstream dependency failure	All messages on a queue fail for a window, then recover; redeliveries spike in bursts	Consumer’s downstream (database, API) health during the redelivery window
Redelivery policy too aggressive	Transient blips become permanent DLQ entries; DLQ messages would have succeeded on retry 7	`maximumRedeliveries` and delay settings on the client `RedeliveryPolicy`
Expired messages routed to DLQ	DLQ grows but `ExpiredCount` on source queues grows in lockstep; no consumer errors	Per-destination `ExpiredCount`; TTL configuration vs actual consumer lag
Broker and consumer redelivery stacked	Messages take far more than `maximumRedeliveries` attempts before DLQ	Whether the broker-side redelivery plugin is enabled in addition to the client policy

Quick checks

All of these are read-only. The Jolokia examples assume the default web console on localhost:8161; adjust credentials and host for your deployment.

# DLQ depth (default shared DLQ)
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ/QueueSize'

# All queues at once: look for depth, dequeue, and inflight anomalies together
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=*/QueueSize'

# Inflight on the suspect queue: redelivering consumers show inflight churn with no progress
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/InFlightCount'

# Expired count: distinguish poison processing failures from TTL expiry feeding the DLQ
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/ExpiredCount'

# Journal file count: is the DLQ already pinning store space?
ls /opt/activemq/data/kahadb/db-*.log | wc -l

# Broker log: deserialization and processing errors
grep -ci "ClassNotFoundException\|InvalidClassException" /opt/activemq/data/activemq.log

Two cautions on interpretation. Dequeue rate alone will lie to you here: DLQ transfers increment DequeueCount on the source queue, so a queue can show a normal dequeue rate while delivering nothing to consumers. And browsing queues via JMX to inspect messages is expensive on deep queues; browse the DLQ (usually shallow relative to application queues) rather than the backlog.

How to diagnose it

Confirm the loop, not just the symptom. Take two readings of EnqueueCount and DequeueCount on the suspect queue a minute apart, alongside DLQ QueueSize. If the DLQ grows by roughly the same amount the source queue “dequeues,” the consumer is not completing work; the broker is discarding to the DLQ.
Catch redelivery before the DLQ. Redelivery has no single clean JMX counter in stock ActiveMQ Classic. The practical sources are the broker log, destination and consumer statistics, and message state: a message with JMSXDeliveryCount greater than 1 has been redelivered. If you can sample in-flight messages or add consumer-side logging of the delivery count, do it; that is your earliest tripwire for the next incident.
Identify the poison class from the DLQ. Browse ActiveMQ.DLQ (or your per-destination DLQ) and inspect message properties. Group messages by source destination and message type: you are looking for the common denominator, one queue, one message shape, one schema version.
Correlate the timing with a change. Poison classes almost always start at a deploy boundary, a schema change, or a downstream dependency failure. Line up the first redelivery spike with producer and consumer deploys. If DLQ growth tracks ExpiredCount instead of consumer errors, the cause is TTL expiry under consumer lag, not processing failure, and the fix is different.
Check whether redelivery is stacked. If both the broker-side redelivery plugin (which requires schedulerSupport="true" on the broker element) and the client-side RedeliveryPolicy are configured, the redelivery budgets multiply rather than replace each other. Symptom: messages take many more attempts than your configured maximumRedeliveries before landing in the DLQ. Pick one layer for redelivery and disable the other.
Assess the blast radius on the store. Count journal files and check StorePercentUsage and disk free on the KahaDB partition. If the DLQ has been growing for days, the cleanup step is part of the incident, not an afterthought.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Redelivery rate	Leading indicator; broker is spending dispatch work on retries	Any sustained increase above baseline; any nonzero on queues that never normally retry
DLQ QueueSize and growth rate	Every entry is a failed business transaction; also a storage leak	Any unexpected sustained growth; growth rate as a fraction of enqueue rate climbing
Dequeue rate vs DLQ enqueue rate	Separates real completion from DLQ discards	Dequeue looks normal while DLQ grows at the same rate
Message age of oldest pending	The latency signal depth alone cannot show	Age climbing while dequeue “looks busy”
ExpiredCount per destination	Expired messages feed the DLQ by default	Expiry and DLQ growth moving together
InFlightCount	Redelivering consumers churn inflight without progress	Inflight near prefetch with stalled completion
Journal file count / StorePercentUsage	DLQ messages pin journal files indefinitely	Journal count growing while application queue depths stay low

Fixes

Quarantine the poison class

Get the failing messages out of the redelivery loop before fixing the root cause. If the poison class is identifiable by a property or destination, stop or scale down the affected consumer (or add a selector so it skips the poisoned shape) so the rest of the queue drains normally. Export or move the DLQ contents aside for later analysis rather than purging them blindly: they are your evidence and, often, transactions you need to replay. Moving or purging messages is a write operation on the broker; on a live system, prefer the web console or JMX move operations during a low-traffic window and confirm the destination before executing.

Fix the root cause, then replay

A deserialization mismatch means producer and consumer must agree on the message contract; a consumer bug means a consumer deploy. Only after the fixed consumer is live should you replay DLQ messages back to the source queue. Replaying before the fix just runs the loop again. For schema mismatches, remember the security dimension: repeated deserialization errors against unexpected classes can also be an attack signature, so check what class actually failed rather than assuming a benign version skew.

Tune the RedeliveryPolicy

The client-side defaults are: maximumRedeliveries of 6, initialRedeliveryDelay of 1000 ms, and exponential backoff disabled by default (the backOffMultiplier of 5.0 only applies once useExponentialBackOff is enabled). Those defaults are reasonable for transient failures and wrong for two common cases. For genuinely transient downstream blips, six retries in about six seconds is not enough runway: enable exponential backoff and set a maximumRedeliveryDelay cap so retries spread over minutes instead of seconds. For deterministic poison (a message that will never parse), six retries is five too many: a low redelivery budget with a short delay gets the message to the DLQ fast and stops wasting dispatch capacity. Per-destination policies via RedeliveryPolicyMap (available since 5.7) let you set these per queue, which is the right granularity since transient versus deterministic failure is a property of the workload.

Version-specific checks

If you use the broker-side redelivery plugin, note that ActiveMQ 5.16.0 and 5.16.1 shipped a regression in which messages scheduled for broker redelivery were deleted from the scheduler instead of redelivered; it was fixed in 5.16.2. If you are on those versions and broker redelivery behaves as if messages vanish, upgrade before tuning anything else.

Prevention

Per-destination DLQs. The default SharedDeadLetterStrategy mixes poison from every queue into one ActiveMQ.DLQ, which makes triage slow and lets a noisy low-priority queue hide failures on a critical one. Configure IndividualDeadLetterStrategy (queue prefix such as DLQ.) via a wildcard policy entry so each queue’s failures are isolated and attributable.
Bound the DLQ’s lifetime and size. DLQ messages have no TTL by default and accumulate forever. Set an expiration on the dead letter strategy, or run a DLQ consumer that logs, categorizes, and alerts on every entry. Do not configure a DLQ whose expired entries forward to another DLQ with expiry, because that creates a loop.
Decide explicitly what happens to expired and non-persistent messages. Expired messages go to the DLQ by default; set processExpired="false" only if you accept silent discard. Non-persistent messages are not dead-lettered by default; set processNonPersistent="true" if losing them silently is worse than DLQ noise.
Alert on redelivery, not just DLQ depth. Redelivery is the leading indicator; DLQ growth means you already lost the messages. Treat any sustained redelivery above baseline as ticket-worthy and any DLQ growth as a bug report.
Separate dequeue from DLQ transfer in dashboards. Because DLQ moves increment DequeueCount, plot DLQ enqueue rate next to source-queue dequeue rate so a discard storm can never impersonate healthy throughput.
Cap dispatch waste with prefetch awareness. A consumer in a redelivery loop holds prefetch slots for messages it cannot process. On queues prone to poison, a smaller prefetch limits how much work one looping consumer can hold hostage.

How Netdata helps

Redelivery rate per destination as a first-class chart, so the leading indicator pages before the first message ever reaches the DLQ.
DLQ depth and growth rate alongside per-queue dequeue rate, making the “dequeue looks healthy but is actually DLQ transfer” confusion visible in one view.
Enqueue/dequeue imbalance and message age on the same dashboard, so stalled useful completion shows up as rising age even while counters move.
KahaDB journal file count, StorePercentUsage, and disk free correlated with DLQ depth, catching the slow store-exhaustion tail of an old poison incident.
Anomaly detection on redelivery and DLQ rates, which catches the onset of a new poison class at a deploy boundary without hand-tuned thresholds per queue.