ActiveMQ.DLQ growing: dead letter queue accumulation and poison messages

The queue named ActiveMQ.DLQ is growing and nobody is consuming from it. That is the default dead letter queue: it has no consumers, no TTL, and no automatic cleanup, so every message that lands there stays there forever.

Every message in the DLQ is two problems at once. It is a failed business transaction that some consumer gave up on after exhausting redeliveries. And it is a storage leak: DLQ messages are never acknowledged, and in KahaDB a single unacknowledged message pins its entire journal file. A DLQ that grows quietly for weeks is a common root cause behind store exhaustion, which eventually halts persistent messaging on the whole broker.

This article covers how to tell what is feeding the DLQ, how to drain it safely, and how to stop it from silently filling your store again.

What this means

ActiveMQ Classic routes a message to the dead letter queue when delivery has definitively failed. By default there is a single shared DLQ named ActiveMQ.DLQ for all destinations (SharedDeadLetterStrategy). Messages arrive there through two main paths:

1. Redelivery exhaustion. A consumer receives a message, fails to process it, and rolls back or nacks. The client redelivers up to RedeliveryPolicy.maximumRedeliveries (default: 6, enforced client-side). After the last attempt, the client sends a poison ack and the broker moves the message to the DLQ.
2. Expiration. Messages whose TTL passes before consumption are routed to the DLQ by default (processExpired="true" on the dead letter strategy). If your producers set TTL and consumers lag, expired messages feed the DLQ continuously without any consumer error at all.

Two details make DLQ growth dangerous rather than merely untidy:

• DLQ messages never expire by default. There is no TTL, no size cap, no cleanup job. Accumulation is permanent until an operator acts.
• Dequeue counts lie. Messages moved to the DLQ are counted as dequeued from the source queue. Your dequeue rate can look perfectly healthy while the broker is actually discarding messages into the DLQ.

flowchart LR
  P[Producer] --> Q[Source queue]
  Q --> C[Consumer]
  C -->|processing fails| R[Redelivery, up to 6 attempts]
  R -->|poison ack| D[ActiveMQ.DLQ]
  Q -->|TTL expires, processExpired=true| D
  D -->|no TTL, never acked| J[KahaDB journal files pinned]
  J -->|files accumulate| S[Store usage climbs to 100%]

If you run ActiveMQ Artemis instead of Classic, the model is different: Artemis uses a dead letter address configured per address in broker.xml, defaults to 10 delivery attempts, and discards undeliverable messages entirely if no dead letter address is configured. The rest of this article is Classic-specific.

Common causes

Quick checks

All read-only. These use the Jolokia JMX bridge on the web console port; adjust credentials and broker name for your environment.

# DLQ depth: the primary signal
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ/QueueSize'

# DLQ enqueue count (cumulative): take two readings to get the growth rate
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ/EnqueueCount'

# Store usage: how close the accumulation is to halting persistent messaging
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/StorePercentUsage'

# Journal file count: DLQ messages pin these
ls /opt/activemq/data/kahadb/db-*.log | wc -l

# Actual disk usage of the store (StorePercentUsage is against the configured limit, not the disk)
du -sh /opt/activemq/data/kahadb/
df -h /opt/activemq/data/

# Expired message counts on application queues (silent DLQ feeder)
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=*/ExpiredCount'

A note on browsing: the JMX browse() operation or the web console’s message view lets you inspect DLQ message properties, which is where the diagnosis lives. Queue browsing is expensive on deep queues and can spike broker memory and CPU. If the DLQ already holds millions of messages, browse a page at a time via the web console rather than a full JMX browse.

How to diagnose it

1. Confirm the growth rate, not just the depth. Take two readings of the DLQ EnqueueCount a few minutes apart. A stable count of 5,000 is a backlog to triage; a rate of 10 messages/minute is an active processing failure happening right now.

2. Inspect DLQ message properties. Browse a sample of DLQ messages and read JMSDestination to see which source queue or topic produced them. DLQ messages also carry the exception information describing why delivery failed (the broker attaches the delivery failure cause as a message property; the exact property name is dlqDeliveryFailureCause in current Classic versions, set by the broker when the message is moved, so it is present regardless of the producer’s client version). Group the failures by source destination and by exception. Usually one or two queues and one dominant error explain almost all of the depth.

3. Separate the two feeders: redelivery vs expiry. If the failing messages show repeated processing exceptions, it is a poison message or consumer bug. If they show no consumer error and the source queues have rising ExpiredCount, the feeder is TTL expiry. Check whether producers set timeToLive and whether the dead letter strategy was left at the default processExpired="true".

4. Check the redelivery leading indicator on source queues. Redelivery happens before the DLQ. Look for messages with JMSXDeliveryCount > 1 on source queues, and watch whether dequeue rate on those queues is real consumption or DLQ transfers. If redelivery is rising and the DLQ has not grown yet, you are early: fix the consumer before the backlog moves.

5. Check the health of the DLQ itself. If the DLQ’s per-destination MemoryPercentUsage has hit its limit, dead-lettering itself starts failing and messages may be discarded entirely. A DLQ that suddenly stops growing while source-queue failures continue is not good news.

6. Quantify the storage damage. Compare DLQ depth and journal file count with StorePercentUsage and actual disk free. DLQ messages are never acknowledged, so every one of them pins its journal file against KahaDB GC. This is what turns a correctness problem into a capacity incident.

Metrics and signals to monitor

Fixes

Stop the active failure first

Consumer bug or poison message: fix or roll back the consumer. Redelivery is for transient failures; a message that fails deterministically will always end up in the DLQ after 6 attempts, and the retries in between are wasted work that delays everything behind it.

Downstream dependency: if consumers fail because a database or API is down, the DLQ growth stops when the dependency recovers, assuming the redelivery policy survives the outage duration. If outages outlast 6 redeliveries, tune the client RedeliveryPolicy (more attempts, backoff) so transient failures do not permanently shed messages.

Expiry feeding the DLQ: if expired messages should be discarded rather than dead-lettered, set processExpired="false" on the dead letter strategy for those destinations. Understand the tradeoff: with processExpired="false" expired messages are silently discarded with no DLQ entry and no advisory. That is the right choice for time-bound data (quotes, heartbeats) and the wrong choice for anything with business value.

Drain the existing backlog

Before purging anything, export or inspect a sample. Every DLQ message is a bug report, and for many businesses the messages themselves have recoverable value (reprocessable orders, payments, events).

Options, in order of preference:

1. Replay after the root cause is fixed. Move messages back to their source queue (the JMSDestination property tells you where each one came from) once the consumer can actually process them. Do this in controlled batches and watch consumer error rates; replaying into a still-broken consumer just re-runs the redelivery-to-DLQ cycle.
2. Export to file for offline analysis, then purge. Keeps the evidence while releasing the store.
3. Purge in place. Via JMX purge() on the DLQ destination or the web console. This discards the messages permanently. It also does not immediately reclaim disk: KahaDB journal cleanup runs periodically, and disk reclamation lags the purge. Do not purge before investigating unless store exhaustion is imminent.

Prevent recurrence with configuration

The default single shared DLQ mixing poison messages from every destination is operationally poor. A per-destination strategy isolates failures and lets you set TTLs:

<!-- activemq.xml: per-destination DLQ with expiry, expired source messages discarded -->
<destinationPolicy>
  <policyMap>
    <policyEntries>
      <policyEntry queue=">">
        <deadLetterStrategy>
          <individualDeadLetterStrategy
            queuePrefix="DLQ."
            useQueueForQueueMessages="true"
            processExpired="false"
            expiration="604800000"/>
        </deadLetterStrategy>
      </policyEntry>
    </policyEntries>
  </policyMap>
</destinationPolicy>

Key choices here:

• IndividualDeadLetterStrategy gives each queue its own DLQ (DLQ.<name>), so a noisy low-priority queue stops contaminating the shared DLQ and you can monitor, alert, and drain per application.
• expiration (supported since 5.12) sets a TTL in milliseconds on DLQ messages, so dead letters eventually self-clean instead of pinning journal files forever. One warning from the official documentation: do not apply DLQ expiration via a wildcard policy that also matches the DLQ destinations themselves, or expired DLQ messages get re-routed back into a DLQ and loop.
• processNonPersistent defaults to false, meaning failed non-persistent messages are silently discarded and never reach the DLQ. If you assume every failure is dead-lettered, non-persistent traffic is the blind spot. Set it to true on the dead letter strategy if you need those failures captured.

Prevention

• Alert on DLQ depth at any non-zero value, and page on growth rate. The DLQ is the most under-monitored destination on most brokers; treat it as a first-class signal, not a black hole.
• Monitor the redelivery rate on application queues as the leading indicator. Redelivery rises before DLQ growth; catching it there saves the backlog.
• Track ExpiredCount on queues where expiry should never happen, and decide explicitly per destination whether expired messages go to the DLQ or are discarded.
• Set DLQ expiration and per-destination DLQs as shown above so no future accumulation is permanent.
• Reconcile dequeue rates against DLQ enqueue rates in dashboards so “healthy consumption” cannot actually be silent dead-lettering.
• Build a DLQ consumer process that logs, categorizes, and raises alerts on new dead letters, even if it does not auto-replay. Automation that reads the failure cause beats an operator browsing the web console at 3 a.m.
• Include the DLQ in store capacity planning. DLQ growth is a leading indicator for KahaDB journal accumulation and store exhaustion; trend it alongside journal file count and disk free.

How Netdata helps

• Per-destination queue depth including ActiveMQ.DLQ collected every second via JMX, so DLQ growth shows up as it starts rather than at the next store-full alert.
• Enqueue/dequeue rate correlation per queue, which exposes the classic DLQ blind spot: dequeue rate looking healthy while the dequeues are actually DLQ transfers.
• ExpiredCount trending per destination, separating the expiry feeder from the poison-message feeder without manual JMX queries.
• Store usage, disk free, and KahaDB journal file count on one dashboard, so you can see the DLQ-to-journal-pinning-to-store-exhaustion chain as one narrative instead of three unrelated alerts.
• Redelivery and inflight signals alongside DLQ depth, letting you catch the poison-message pattern at the redelivery stage, before the backlog moves to the DLQ and starts pinning journal files.

Related guides

• ActiveMQ monitoring checklist: the signals every production broker needs
• ActiveMQ KahaDB journal files not deleted: one unacked message pinning a 32MB log
• ActiveMQ disk full on the KahaDB partition: write failures and store corruption risk
• ActiveMQ consumers connected but not acknowledging: the zombie consumer
• ActiveMQ InFlightCount high: prefetch full, acks stalled, and zombie consumers
• ActiveMQ enqueue outpacing dequeue: reading the rate imbalance before the backlog
• ActiveMQ oldest message age: the queue latency depth alone cannot show
• How ActiveMQ Classic actually works in production: a mental model for operators