ActiveMQ DLQ never expires: setting TTL so the dead-letter queue stops leaking storage

Your ActiveMQ.DLQ has been sitting at some non-zero depth for months. Nobody looks at it because “it’s just the DLQ.” Meanwhile StorePercentUsage creeps up a fraction of a percent a day, the db-*.log journal files keep accumulating, and one morning the broker hits 100% store usage and blocks every persistent producer on the bus.

This is the default behavior of ActiveMQ Classic, not a bug. Messages sent to the dead-letter queue have no TTL. They accumulate forever, and every one of them is a live reference that pins KahaDB journal files against garbage collection. The fix has two parts: configure expiration on the DLQ so the leak stops, and drain-and-investigate the existing backlog instead of just deleting it. Every DLQ message is a failed business transaction; purging blind throws away the evidence.

This article covers the diagnosis, the exact configuration, and the two traps that turn this fix into a new incident: DLQ-to-origin forwarding loops and wildcard policy entries that apply expiration where you did not intend it.

What this means

In ActiveMQ Classic, when a message exhausts its redelivery attempts (default: 6, set client-side via RedeliveryPolicy.maximumRedeliveries) or expires on its origin queue, the broker moves it to the dead-letter queue. By default that is a single shared queue, ActiveMQ.DLQ, governed by SharedDeadLetterStrategy.

Three defaults combine to make this a silent storage leak:

1. No TTL on DLQ messages. Once a message lands in the DLQ, nothing ever removes it unless an operator does.
2. processExpired="true" by default. Messages that hit their TTL on origin queues are forwarded into the DLQ too, so the DLQ also collects expiry traffic, not just poison messages.
3. Journal pinning. KahaDB reclaims a db-*.log journal file only when every message in it has been acknowledged. DLQ messages are never acknowledged. One old DLQ message pins an entire 32MB journal file, and DLQ traffic spread across weeks of journals pins many of them.

The cascade looks like this:

flowchart TD
  A[Poison or expired messages] --> B[Redeliver to max retries]
  B --> C[Moved to ActiveMQ.DLQ]
  C --> D[No TTL: never expires]
  D --> E[DLQ depth grows or churns in place]
  E --> F[Journal files pinned: unacked msgs block GC]
  F --> G[StorePercentUsage climbs over weeks]
  G --> H[Store 100%: persistent producers blocked]

One more subtlety: a stable DLQ depth is not proof the leak stopped. “It’s always been around 500” can hide continuous churn, with new failures arriving as fast as old ones are dealt with (or expired on the origin side and re-arriving). Watch the DLQ enqueue rate, not just the depth. Also note that messages moved to the DLQ are counted as dequeued from the source queue, so your origin queue’s dequeue rate can look healthy while messages are actually being discarded.

Common causes

Quick checks

All read-only. These assume the Jolokia HTTP bridge on the web console (default port 8161, bound to 127.0.0.1 since 5.16) and default credentials; adjust for your environment.

# DLQ depth right now
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ/QueueSize'

# DLQ enqueue and dequeue counters (take two readings to get a rate)
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ/EnqueueCount,DequeueCount,ExpiredCount'

# Store pressure
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/StorePercentUsage'

# Journal file count and total store footprint on disk
ls /opt/activemq/data/kahadb/db-*.log | wc -l
du -sh /opt/activemq/data/kahadb/
df -h /opt/activemq/data/

# Which origin queues are feeding the DLQ: browse and inspect JMSDestination
# (browse is expensive on deep queues; the web console's first page is safer)
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/exec/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ/browse'

Two things to note on the last command: queue browsing is expensive on deep queues, so prefer the web console’s paged view over a full browse() on a DLQ with hundreds of thousands of messages. And JMSTimestamp is set by the producer’s clock, so age calculations are only as good as your NTP discipline.

How to diagnose it

1. Establish whether the DLQ is leaking or just old. Take two readings of the DLQ EnqueueCount an hour apart. A rising enqueue count means active failure traffic, and that root cause matters more than the storage. A flat enqueue count with a large depth means a historical backlog to drain.
2. Attribute DLQ messages to their origins. Browse a sample and read the JMSDestination property plus any exception/redelivery headers. Group by origin queue. Usually one or two queues supply almost all DLQ traffic.
3. Check whether expiry is feeding the DLQ. Look at ExpiredCount on the origin queues. If expired messages are a large share of DLQ arrivals and you do not need expired messages preserved, that traffic can be shut off at the strategy level (see Fixes).
4. Quantify the store impact. Compare journal file count and StorePercentUsage trend against DLQ depth. If the DLQ is the main unacked population, its messages are what pins the journals. Remember the pinning is not one-file-per-message: DLQ messages interleaved through many journals can hold a large fraction of the store.
5. Check for non-persistent blind spots. By default processNonPersistent="false", so failed non-persistent messages never reach the DLQ at all. If your workload is non-persistent, your DLQ depth understates your failure rate.
6. Rule out a prior misconfiguration. If someone already tried to fix this, look for expiration set on a wildcard or default policyEntry. That is the loop trap described below.

Metrics and signals to monitor

Fixes

Set an expiration on the DLQ strategy

Since ActiveMQ 5.12, deadLetterStrategy supports an expiration attribute (milliseconds) that stamps a TTL on messages as they are dead-lettered. A 7-day retention looks like this in activemq.xml:

<destinationPolicy>
  <policyMap>
    <policyEntries>
      <policyEntry queue=">">
        <deadLetterStrategy>
          <sharedDeadLetterStrategy processExpired="false" expiration="604800000"/>
        </deadLetterStrategy>
      </policyEntry>
    </policyEntries>
  </policyMap>
</destinationPolicy>

Two deliberate choices in that example:

• processExpired="false" stops expired messages from being forwarded into the DLQ at all. If your expired traffic is expected shedding (TTLs doing their job), this removes a whole class of DLQ arrivals. If you need expired messages preserved for audit, leave it true and rely on expiration to bound their lifetime instead.
• expiration on the dead-letter strategy, not on the queue policy. This is the distinction that matters.

The broker checks message expiry periodically on queues, controlled by expireMessagesPeriod on the policy entry (default 30000ms; 0 disables the check). So DLQ messages are reaped within roughly one sweep interval of their TTL, not at the exact millisecond.

Tradeoff: once a DLQ message expires it is gone. If your organization needs failed messages retained longer for audit or replay, set the TTL to match that retention requirement and build a drain-and-archive consumer instead of relying on the queue as cold storage.

Never set expiration on a wildcard or default policy entry

The official documentation is blunt about this: do not apply expiration to your DLQ destinations by setting it on a default or inclusive wildcard policy entry. Two concrete failure modes:

• DLQ-to-origin loops. If an expired DLQ message is forwarded onward (the official documentation warns that a DLQ entry which expires and forwards to the same or another DLQ with expiry can introduce a loop) and your strategy audit is disabled or its sliding window is exceeded, you can build a forwarding cycle that never terminates.
• Advisory queue explosion. Applying message expiry through a wildcard like queue=">" has been observed to interact badly with advisory destinations, producing unexpected ActiveMQ.Advisory.* queues with doubled message populations. Scope the policy entry tightly and enumerate destinations after any policy change.

Use per-destination DLQs (IndividualDeadLetterStrategy) if you need different retention per queue, and set expiration on the strategy for those specific entries.

Drain and investigate the existing backlog

Setting expiration does not retroactively stamp messages already sitting in the DLQ. The existing backlog needs an explicit decision:

1. Export or browse a representative sample per origin queue. Identify the common failure: schema change, deserialization error, downstream outage window.
2. Fix or ticket the root cause. The DLQ is a bug report queue; treat the top categories as work items.
3. Replay what is still valid, then purge the rest. Purging is destructive and irreversible: confirm the export is complete and the business owners have signed off before you run it.

After the drain, journal reclamation is not instant. KahaDB cleanup runs periodically, so StorePercentUsage and the journal file count will step down over the following minutes rather than immediately.

One caveat if you run JBoss A-MQ or an older Classic line: the Red Hat knowledge base documents a “ghost message” problem when forcing DLQ expiry via the TimeStampPlugin, where the message count does not decrease and KahaDB space is not released until an explicit purge(). That behavior is documented for the TimeStampPlugin approach in the JBoss A-MQ 6.x line; the built-in expiration attribute on deadLetterStrategy used in this guide is not documented to suffer from it.

Prevention

• Alert on any non-zero DLQ depth. Every DLQ message is a processing failure. Ticket-level on depth, escalate on growth rate relative to origin enqueue rate.
• Treat DLQ expiration as a standard part of broker provisioning, with a retention value chosen deliberately, and always scoped through the dead-letter strategy rather than wildcard destination policies.
• Monitor ExpiredCount on origin queues so you know whether the DLQ is receiving poison messages or expiry shedding; the fix for each is different.
• Watch journal file count alongside store usage. It is the earliest visible symptom of pinning and has no JMX equivalent, so it needs a filesystem-level check.
• Run a DLQ consumer or scheduled export job that categorizes arrivals by origin and exception, so “investigate the DLQ” is a standing process rather than an incident activity.
• Enumerate destinations after any policy change to catch advisory-queue side effects early.

How Netdata helps

• Netdata collects ActiveMQ destination metrics via JMX, so QueueSize, EnqueueCount, DequeueCount, and ExpiredCount for ActiveMQ.DLQ sit on the same dashboard as the broker-level store signals, making the DLQ-to-store correlation visible without manual Jolokia polling.
• Per-second collection on StorePercentUsage and memory usage lets you see the slow multi-week climb that threshold-only checks miss, and confirm the step-down after a drain.
• Correlating DLQ enqueue rate against origin queue ExpiredCount and dequeue rate distinguishes poison-message churn from expiry shedding, which determines whether the fix is in the strategy config or the consumer code.
• Filesystem metrics on the KahaDB partition (disk usage, file counts) sit next to broker metrics, closing the store-limit-vs-physical-disk gap that the broker’s own accounting cannot see.
• Alerting on DLQ depth and DLQ/origin enqueue ratio turns the “black hole” into a ticket queue, which is the actual prevention for this failure mode.

Related guides

• ActiveMQ.DLQ growing: dead letter queue accumulation and poison messages
• ActiveMQ disk full on the KahaDB partition: write failures and store corruption risk
• ActiveMQ KahaDB journal files not deleted: one unacked message pinning a 32MB log
• ActiveMQ expired message count climbing: TTL expiry and silent correctness loss
• ActiveMQ enqueue outpacing dequeue: reading the rate imbalance before the backlog
• ActiveMQ memory limit reached: MemoryPercentUsage at 100% and the flow-control cliff
• How ActiveMQ Classic actually works in production: a mental model for operators