ActiveMQ per-destination DLQ: IndividualDeadLetterStrategy vs the shared ActiveMQ.DLQ

Out of the box, ActiveMQ Classic routes every undeliverable message from every queue and topic into a single queue: ActiveMQ.DLQ. One noisy destination with a poison-message problem mixes its failures with everyone else’s, and the only way to attribute a message back to its source is to browse the DLQ and inspect the JMSDestination property on each entry. On a busy broker that is slow, expensive, and usually done too late.

IndividualDeadLetterStrategy routes each source destination’s failures to its own dead letter queue, typically named DLQ.<source>. Failures become attributable at a glance, each DLQ gets its own JMX MBean and metrics, and you can alert, drain, and set expiry per source instead of globally.

The tradeoff is real: every per-destination DLQ is a real destination with its own MBeans, advisory topics, and memory accounting. This article covers how the two strategies differ, how to configure the individual strategy correctly (including the defaults that bite people), and how to monitor the resulting DLQ.* queues so they do not become a silent storage leak.

What the shared ActiveMQ.DLQ does by default

The default SharedDeadLetterStrategy sends every message that exhausts its redelivery attempts to one queue named ActiveMQ.DLQ. Redelivery itself is controlled client-side by the JMS RedeliveryPolicy, which defaults to 6 redeliveries before the broker dead-letters the message.

Two properties of the shared DLQ matter operationally:

• Everything lands in one place. Poison messages from a low-priority batch queue sit next to failures from your payment flow. Attribution requires browsing messages and reading the original JMSDestination header, and queue browsing is expensive on deep queues.
• Messages never expire. DLQ messages have no TTL by default. They accumulate forever, each one pinning space in KahaDB journal files. This is a common path to store exhaustion: application queues look fine, but StorePercentUsage climbs for weeks because the DLQ is quietly growing.

There is also a subtle accounting effect: messages moved to the DLQ are counted as dequeued from the source queue. A source queue can show a healthy dequeue rate while every “consumed” message was actually discarded to the DLQ. If you only watch dequeue rates, a poison-message incident looks like normal processing.

How IndividualDeadLetterStrategy works

IndividualDeadLetterStrategy is configured per destination policy entry, inside <destinationPolicy> in activemq.xml. When a message exhausts redelivery on a matching destination, the broker routes it to a dedicated dead letter destination whose name is built from a prefix, the source destination name, and an optional suffix. With the conventional queuePrefix="DLQ.", a poison message on queue orders.incoming lands in DLQ.orders.incoming.

flowchart LR
  A[orders.incoming] -->|redeliveries exhausted| S{deadLetterStrategy}
  B[billing.events] -->|redeliveries exhausted| S
  S -->|SharedDeadLetterStrategy default| D1[ActiveMQ.DLQ mixed failures]
  S -->|IndividualDeadLetterStrategy| D2[DLQ.orders.incoming]
  S -->|IndividualDeadLetterStrategy| D3[DLQ.billing.events]

The strategy is broker-side only. There is no way to set or override it from a JMS client; it lives entirely in the destination policy.

Each resulting DLQ is a first-class destination. That is the point: it gets its own QueueSize, EnqueueCount, DequeueCount, memory accounting, and MBean, so per-source failure rates become directly measurable instead of buried in a shared bucket.

Configuring per-destination dead letter queues

The canonical configuration applies the strategy to all queues with a wildcard policy entry:

<broker xmlns="http://activemq.apache.org/schema/core">
  <destinationPolicy>
    <policyMap>
      <policyEntries>
        <policyEntry queue=">">
          <deadLetterStrategy>
            <individualDeadLetterStrategy
                queuePrefix="DLQ."
                useQueueForQueueMessages="true"/>
          </deadLetterStrategy>
        </policyEntry>
      </policyEntries>
    </policyMap>
  </destinationPolicy>
</broker>

You can scope it more narrowly. A policy entry per queue family lets different flows keep different DLQ policies:

<policyEntry queue="orders.>">
  <deadLetterStrategy>
    <individualDeadLetterStrategy queuePrefix="DLQ."/>
  </deadLetterStrategy>
</policyEntry>

Changing activemq.xml requires a broker restart to take effect. There is no runtime reload of destination policy, so plan the cutover.

The attributes that matter

Three of these defaults deserve explicit decisions rather than acceptance:

• Set queuePrefix explicitly. The source-code default is ActiveMQ.DLQ.Queue., not empty. If you configure other attributes but leave the prefix alone, your DLQs come out named like ActiveMQ.DLQ.Queue.orders.incoming instead of the DLQ.orders.incoming you expected. Always set queuePrefix="DLQ." (or your convention) by hand.
• Decide on processExpired. The default true sends every TTL-expired message to the DLQ alongside genuine poison messages. On a system with aggressive TTLs, that floods the DLQ with non-error traffic and drowns the signal. If expired messages are expected shedding rather than failures, set processExpired="false" and monitor ExpiredCount on the source destinations instead. If expired messages are correctness loss you must inspect, keep them in the DLQ but plan for the volume.
• Decide on processNonPersistent. The default false means non-persistent messages that exhaust redelivery are discarded, not dead-lettered. The rationale in the official documentation is that if the application did not care enough to make the message persistent, recording its failure has little value. If your non-persistent flows still need failure visibility, set it to true deliberately.

Tradeoffs and operational gotchas

• Destination and MBean growth. Each per-destination DLQ is a real destination, and each destination creates at least 4 MBeans plus advisory topics. Switching a broker with 300 application queues to individual DLQs adds roughly 300 more destinations. That is normally fine, but if your broker already has destination-count pressure, watch the total destination count after the change and consider gcInactiveDestinations with schedulePeriodForDestinationPurge so empty, inactive DLQs get cleaned up.

• The shared DLQ does not go away automatically. Switching strategy changes routing for new dead-letters; anything already sitting in ActiveMQ.DLQ stays there until you drain it. After cutover, verify nothing new is arriving in the shared queue.

• DLQ messages still have no TTL by default. Per-destination naming fixes attribution, not retention. Each DLQ.* queue accumulates forever unless you set the expiration attribute (in milliseconds) on the strategy, which discards DLQ entries after that time.

• Do not create a DLQ expiry loop. If a wildcard policy entry also matches your DLQ destinations and gives them an expiring dead letter strategy, an expired DLQ message can be routed into another (or the same) DLQ and loop. The official documentation calls this out directly. Keep DLQ destinations out of the scope of policies that apply their own dead letter strategy with expiry.

• Dequeue metrics hide the transfer. As with the shared DLQ, a message moved to DLQ.orders.incoming increments the dequeue count of orders.incoming. Correlate source dequeue rate with DLQ enqueue rate before concluding consumption is healthy.

• Durable subscribers are a special case. With destinationPerDurableSubscriber="true", each durable subscriber of a topic gets its own DLQ. Note that the broker’s message audit (the strategy’s enableAudit attribute, enabled by default) can prevent the same message from being added to multiple subscriber DLQs when it fails for more than one subscriber.

Monitoring the per-destination DLQs

The operational win of the individual strategy is that every source’s failure stream becomes a separately measurable destination. Query them with a wildcard over the naming prefix via Jolokia:

# Depth of every per-destination DLQ at once
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=DLQ.*/QueueSize'

# Enqueue count (cumulative; take deltas for rate) on each DLQ
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=DLQ.*/EnqueueCount'

# If the shared DLQ still exists, watch it too; it should stay flat after cutover
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ/QueueSize'

When a specific DLQ grows, browse it to see what is failing. DLQ messages retain their original properties, so JMSDestination, the redelivery count, and any exception information recorded by the client or broker tell you what failed and where. Browse sparingly: a full browse() on a deep queue spikes broker memory and CPU.

Alerting posture: any non-zero DLQ depth is ticket-worthy, because each message is a processing failure someone should look at. Page only when the DLQ growth rate relative to the source enqueue rate breaches a queue-specific threshold and the main workflow is actually impaired. Escalate on storage regardless of failure semantics: a DLQ consuming meaningful store space is a leak, and KahaDB can only reclaim a journal file when every message in it is acknowledged, so old DLQ entries pin disk out of proportion to their size.

How Netdata helps

• Netdata charts each DLQ.* destination’s QueueSize, EnqueueCount, and DequeueCount separately via JMX, so per-source failure attribution is a dashboard view rather than a queue-browsing session.
• Cumulative counters like EnqueueCount are converted to rates, which makes the DLQ-rate-to-enqueue-rate ratio practical to alert on per source queue.
• Source dequeue rate, DLQ enqueue rate, and redelivery indicators sit on the same dashboard, exposing the “healthy dequeue that is actually DLQ transfer” trap.
• StorePercentUsage, KahaDB journal file count, and disk free on the store partition are correlated in one place, catching the slow DLQ-driven storage leak weeks before the store hits 100%.
• Total destination count tracking surfaces the MBean growth side effect of enabling per-destination DLQs on brokers with many queues.

Related guides

• ActiveMQ KahaDB journal files not deleted: one unacked message pinning a 32MB log
• ActiveMQ disk full on the KahaDB partition: write failures and store corruption risk
• ActiveMQ expired message count climbing: TTL expiry and silent correctness loss
• ActiveMQ InFlightCount high: prefetch full, acks stalled, and zombie consumers
• ActiveMQ enqueue outpacing dequeue: reading the rate imbalance before the backlog
• How ActiveMQ Classic actually works in production: a mental model for operators