$ guides / activemq / activemq-redelivery-rate-high ▌

Operations Guides

ActiveMQ redelivery rate climbing: rollbacks, nacks, and retry storms

Your dequeue rate looks busy, but useful work is not getting done. Messages are dispatched, rolled back or nacked, and dispatched again. The broker spends its cycles retrying instead of making progress, and the redelivery counters climb. This is the poison message replay storm pattern in its early stage, and it leads two things you do not want: dead letter queue growth and silent correctness loss.

Redelivery is normal in small doses. A consumer restart, a transient downstream timeout, a transacted session rolled back once: all bump redelivery counters briefly. What matters is a sustained pattern. High dispatch rate combined with an equally high redelivery rate is net-zero progress, and if you only watch dequeue-adjacent throughput the broker can look healthy while nothing completes.

This article covers ActiveMQ Classic 5.x. Artemis handles redelivery broker-side with different settings and is called out only where the contrast matters.

What this means

In ActiveMQ Classic, redelivery is primarily a client-side mechanism. When a consumer rolls back a transacted session, calls recover(), nacks a message, or dies holding unacknowledged messages, the client (or broker, on abrupt disconnect) marks the message for redelivery. The client-side RedeliveryPolicy on the connection factory controls how many times this happens before the client gives up and sends a “poison ACK” to the broker, which routes the message to the dead letter queue, ActiveMQ.DLQ by default.

The default RedeliveryPolicy is: maximumRedeliveries=6, initialRedeliveryDelay=1000 ms, useExponentialBackOff=false, backOffMultiplier=5.0. Wrappers and frameworks around the client library sometimes override these, so verify against your actual connection factory configuration rather than assuming.

The key operational detail: redelivery happens before the DLQ. A rising redelivery rate is the early warning. DLQ growth is the lagging confirmation that retries were exhausted. If you only alert on DLQ depth, you find out about poison messages after six or more rounds of wasted dispatch work per message.

flowchart LR
  A[Broker dispatches message] --> B{Consumer processes?}
  B -->|ack| C[Dequeue count increments]
  B -->|rollback / nack / timeout| D[Redelivery counter +1]
  D --> E{Attempts left?}
  E -->|yes, after redelivery delay| A
  E -->|no| F[Poison ACK sent]
  F --> G[Message moved to ActiveMQ.DLQ]
  G --> H[Counted as dequeued from source queue]

Note the last step: messages moved to the DLQ count as dequeued from the source queue. Your dequeue rate can look healthy while every “dequeued” message was actually discarded as a failure.

Common causes

Cause	What it looks like	First thing to check
Poison message (malformed payload, schema mismatch, deserialization failure)	Redelivery concentrated on one queue; the same message(s) cycle; DLQ grows after max attempts	Browse the queue or inspect DLQ message properties (`JMSDestination`, exception info, `JMSXDeliveryCount`)
Downstream dependency failure (database, API)	Redelivery across many consumers at once; processing errors in consumer logs; inflight high, dequeue stalled	Consumer application logs for repeated exceptions; downstream health
Consumer restart or deployment	Brief redelivery spike, then back to baseline; correlates with deploy events	Consumer count and connection count timeline around the spike
Blocking redelivery stalling a consumer (Classic)	One consumer connected but not progressing while the redelivery delay blocks its receive thread; looks like a zombie consumer	`nonBlockingRedelivery` on the connection factory; per-consumer `DispatchedQueueSize`
Retry storm against a failing downstream	All consumers fail simultaneously and retry immediately (default 1s delay, no backoff); redelivery tracks dispatch rate almost 1:1	Redelivery delay and backoff settings; downstream error rate
Redelivery policy misconfiguration	Transient failures pushed to DLQ too fast (retries too low), or poison messages cycling too long (retries too high, no backoff)	`maximumRedeliveries`, `redeliveryDelay`, `useExponentialBackOff` on the client

Quick checks

All read-only. The Jolokia paths assume the default web console on 8161 with default credentials; adjust for your deployment, and treat default credentials in production as its own finding.

# 1. DLQ depth: is redelivery already exhausting into the dead letter queue?
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ/QueueSize'

# 2. Per-queue inflight: are consumers holding messages they never ack?
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/InFlightCount'

# 3. Dequeue count, two readings apart: is real completion happening?
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/DequeueCount'

# 4. Expired count: is some of the "drain" actually TTL expiry?
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/ExpiredCount'

# 5. Broker log for rollback and redelivery evidence
grep -i "rollback\|redeliver\|poison" /opt/activemq/data/activemq.log | tail -30

Two cautions on deeper inspection. First, browsing a queue via JMX to check JMSXDeliveryCount on individual messages is expensive on deep queues; do it briefly and on the suspect queue only. Second, JMSTimestamp and delivery-count properties are set from the producer and client side, so clock skew and client version differences can distort what you see.

How to diagnose it

Establish whether the pattern is sustained. Take redelivery-related readings (DLQ depth, dequeue count, inflight) two or three times over five to ten minutes. A single spike that correlates with a consumer restart or deployment is transient noise. A monotonic climb is the incident.
Localize it. One queue or many? One queue points to a poison message or a queue-specific consumer bug. Many queues at once points to a shared downstream dependency or a shared client configuration.
Compare dispatch work against completion. If messages are dispatched at a high rate but dequeue (acknowledgment) is not keeping pace, and inflight sits near consumer_count x prefetch_size, consumers are receiving and failing, not receiving and processing.
Identify the failing message or the failing dependency. Inspect DLQ messages if any have arrived: JMSDestination gives you the source queue, and the redelivery count and exception information tell you what failed. If nothing has reached the DLQ yet, check consumer application logs for the repeating exception. That exception is your root cause in almost every case.
Check for the blocking-redelivery stall. On Classic with default settings, a consumer that rolls back a message can block its receive thread for the entire redelivery delay. If one consumer appears connected but makes no progress while others are fine, you may be looking at a redelivery-blocked consumer rather than a dead one. See the zombie consumer guide for that differential.
Rule out a retry storm. If all consumers began failing at the same moment and redelivery tracks dispatch rate, look downstream first. Immediate retry with no backoff against a degraded dependency is a self-inflicted denial of service: every message gets attempted, fails, and is re-attempted in quick succession, consuming broker and downstream CPU while nothing recovers.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Redelivery rate (destination/consumer stats, `JMSXDeliveryCount > 1` on sampled messages)	Leading indicator for poison messages and DLQ growth	Any sustained rate above the normal error baseline; any nonzero on queues that should never retry
DLQ depth and DLQ growth rate	Lagging confirmation that retries are exhausted; each message is a failed business transaction	Any non-zero depth; growth rate relative to enqueue rate
Dequeue rate vs dispatch activity	Dequeue increments on ack, and DLQ transfers count as dequeues; healthy-looking dequeue can hide failure	Dequeue steady while DLQ grows or real completion stalls
Inflight count vs prefetch	Consumers holding messages they fail and redeliver sit near prefetch saturation	Inflight pinned at `consumer_count x prefetch_size` sustained
Age of oldest pending message	Rising age plus high redelivery means the backlog is aging while the broker churns	Oldest message age climbing despite active consumers
Consumer count	Separates “consumers gone” from “consumers present but failing”	Count stable while redelivery climbs

Fixes

Poison message on one queue

Fix the consumer bug or the producer payload, then deal with the stuck message. If one message is cycling, move it to the DLQ manually via the web console or JMX after capturing it for analysis; that unblocks the messages behind it. Do not purge the DLQ without inspecting it first. Every DLQ message is a bug report with the evidence attached.

Downstream dependency failure

Fix the downstream, not the broker. While the dependency is down, the kindest thing your consumers can do is back off. With the default policy (1s delay, no exponential backoff), every rollback hammers the failing system again. Enabling useExponentialBackOff=true with a sensible maximumRedeliveryDelay reduces load on the struggling dependency and gives it room to recover.

Retry storm from immediate retries

Same lever: redelivery delay and exponential backoff on the client-side RedeliveryPolicy. This is a client configuration change, so it requires a consumer application deploy, not a broker restart. Per-destination policies via RedeliveryPolicyMap let you tune aggressive queues without touching well-behaved ones.

Consumer stalled by blocking redelivery

On Classic, setting nonBlockingRedelivery=true on the connection factory lets a consumer keep processing other messages while a redelivery is pending, instead of blocking its receive thread for the whole delay. The tradeoff is relaxed message ordering on that consumer; per the client API, enabling it also forces TransactedIndividualAck on. If ordering matters, fix the rollback cause rather than unblocking the consumer.

Redelivery policy misconfiguration

If transient failures land in the DLQ before a retry could plausibly succeed, raise maximumRedeliveries or add delay. If poison messages cycle for minutes before being quarantined, lower it for that destination. There is no universally right number; match the retry budget to the expected recovery time of your downstreams.

Artemis contrast

Artemis handles redelivery broker-side via address-settings (max-delivery-attempts, redelivery-delay, redelivery-delay-multiplier). The significant gotcha for mixed environments: an OpenWire client connecting to Artemis applies its own client-side RedeliveryPolicy, which can override the broker’s attempt count, so messages reach the dead letter address after the client’s limit rather than the broker’s. If you run OpenWire clients against Artemis, verify which limit is actually in effect: the broker-side default for max-delivery-attempts in address-settings is 10.

Prevention

Track redelivery as a first-class signal. Separate from dequeue rate, with a baseline per queue. This is the most commonly skipped signal and the earliest warning you get.
Alert on the bundle, not the single metric. Page when rapid redelivery growth coincides with dequeue collapse and rising message age on a critical queue. Ticket on any sustained increase above baseline.
Set redelivery delay and exponential backoff deliberately. The defaults are chosen for development convenience, not production failure handling.
Monitor the DLQ and give messages a TTL. Redelivery’s end state is DLQ growth, and DLQ messages never expire by default. See the related guides on DLQ-adjacent storage growth and expired messages.
Configure per-destination DLQs so poison messages from a noisy queue do not mix with failures from critical workflows.
Design consumers to fail fast and classify errors. A permanent failure (bad payload) should go to the DLQ quickly; a transient failure (downstream timeout) deserves backoff and retries. Treating both the same produces either retry storms or premature quarantine.

How Netdata helps

Redelivery alongside dequeue and dispatch rates, so you can see net-zero progress (high delivery, high redelivery, flat completion) in one view instead of inferring it from separate dashboards.
DLQ depth and growth rate as the lagging confirmation, on the same timeline as redelivery so you can watch the leading-to-lagging sequence during an incident.
Inflight count against prefetch, distinguishing consumers that are failing-and-retrying from consumers that are genuinely stuck.
Consumer count and connection churn, to separate deployment-related transient spikes from sustained failure patterns.
Message age on critical queues, the business-relevant signal that tells you whether the churn is hurting your SLA.
Per-destination breakdowns, to localize a poison-message incident to one queue versus a broker-wide downstream failure in seconds.