ActiveMQ consumers connected but not acknowledging: the zombie consumer

The queue has consumers. ConsumerCount is exactly where it should be. But dequeue rate has collapsed to near zero, InFlightCount is pinned at the prefetch limit, and the backlog keeps growing. From the broker’s perspective everything is subscribed; from the business’s perspective nothing is being processed.

This is the zombie consumer: a consumer that holds a live connection and a full prefetch buffer but never acknowledges. ActiveMQ dispatched the prefetch window of messages to it, and the broker will not send that consumer more until acks come back. If it is the only consumer, the queue stalls while looking fully staffed.

The operational trap is that the two signals most teams watch, consumer count and queue depth, both look ambiguous here. ConsumerCount > 0 looks fine. QueueSize can even be low, because QueueSize includes inflight messages: if QueueSize = 1000 and InFlightCount = 1000, the broker considers the queue drained while the messages sit in the consumer’s prefetch buffer, unprocessed. You only see the failure when you correlate three signals: ConsumerCount, InFlightCount, and dequeue rate.

What this means

ActiveMQ dispatches messages to a consumer up to its prefetch window (default 1000 for queues and 100 for persistent topics, up to Short.MAX_VALUE (~32k) for non-persistent topics) and then waits for acknowledgments before topping up. Dequeue counters increment on acknowledgment, not on dispatch. So the signature of a zombie consumer is precise:

• ConsumerCount is at its expected value (TCP connection and JMS session are alive).
• Dequeue rate is zero or near zero while messages are pending or enqueues continue.
• InFlightCount equals consumer_count x prefetch_size, sustained. Every prefetch slot is occupied and none are being freed.

The connection being alive tells you nothing about the processing path. The consumer’s message listener thread may be deadlocked, its executor pool may be exhausted, it may be blocked indefinitely on a downstream database or API, or it may never call acknowledge() because the ack mode was misconfigured. A consumer left over from a stale deployment can also hold a connection and mis-handle messages, which looks identical from the broker. Connected does not mean processing.

flowchart TD
  A[Dequeue rate collapsed] --> B{ConsumerCount > 0?}
  B -- No --> C[Consumer absence: different incident]
  B -- Yes --> D{InFlightCount pinned at prefetch?}
  D -- No --> E[Slow but processing: check downstream latency]
  D -- Yes --> F[Zombie consumer]
  F --> G{Ack mode by design? CLIENT_ACK / transacted}
  G -- Yes --> H[High inflight expected: check ack/commit logic]
  G -- No --> I[Thread dump consumer: deadlock, pool exhaustion, blocked downstream call]
  I --> J[Restart or disconnect consumer: inflight redelivered]

Common causes

Quick checks

All read-only. Run against the broker via Jolokia (default web console port 8161) or any JMX client.

# 1. Confirm consumers are attached
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/ConsumerCount'

# 2. Check inflight against prefetch x consumer count
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/InFlightCount'

# 3. Read dequeue counter twice, 60s apart, to derive the rate
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/DequeueCount'

# 4. Queue depth: remember QueueSize INCLUDES inflight
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/QueueSize'

If InFlightCount equals ConsumerCount x prefetch_size and the dequeue delta over 60 seconds is zero, you have confirmed the zombie pattern. EnqueueCount and DequeueCount are cumulative counters; you always need two readings to get a rate.

Also check the broker log for slow-consumer advisories, which only fire if a slow consumer strategy is configured:

grep -i "slow consumer" /opt/activemq/data/activemq.log | tail -20

How to diagnose it

1. Confirm the bundle, not one signal. ConsumerCount at the expected value, InFlightCount == consumer_count x prefetch sustained over several minutes, dequeue rate near zero, and pending or enqueued work still arriving. A single reading of high inflight means little: transacted and client-ack consumers hold inflight by design.
2. Identify which consumer is stuck. Enumerate the per-consumer subscription MBeans under the destination (endpoint=Consumer) and read DispatchedQueueSize / MessageCountAwaitingAcknowledge. The zombie is the one sitting at its prefetch ceiling. The MBean path also carries the client ID, which maps to the application instance.
3. Check whether the inflight level is by design. If the consumer uses CLIENT_ACKNOWLEDGE or SESSION_TRANSACTED, high inflight is expected between acks or commits. The question becomes whether acks or commits ever arrive. If the dequeue delta over a full processing cycle is zero, they do not.
4. Rule out DLQ churn masking the stall. Messages moved to the DLQ count as dequeued from the source queue. If the dequeue rate is low but not zero, check ActiveMQ.DLQ depth growth before concluding consumers are processing. You may have a poison-message redelivery loop instead of a zombie.
5. Get a thread dump from the consumer JVM. This is the single most valuable artifact. Deadlocks show as mutually blocked threads; pool exhaustion shows all worker threads busy with a growing task queue; a dead downstream shows every thread parked in the same socket read or JDBC call. The stack traces will name the dependency.
6. Verify the deployment. Compare the stuck consumer’s client ID and source IP against the expected deployment inventory. A stale instance from an old release counts as a consumer but may fail on every message in a way that never acks.
7. Check expiry as a confounder. A queue with stable depth and zero dequeue can be expiring messages as fast as they arrive. Check ExpiredCount before blaming the consumer.

Metrics and signals to monitor

Fixes

Restart or disconnect the stuck consumer

When a consumer disconnects, its inflight messages are redelivered to the remaining consumers. For a single-consumer queue, restarting the consumer process returns the prefetch-buffered messages to the queue for redispatch. This is the fastest recovery, but capture a thread dump first, or you lose the evidence and the incident recurs. If the queue has competing consumers, disconnecting the zombie rebalances its inflight to the healthy ones without a full application restart.

Fix the downstream dependency

If the thread dump shows every processing thread parked on the same database or HTTP call, the broker is a bystander. Recover the dependency, and the consumer will drain its prefetch and ack normally. Add timeouts and circuit breakers on the consumer’s downstream calls so a hung dependency cannot hold prefetch slots indefinitely.

Correct the ack mode

If the consumer uses CLIENT_ACKNOWLEDGE, verify that acknowledge() is actually called on the success path, including exception paths. If transacted, verify commits happen and rollback loops are bounded by a redelivery policy that eventually sends to the DLQ. If optimizeAcknowledge is in play, acks are batched (fired at 65% of prefetch or a 300ms timer), which delays dequeue increments and can leave the tail batch unacked when flow stops.

Size prefetch to processing capacity

Prefetch is a buffer, not a performance feature. A prefetch of 1000 against a consumer that processes 10 messages per second means a stuck consumer can pin 100 seconds of work. Lowering prefetch limits the blast radius of any single zombie and rebalances work to healthy consumers faster. For pooled consumer patterns (for example Spring’s caching connection factory), pooled-but-idle consumers can hold prefetched messages until reused; disable consumer caching or use prefetch 0 (pull mode) for those pools. Note that prefetch 0 disables asynchronous MessageListener delivery.

Disconnect slow consumers automatically

ActiveMQ supports slow-consumer strategies on destination policies. AbortSlowConsumerStrategy aborts consumers whose prefetch buffer stays full, and AbortSlowAckConsumerStrategy (5.9+) aborts on time-since-last-ack instead, which works better for low-prefetch consumers. Both default to closing the consumer without forcibly killing the TCP connection (abortConnection="false"); set abortConnection="true" if you need the socket gone. These are blunt instruments: a consumer aborted during a legitimate slow patch will reconnect and rejoin, so pair them with alerting rather than treating them as the fix. The XML element and attributes are unchanged in ActiveMQ 6.x (abortSlowAckConsumerStrategy with abortConnection, maxTimeSinceLastAck, and checkPeriod).

Prevention

• Alert on the bundle, not the parts. Page when inflight is pinned at prefetch AND dequeue has collapsed AND message age is rising, sustained, on a critical queue. Ticket when inflight equals prefetch for any single consumer beyond a couple of minutes. Raw consumer count alerts alone will never catch this.
• Track message age, not just depth. A shallow queue with an old oldest-message is worse than a deep fresh one, and a zombie consumer produces exactly that shape. Derive age from the JMSTimestamp of the oldest pending message; browse sparingly on deep queues because browsing is expensive.
• Per-consumer inflight monitoring. Broker-aggregated InFlightCount hides which consumer is stuck. Track DispatchedQueueSize per subscription on critical queues so diagnosis is a lookup, not a hunt.
• Consumer-side instrumentation. Thread-pool saturation, processing latency, and downstream call latency in the consumer application are where zombie consumers are born. The broker can only show you the result.
• Bound every downstream call. No unbounded JDBC or HTTP waits in the message listener. A hung call with no timeout is the most common way a healthy consumer becomes a zombie.
• Deployment hygiene. Versioned client IDs, and a check that connected consumer identities match the current deployment, catch stale-instance zombies that count-based monitoring waves through.

How Netdata helps

• Netdata collects ActiveMQ JMX metrics per destination, so ConsumerCount, InFlightCount, QueueSize, and enqueue/dequeue rates are on one dashboard, making the zombie signature (consumers present, inflight pinned, dequeue flat) visible as a shape rather than three separate checks.
• Rate derivation from cumulative counters like EnqueueCount and DequeueCount is automatic, so a dequeue collapse shows up immediately without manual delta math.
• Per-second collection catches the transition: the moment dequeue flatlines while inflight saturates is exactly the moment to start the clock on your stuck-consumer threshold.
• Correlating broker signals with consumer-side host metrics (CPU flat, threads blocked, downstream latency spiking) on the same timeline shortens the path from “queue stalled” to “consumer pool exhausted on the database call.”
• DLQ depth and expired-count charts alongside dequeue expose the look-alike cases where messages are being discarded or expired rather than processed.
• Alerts on composite conditions let you encode the safe paging bundle (inflight pinned AND dequeue collapsed AND age rising) instead of paging on any single noisy signal.

Related guides

• ActiveMQ InFlightCount high: prefetch full, acks stalled, and zombie consumers
• ActiveMQ oldest message age: the queue latency depth alone cannot show
• ActiveMQ MemoryPercentUsage climbing: reading the flow-control leading indicator
• ActiveMQ memory limit reached: MemoryPercentUsage at 100% and the flow-control cliff
• ActiveMQ KahaDB journal files not deleted: one unacked message pinning a 32MB log
• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ monitoring checklist: the signals every production broker needs