You have four consumers on a queue, producers are sending steadily, and yet throughput is a quarter of what you expect. ConsumerCount says 4. QueueSize looks small. DequeueCount is crawling. The broker looks healthy and the consumers look connected, but three of them are doing nothing.

This is the classic prefetch mismatch. The ActiveMQ prefetch limit controls how many messages the broker pushes to a consumer before it requires acknowledgments back. It is a client-side buffer the broker fills eagerly. With the default of 1000 for queues, the first consumer to connect can absorb the entire visible backlog into its prefetch buffer, where those messages are invisible to load balancing, no longer read as pending depth, and pinned against broker memory until they are acked.

What the prefetch limit actually controls

The prefetch limit is the maximum number of messages the broker will dispatch to a consumer without waiting for acknowledgments. It exists to keep consumers busy: without prefetching, every message would require a round trip, and consumer throughput would collapse to network latency per message.

Defaults in ActiveMQ Classic 5.x:

Destination typeDefault prefetch
Queue (persistent and non-persistent)1000
Topic (persistent)100
Topic (non-persistent)32766 per the official docs; some sources and versions report 32767 (Short.MAX_VALUE)

The key operational facts:

  • Prefetched messages are dispatched but not acknowledged. They appear in InFlightCount, and QueueSize includes inflight messages. A queue with QueueSize=1000 and InFlightCount=1000 is, from the broker’s dispatch perspective, drained. Everything is sitting in consumer buffers.
  • Prefetched messages still count against broker memory accounting until acknowledged. A large prefetch across many consumers is a real memory commitment, not a free performance knob.
  • Once a consumer’s prefetch buffer is full, the broker stops dispatching to it until acks free up slots. Per the official documentation, dispatch resumes in top-up batches once the consumer has acked roughly half of the prefetched messages.
  • Setting prefetch to 0 switches the consumer to polling mode: it pulls one message at a time instead of receiving pushed messages. This changes the performance profile completely and, as a side effect reported by operators, can inflate the console’s Enqueues counter.

For the broader picture of how dispatch, cursors, and memory accounting fit together, see how ActiveMQ Classic actually works in production.

How dispatch and prefetch interact

Queue dispatch in ActiveMQ Classic is competing-consumer: each message goes to exactly one consumer. The dispatch thread is nominally round-robin, but there is a subtlety that bites operators constantly: the broker fills one consumer’s prefetch buffer before moving on to the next. The official dispatch policies documentation is explicit that with many consumers, a high prefetch value, and a small number of messages, load balancing is not fair.

flowchart LR
  P[Producers] --> Q[Queue pending cursor]
  Q --> D[Dispatch thread]
  D -->|fills prefetch 1000| CA[Consumer A buffer FULL]
  D -.->|nothing left to push| CB[Consumer B idle]
  D -.->|nothing left to push| CC[Consumer C idle]
  CA -->|acks free slots| D

Consumer A is not greedy or buggy. It connected first, and the broker did exactly what the prefetch setting told it to do. The messages in A’s buffer are now exclusive to A: they cannot be redispatched to B or C unless A disconnects or its acks free slots and new messages arrive.

The two failure modes

Unacked hoarding

One consumer holds a large number of dispatched-but-unacked messages while processing them slowly. Consequences:

  • Hidden backlog. Queue depth looks modest because the messages are inflight, not pending. Operators watching only QueueSize conclude the queue is fine while hundreds of messages sit unprocessed in one consumer’s buffer. The extreme version is the zombie consumer pattern described in InFlightCount high: QueueSize near zero, InFlightCount high, dequeue rate near zero.
  • Pinned broker memory. Unacked messages remain charged against memory accounting. A consumer with prefetch 1000 holding large messages can push a destination toward its memory limit on its own.
  • Pinned store. For persistent messages, journal files are only reclaimable when every message in the file is acknowledged. A hoarded message that never gets acked pins its journal file, which is how prefetch misconfiguration feeds the journal files not deleted problem.
  • Slow failure detection. The consumer is connected and the broker sees no error. Only the inflight-versus-dequeue correlation reveals it.

Idle consumers

The mirror image: prefetch too large relative to message volume, so the dispatch thread drains everything into whichever consumer it visits first and the rest starve. You pay for four consumer instances and get the throughput of one. This is worst when message volume is low or bursty and processing time is long, because the hoarding consumer never drains its buffer fast enough for the imbalance to self-correct.

The failure mode in the other direction is prefetch too small: consumers finish each message, then wait for the ack round trip and the next dispatch before they have more work. Throughput drops even though CPU on the consumer side is idle. You see this as a low, steady dequeue rate with healthy consumers and a queue that never drains as fast as it should.

Acknowledgment mode changes the math

The prefetch limit interacts with ack mode, and you cannot reason about one without the other.

  • AUTO_ACKNOWLEDGE. The client acks automatically, so InFlightCount stays lower and prefetch slots free quickly. The official performance tuning documentation notes that acks are batched, with the batch size tied to a fraction of the prefetch limit. The trap: acking is decoupled from how fast your downstream processing actually drains, so the broker keeps a consumer’s buffer topped up even while that consumer’s work is backing up and a peer sits idle. Prefetch hoarding can still happen; it just does not always show up as high broker inflight.
  • CLIENT_ACKNOWLEDGE and transacted sessions. Inflight runs high by design, because acks (or commits) lag dispatch deliberately. High inflight here is not a stuck consumer by itself; the warning sign is inflight pinned at exactly consumer_count x prefetch with a collapsing dequeue rate.
  • optimizeAcknowledge. Changes when batch acks are sent, which changes how quickly prefetch slots free. If you enable it, verify the interaction with your prefetch size empirically.

Sizing prefetch against your workload

There is no universally correct number. Size against message processing time, message size, and how evenly work must fan out.

WorkloadSuggested starting pointWhy
Fast, uniform processing (sub-100ms), high volume100-1000 (default is fine)Batching amortizes round trips; imbalance self-corrects because buffers drain quickly
Slow, uneven processing (seconds per message), must fan out evenly1-10Each consumer holds almost nothing, so dispatch stays fair and a stuck consumer pins almost nothing
Large messagesLower than defaultPrefetched bytes are charged to broker memory; 1000 large messages is a serious memory commitment
STOMP or scripting-language consumers that cannot buffer1The official docs recommend prefetch 1 for consumers that cannot cache prefetched messages
Pooled/cached consumers (e.g. caching connection factories)0 (polling) or disable consumer cachingPooled consumers defer close, so prefetched messages sit unconsumed in a buffer nobody is draining

The prefetch=1 recommendation for slow, uneven workloads deserves emphasis. Operators often raise prefetch chasing throughput, when the actual problem is that one slow message blocks 999 others in the same buffer. With prefetch 1, a slow message delays exactly one message, and the remaining work flows to the other consumers.

Where this shows up in production

Concrete checks that distinguish hoarding from starving on a live broker:

# Queue depth vs inflight: are messages pending or sitting in consumer buffers?
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/QueueSize'

curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/InFlightCount'

# How many consumers are supposed to be sharing the load?
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/ConsumerCount'

Reading the results:

  • QueueSize roughly equal to InFlightCount, with ConsumerCount > 1 and a low dequeue rate: one or few consumers are hoarding. Check per-consumer DispatchedQueueSize or MessageCountAwaitingAcknowledge on the subscription MBeans to find which one.
  • InFlightCount near zero, dequeue rate low, consumers healthy and idle: prefetch may be too small, or the bottleneck is downstream of the broker entirely.
  • InFlightCount pinned at ConsumerCount x prefetch sustained: every consumer’s buffer is full. If dequeue is not keeping pace, the consumers are saturated or stuck, which is the pattern covered in InFlightCount high.

Broker-side memory pressure is the other tell. If hoarded messages are large, watch MemoryPercentUsage on the destination and the broker. Prefetch-driven hoarding is one of the less obvious paths to the flow-control cliff described in MemoryPercentUsage climbing.

Signals to watch in production

SignalWhy it mattersWarning sign
InFlightCount per destinationMessages sitting in consumer prefetch buffersEquals consumer_count x prefetch sustained, or high while QueueSize is near zero
InFlightCount / QueueSize ratioShows how much of the visible depth is actually hoarded vs pendingRatio near 1.0 with multiple consumers: load balancing is defeated
Per-consumer DispatchedQueueSizeIdentifies which specific consumer is hoardingOne consumer at its prefetch limit while peers are at zero
Dequeue rate vs consumer countReveals whether all consumers are actually workingThroughput of one consumer while N are connected
MemoryPercentUsage (destination and broker)Hoarded unacked messages still cost broker memoryClimbing memory with modest queue depth
Message age of oldest pending messageThe latency symptom hoarding hidesOld messages despite shallow queue depth; see oldest message age

A useful composite alert: page only when inflight is pinned near total prefetch AND dequeue has collapsed AND message age is rising, sustained on a critical queue. Any one of those alone false-pages on transacted or delayed-ack consumers.

How Netdata helps

  • Netdata’s ActiveMQ collector pulls JMX destination metrics including QueueSize, InFlightCount, ConsumerCount, and enqueue/dequeue counts, so the hoarding signature (depth equal to inflight, dequeue collapsed, consumers connected) is visible on one dashboard instead of three curl commands.
  • Per-second collection catches the transient version of this problem: a consumer that hoards for 30 seconds during a GC pause and then recovers, which minute-scrape monitoring smooths into invisibility.
  • Correlating InFlightCount against MemoryPercentUsage on the same timeline shows when a prefetch buffer, not a true backlog, is what is pushing the broker toward flow control.
  • Alerting on the ratio of inflight to queue size, and on dequeue rate per connected consumer, turns the idle-consumer failure mode into a detectable condition rather than a quiet waste of consumer capacity.
  • Historical retention lets you verify a prefetch change actually worked: compare inflight distribution and dequeue rate before and after, rather than trusting the first five minutes.