$ guides / activemq / activemq-queue-size-growing ▌

Operations Guides

ActiveMQ QueueSize growing: reading the broker's backlog gauge correctly

QueueSize is the first number every operator looks at, and the one most often misread. The common mental model is “messages waiting in the queue,” but that is not what the broker reports. QueueSize is a broker-maintained gauge, not enqueued-minus-dequeued, and it includes messages already dispatched to consumers but not yet acknowledged. Two queues showing QueueSize=5000 can be in completely different states, and a queue showing QueueSize=0 can still be in trouble.

This article explains what the gauge counts, the specific ways it misleads during an incident, and the derived signals (growth rate, time-to-clear, message age) that are safe to alert on. For the broader broker mental model, see How ActiveMQ Classic actually works in production.

What QueueSize actually counts

The per-destination QueueSize attribute (JMX: org.apache.activemq:type=Broker,brokerName=<name>,destinationType=Queue,destinationName=<q>) is the broker’s count of messages not yet acknowledged by a consumer. Two consequences surprise people:

It is not a delta of counters. EnqueueCount and DequeueCount are cumulative lifetime counters that reset on broker restart. You cannot subtract one from the other and expect QueueSize, especially across restarts. QueueSize is a point-in-time gauge maintained by the broker itself.

It includes inflight messages. Messages dispatched to a consumer’s prefetch buffer but not yet acknowledged still count. The separate InFlightCount attribute tells you how many of those exist:

QueueSize = pending (undispatched) + inflight (dispatched, unacked)
QueueSize=1000 with InFlightCount=1000 means the broker has nothing left to hand out. Every message is sitting in consumer prefetch buffers. From the broker’s perspective the queue is drained; from the business’s perspective nothing has been processed.
QueueSize=0 with InFlightCount=500 is the zombie-consumer pattern: the queue looks empty, but 500 messages are stuck in consumer limbo, received but never acked.

Reading QueueSize without InFlightCount is reading half the gauge.

The message lifecycle behind the gauge

The gauge only makes sense against the dispatch path:

flowchart LR
  P[Producer send] --> S[Store / pending cursor]
  S -->|dispatch up to prefetch| I[Consumer prefetch buffer - inflight]
  I -->|ack| D[Dequeued - store cleanup]
  I -.->|consumer disconnects| S
  Q["QueueSize = pending + inflight"]
  S -.counts.- Q
  I -.counts.- Q

Key mechanics:

Dispatch is push-based, bounded by prefetch. The dispatch thread pushes messages to each consumer up to its prefetch window (default 1000 for queues). The broker stops pushing when the window is full and tops up as acks arrive.
DequeueCount increments on acknowledgment, not dispatch. With CLIENT_ACKNOWLEDGE or transacted sessions, the gap between dispatch and dequeue can be long by design. With AUTO_ACKNOWLEDGE, inflight is normally very low.
Abrupt consumer disconnect redelivers. Inflight messages from a dead consumer return to pending and are redispatched, which shows up as a transient QueueSize bump plus redeliveries.
DLQ transfers count as dequeues. A message moved to ActiveMQ.DLQ increments the source queue’s DequeueCount. Dequeue rate can look healthy while messages are actually being discarded.

How the gauge misleads in production

High prefetch hides backlog in client memory. With the default prefetch of 1000, a fleet of 20 consumers can hold 20,000 messages in their own buffers. Those messages are invisible to other consumers and to any naive “is the queue empty” check, yet they are unprocessed work. The broker can look nearly drained while the real backlog sits in consumer memory. This is the most common misreading: QueueSize drops, the alert clears, and processing latency is still climbing.

Absolute depth means nothing without a baseline. 5000 pending messages is a rounding error for a queue processing 2000 msg/s, and a full consumer outage for a queue that normally sits at 10. Fixed thresholds on raw depth either page on noise or miss real failures, depending on the queue.

Stable depth can hide expiry churn. If messages arrive and expire at similar rates, QueueSize stays flat and DequeueCount stays near zero while ExpiredCount climbs. The queue looks calm; business events are being silently lost.

Growing depth can be benign. Batch workloads, scheduled delivery, and post-restart catch-up all produce legitimately deep queues. Depth oscillation in bursty workloads is normal; sustained monotonic growth is the pattern that matters.

Inflight counts can go weird under prefetch races. When consumers close with unconsumed prefetched messages, inflight accounting can transiently report odd values, including negative counts reported against past releases (AMQ-2262 for inflight going negative when messages expire, AMQ-1940 for negative queue size), each fixed in its own 5.x release. If your alerting does math on InFlightCount, guard against transient negative or implausible readings rather than paging on them.

Reading it correctly: the signals that matter

Rate of growth, not depth

enqueue_rate - dequeue_rate sustained positive is the real backlog signal. QueueSize is the integral of that difference; the difference itself tells you the trajectory and gives you lead time. Alert on sustained positive growth (for example, depth exceeding 2x baseline for more than 15 minutes), not on crossing a static count.

Time-to-clear

time_to_clear = QueueSize / dequeue_rate converts depth into the unit the business cares about: how long until a newly arrived message gets processed if nothing changes. “This queue has 45 minutes of work backed up” is an SLA statement; “this queue has 90000 messages” is not. When dequeue rate collapses toward zero with non-zero depth, time-to-clear goes to infinity. That is the actual incident.

Message age

The age of the oldest pending message is the most business-relevant latency signal. A shallow queue with hour-old messages is worse than a deep queue of fresh ones. Caveats: there is no reliable direct JMX attribute for this. You must browse the queue and read JMSTimestamp of the first message, browsing is expensive on deep queues, and JMSTimestamp comes from the producer’s clock, so clock skew distorts it.

Inflight-to-prefetch saturation

InFlightCount approaching consumer_count x prefetch_size sustained means every consumer’s buffer is full and nobody is acking: consumers are stuck or overloaded. Combined with a falling dequeue rate, this distinguishes “consumers dead” (ConsumerCount=0) from “consumers present but wedged” (ConsumerCount normal, inflight pinned, dequeue collapsed), which is the zombie-consumer case.

Quick read-only checks

# QueueSize, InFlightCount, ConsumerCount for one queue via Jolokia
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/QueueSize,InFlightCount,ConsumerCount,DequeueCount,EnqueueCount,ExpiredCount'

# All queues at once (bulk read, cheaper than per-queue polling)
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=*/QueueSize,InFlightCount,ConsumerCount'

Both commands are read-only; the only cost is JMX read load on the broker. Two operational notes:

EnqueueCount and DequeueCount are cumulative counters. Take two readings with a known interval and derive rates; never alert on the counters themselves.
Heavy JMX polling is not free on brokers with many destinations. Prefer bulk reads and reasonable intervals over per-attribute polling at high frequency.

Signals to watch in production

Signal	Why it matters	Warning sign
QueueSize (per destination)	Canonical backlog gauge, but includes inflight	Sustained monotonic growth; depth >2x baseline for >15 min
InFlightCount	Exposes prefetch-hidden backlog and stuck consumers	Pinned near consumer_count x prefetch while dequeue collapses
Enqueue vs dequeue rate	The trajectory; gives lead time before depth becomes a problem	Sustained positive gap; dequeue=0 with enqueue>0
Time-to-clear (depth / dequeue rate)	Backlog expressed as SLA time	Exceeds per-queue processing SLO
Oldest message age	True end-to-end latency from the broker’s view	Oldest message older than queue SLA
ConsumerCount	Zero consumers means nothing drains regardless of depth	Critical queue at zero with depth or enqueue present
ExpiredCount	Flat depth can hide expiry churn	Non-zero sustained expiry on queues designed for none
DLQ QueueSize	DLQ transfers count as dequeues on the source queue	Any sustained growth; check store usage on the DLQ destination

How Netdata helps

Netdata collects the per-destination JMX gauges (QueueSize, InFlightCount, ConsumerCount) alongside enqueue/dequeue counters and derives rates, so depth and trajectory appear on the same chart instead of computing deltas by hand.
Plotting QueueSize next to InFlightCount makes the prefetch-hiding effect visible: depth dropping while inflight rises is backlog moving into consumer memory, not backlog clearing.
Rate-of-change views on queue depth surface sustained monotonic growth, the alertable pattern, while bursty oscillation stays visibly benign.
Correlating dequeue rate with DLQ depth catches the “healthy dequeue that is actually DLQ transfers” case, and correlating flat depth with rising ExpiredCount catches silent expiry.
Because Netdata samples per second and keeps per-destination series, you can alert on derived ratios (growth rate, inflight saturation) rather than raw counts that need per-workload tuning.