$ guides / activemq / activemq-enqueue-dequeue-imbalance ▌

Operations Guides

ActiveMQ enqueue outpacing dequeue: reading the rate imbalance before the backlog

On a healthy ActiveMQ Classic broker, enqueue rate and dequeue rate track each other within normal burst variability. When enqueue outpaces dequeue for more than a few minutes, the broker is accumulating a backlog whether or not QueueSize has moved enough to alarm yet. The rate delta is the leading indicator; QueueSize, MemoryPercentUsage, and StorePercentUsage are the lagging confirmations. If you wait for the backlog gauges to fire, you have already lost your runway.

The catch is that both signals are cumulative JMX counters with accounting behaviors that produce misleading readings: advisory topics inflate the broker-level enqueue count, dequeue increments on acknowledgment rather than dispatch, and messages moved to the DLQ count as dequeued from the source queue. Reading the imbalance correctly means deriving per-destination rates and knowing which of these gotchas applies to your broker.

This article covers how to compute the rates, how to interpret a sustained positive delta, and what to check before you scale consumers or touch the broker.

What this means

TotalEnqueueCount and TotalDequeueCount on the Broker MBean (org.apache.activemq:type=Broker,brokerName=<name>) are cumulative counters that reset only on broker restart. They are not rates. A single reading tells you nothing about current behavior; you need two readings with a known time delta:

enqueue_rate = (TotalEnqueueCount[t2] - TotalEnqueueCount[t1]) / (t2 - t1)
dequeue_rate = (TotalDequeueCount[t2] - TotalDequeueCount[t1]) / (t2 - t1)

The balance equation is enqueue_rate - dequeue_rate. A sustained positive value means the backlog will grow at roughly that rate, and QueueSize will confirm it later. The magnitude tells you how fast you are approaching trouble: the broker memory runway in seconds is approximately (memory_limit - current_usage) / ((enqueue_rate - dequeue_rate) * avg_message_size).

Broker-level counters are the wrong granularity for diagnosis. Use them for a quick health glance, then drop to per-destination EnqueueCount and DequeueCount on the Queue/Topic MBeans to find which destination is actually imbalanced.

The cascade you are trying to intercept looks like this:

flowchart TD
  A[enqueue rate > dequeue rate sustained] --> B[QueueSize starts growing]
  B --> C[MemoryPercentUsage climbs]
  B --> D[StorePercentUsage climbs]
  C --> E[100 percent: producer flow control, send blocks silently]
  D --> F[100 percent: persistent messaging halts]
  A -.leading read.-> G[rate delta alarms here]
  B -.lagging confirmation.-> G

The point of monitoring the rate delta is to act at the top of that cascade, not at the bottom where producers are already blocked.

Common causes

Cause	What it looks like	First thing to check
Consumer application down or crashed	Dequeue rate at or near zero, ConsumerCount dropped, enqueue steady	ConsumerCount per destination
Consumer processing bottleneck (slow DB, downstream API)	Dequeue rate degraded but nonzero, InFlightCount pinned at prefetch	InFlightCount vs consumer_count x prefetch_size
Traffic spike or replay burst	Enqueue rate 3x+ baseline, dequeue unchanged, drains once burst ends	Enqueue baseline comparison; is the delta shrinking?
DLQ transfer masquerading as healthy consumption	Dequeue rate looks fine, but DLQ QueueSize is growing	EnqueueCount on ActiveMQ.DLQ
Silent expiry	Stable or slowly moving QueueSize, low dequeue, ExpiredCount climbing	ExpiredCount on the destination
Selector mismatch or zombie consumer	Consumers connected, InFlightCount high or zero, dequeue near zero	InFlightCount and ConsumerCount together

Quick checks

All of these are read-only. The examples use the Jolokia endpoint on the web console (port 8161) with the default broker name localhost; adjust the broker name and credentials for your deployment.

# Two readings 60s apart to derive broker-level rates
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/TotalEnqueueCount,TotalDequeueCount'
sleep 60
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/TotalEnqueueCount,TotalDequeueCount'

# Per-destination rates: pull EnqueueCount and DequeueCount for all queues twice
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=*/EnqueueCount,DequeueCount'

# Backlog and consumer state on the suspect queue
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/QueueSize,InFlightCount,ConsumerCount,ExpiredCount'

# Is the dequeue rate actually DLQ transfer?
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ/QueueSize,EnqueueCount'

# How much runway before flow control?
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/MemoryPercentUsage,StorePercentUsage'

Two cautions. JMX queries serialize MBeans and are not free on brokers with many destinations; keep polling intervals reasonable. And do not use the browse() operation to inspect a deep queue during an incident: a full browse spikes broker memory and CPU.

How to diagnose it

Compute per-destination rates, not broker-level rates. The broker-level TotalEnqueueCount includes advisory topic traffic, which inflates the number without representing application throughput. In a Network of Brokers, forwarded messages also count as enqueues on the receiving broker, so do not double-count across the topology. Drop to per-destination counters before concluding anything.
Confirm the imbalance is sustained, not a burst. Brief windows where enqueue exceeds dequeue are normal in bursty workloads; queue depth oscillation is expected. What matters is the delta sustained over 10 minutes or more. Also check whether the delta is shrinking on its own: a spike with a recovering dequeue rate is a burst draining, not an incident.
Rule out restart artifacts. After a broker restart with a stored backlog, dequeue rate spikes as the broker dispatches and consumers ack the replayed messages. That is catch-up, and memory jumping to 50-70% as cursors page stored messages in is normal cursor behavior. Check uptime before alarming on either rate.
Validate the dequeue number before trusting it. DequeueCount increments on acknowledgment, not dispatch. With CLIENT_ACKNOWLEDGE or transacted sessions there is a real delay between dispatch and the counter moving, so check InFlightCount: high inflight with low dequeue means consumers received messages but are not acking (stuck or overloaded). Separately, check the DLQ: messages moved to the DLQ count as dequeued from the source queue, so a healthy-looking dequeue rate can be entirely DLQ transfer. If DLQ EnqueueCount is growing at roughly your source queue’s dequeue rate, your consumers are failing, not keeping up.
Check for silent expiry. If QueueSize is oddly stable while dequeue is near zero, look at ExpiredCount. Messages may be expiring as fast as they arrive, which is correctness loss wearing a healthy-looking queue depth.
Classify the consumer state. Combine three readings: ConsumerCount (are consumers connected?), InFlightCount (have they received but not acked?), dequeue rate (are they completing?). Consumers connected plus inflight pinned at prefetch plus collapsed dequeue is the stuck-consumer pattern. Zero consumers plus growing enqueue is the page-immediately condition.
Estimate runway. With the confirmed net accumulation rate, compute time-to-flow-control against MemoryPercentUsage and time-to-store-full against StorePercentUsage and actual disk free. This decides whether you are in a “fix the consumer this afternoon” situation or a “shed load now” situation.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
enqueue_rate - dequeue_rate per destination	The leading read; predicts backlog before QueueSize confirms	Sustained positive delta over 10 minutes
Enqueue/dequeue ratio	Normalized balance independent of volume	Greater than 1.5 sustained; greater than 3.0 is an active shortfall
QueueSize	Backlog confirmation; includes inflight messages	Growth rate, not absolute depth; compare against baseline
InFlightCount	Separates “not dispatched” from “dispatched but not acked”	Pinned at consumer_count x prefetch_size with low dequeue
ConsumerCount	Zero consumers means nobody is draining	Below expected minimum, or zero with enqueue still flowing
DLQ EnqueueCount / QueueSize	Exposes dequeue rates that are really DLQ transfer	Growing in step with source queue dequeue rate
ExpiredCount	Detects silent loss behind a stable QueueSize	Any unexpected sustained expiry
MemoryPercentUsage / StorePercentUsage	The cliff the imbalance is driving toward	Climbing in step with the rate delta; 100% is flow control

A workable severity ladder: ticket when the ratio exceeds 1.5 sustained for over 10 minutes, page when the ratio exceeds 3.0 sustained, and page immediately when dequeue rate is zero while enqueue rate is positive, QueueSize is nonzero, consumer count is zero, and broker uptime excludes restart noise.

Fixes

Restore consumer capacity

If consumers crashed or are undersized, that is the fix path: restart or scale the consumer application. While they recover, watch MemoryPercentUsage, because the backlog drain itself pages messages into memory and can transiently push usage up.

Unstick consumers that are connected but not acking

Inflight pinned at prefetch with collapsed dequeue means the consumer’s prefetch buffer is full and processing is stalled, usually on a slow downstream dependency or thread pool exhaustion. Restarting the slow consumer releases its inflight messages for redelivery to healthy consumers. Reducing prefetch limits how much a single slow consumer can hoard. As a longer-term safety valve, a configured slow consumer strategy (including AbortSlowConsumerStrategy) can disconnect chronically slow consumers automatically, but that is aggressive and should be a deliberate policy choice.

Stop poison-message churn

If the “healthy” dequeue rate was DLQ transfer, the real fix is in the consumer or the message, not the broker. Inspect DLQ messages for JMSDestination and exception properties to find the source queue and failure mode, fix the consumer, then decide whether to replay or purge the DLQ. Do not just purge and move on: the poison messages will return.

Absorb a legitimate burst

If the imbalance is a real traffic spike and consumers are healthy, the options are headroom or time. Confirm runway against memory and store limits, and let the backlog drain. If bursts are recurring, the fix is consumer capacity or producer-side rate control, not broker tuning.

Do not restart the broker as a first move. A restart triggers KahaDB recovery and a replay burst, loses non-persistent messages, and converts a diagnosable consumer problem into an availability event.

Prevention

Alert on the rate delta, not just QueueSize. Per-destination enqueue minus dequeue, sustained over 10 minutes, is the earliest reliable signal. QueueSize and memory alerts are the safety net, not the tripwire.
Compute time-to-clear, not raw depth. Backlog divided by dequeue rate gives you an SLA-relevant number that pages only when the business cares.
Watch the DLQ as a first-class destination. Any non-zero DLQ depth is a processing failure; DLQ growth invalidates dequeue-rate health on the source queues.
Keep consumer expectations explicit. Per-queue expected consumer counts and prefetch-aware inflight thresholds catch the zombie-consumer pattern before the rate delta does.
Correlate with memory and store. The rate delta plus a climbing MemoryPercentUsage is a flow-control incident in progress; the rate delta with flat memory is a backlog you have time to fix.

How Netdata helps

Netdata collects the ActiveMQ JMX counters and derives enqueue and dequeue rates automatically, so you are reading rates instead of differencing cumulative counters by hand.
Per-destination charts let you see which queue’s delta is driving the broker-level imbalance without querying MBeans one at a time.
Correlating the rate delta with QueueSize, InFlightCount, and ConsumerCount on one dashboard is what separates “consumers down” from “consumers stuck” from “DLQ churn” in one glance.
Plotting the imbalance against MemoryPercentUsage and StorePercentUsage turns the runway estimate into a visible trend line instead of a back-of-envelope calculation.
Alerting on a sustained positive delta gives you the leading alarm this article describes, ahead of the backlog gauges.