ActiveMQ per-destination memory usage: one noisy queue blocking every producer

Every producer on the broker is stuck in send(). Broker-level MemoryPercentUsage reads 100, the broker log shows the usage manager hitting its memory limit, and upstream services are timing out. But when you list queue depths, almost every queue is empty. One queue holds nearly all the pending messages.

That one queue has consumed the broker’s shared memory pool. Every pending message is charged against its destination’s memory accounting and against the broker-wide system memory limit. If either limit is reached, flow control activates. With no per-destination memory limit set, ActiveMQ Classic does not throttle just that queue’s producers: it throttles everyone’s. This is the default behavior, not a bug.

This article covers how to confirm which destination is responsible, why the blocking is silent, and how per-destination limits contain the blast radius. For the broader failure model, see How ActiveMQ Classic actually works in production.

What this means

During the memory accounting step of the message flow, each message’s footprint is charged against two budgets: the destination’s and the broker’s system memory limit (configured via <systemUsage><memoryUsage> in activemq.xml). When either budget is exhausted, the broker stops reading from the producer’s socket, creating TCP backpressure. From the producer’s perspective, send() simply blocks. There is no exception, no log entry on the producer side, and no timeout unless you configure one.

There are two levels of flow control, and they have very different blast radii:

• Per-destination flow control. A destination with its own memoryLimit blocks only producers sending to that destination.
• Broker-level flow control. When the shared pool is exhausted, every producer on every destination blocks, regardless of which destination caused it.

Per-destination MemoryPercentUsage is the signal that tells you which level you are dealing with and who is responsible. On a multi-tenant broker, where unrelated applications share one JVM, this is the difference between “tenant A is throttled” and “the whole platform is down.”

flowchart TD
  A[Consumer on queue A stalls] --> B[Messages pile up in queue A]
  B --> C[Queue A memory usage climbs toward 100 percent]
  C --> D[Shared broker memory pool fills]
  D --> E[Broker-level flow control engages]
  E --> F[Producers on all destinations block in send]
  C -.->|with per-destination memoryLimit| G[Flow control scoped to queue A only]
  G --> H[Other destinations keep flowing]

One mechanism worth knowing: because flow control works by the broker not reading from the producer’s socket, it operates at connection granularity. If several producer sessions share one JMS connection and any of them hits flow control, sends on that connection stall together.

Common causes

The first two causes usually appear together: the slow consumer is the trigger, the missing per-destination limit is the structural weakness that turns one team’s problem into everyone’s incident.

Quick checks

These are read-only. They assume the embedded web console is reachable on port 8161 (bound to 127.0.0.1 by default since 5.16) and use the default credentials from the stock config. Adjust brokerName and credentials for your deployment.

# 1. Is broker-level flow control active?
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/MemoryPercentUsage'

# 2. Which queues are consuming the shared pool?
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=*/MemoryPercentUsage'

# 3. Check topics as well (offline durable subscribers accumulate silently)
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Topic,destinationName=*/MemoryPercentUsage'

# 4. Depth on the suspect queue
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/QueueSize'

# 5. Are consumers attached to it?
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/ConsumerCount'

# 6. Are consumers holding messages without acking?
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/InFlightCount'

# 7. Broker log: flow control and usage manager messages (path varies by install)
grep -i "memory limit" /opt/activemq/data/activemq.log | tail -20

# 8. Rule out a different saturation path
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/StorePercentUsage'
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/TempPercentUsage'

If check 1 returns 100 while exactly one destination in checks 2-3 is near its limit and everything else is idle, you have the noisy-neighbor pattern this article is about.

How to diagnose it

1. Confirm broker-level flow control. Broker MemoryPercentUsage at 100 with active producers, plus Usage Manager ... reached memory limit lines in the broker log, means producers are being blocked right now. When an individual destination fills, the broker also publishes to ActiveMQ.Advisory.FULL.Queue.<name> (or the topic equivalent), which tells you which destination tripped first.
2. Rank destinations by MemoryPercentUsage. The noisy destination stands out: near 100 while peers are idle. Note that per-destination usage can briefly exceed 100 during bursts before flow control engages. Treat a value over 100 as a burst in progress, not a broken metric.
3. Explain the accumulation. ConsumerCount of zero means nothing is draining the queue. Consumers connected but InFlightCount equal to consumer count times prefetch (default 1000 for queues) means consumers are holding messages without acknowledging them: stuck or overloaded, not absent.
4. Check the topic case separately. A durable subscriber that went offline without unsubscribing accumulates every published message indefinitely. Look for a subscription with growing PendingQueueSize and no active consumer.
5. Check JVM heap independently. java.lang:type=Memory HeapMemoryUsage is not the same thing as ActiveMQ’s memory accounting. Either can fill first, and they fail differently: one triggers flow control, the other triggers GC stalls or an OOM kill.
6. Decide the immediate action. Drain (fix or restart the stalled consumer), shed (purge the queue, destructive: messages are discarded, so confirm with the owning team first), or cap (apply a per-destination limit so it cannot recur this way).

Metrics and signals to monitor

The degradation curve here is a cliff, not a slope. At 99 percent everything works; at 100 producers block instantly. Alerts that fire at 100 arrive after the incident has already started.

Fixes

Set per-destination memory limits (the structural fix)

Give destinations their own budget with the memoryLimit attribute on a <policyEntry>:

<destinationPolicy>
  <policyMap>
    <policyEntries>
      <policyEntry queue=">" producerFlowControl="true" memoryLimit="100mb"/>
    </policyEntries>
  </policyMap>
</destinationPolicy>

With memoryLimit set, a destination that fills its own budget flow-controls only its own producers. The noisy queue is contained; every other destination keeps flowing. When a destination has its own limit, its cursor high-water mark (cursorMemoryHighWaterMark) is evaluated against that per-destination limit rather than the broker-wide pool, so capped destinations also start paging toward the store earlier. See the per-destination policies documentation for the full attribute list. Add stricter entries for known-hot queues and looser wildcard entries for the rest.

Sizing tradeoff: too small and you throttle a healthy destination during normal peaks; too large and the cap never protects the shared pool. Size from each destination’s observed baseline backlog plus headroom for its worst legitimate burst. On multi-tenant brokers, every destination should have a cap; an uncapped wildcard is the hole the next incident walks through.

Unblock the stalled consumer (the root cause fix)

Per-destination limits contain the blast radius, but the stalled consumer is why memory filled at all. Find the consumer whose inflight count is pinned at its prefetch and restart or rebalance it. Disconnecting a slow consumer forces its inflight messages to be redelivered to the remaining consumers, which often restores drain immediately. That is disruptive (messages redeliver, ordering within the affected stream can change), so prefer it over a broker restart, not over a clean consumer fix. Once the queue drains, memory accounting releases and flow control lifts on its own.

Make blocking visible to producers

Silent blocking is what turns a broker problem into a mystery across five upstream services. Configure sendFailIfNoSpaceAfterTimeout so a producer that would block instead receives a ResourceAllocationException after the timeout, which it can log, alert on, and retry. It can be set on the connection factory and, since 5.16.0, per destination via the policy entry. Tradeoff: your producers must actually handle the exception; an unhandled exception in a fire-and-forget send path is its own kind of silent failure.

Turning flow control off is a different failure mode, not a fix

Setting producerFlowControl="false" on a destination changes blocking into spooling to the temp store or, depending on sendFailIfNoSpace, dropping messages. If the temp store fills with flow control disabled, non-persistent messages can be discarded silently. Use this only on destinations where loss is acceptable, and pair it with sendFailIfNoSpace="true" so producers at least get an error.

What not to do first

• Do not restart the broker. On restart the store replays the backlog and cursors page messages back into memory, so the noisy queue refills the pool and you are back at 100 percent, minus the time you lost. Nothing structural has changed.
• Do not just raise the broker memoryUsage. A bigger shared pool with no per-destination caps is the same incident on a longer fuse. Also keep the broker memory limit around 60 to 70 percent of JVM max heap; push it higher and you trade flow control for GC stalls and OOM risk, which are worse.

Prevention

• Per-destination memory limits. Set memoryLimit on every policyEntry, with tighter caps on shared and multi-tenant brokers, so no single destination can drain the shared pool.
• Alerting below the cliff. Ticket above 80 percent on both per-destination and broker-level memory. Flow control is a cliff-edge at 100, so threshold alerts at 100 only confirm an outage already in progress.
• Loud producers. Configure sendFailIfNoSpaceAfterTimeout so blocked sends surface as exceptions in application logs instead of silent hangs.
• Leading indicators. Track the enqueue/dequeue ratio and queue depth growth on busy destinations. A ratio sustained above 1.0 is your runway warning long before memory is involved.
• Durable subscription hygiene. Unsubscribe decommissioned durable subscribers; an offline durable subscription accumulates every published message forever and shows up later as unexplained topic memory pressure.
• Heap alignment. Keep broker memoryUsage near 60 to 70 percent of JVM max heap and monitor both numbers independently, because either one can fill first.

How Netdata helps

• Netdata’s ActiveMQ monitoring charts per-destination MemoryPercentUsage alongside broker-level memory, so ranking destinations during an incident is a glance at a dashboard instead of a sequence of JMX queries.
• Correlating destination memory with QueueSize, ConsumerCount, and dequeue rate on the same view is what lets you distinguish a stalled consumer from a producer burst in minutes.
• Alerts at 80 percent catch the climb before the cliff-edge at 100 where flow control engages, which is the difference between a ticket and a page.
• JVM heap is charted next to ActiveMQ’s internal memory accounting, which matters because the two fail differently and the most common misdiagnosis here is confusing them.
• Per-second history shows whether the growth was a spike (replay or traffic event) or a slow burn (consumer gradually falling behind), and that shape determines which fix applies.

Related guides

• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ monitoring checklist: the signals every production broker needs
• ActiveMQ monitoring maturity model: from survival to expert
• ActiveMQ producer flow control: why send() hangs and producers block silently