ActiveMQ memory limit reached: MemoryPercentUsage at 100% and the flow-control cliff

Your producers have stopped sending. No exceptions, no timeouts, no errors on the producer side. The send() calls just hang, and upstream services pile up threads waiting on the broker. In the broker log you find the line operators search for at 3 a.m.:

Usage Manager Memory Usage ... reached memory limit

MemoryPercentUsage on the Broker MBean is at 100%. This is ActiveMQ Classic’s own memory accounting, not JVM heap, and 100% is not a slowdown. It is a cliff edge. At 99% everything works. At 100% every producer is flow-controlled: the broker stops reading from producer sockets, TCP backpressure builds, and sends block silently until memory frees up.

This guide covers how to confirm what is actually full, find the consumer or message pattern responsible, relieve the pressure without making things worse, and put guardrails in so the next climb gets caught at 80% instead of 100%.

What this means

ActiveMQ tracks message memory separately from JVM heap. Every pending message is charged against two limits: the destination’s memory limit and the broker-wide system memory limit configured via <systemUsage><memoryUsage> in activemq.xml (default: 70% of JVM heap, or an absolute value). MemoryPercentUsage on the Broker MBean reports consumption against that broker-wide budget.

When either limit is reached, producer flow control activates. By default there is no exception and no producer-side log entry. The producer’s send() simply blocks. From the outside this looks like “the application is slow,” and teams routinely burn hours debugging upstream services before anyone looks at the broker.

The mechanism almost always traces back to one side of a simple imbalance:

flowchart TD
  A[Consumer slows or stops] --> B[Prefetch fills: InFlightCount pinned at prefetch]
  B --> C[Messages accumulate in destination memory]
  C --> D[MemoryPercentUsage climbs]
  D --> E{Which limit is hit?}
  E -->|Destination limit| F[That destination's producers block]
  E -->|Broker limit at 100%| G[All producers flow-controlled]
  F --> H[send hangs silently]
  G --> H
  H --> I[Upstream stalls, retry loops amplify]

One important nuance: broker-level memory is a shared pool. If you have not set per-destination limits, one runaway queue can consume the entire broker budget and flow-control producers sending to completely unrelated, healthy queues.

Common causes

The two you will see most often: a slow or stuck consumer, and a non-persistent flood held in memory by VM cursors. Everything else is a variant.

Quick checks

All read-only. These use the Jolokia HTTP/JMX bridge on the web console port; adjust host, port, and credentials for your deployment.

# Broker-level memory usage: 100 means flow control is active
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/MemoryPercentUsage'

# Per-destination memory usage across all queues: find the hot one
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=*/MemoryPercentUsage'

# Depth, consumers, and inflight for a suspect queue
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/QueueSize,ConsumerCount,InFlightCount,DequeueCount'

# Confirm flow-control events in the broker log
grep -i "memory limit" /opt/activemq/data/activemq.log | tail -20

# JVM heap: is the JVM itself also under pressure, or is this purely broker accounting?
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/java.lang:type=Memory/HeapMemoryUsage'

# Temp store: non-persistent overflow spilling to disk points at a VM-cursor problem
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/TempPercentUsage'

How to diagnose it

1. Confirm which limit is hit. Broker MemoryPercentUsage at 100% blocks everyone. A single destination at 100% (with the broker below 100%) blocks only that destination’s producers, assuming per-destination limits are configured. The per-destination wildcard query above answers this in one call. Note that a destination can briefly exceed 100% during a burst before flow control engages.

2. Rank destinations by memory consumption. One or two destinations will usually dominate. If memory is spread evenly across many queues, suspect a broad consumer outage or a producer burst rather than a single stuck consumer.

3. For the hot destination, check consumer health. ConsumerCount at zero means nobody is draining. ConsumerCount normal but DequeueCount flat and InFlightCount pinned at the prefetch limit means consumers are connected but not acknowledging: the zombie consumer pattern. Correlate with the consumer application’s own health (its downstream database, its GC, its thread pool) before blaming the broker.

4. Check message persistence and cursor type. Queues default to store-based cursors, which page messages from disk and keep memory usage low. If memory is climbing on a queue anyway, either the cursor cannot page effectively or something is holding message references. Non-persistent messages on VM cursors bypass the store and live entirely in memory; this is the most common path to memory exhaustion. Rising TempPercentUsage alongside confirms non-persistent overflow.

5. Read the actual configuration. Check <systemUsage><memoryUsage> in activemq.xml and the JVM’s -Xmx. A limit set to an absolute value years ago, while traffic has tripled, is a misconfiguration incident, not a capacity incident.

6. Rule out heap confusion. If MemoryPercentUsage is at 100% but JVM heap usage after GC is comfortable, the broker limit is simply too small for the workload. If heap is also above 85-90% after GC, you have a broader memory problem and raising the broker limit will make things worse.

Metrics and signals to monitor

Fixes

Stuck or slow consumer

Restart the slow consumer application, or disconnect that consumer so messages rebalance to healthy consumers on the same queue. This is usually the fastest way to get memory draining. Tradeoffs: inflight messages from the dead consumer are redelivered, so expect a redelivery spike; if the consumer was slow because of a downstream dependency, the rebalance just moves the bottleneck. Reducing prefetch on the consumer (default 1000 for queues) lowers how much memory one stalled consumer can hold hostage, at the cost of more dispatch round trips.

Broker memory limit too small

Raise <memoryUsage> in activemq.xml. Keep it to roughly 60-70% of JVM max heap; the broker needs heap for connection state, MBeans, and other non-message objects, so setting the limit equal to -Xmx invites an OOM. If you raise the limit, check whether -Xmx also needs to rise. The broker reads this at startup, so the change requires a restart. Do not treat a restart as the first move during the incident; drain the backlog first.

Isolate destinations from each other

Configure per-destination memory limits via <policyEntry> so one runaway queue cannot consume the entire broker pool and flow-control unrelated producers. Tradeoff: per-destination caps mean individual destinations hit flow control sooner under their own bursts, so size them against real per-queue backlog expectations.

Make flow control visible

The default silent block is the worst part of this failure mode. Configure sendFailIfNoSpaceAfterTimeout on the destination policy so producers get a javax.jms.ResourceAllocationException after a timeout instead of hanging forever, or sendFailIfNoSpace="true" to fail immediately. Tradeoff: you are converting a hidden stall into visible producer-side errors, which is what you want, but your producers must handle and retry that exception correctly or you have moved the incident upstream.

Non-persistent VM-cursor flood

If the pressure is non-persistent traffic living in memory, decide whether those messages actually need to be non-persistent. If they do, the temp store is your overflow valve: size <tempUsage> for realistic bursts and monitor TempPercentUsage, because a full temp store breaks non-persistent messaging. If producerFlowControl="false" is set anywhere on this path, verify what happens when space runs out; depending on sendFailIfNoSpace, messages may be dropped silently instead of blocking, which is a data-loss incident wearing a healthy broker’s clothes.

Prevention

• Alert at 80%, page at 100% with conditions. MemoryPercentUsage above 80% sustained for more than 5 minutes is a ticket. At 100% with active producers, it is a page. Rate of climb matters more than the absolute value; 5% per minute is minutes of runway.
• Set per-destination limits so multi-tenant brokers degrade one queue instead of all of them.
• Keep the broker limit at 60-70% of heap and monitor JVM heap independently. Low MemoryPercentUsage with high heap usage is its own failure mode.
• Subscribe to or alert on FULL advisories. The broker publishes to ActiveMQ.Advisory.FULL.Queue.<name> when a destination’s memory is full. Caveat: advisories are themselves messages and can be lost when the broker is under memory pressure, so treat them as a secondary signal, never the only one.
• Configure sendFailIfNoSpaceAfterTimeout on production destinations so the next event shows up as producer exceptions with timestamps instead of a mystery hang.
• Watch the leading indicators: enqueue/dequeue ratio above 1.0 sustained, InFlightCount pinned at prefetch, consumer count drifting down. Flow control at 100% is the last signal to fire, not the first.
• Size the temp store deliberately if you run meaningful non-persistent traffic, and alert on any sustained TempPercentUsage above 50%.

How Netdata helps

• Netdata’s JMX collection tracks broker and per-destination MemoryPercentUsage per second, so you see the climb and the rate of climb, not just the 100% cliff after the fact.
• Correlating memory usage with InFlightCount, DequeueCount, and ConsumerCount on the same dashboard turns “producers are blocked” into “consumer X on queue Y stopped acking at 03:12” in one view.
• JVM heap and GC pause charts sit next to broker memory charts, which makes the “broker accounting vs actual heap” distinction concrete instead of a recurring postmortem confusion.
• Temp store, store usage, and disk-free metrics on the KahaDB partition are collected together, so a non-persistent flood shows its full path: memory, temp store, disk.
• Alerts can be set on the 80% warning threshold and on anomaly detection for enqueue/dequeue imbalance, catching the imbalance while there is still runway.

Related guides

• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ monitoring checklist: the signals every production broker needs
• ActiveMQ monitoring maturity model: from survival to expert
• ActiveMQ producer flow control: why send() hangs and producers block silently