ActiveMQ memoryUsage vs JVM heap: the two memory budgets teams confuse

An ActiveMQ Classic broker can hit 100% MemoryPercentUsage and silently block every producer while the JVM heap sits at 40%. The same broker, on a different day, can die from OutOfMemoryError while MemoryPercentUsage shows 50% and every queue-depth dashboard looks calm. Both incidents look contradictory until you understand that ActiveMQ tracks two separate memory budgets, and only one of them is the JVM’s.

Teams monitor one budget, assume it represents the other, and get surprised by whichever one they ignored. This article covers what each budget counts, how they drift apart, how to size them relative to each other, and which signals detect the drift.

Scope: ActiveMQ Classic 5.x/6.x. Artemis has a fundamentally different memory model (paging and global-max-size rather than usage-manager flow control) and nothing below applies to it.

What each budget actually is

JVM heap is the real memory. It is the Java heap bounded by -Xmx, holding everything the broker’s JVM allocates: message bodies sitting in cursors, destination metadata, connection and session state, JMX MBean trees, pending acknowledgment records, KahaDB index structures, and every non-ActiveMQ object the runtime creates. When this budget is exhausted, the JVM throws OutOfMemoryError or dies in a GC thrash spiral. Nothing in ActiveMQ configuration can save you from it.

ActiveMQ memoryUsage is internal accounting on top of the heap. It is a counter, configured in activemq.xml inside the <systemUsage> block, that tracks the bytes charged for pending (undispatched and unacknowledged) messages across all destinations. It is not a reservation and it is not a measurement of heap. It is the broker’s own bookkeeping of “how much message data am I holding,” maintained separately from what the JVM has actually allocated.

The distinction matters because the two budgets enforce completely different failure behaviors:

• memoryUsage hitting 100% triggers producer flow control. The broker stops reading from producer sockets. Producer send() calls block silently, with no exception and no log entry on the producer side by default. The broker stays alive. Upstream services hang.
• JVM heap hitting its ceiling triggers GC collapse and OOM. Long GC pauses freeze all threads, clients hit their wireFormat.maxInactivityDuration heartbeat timeout (default 30000ms) and disconnect, reconnection storms pile more objects onto the heap, and the cycle accelerates until the process dies or is OOM-killed. The broker does not stay alive.

flowchart TD
  HEAP["JVM heap (-Xmx)
real memory"]
  MU["memoryUsage counter
message bytes only
limit: percentOfJvmHeap or absolute"]
  OTHER["Non-message heap
MBeans, connection state,
destination metadata, index, sessions"]
  FC["memoryUsage = 100%
producer flow control
send() blocks silently"]
  OOM["heap exhausted
GC spiral, OOM kill,
broker down"]

  HEAP --> MU
  HEAP --> OTHER
  MU -->|"limit reached"| FC
  OTHER -->|"grows unchecked"| OOM
  MU -.->|"does not count"| OTHER

memoryUsage is a subset of heap in spirit but not in accounting: the bytes it counts live in the heap, but the heap also holds everything the counter ignores. Either side can run out first.

How the two budgets drift apart

The drift is not a bug. It falls out of what each budget counts. There are two failure directions, and both are common in production.

Direction 1: heap exhausted while memoryUsage shows headroom

The memoryUsage counter only charges message bytes. It does not count:

• JMX MBeans and their metadata. Every destination creates several MBeans. A broker with thousands of dynamically created destinations carries tens of thousands of MBeans, all in heap, none charged to memoryUsage. This is the destination-explosion failure pattern: heap and GC pressure climb while message memory stays low.
• Connection and session state. Thousands of connections, each with sessions, consumers, producers, and prefetch bookkeeping, consume heap outside the counter.
• Advisory destinations. Each advisory topic is a real destination with real MBeans and state, created silently on top of your application destinations.
• KahaDB index structures and page cache effects in the JVM.
• Redelivery state, transaction state, scheduler state, network bridge bookkeeping in Network of Brokers deployments.

This is the silently catastrophic pattern: low MemoryPercentUsage with high heap usage means the two budgets are misaligned and OOM is possible. Your message-flow dashboards look green while the JVM is dying of metadata bloat. It is also why a broker can enter the GC pause death spiral (long pauses, heartbeat timeouts, reconnection storms) with MemoryPercentUsage nowhere near 100%.

Direction 2: memoryUsage full while the heap has room

This is the reverse misconfiguration, and it is self-inflicted. If memoryUsage is set far below what the heap could carry, or if per-destination memory limits on <policyEntry> elements are set too small, the broker flow-controls producers while gigabytes of heap sit idle. Symptoms:

• Producer send() calls hang with no exception. Upstream services stall and their owners blame their own code, connection pools, or the network.
• Broker-level MemoryPercentUsage is at 100%, or one destination’s MemoryPercentUsage is at 100% while the broker-level counter is fine.
• JVM heap usage is moderate. GC is calm. Nothing looks “broken” except that throughput has collapsed.

Because producer flow control is silent by default, the first team to notice is usually the one whose HTTP API started timing out, three services upstream of the actual producer. The related guide on producer flow control covers the diagnosis of the blocked-send side in detail.

Sizing: why memoryUsage must be 60-70% of heap, never 100%

The rule is absolute: never set memoryUsage equal to max heap. The broker needs heap for everything the counter does not track. If the message counter is allowed to consume the entire heap, the first burst of messages leaves zero room for MBeans, connection state, and dispatch bookkeeping, and the broker OOMs at exactly the moment it is under the most load.

The shipped default in current 5.x configurations is:

<systemUsage>
  <systemUsage>
    <memoryUsage>
      <memoryUsage percentOfJvmHeap="70"/>
    </memoryUsage>
    ...
  </systemUsage>
</systemUsage>

Seventy percent is a rule of thumb, not a guarantee. The right value depends on how much non-message heap your workload carries:

• Many destinations, many connections, NoB, heavy advisory traffic: non-message heap is large. Keep memoryUsage at 50-60% of heap and treat the default 70% as too generous.
• Few destinations, few connections, large message bodies: message bytes dominate. 70% is reasonable.
• Very large heaps: the absolute headroom (heap minus memoryUsage limit) is what matters. On a 32GB heap, 70% leaves ~9.6GB for non-message state, comfortable for almost any deployment. On a 2GB heap, 70% leaves ~600MB, which a destination explosion can eat in an afternoon.

You can also set an absolute limit (limit="2 gb") instead of percentOfJvmHeap. Absolute limits are safer when the same activemq.xml is deployed to hosts or containers with different -Xmx values, because a percentage silently rescales with heap and can become wrong without anyone editing the file. If you run the broker in a container, remember that the container memory limit and the JVM heap interact independently of all of this: if -Xmx approaches the container limit, the OOM killer terminates the process before the JVM throws OutOfMemoryError, and neither budget’s metrics will tell you why.

Two adjacent limits deserve the same skepticism. storeUsage is a configured limit compared against KahaDB consumption, not against physical disk; if the store limit exceeds real disk capacity, the filesystem fills first. tempUsage caps non-persistent overflow spill. All three systemUsage values are accounting limits layered on real resources, and each can be misaligned with the resource it proxies.

Signals to watch in production

Monitor both budgets independently, plus the ratio between them:

All of these are JMX attributes, reachable via Jolokia on the web console (default port 8161) or any JMX client. HeapMemoryUsage comes from java.lang:type=Memory; MemoryPercentUsage and MemoryLimit come from the broker MBean (org.apache.activemq:type=Broker,brokerName=<name>). Pull both on the same polling interval so the gap between them is a real comparison, not two unrelated time series.

Checking alignment by hand

A quick manual audit takes three queries. Compare what the broker thinks it is using for messages against what the JVM is actually holding:

# Broker message-memory budget
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/MemoryPercentUsage'

curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/MemoryLimit'

# Real JVM heap
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/java.lang:type=Memory/HeapMemoryUsage'

From the three results, derive:

1. Effective message budget = MemoryLimit (or percentOfJvmHeap x -Xmx if configured as a percentage).
2. Non-message heap pressure = heap used after a major GC, minus roughly the message bytes in flight. If heap-after-GC is high while MemoryPercentUsage is low, the difference is MBeans, connection state, and metadata, and it will not be freed by draining queues.
3. Headroom check = is MemoryLimit between 60% and 70% of heap max? If it equals heap max, fix it before the next traffic burst does it for you.

If non-message heap is the growing half, check the usual suspects in order: destination count (the Queues and Topics arrays on the broker MBean), connection count (CurrentConnectionsCount), and orphaned durable subscriptions accumulating pending messages and state.

Common misuses, condensed

• Setting memoryUsage equal to -Xmx. Leaves no heap for anything that is not a message body. The broker OOMs under load. Keep it at 60-70%.
• Monitoring only MemoryPercentUsage. Catches flow control, misses every non-message heap failure: destination explosion, MBean bloat, connection-state growth.
• Monitoring only JVM heap. Catches OOM risk, misses flow control entirely. Producers block at 100% memoryUsage with the heap half empty.
• Trusting the 70% default blindly on small heaps. On a 1-2GB heap the remaining 30% is small in absolute terms and easily consumed by a few thousand destinations.
• Percentage limits with heterogeneous -Xmx across environments. The same XML silently means different byte budgets in staging and production.
• Assuming Artemis semantics. Artemis pages to disk and uses global-max-size; the flow-control-at-100% model is Classic-specific.

How Netdata helps

The two-budget problem is a correlation problem, and correlation is where per-second monitoring earns its place:

• Both budgets on one timeline. Netdata collects ActiveMQ broker metrics alongside JVM heap metrics from the same host, so MemoryPercentUsage and heap-after-GC are directly comparable instead of living in two tools.
• The gap as an anomaly. When heap usage climbs while broker memory usage stays flat, ML anomaly scoring flags the divergence before the OOM, which is exactly the misalignment pattern described above.
• GC pauses next to connection drops. Long GC pauses and the resulting client disconnect/reconnect sawtooth appear on adjacent charts, making the heap-side failure mode recognizable in seconds.
• Destination and connection counts next to heap. The usual causes of non-message heap growth (destination explosion, connection leaks) are visible on the same dashboard as the heap curve they produce.
• Per-destination memory usage. When broker-level usage rises, per-destination breakdowns identify the noisy queue without JMX archaeology.

Related guides

• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ monitoring checklist: the signals every production broker needs
• ActiveMQ monitoring maturity model: from survival to expert
• ActiveMQ producer flow control: why send() hangs and producers block silently