ActiveMQ MemoryPercentUsage climbing: reading the flow-control leading indicator

Your ActiveMQ broker’s MemoryPercentUsage gauge is at 65% and climbing. Nothing is broken yet. Producers are sending, consumers are consuming, queue depths look tolerable. But the trend line only goes one direction, and you know what happens at 100%: producer flow control activates, every producer’s send() call blocks silently, and upstream services start hanging with no error message anywhere.

This is the right moment to act, and the gauge gives you more information than most operators extract from it. The absolute value tells you where you are. The rate of climb tells you how much time you have. The surrounding signals tell you why. This article is about reading all three before the cliff edge.

If you need the broader picture of how memory accounting fits into the broker’s message path, see How ActiveMQ Classic actually works in production. This page stays narrowly on the climbing gauge.

What this means

MemoryPercentUsage is ActiveMQ’s internal accounting of how much of its configured memory budget is consumed by pending messages. It exists at two levels:

• Broker level: percentage of the memoryUsage limit configured under <systemUsage> in activemq.xml (default 70% of JVM heap, or an absolute value), consumed across all destinations.
• Per-destination level: percentage of an individual queue or topic’s memory limit, if one is set via <policyEntry>.

Two facts about this gauge trip up even experienced operators:

It is not JVM heap usage. ActiveMQ memory accounting is a subset of heap. You can sit at 50% MemoryPercentUsage with JVM heap at 95% (MBean metadata, connection state, advisory overhead), or at 100% with heap to spare (memory limit set too low). Monitor both independently.

The impact curve is a cliff, not a slope. At 99% everything works. At 100% producer flow control engages and the broker stops reading from producer sockets. There is no graceful degradation zone. And flow control is silent by default: no exception, no producer-side log line, no timeout unless you configured sendFailIfNoSpaceAfterTimeout. The first symptom upstream is usually “the service is slow” with nothing in its logs.

That is why this gauge is the leading indicator that matters. By the time anyone pages you for blocked producers, the gauge has been telling you about it for minutes, hours, or days.

The rate of climb is the real signal

An absolute reading of 70% means almost nothing without the trend. 70% stable for six months is a sizing fact. 70% having climbed from 30% in the last hour is an incident in progress.

The playbook rule of thumb: climbing 5% per minute means minutes of runway. At that rate from 70%, you have roughly six minutes before flow control. A climb of 0.5% per hour from a slow consumer drain gives you days, and you should treat it as a capacity-planning ticket, not a page.

Estimate time-to-flow-control directly:

runway = (memory_limit - current_usage) / (enqueue_rate - dequeue_rate) / avg_message_size

This is approximate. Message sizes vary and the accounting is not perfectly linear. But it converts “the gauge is going up” into “we have about 40 minutes,” which is the number you need to decide whether to page, to roll back a deployment, or to watch.

flowchart TD
  A[Slow or absent consumers] --> B[Enqueue rate exceeds dequeue rate]
  B --> C[Messages accumulate in pending cursors]
  C --> D[MemoryPercentUsage climbs]
  D --> E{Rate of climb?}
  E -->|Fast, percent per minute| F[Minutes of runway: page now]
  E -->|Slow, percent per hour| G[Days of runway: ticket and plan]
  F --> H[100 percent: producer flow control]
  G --> H
  H --> I[Producers block silently, upstream hangs]

The precursors to watch

MemoryPercentUsage rarely starts climbing without earlier symptoms. Three signals precede it, and they are all cheaper to act on than the memory climb itself:

If you catch the first or second, you often fix the problem before the memory gauge moves noticeably. If you are reading the memory gauge at 85%, the precursors already fired and someone missed them.

Common causes

Quick checks

All of these are read-only. They assume the Jolokia HTTP bridge on the embedded web console (default port 8161). Adjust credentials, broker name, and destination names for your environment.

# Broker-level memory percent and limit
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/MemoryPercentUsage,MemoryLimit'

# Every queue's memory percent: find which destination is eating the budget
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=*/MemoryPercentUsage'

# Enqueue/dequeue counters: take two readings 60s apart, derive rates
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/TotalEnqueueCount,TotalDequeueCount'

# Consumer count and inflight on the suspect queue
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/ConsumerCount,InFlightCount,QueueSize'

# JVM heap, because it is not the same thing
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/java.lang:type=Memory/HeapMemoryUsage'

# Temp store: non-persistent overflow already spilling to disk?
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/TempPercentUsage'

# Broker log: flow control events already happening on some destination?
grep -i "memory limit" /opt/activemq/data/activemq.log | tail -20

Two cautions. JMX reads are not free: on a broker with thousands of destinations, per-destination wildcard queries serialize a large MBean tree and can take seconds. And if MemoryPercentUsage is already at 100%, advisory messages (including ActiveMQ.Advisory.FULL.*) can themselves be dropped under memory pressure, so do not rely on advisories as confirmation.

How to diagnose it

1. Establish the rate. Take two broker-level readings a known interval apart, or read your monitoring history. Compute percent per minute. Convert to runway with the formula above. This single number decides your urgency.
2. Isolate the destination. Pull per-destination MemoryPercentUsage for all queues. In most incidents one or two destinations account for the climb. If the rise is uniform across everything, suspect a broker-wide cause (traffic spike, many consumers down) rather than a single slow consumer.
3. Check the balance equation. Derive enqueue and dequeue rates for the suspect destination. Enqueue/dequeue ratio above 1.0 sustained confirms accumulation rather than an accounting artifact.
4. Check consumers on that destination. ConsumerCount below expected points to a consumer outage. Count normal but InFlightCount pinned at prefetch points to stuck consumers that are connected but not processing.
5. Rule out the lookalikes. Post-restart cursor paging legitimately pushes memory to 50-70% while a stored backlog is paged in for dispatch; that is warmup, not an incident. Bursty workloads oscillate; a sine wave is normal, a monotonic climb is not.
6. Check temp store and DLQ. TempPercentUsage rising alongside memory means non-persistent overflow is already spilling to disk. DLQ growth means messages are being discarded there and may be inflating dequeue while feeding accumulation elsewhere.
7. Verify configuration. Confirm the actual memoryUsage limit, whether per-destination limits exist via <policyEntry>, and whether producerFlowControl is true. If flow control is disabled without sendFailIfNoSpace, the failure mode at 100% is silent message drop, not blocked producers.

Metrics and signals to monitor

Alerting posture from the playbook: page at 100% with active producers, ticket above 80% sustained, plan above 60% sustained. The 60% line exists so this conversation happens in a planning meeting, not at 3 a.m.

Fixes

Restore consumption. The correct fix for most climbs is on the consumer side: restart or scale the stuck consumer, fix its slow downstream dependency, or disconnect a wedged consumer so messages rebalance. Reducing an oversized prefetch releases memory back to the broker faster. This is the only fix that addresses root cause when consumers are the problem.

Buy time at the broker. Raising the memoryUsage limit or adding per-destination limits buys runway, but only if JVM heap has real headroom above the ActiveMQ limit. Keep the broker limit at roughly 60-70% of max heap. Raising the limit against a heap that is already tight converts a flow-control event into an OOM kill, which is worse.

Make flow control visible. If you take one permanent action from this article: configure sendFailIfNoSpaceAfterTimeout so blocked producers get an exception after a bounded wait instead of hanging forever. Silent blocking is why these incidents surface as “everything is slow” instead of “the broker is full.”

Tame non-persistent traffic. VM-cursor non-persistent messages live entirely in memory and are the most common fast path to exhaustion. If that traffic is burst-prone, either size memory and temp store for the burst, or accept persistence for those destinations.

Prevention

• Alert on rate, not just level. A threshold at 80% catches slow leaks late. A derivative alert (percent-per-minute above a bound) catches fast climbs while you still have runway.
• Set per-destination limits. Without <policyEntry> memory limits, one runaway destination consumes the shared broker pool and flow-controls everyone.
• Watch the precursors. Enqueue/dequeue ratio, consumer count, and inflight-vs-prefetch fire before the memory gauge. Alert on them directly.
• Size deliberately. Know your memoryUsage value, your JVM max heap, and your worst-case consumer-outage duration. The headroom rule: under 70% comfortable, 70-85% acceptable at peak, above 85% needs attention.
• Handle the DLQ. No TTL by default means silent accumulation feeding both store and memory pressure. TTL, alerting, and per-destination DLQs.

How Netdata helps

Netdata’s ActiveMQ monitoring surfaces the signals this article depends on, at the granularity where the precursors are visible:

• Broker and per-destination MemoryPercentUsage as time series, so rate of climb is a visual fact rather than a mental subtraction between two JMX reads.
• Enqueue and dequeue rates derived from the cumulative counters, making the enqueue/dequeue ratio and backlog growth readable at a glance.
• Per-destination consumer count, queue depth, and inflight count on the same dashboard, so the “consumers gone vs consumers stuck” distinction takes seconds.
• TempPercentUsage and store usage alongside memory, so you see overflow spilling to disk before non-persistent messaging breaks.
• JVM heap and GC pause charts from the same broker, keeping the two memory systems visibly separate instead of conflated.
• Threshold alerts you can set at the playbook levels (plan at 60%, ticket at 80%, page at 100%) plus derivative alerts on rate of climb.

Related guides

• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ monitoring checklist: the signals every production broker needs
• ActiveMQ monitoring maturity model: from survival to expert
• ActiveMQ producer flow control: why send() hangs and producers block silently