ActiveMQ JVM heap exhaustion: OutOfMemoryError and the OOM kill

The broker was fine an hour ago. Now the process is gone, or it is alive but frozen: clients disconnected, the web console hangs, and the log shows java.lang.OutOfMemoryError or, if it runs in a container, nothing at all because the kernel killed the JVM with SIGKILL (exit code 137, i.e. 128+9).

This is ActiveMQ JVM heap exhaustion. It is distinct from the broker’s internal memoryUsage hitting 100%. That limit triggers producer flow control and blocks send() calls; this one kills or freezes the JVM itself. The confusing part is that the two are independent counters. You can have MemoryPercentUsage at 50% with plenty of headroom while JVM heap is at 97% and the next full GC never finishes. Non-message allocations (destination metadata, MBeans, connection and session state, cursor overhead, transport buffers) live on the heap too, and ActiveMQ’s flow control accounting does not protect them.

The failure is also abrupt at the end. Heap pressure degrades gradually until about 85-90% used after GC, then the JVM tips into a cliff: GC frequency explodes, pauses exceed the client inactivity timeout, clients disconnect and reconnect in a storm, the reconnection objects add more heap pressure, and the process either throws OutOfMemoryError or gets OOM-killed by the kernel or the container runtime.

What this means

The JVM heap holds everything the broker allocates: message bodies in cursors, destination and subscription metadata, connection state, JMX MBeans, pending ack records, transport state. The metric that matters is HeapMemoryUsage from java.lang:type=Memory, specifically the used value measured after a major GC. Instantaneous usage is meaningless: it oscillates constantly between minor GCs.

Two shapes of the problem:

• Heap-after-GC trending upward over hours or days. Each major GC frees less than the last. This is a leak or a structural growth problem (destination explosion, unclosed resources). You have runway, but the direction is fixed.
• Heap-after-GC pinned near max with GC distress. The JVM spends most of its time collecting and frees almost nothing. Once full GC pauses approach 30s they cross the default OpenWire wireFormat.maxInactivityDuration of 30000ms, and clients start timing out. This is the GC pause death spiral, and it ends in OOM or an unresponsive broker.

In containers there is a third shape: the JVM is fine by its own accounting, but total process memory (heap plus metaspace, thread stacks, direct buffers, JVM overhead) exceeds the container memory limit, and the kernel OOM killer SIGKILLs the process mid-write. No exception, no heap dump, no warning in the broker log. If the store was mid-write, you may also have a KahaDB recovery waiting for you on restart.

flowchart TD
  A[Heap pressure grows] --> B[Longer, more frequent GC pauses]
  B --> C{Pause > client inactivity timeout?}
  C -- no --> B
  C -- yes --> D[Mass client disconnects]
  D --> E[Reconnection storm]
  E --> F[New connection, session, subscription objects]
  F --> A
  A --> G[OutOfMemoryError or kernel OOM kill]
  G --> H[Unclean shutdown, possible KahaDB recovery on restart]

Common causes

Quick checks

All read-only. Run these before touching anything.

# 1. Is the process alive, and how long has it been up?
ps -o pid,etime,rss,cmd -p $(pgrep -f activemq)

# 2. Was it OOM-killed? (kernel log, needs root)
dmesg -T | grep -i "killed process" | tail -5

# 3. Current heap usage via Jolokia
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/java.lang:type=Memory/HeapMemoryUsage'

# 4. GC activity: are we in GC distress?
jstat -gcutil $(pgrep -f activemq) 1000 5

# 5. ActiveMQ's own memory accounting (compare against heap!)
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/MemoryPercentUsage'

# 6. How many destinations exist?
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/Queues' \
  | python3 -c "import json,sys; print(len(json.load(sys.stdin)['value']))"

# 7. Thread count (leaks show here too)
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/java.lang:type=Threading/ThreadCount'

# 8. Recent GC log lines, if GC logging is enabled
# (path is wherever your -Xlog:gc configuration points)
tail -50 /opt/activemq/data/gc.log

Check 5 is the one that surprises people. If MemoryPercentUsage is at 40% while heap-after-GC is at 95%, the broker’s message accounting has headroom and flow control will never fire, but the JVM is about to die. The two limits guard different things.

How to diagnose it

1. Confirm which failure you had. OutOfMemoryError in activemq.log means the JVM threw and may still be running (badly). No exception plus a missing process plus exit code 137 or a Killed process line in dmesg means the kernel or container runtime did it. The response differs: a JVM OOM leaves you a live process to inspect; an OOM kill leaves you nothing but logs and a possibly dirty KahaDB.

2. Check uptime before trusting any reading. Heap readings in the first 10 minutes after start are not diagnostic. Cursors page stored messages into memory after restart, and a single post-start reading above 90% is not evidence of a leak worth paging on. You need consecutive post-GC windows.

3. Plot heap-after-major-GC, not instantaneous heap. From jstat -gcutil output, the OU (old gen used) value right after a full GC is the floor. If that floor rises across successive full GCs, you have a leak or structural growth. If it is flat but high, the heap is simply small.

4. Correlate heap growth with a candidate driver. Pull destination count, connection count, and thread count over the same window. Heap rising in lockstep with destination count points at destination explosion (each destination adds its own MBeans plus internal structures). Heap rising with connection count points at a connection or session leak. Heap rising with neither points at message-path allocations (VM cursors, large messages) or a leak inside retained objects.

5. Compare against ActiveMQ’s internal accounting. Read MemoryPercentUsage. If it is also near 100%, message backlog is the driver and flow control is either active or imminent; treat it as a memory-pressure incident first (see the memory limit guide). If it is low, the heap consumer is non-message state: destinations, MBeans, connections, sessions, or leaked objects.

6. Capture evidence before restarting. If the process is alive, take a heap dump and a thread dump now. A restart destroys the evidence and, with a leak, only resets the clock.

   # Heap dump (large file; pauses the JVM, and the :live option forces a
   # full GC first. On a broker already in GC distress this can be the
   # final straw, so plan the timing.)
   jmap -dump:live,format=b,file=/tmp/broker-heap.hprof $(pgrep -f activemq)
   
   # Thread dump (cheap, safe)
   jstack $(pgrep -f activemq) > /tmp/broker-threads.txt
   

   In the heap dump, thousands of retained producer, consumer, or session objects point at unclosed JMS resources. Destination or MBean objects dominating retained heap point at destination explosion.

7. If it was a container OOM kill, audit the sizing. The JVM footprint is -Xmx plus metaspace, thread stacks, direct buffers, and JVM internals. If the container limit is at or below -Xmx, the kernel will kill the process under load even when the heap itself is healthy.

Metrics and signals to monitor

A workable threshold ladder: after a major GC, you want at least 30% of max heap free. Under 20% free is yellow, under 10% is critical. PAGE only when heap exceeds 95% after GC across multiple consecutive major-GC windows with GC distress present and uptime over 600s. A single reading, or any reading right after startup, pages nobody correctly.

Fixes

If it is happening right now (process alive, in GC distress)

The spiral feeds on reconnections. If you can shed load, do it: disconnect non-essential consumers and producers to cut the allocation rate. Do not restart as a first move unless the broker is already unrecoverable, because you lose the heap dump and thread dump that tell you why it happened. If clients have already mass-disconnected and the broker is oscillating between GC pauses and reconnection storms, a restart is usually the only way back, so capture diagnostics first, then restart, then expect KahaDB recovery time proportional to journal and index size if the shutdown was unclean.

Heap too small

Raise -Xmx to match the actual workload (connections, destinations, message throughput). This requires a restart. Keep ActiveMQ’s memoryUsage limit at roughly 60-70% of JVM max heap; the remaining 30-40% is exactly the headroom that non-message allocations need. Setting memoryUsage equal to max heap removes the buffer and makes JVM OOM more likely, not less.

Memory leak from unclosed JMS resources

The heap dump tells you which objects are retained. The fix is in the application: close sessions, producers, and consumers, or fix the pooling configuration that pins them. Broker-side restarts only reset the timer.

Destination explosion

Identify the naming pattern (UUIDs, user IDs, timestamps), fix the application to stop creating a destination per entity, then clean up: configure gcInactiveDestinations and schedulePeriodForDestinationPurge so empty inactive destinations are removed, and purge the existing empty ones via JMX. Advisory topics multiply the count; disable advisories where you do not use them.

Container OOM kills

Set the container memory limit comfortably above the JVM’s total footprint, not above -Xmx alone. Leave room for metaspace, thread stacks (one transport thread per connection with the default TCP transport), and direct buffers. If you cannot raise the limit, lower -Xmx so the JVM stays inside the cgroup.

GC tuning

For heaps above 4GB, G1 is the standard recommendation. Whatever collector you run, enable GC logging (-Xlog:gc*:file=gc.log:time on JDK 11+); post-incident, the GC log is the most useful artifact you will have. Tuning GC pauses treats the symptom; if heap-after-GC keeps ratcheting up, only fixing the driver stops the incident from recurring.

Prevention

• Alert on the floor, not the spikes. Track heap-used-after-major-GC as a first-class metric. Page at >95% sustained with GC distress; ticket at >85%; plan at >70%.
• Monitor both memory signals independently. JVM heap and MemoryPercentUsage answer different questions. Alerting on only one is the classic mistake.
• Cap destination growth. Alert when destination count exceeds 2x expected, and enable inactive-destination GC in activemq.xml.
• Enable GC logging permanently. You cannot reconstruct GC history after an OOM without it.
• Size containers for JVM footprint. Limit = -Xmx plus overhead, with margin. Validate with a load test that pushes connection and destination counts to production levels.
• Load-test for the leak shape. A soak test over hours, watching heap-after-GC, catches resource leaks that functional tests never see.

How Netdata helps

• Netdata charts JVM heap usage per memory pool, so you can watch the post-GC floor trend rather than eyeballing instantaneous usage spikes.
• GC collection count and time per collector are charted alongside heap, making the “frequency rising, pause duration rising” distress pattern visible in one view.
• Broker-level metrics like MemoryPercentUsage, destination count, and connection count sit on the same dashboard as JVM metrics, which is the exact correlation you need to separate message-driven pressure from heap-driven pressure.
• Netdata’s ML anomaly detection flags the slow upward ratchet of heap-after-GC days before it crosses a static threshold, which is when you still have runway to act.
• Process and cgroup memory charts catch the container case: RSS plus non-heap drifting toward the cgroup limit while JVM heap looks fine.
• Alerts with hysteresis let you encode the page-safe rule (sustained, post-GC, uptime-gated) instead of paging on a single noisy reading.

Related guides

• ActiveMQ memory limit reached: MemoryPercentUsage at 100% and the flow-control cliff
• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ enqueue outpacing dequeue: reading the rate imbalance before the backlog
• ActiveMQ consumers connected but not acknowledging: the zombie consumer
• ActiveMQ InFlightCount high: prefetch full, acks stalled, and zombie consumers
• ActiveMQ KahaDB corruption: the broker won’t start after an unclean shutdown
• ActiveMQ disk full on the KahaDB partition: write failures and store corruption risk
• ActiveMQ.DLQ growing: dead letter queue accumulation and poison messages
• ActiveMQ offline durable subscriber pending messages: the silent storage leak
• ActiveMQ expired message count climbing: TTL expiry and silent correctness loss
• ActiveMQ KahaDB db.data index bloat: slow lookups and slow startup recovery
• ActiveMQ KahaDB journal files not deleted: one unacked message pinning a 32MB log