$ guides / activemq / activemq-destination-explosion ▌

Operations Guides

ActiveMQ destination explosion: dynamic destinations, MBean bloat, and GC pressure

An ActiveMQ Classic broker that has been running for months shows a specific signature: heap usage climbs steadily even when message volume is flat, GC pauses get longer and more frequent, the web console and JMX queries feel sluggish, and eventually the broker OOMs or slides into a GC death spiral. Queue depths look normal. Enqueue and dequeue rates look normal. Nothing about the messaging workload explains it.

The explanation is usually the destination registry. ActiveMQ Classic creates a destination automatically the first time a client produces to or consumes from a name that does not exist. If an application generates destination names dynamically (per request ID, per user, per session, per tenant) and nothing removes them, the registry grows without bound. Every destination registers at least four JMX MBeans and holds in-memory structures in the broker’s heap. At thousands of destinations, JMX itself becomes slow. At tens of thousands, the accumulated metadata is a first-order heap consumer, and the broker degrades or dies.

This guide covers ActiveMQ Classic 5.x. Artemis has a different address model and is out of scope.

What this means

The destination registry is not a lightweight name list. Each queue or topic carries dispatch state, subscription bookkeeping, memory accounting, store index references, and an MBean tree. Advisory topics multiply the count: advisory support is enabled by default, advisory destinations are real topics with real MBeans, and destination-specific advisories (consumer, producer, DLQ notifications) are created on demand, so a broker with 1,000 application queues can also carry a large population of ActiveMQ.Advisory.* topics.

flowchart TD
  A[App creates destination names dynamically] --> B[Broker auto-creates each destination]
  B --> C[Registry grows: queues, topics, advisory topics]
  C --> D[4+ MBeans per destination plus metadata]
  D --> E[Heap grows, GC frequency rises]
  E --> F[JMX queries slow, GC pauses lengthen]
  F --> G[Heartbeat timeouts, disconnects, OOM]

The failure is slow. Nothing breaks the day the application deploys. The count grows with application traffic, and by the time heap pressure shows up, the broker may hold tens of thousands of dead destinations, most with zero producers, zero consumers, and zero messages. The distinguishing feature versus a normal memory leak: heap growth correlates with destination count growth, not with message volume.

Common causes

Cause	What it looks like	First thing to check
Destination-per-entity antipattern	Destination names contain UUIDs, timestamps, user IDs, or tenant IDs; count tracks application traffic	Enumerate queue/topic names and look for the pattern
Request-reply with uncollected reply queues	Many queues with zero consumers and zero enqueue activity	Check whether the application deletes reply destinations after use
Missing auto-cleanup configuration	Destinations persist forever; the default broker config never removes inactive ones	Check `policyEntry` for `gcInactiveDestinations` and the broker for `schedulePeriodForDestinationPurge`
Advisory topic multiplication	Topic count far exceeds application topic count, dominated by `ActiveMQ.Advisory.*` names	Count topics matching the advisory prefix
Temporary destination accumulation	Temporary queue/topic counts climb monotonically	Check client connection pooling and whether connections (and their temp destinations) are ever closed

Quick checks

All read-only. The JMX examples use Jolokia on the default web console port; adjust the broker name and credentials for your deployment.

# Count registered queues on the broker
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/Queues' \
  | python3 -c "import json,sys; print(len(json.load(sys.stdin)['value']))"

# Count registered topics
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/Topics' \
  | python3 -c "import json,sys; print(len(json.load(sys.stdin)['value']))"

# Count temporary destinations
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/TemporaryQueues' \
  | python3 -c "import json,sys; print(len(json.load(sys.stdin)['value']))"

Rough severity bands for the totals: under 100 destinations is typical for a small deployment, 100-1000 is manageable, above 1000 needs explicit management, and above 10000 is almost always a design problem rather than a workload.

# List destination names to find the naming pattern
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/Queues' \
  | python3 -c "import json,sys; [print(q['objectName']) for q in json.load(sys.stdin)['value']]" | head -50

# JVM heap usage
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/java.lang:type=Memory/HeapMemoryUsage'

# GC collection counts and time
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/java.lang:type=GarbageCollector,name=*/CollectionTime'

These JMX reads are not free. On a broker that already has tens of thousands of destinations, a wildcard or array read serializes a large MBean tree and can take seconds. Poll at a modest interval and prefer a few targeted reads over broad enumeration loops. If you run a JMX-to-Prometheus exporter, increase the scrape interval rather than scraping a bloated broker aggressively.

How to diagnose it

Establish the growth curve. Take the queue and topic counts now and again in an hour. A count that climbs with application traffic and never goes down confirms creation without cleanup. A static high count means the leak may already be fixed but the broker was never restarted or purged.
Identify the naming pattern. Dump destination names and group them. Dynamic destinations almost always embed a variable: UUID, epoch timestamp, session ID, user ID. The pattern points directly at the producing application and often at the exact line of code.
Separate live destinations from dead ones. For a sample of the patterned names, check ConsumerCount, EnqueueCount, and QueueSize on the destination MBeans. In a classic explosion, the overwhelming majority have zero consumers, zero producers, and zero messages. That is your purge candidate set.
Quantify the advisory overhead. Count topics matching ActiveMQ.Advisory.*. If advisory support is enabled and you have thousands of destinations, advisories inflate the topic count substantially. Decide whether you actually consume advisory messages anywhere; many deployments do not.
Confirm the heap and GC link. Plot heap-used-after-GC against destination count over days. In a destination explosion the two curves track each other while enqueue volume stays flat. Also check GC pause durations: pauses growing toward the OpenWire wireFormat.maxInactivityDuration default of 30000 ms mean clients will start timing out and disconnecting, which turns slow degradation into a reconnect-storm incident.
Rule out the lookalikes. A broker memory usage (MemoryPercentUsage) problem is a different failure: pending messages against the configured limit, which triggers producer flow control. Destination explosion shows up in JVM heap and GC, not in MemoryPercentUsage. If your symptom is producers blocking, check the memory and flow-control signals first, not the destination count.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Queue and topic counts (JMX `Queues`/`Topics` arrays)	Direct measure of registry growth	Count >2x expected baseline, or monotonic growth tracking traffic
Temporary destination count	Detects request-reply temp destination leaks	Sustained growth or count above ~100
JVM heap used after major GC	MBeans and destination metadata live in heap	Upward trend after each major GC while message volume is flat
GC pause duration and frequency	The mechanism by which bloat becomes an outage	Full GCs more often than 1 per 5 minutes, pauses over 2 s
JMX query response time	Early, cheap indicator of MBean bloat	Queries that used to return in milliseconds now take seconds
Advisory topic count	Silent multiplier of the destination population	Advisory topics vastly outnumbering application topics

Page on the resource symptoms (heap after GC, GC pauses, connection drops), not on the raw destination count. The count is a ticket-level signal during growth; the GC and heap signals are what make it a page.

Fixes

Purge the dead destinations now

If the registry is already bloated, configuration changes alone will not reclaim the heap; you must remove the accumulated destinations. Identify the dead set (zero consumers, zero pending messages, no recent enqueue activity) and remove it via the broker MBean’s removeQueue/removeTopic operations, executable through Jolokia.

This is destructive and irreversible. Verify each destination is genuinely unused before removing it, script the removal in batches rather than deleting thousands in one call, and expect heap relief only after the next major GC. If the broker is already near OOM, removing in small batches with pauses is safer than one large sweep.

Enable automatic cleanup of inactive destinations

ActiveMQ does not garbage-collect inactive destinations by default. Enable it on the destination policy so the problem cannot recur:

<broker ... schedulePeriodForDestinationPurge="10000">
  <destinationPolicy>
    <policyMap>
      <policyEntries>
        <policyEntry queue=">" gcInactiveDestinations="true"
                     inactiveTimoutBeforeGC="60000" />
      </policyEntries>
    </policyMap>
  </destinationPolicy>
</broker>

schedulePeriodForDestinationPurge controls how often the broker scans for purgeable destinations, and inactiveTimoutBeforeGC (default 5000 ms; the misspelled attribute name is in the ActiveMQ source) defines how long a destination must be empty and consumer-less before it qualifies. Two operational caveats:

Empty and inactive only. This deletes destinations with no pending messages and no consumers. It will not drain a queue with unconsumed messages, so do not expect it to fix backlog problems.
Topics with active or durable subscribers. Applying a GC policy broadly with topic=">" can fight against topics that have subscribers. Scope the policy with a more specific wildcard covering the dynamically-created namespace rather than the whole broker.

Reduce the MBean footprint

Since ActiveMQ 5.12, the suppressMBean broker attribute can suppress MBean registration for dynamic producers, consumers, connections, and advisory destinations, for example suppressMBean="endpoint=dynamicProducer,endpoint=Consumer,connectionName=*,destinationName=ActiveMQ.Advisory.*". This reduces JMX bloat but does not remove the destinations themselves, so treat it as a complement to cleanup, not a substitute.

If you do not consume advisory messages, disable advisories with advisorySupport="false" on destinations where they are not needed. This stops the advisory topic population from growing alongside every new destination.

Fix the application

The durable fix is on the client side. Reuse a bounded set of destinations with message selectors or correlation IDs instead of minting a queue per entity. Delete reply destinations after use. Close JMS connections, sessions, consumers, and producers explicitly; pooled connections that never close also hold their temporary destinations open. If dynamic creation is truly unnecessary, restrict it with authorization policy so only approved naming patterns can be created.

Prevention

Alert on destination count growth. Track queue, topic, and temp destination counts against baseline. A count growing in step with application traffic is a ticket before it is ever a page.
Set auto-cleanup at provisioning time. gcInactiveDestinations and a purge schedule belong in the baseline broker configuration for any environment where clients create destinations, not just production.
Whitelist destination names. Periodically enumerate destinations and compare against expected patterns. Unexpected creation is both an operational and a security signal.
Watch heap-after-GC as a leak detector. A rising post-GC floor with flat message volume is the earliest reliable warning, weeks before the count itself looks scary.
Bound JMX cost. On high-destination brokers, keep monitoring poll intervals modest and avoid broad wildcard queries in tight loops; the monitoring itself can worsen the bloat’s impact.

How Netdata helps

JVM heap after GC and GC pause metrics collected from the broker’s JVM show the post-GC floor rising and pauses lengthening, which is the signal chain that separates destination explosion from a traffic problem.
Per-second connection counts expose the downstream consequence: when GC pauses approach the inactivity timeout, you see the sawtooth of disconnects and reconnect storms on the same dashboard as the GC metrics.
Destination and broker JMX metrics let you track queue and topic counts, enqueue/dequeue rates, and memory percent usage together, so you can confirm that heap growth tracks destination count rather than message volume.
Disk and store metrics rule out the lookalike failure: if store usage and journal growth are flat while heap climbs, the problem is metadata bloat, not message backlog.
Long retention matters here because this failure develops over weeks. Comparing this month’s destination count and heap floor against last month’s is often the entire diagnosis.