ActiveMQ GC pause death spiral: long pauses, heartbeat timeouts, and reconnect storms

The broker is up. The port is listening. But clients keep dropping, reconnecting, dropping again, and every cycle makes things worse. Connection count charts show a sawtooth: a sudden drop, a spike, another drop. In the JVM, heap usage sits above 90% after every major GC and full GC pauses are climbing into the multi-second range.

This is the GC pause death spiral, a positive-feedback loop and one of the characteristic failure archetypes of ActiveMQ Classic. Heap pressure produces long GC pauses. A long pause freezes every broker thread, including the ones answering client keepalives. Clients exceed wireFormat.maxInactivityDuration (default 30000 ms) and disconnect. After the pause ends, every client reconnects at once, and each reconnect allocates new connection, session, consumer, and subscription objects. That allocation spike raises heap pressure further, the next GC pause is longer, and more clients time out. The broker oscillates between being frozen in GC and being hammered by reconnect storms until it is effectively down.

The insidious part: from the client’s perspective, a GC pause is indistinguishable from a dead broker or a network failure. The InactivityMonitor does not care why it heard nothing for 30 seconds. It closes the transport either way.

What this means

ActiveMQ Classic is a JVM. Every pending message in a VM cursor, every connection, every destination MBean, every session lives in heap. When heap pressure builds, the garbage collector stops the world to reclaim memory, and during that stop the broker cannot read sockets, dispatch messages, or respond to keepalives.

OpenWire clients and the broker exchange keepalive traffic governed by wireFormat.maxInactivityDuration, which defaults to 30000 ms and is negotiated to the shortest value between client and broker at connection startup. Any GC pause approaching or exceeding that window looks like a broker failure to every connected client. Clients using the failover transport then reconnect, and the failover defaults mean the first attempt happens almost immediately, with exponential backoff only kicking in afterward. With hundreds of clients, that is a thundering herd landing on a broker that has barely finished its last collection.

flowchart TD
  A[Heap pressure builds] --> B[Long stop-the-world GC pause]
  B --> C[Clients exceed maxInactivityDuration 30s]
  C --> D[Mass client disconnect]
  D --> E[Reconnect storm hits broker]
  E --> F[New connection, session, subscription objects allocated]
  F --> A
  B -.->|enqueue and dequeue drop to zero| G[Messages backlog grows]
  G -.->|more memory charged| A

Two thresholds matter for reading GC health on the broker:

A pause over 10 seconds is not yet past the default 30-second inactivity window, but it is close enough that stacked pauses, slow clients, or a lowered maxInactivityDuration on either side will push connections over the edge. A pause over 30 seconds guarantees it.

Common causes

Quick checks

All read-only. Run these before touching anything.

# 1. Heap usage after GC via Jolokia (adjust credentials, broker name, and port)
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/java.lang:type=Memory/HeapMemoryUsage'

# 2. GC cumulative time and count per collector
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/java.lang:type=GarbageCollector,name=*/CollectionTime'

# 3. Live GC sampling with jstat (if JDK tools are on the host)
jstat -gcutil $(pgrep -f activemq) 1000 10

# 4. Current connection count: look for the sawtooth
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/CurrentConnectionsCount'

# 5. Total destination count: rule out a destination explosion
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/Queues' \
  | python3 -c "import json,sys; print(len(json.load(sys.stdin)['value']))"

# 6. Broker log for InactivityMonitor and GC-adjacent errors
grep -i "inactivity\|InactivityIOException\|channel inactive" /opt/activemq/data/activemq.log | tail -20

# 7. GC log, if enabled (it should be)
tail -100 /opt/activemq/data/gc.log

CollectionTime and CollectionCount are cumulative counters. You need two readings with a time delta to compute average pause duration: divide the CollectionTime delta by the CollectionCount delta for each collector (names vary by GC algorithm: G1 Young Generation, G1 Old Generation, PS MarkSweep, and so on).

How to diagnose it

1. Confirm the sawtooth. Pull CurrentConnectionsCount over the incident window. Repeated sudden drops followed by immediate spikes back to (or above) baseline are the disconnect/reconnect signature. Correlate the drop timestamps with enqueue and dequeue rates: both should crater during the pauses.

2. Correlate drops with GC pauses. For each connection-count drop, find the matching full GC in the GC log or in the G1 Old Generation (or equivalent) CollectionTime deltas. If every drop lines up with a multi-second collection, the mechanism is confirmed. If drops happen with no GC activity, you are looking at a network or load balancer problem instead.

3. Determine whether heap can be freed. The decisive reading is heap used after a major GC, not instantaneous usage. If post-GC heap stays above 90%, the heap is genuinely full of live objects and no amount of GC tuning will save you; the broker needs more heap or fewer live objects. If post-GC heap drops well below 70%, the pauses may be tunable.

4. Find what is filling the heap. Check total destination count first (each destination creates several MBeans plus real per-destination state). Then check whether ActiveMQ MemoryPercentUsage tracks JVM heap. High ActiveMQ memory with high heap points at messages in VM cursors (typically non-persistent traffic). High heap with low ActiveMQ memory points at non-message objects: destination metadata, MBeans, connection state, advisory topics.

5. Check the client side. Look at producer and consumer logs for InactivityIOException and reconnect attempts. Note whether clients use the failover transport with default settings: initialReconnectDelay=10 with exponential backoff (reconnectDelayExponent=2.0, maxReconnectDelay=30000). The near-instant first attempt is what turns a pause into a herd. Also check whether any client lowered maxInactivityDuration below 30000, since the monitor negotiates the shortest value.

6. Capture a thread dump if the broker hangs. During a hang, jstack <pid> or kill -3 <pid> is the single most valuable artifact. Transport threads are named with client IPs, which tells you who is hammering the broker.

Metrics and signals to monitor

Fixes

Break the loop first (incident response)

The spiral feeds on reconnections. Reducing inbound connection pressure buys time.

• Shed non-essential clients. Stop or disconnect non-critical producers and consumers. Fewer connections means fewer objects allocated per reconnect cycle and fewer keepalives to answer.
• Do not restart as a first move. A restart interrupts all in-flight work, and on a large KahaDB store the recovery phase (journal replay, index rebuild) can take minutes to hours while the port is open but not serving. Restart is often eventually necessary here, but make it a deliberate decision, not a reflex. If clients use the failover transport with maxReconnectAttempts=-1 (retry forever, the default from 5.6 onward), they will queue up and storm the broker the moment it returns.
• Capture diagnostics before any restart: GC log, thread dump, and the Jolokia readings above. Post-restart, the evidence is gone.

Right-size the heap

If post-GC heap is above 90% and destinations and connections are legitimate, the heap is too small. Increase -Xmx (requires a restart), and set -Xms equal to -Xmx so the JVM does not waste time on heap resizing. Keep ActiveMQ’s memoryUsage limit at roughly 60-70% of JVM max heap so the broker retains room for connection state, MBeans, and dispatch machinery. After the increase, target at least 30% free heap after major GC.

Tune or switch the garbage collector

For heaps above 4 GB, use G1GC (-XX:+UseG1GC). Note that CMS was removed in JDK 14 (JEP 363), and current ActiveMQ Classic releases require Java 11+. Any deployment that still passes -XX:+UseConcMarkSweepGC is either pinned to an old JDK or running with the flag rejected or ignored; check which. If your pause requirements are stricter than G1 can deliver, ZGC (production-ready from JDK 15) or Shenandoah can largely eliminate stop-the-world pauses; Shenandoah availability depends on the JDK build, so confirm your distribution ships it. Whatever collector you run, always enable GC logging (-Xlog:gc*:file=gc.log:time on JDK 11+). Post-incident, the GC log is the most useful diagnostic artifact you will have.

Fix the root allocation pressure

• Destination explosion: identify the naming pattern, fix the application to reuse destinations, and configure schedulePeriodForDestinationPurge and gcInactiveDestinations so empty destinations are cleaned up automatically. Purge existing dead destinations via JMX.
• Non-persistent floods in VM cursors: either move the traffic to persistent messaging (store-based cursors page to disk) or accept the memory cost explicitly by sizing for it.
• Advisory overhead: with thousands of destinations, advisory topics can substantially increase destination and MBean count. Set advisorySupport="false" where advisories are not consumed.

Blunt the reconnect storm on the client side

You cannot stop clients from reconnecting, but you can stop them from doing it all at once. The failover transport supports backoff tuning: initialReconnectDelay, maxReconnectDelay, and reconnectDelayExponent are all configurable in the client URI. Raising the initial delay and adding jitter across client fleets spreads the storm out. Raising wireFormat.maxInactivityDuration above 30000 gives the broker more room to survive a long pause, at the cost of slower detection of genuinely dead brokers; that is a real tradeoff, not a free win. Setting maxInactivityDuration=0 or transport.useInactivityMonitor=false disables the monitor entirely, which hides GC freezes but also hides real network failures. Use it only if you understand what you are giving up.

Prevention

• Track heap-after-major-GC as a trend, not a point reading. The spiral has a long fuse: weeks of rising post-GC heap and climbing GC frequency before the first disconnect. Alerting on pauses alone catches the fire, not the fuel.
• Alert on the composite, not any single signal. A single 3-second GC pause is not an incident. A pause plus a connection-count drop plus an accept-rate spike is the pattern. Page on the correlation, ticket on the individual threshold breaches.
• Size with headroom. At least 30% heap free after major GC in steady state, and ActiveMQ memoryUsage at 60-70% of max heap.
• Enable GC logging everywhere, permanently. It costs almost nothing and is the difference between a ten-minute postmortem and a week of guessing.
• Watch destination count growth. It is the most common slow leak into heap exhaustion, and it is invisible if you only watch message rates.
• Exclude startup from paging. KahaDB recovery and initial cursor paging legitimately produce elevated memory and GC activity. Suppress spiral alerts until broker uptime exceeds 600 seconds.
• In containers, reconcile limits. -Xmx must fit inside the container memory limit with room for thread stacks and off-heap use, and liveness probes must tolerate multi-second pauses or they will restart the broker mid-spiral and make everything worse.

How Netdata helps

Netdata’s strength on this failure mode is correlation across the boundary between the JVM and the broker:

• JVM GC and heap metrics collected per collector (matching the CollectionCount/CollectionTime MBeans) so pause duration and frequency are visible without SSH-ing for jstat.
• ActiveMQ connection count and per-connector state alongside GC charts, making the sawtooth and its alignment with full GCs a one-screen diagnosis instead of a log-matching exercise.
• Enqueue/dequeue rates and MemoryPercentUsage on the same dashboard as heap usage, so you can immediately tell message-driven heap pressure from metadata-driven pressure.
• Anomaly detection on connection count and accept rate, which flags the reconnect-storm phase of the loop even before absolute thresholds are breached.
• Per-second granularity, which matters here because the interesting events (a 5-second pause, a 200-connection drop, an accept spike) fit inside a single minute and vanish in coarse-grained monitoring.

Related guides

• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ memory limit reached: MemoryPercentUsage at 100% and the flow-control cliff
• ActiveMQ consumers connected but not acknowledging: the zombie consumer
• ActiveMQ InFlightCount high: prefetch full, acks stalled, and zombie consumers
• ActiveMQ enqueue outpacing dequeue: reading the rate imbalance before the backlog
• ActiveMQ KahaDB corruption: the broker won’t start after an unclean shutdown
• ActiveMQ KahaDB journal files not deleted: one unacked message pinning a 32MB log
• ActiveMQ.DLQ growing: dead letter queue accumulation and poison messages
• ActiveMQ offline durable subscriber pending messages: the silent storage leak
• ActiveMQ expired message count climbing: TTL expiry and silent correctness loss
• ActiveMQ disk full on the KahaDB partition: write failures and store corruption risk
• ActiveMQ KahaDB db.data index bloat: slow lookups and slow startup recovery