ActiveMQ Network of Brokers replay storm: burst forwarding after a bridge reconnect

A network bridge between two ActiveMQ brokers drops. Producers keep sending to the origin broker, and because ActiveMQ networks do reliable store-and-forward, the messages pile up locally. When the bridge reconnects, the accumulated backlog is replayed across the bridge in a burst. The receiving broker, idle a second ago, is now ingesting the backlog at wire speed. Its memory usage spikes, and if the spike reaches 100%, producer flow control activates on the receiver and producers there block silently.

This is not a bug; it is store-and-forward doing exactly what it was designed to do, at a rate the receiving broker cannot absorb. The partition itself is usually a TICKET-level event. The pageable incident starts when replay triggers flow control or breaches backlog/age SLOs on the receiving broker.

This guide covers how to recognize the pattern, confirm it with JMX and logs, contain an in-progress replay, and size the bridge so the next partition does not become an incident.

What this means

In a Network of Brokers, each network connector creates a demand-forwarding bridge. Consumer demand propagates across the bridge via advisory messages, and messages flow toward that demand. When the bridge goes down, two things happen:

1. The origin broker keeps accepting messages. Persistent messages are written to KahaDB as usual. Durability is retained because the source is durable.
2. Forwarding stops. With no bridge, the backlog grows on the origin broker for every destination whose consumers live on the remote side.

When connectivity returns, the bridge reconnects (network connectors reconnect by default), demand re-converges, and the stored backlog is forwarded as fast as the bridge and the receiving broker will take it. On the receiving broker, forwarded messages count as enqueues. If the backlog is large relative to the receiver’s memory limit and its consumers cannot drain fast enough, memory climbs to 100% and flow control activates:

flowchart LR
  A[Network partition] --> B[Bridge down: backlog grows on origin broker]
  B --> C[Bridge reconnects]
  C --> D[Store-and-forward replay burst]
  D --> E[Receiving broker memory spikes]
  E --> F{Memory at 100%?}
  F -->|no| G[Backlog drains, incident over]
  F -->|yes| H[Flow control on receiver: producers block silently]

The operationally confusing part: the receiver’s producers start blocking even though nothing about its local workload changed. Operators looking only at the receiver see a classic slow-consumer cascade and hunt for a stuck consumer that does not exist. The bridge replay is the cause; the memory spike is the symptom.

Common causes

The last row is the inverse failure: a bridge can be “connected” but not forwarding. Messages accumulate on the origin while consumers wait idle on the remote. When demand finally propagates, you get the same burst.

Quick checks

Read-only checks for a suspected replay event. The Jolokia examples assume the default web console on 8161 with default credentials and default log paths; adjust for your environment.

# Check receiving broker memory usage (the primary containment signal)
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/MemoryPercentUsage'

# Enumerate network bridge MBeans on the origin broker
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,connector=networkConnectors,networkConnectorName=*'

# Watch the receiver's enqueue rate (cumulative counter; take two readings)
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/TotalEnqueueCount'

# Check per-destination memory on the receiver to find which queues the replay is hitting
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=*/MemoryPercentUsage'

# Bridge events in the origin broker log
grep -i "network" /opt/activemq/data/activemq.log | tail -30

• Receiving broker memory percent. The number that decides whether the replay is an inconvenience or an incident. At 100%, flow control is active.
• Bridge MBean enumeration. There is no single bridge count attribute; per-bridge EnqueueCounter/DequeueCounter live on bridge MBeans whose object name appends networkBridge=<remote-address> to the network connector’s name. Query connector=networkConnectors,networkConnectorName=<name>,networkBridge=* and read their counters to confirm whether the replay is flowing and at what rate.
• Receiver enqueue rate. Forwarded messages count as enqueues on the receiving broker. A sharp enqueue spike with no local producer burst is the replay signature.
• Broker log network lines. Bridge connect/disconnect events with timestamps let you size the partition window, which predicts the backlog volume.

How to diagnose it

1. Establish the timeline. In the origin broker’s log, find the bridge disconnect and reconnect timestamps. The gap is the accumulation window. Multiply the window by the normal enqueue rate on affected destinations to estimate the backlog the receiver is about to absorb.
2. Confirm accumulation on the origin. During the partition, queue depth on the origin grows for destinations whose consumers are remote. Consumer count on those destinations may show only the bridge’s demand-forwarding subscription, or zero once the bridge is gone.
3. Confirm the burst on the receiver. After reconnect, take two readings of TotalEnqueueCount 30 seconds apart on the receiver. If the derived rate is far above baseline while local producer traffic is normal, the replay is in flight.
4. Watch the receiver’s MemoryPercentUsage continuously. Rate of climb matters more than the current value. Climbing several percent per minute means you have minutes of runway before flow control.
5. If memory hits 100%, verify flow control impact. The broker log shows memory limit messages, and advisory topics ActiveMQ.Advisory.Full.Queue.<name> fire for full destinations. Producers with no send timeout will be silently blocked in send().
6. Check the receiver’s consumers are actually draining. Replay plus healthy consumers usually drains without flow control. If dequeue rate is low while memory climbs, the replay exposed a consumer capacity problem, not just a burst problem. Check inflight counts against prefetch.
7. Rule out the lookalike: bridge connected but not forwarding. If the origin backlog keeps growing after “reconnect,” check per-bridge counters. Zero bridge throughput with a connected bridge is a demand-forwarding failure, a different incident.

Metrics and signals to monitor

Severity guidance: the partition itself is TICKET. Page only when replay triggers flow control on the receiving broker, or when backlog depth or message age on the receiver breaches a critical SLO.

Fixes

During the replay: contain the burst

Monitor first, act second. If the receiver’s memory is climbing but has headroom and consumers are draining, the correct action is to watch it. The backlog will drain and the incident ends on its own.

Slow the bridge if memory is heading for 100%. Reduce the prefetch on the network connector so the bridge pulls the backlog in smaller chunks instead of at wire speed. Network connector consumers have a default prefetch of 1000, and it must stay above zero because network consumers do not poll. Lowering it throttles the replay rate at the cost of a longer drain. This is a configuration change on the origin broker’s network connector, so plan it as a controlled change, not a mid-page edit.

If flow control is already active on the receiver, the standard slow-consumer-cascade responses apply: temporarily increase the receiving broker’s memory limit (requires restart), shed non-critical producers or consumers on the receiver, or disconnect a genuinely stuck consumer if the replay exposed one. Do not restart the receiving broker as a first move: the backlog is durable on the origin, but restarting the receiver extends the bridge churn and can re-trigger replay against a broker that is also doing KahaDB recovery.

After the event: fix the root cause

Fix why the bridge dropped. Partitions from rolling restarts are expected and brief. Partitions from GC pauses mean the broker that paused needs heap/GC attention (see the GC death spiral guide). Partitions from network infrastructure mean the bridge reconnect delay and your alerting both need tuning.

Right-size receiver capacity for the worst-case partition. Estimate the largest realistic accumulation window (longest plausible partition times peak enqueue rate) and verify the receiver’s memory limit and consumer capacity can absorb that replay without hitting 100%. If it cannot, either raise the receiver’s headroom or permanently lower the network connector prefetch so replays are always paced.

Prevention

• Alert on bridge state. Any deviation from expected bridge topology is a TICKET. Catching the partition early is what keeps the backlog small.
• Alert on receiver memory rate-of-climb during bridge reconnects. Correlate bridge reconnect events with MemoryPercentUsage slope. This is the earliest reliable replay-storm warning.
• Set per-destination memory limits. Without them, one replayed destination can consume the shared broker-level pool and flow-control unrelated producers on the receiver.
• Size prefetch deliberately on network connectors. Default 1000 is fine for steady state; where partitions are common and backlogs are large, a lower prefetch paces every replay.
• Keep TTL and expiry policy intentional. Messages that expire while waiting out a partition still consume store and memory until the expiry sweep, and expired messages go to the DLQ by default.
• If you use replayWhenNoConsumers, note the version-specific gotcha from the official documentation: it applies only before 5.9. On those versions you must also set enableAudit="false" on the destination policy, or the cursor’s duplicate audit can mark replayed messages as duplicates; on 5.9 and later the requirement no longer applies.
• Cross-broker dashboards. The replay storm is invisible from either broker alone. The origin looks fine (it is just storing messages); the receiver looks like a slow-consumer incident. Only the correlation shows the bridge replay as the cause.

How Netdata helps

• Bridge and broker state in one view. Netdata’s ActiveMQ collector pulls JMX metrics from every broker in the topology, so a bridge drop on the origin and a memory spike on the receiver appear on the same timeline instead of in two separate consoles.
• Enqueue rate anomaly detection. A replay burst is a sharp enqueue-rate deviation with no local producer cause. Per-second collection and ML anomaly flags catch the spike the moment the bridge reconnects, before memory reaches 100%.
• Memory usage with rate-of-climb visibility. Watching MemoryPercentUsage at one-second resolution during a replay shows slope, not just level, which is what determines your runway to flow control.
• Cross-signal correlation. Overlaying queue depth growth on the origin during the partition window with the receiver’s memory curve confirms the replay mechanism and rules out the slow-consumer lookalike.
• Alerting matched to the severity split. TICKET-level alerts on bridge state and origin backlog growth; PAGE-level alerts only when receiver memory approaches flow control or message age breaches SLO, which keeps the pattern from paging during harmless drains.

Related guides

• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ broker down: telling a crashed broker from a hung one
• ActiveMQ InactivityIOException: Channel was inactive for too long
• ActiveMQ enqueue outpacing dequeue: reading the rate imbalance before the backlog
• ActiveMQ GC pause death spiral: long pauses, heartbeat timeouts, and reconnect storms
• ActiveMQ disk full on the KahaDB partition: write failures and store corruption risk