ActiveMQ network bridge down: a Network of Brokers partition and store-and-forward backlog

In a Network of Brokers, messages cross between ActiveMQ Classic brokers over network bridges. When a bridge drops, the failure is asymmetric: the origin broker keeps accepting messages for destinations whose consumers live on the remote broker, and those messages pile up locally. The remote broker keeps running with consumers connected and nothing to consume. Both brokers look healthy in isolation. Only cross-broker correlation reveals the partition.

The second phase is often worse than the first. When the bridge reconnects, the store-and-forward backlog replays in a burst. The receiving broker gets the entire accumulated backlog at wire speed, which can spike its memory into producer flow control and take down traffic that was fine all along.

This guide covers how to confirm a bridge is actually down (not just idle), how to find why, and how to get through reconnection and replay without a second incident.

What this means

ActiveMQ Classic network connectors use demand forwarding with store-and-forward semantics. A bridge forwards messages to a remote broker only when consumer demand for that destination exists on the remote side. Demand is propagated via advisory messages. When the bridge connection drops:

• Messages produced on the origin broker for remotely-consumed destinations have nowhere to go. They accumulate in the origin broker’s store and memory.
• Consumers on the remote broker stay connected but receive nothing.
• Queue depth grows on the origin broker while the remote broker shows consumers with zero dequeue rate.

Bridges are directional unless the network connector sets duplex="true". Broker A forwarding to broker B is a separate bridge from B forwarding to A. One direction can be up while the other is down, which produces a one-way partition that is easy to misread as a consumer problem.

There is no single broker-level counter that tells you how many bridges are up. You enumerate the bridge MBeans under each network connector and read their state and counters individually. Alerting on “broker up” will never catch this; alerting on cross-broker queue asymmetry will.

Bridge reconnection is not instant. Reconnect attempts back off, and after the TCP connection is re-established, demand-forwarding subscriptions still need to reconverge before messages flow. Expect minutes, not seconds.

flowchart TD
  A[Network fault, remote broker down, or auth/DNS failure] --> B[Bridge drops between broker A and B]
  B --> C[Messages accumulate on origin broker A]
  B --> D[Consumers on broker B idle, dequeue rate zero]
  C --> E[Origin broker memory and store usage climb]
  B --> F[Bridge reconnects after backoff, minutes]
  F --> G[Demand forwarding reconverges]
  G --> H[Store-and-forward replay burst]
  H --> I[Receiving broker memory spike, possible flow control]

Common causes

One known operational gap: after the remote broker’s transport connector itself restarts (for example, after an internal shutdown handler closes and reopens it), the network bridge does not always re-establish automatically. The remote broker looks fully healthy, clients reconnect fine, but the bridge stays down until the network connector is restarted on the origin side. If your bridge is down and the remote broker recently bounced its transport, suspect this first.

Quick checks

All checks are read-only. Run them on the origin broker first, then the remote.

# 1. Enumerate network connectors and their bridge MBeans via Jolokia
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,connector=networkConnectors,networkConnectorName=*'

# 2. Enumerate child bridge MBeans and their counters
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,connector=networkConnectors,networkConnectorName=*,networkBridge=*'
# Look at EnqueueCounter and DequeueCounter per bridge. Take two readings a
# minute apart: flat counters on a bridge that should carry traffic is a finding.

The broker registers each network connector under connector=networkConnectors,networkConnectorName=<name> and each child bridge under the same name plus networkBridge=<remote address> (duplex connectors use connector=duplexNetworkConnectors). The networkBridge=* wildcard in the Jolokia queries above is therefore valid for enumerating bridges.

# 3. Check bridge connect/disconnect events in the broker log
grep -i "network" /opt/activemq/data/activemq.log | tail -20

# 4. From the origin host, verify the remote broker's transport port is reachable
nc -z -w 3 remote-broker-host 61616 && echo "REACHABLE" || echo "UNREACHABLE"

# 5. Check the remote broker for auth failures from the origin broker's network user
grep -i "authentication failed\|invalid credentials\|not authorized" /opt/activemq/data/activemq.log | tail -20

# 6. Queue depth on affected destinations on the origin broker
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=YOUR.QUEUE/QueueSize'

# 7. Consumer count for the same destination on the remote broker
# (run against the remote broker's Jolokia endpoint)
curl -s -u admin:admin \
  'http://remote-broker:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=remote,destinationType=Queue,destinationName=YOUR.QUEUE/ConsumerCount'

Adjust the Jolokia paths, MBean broker names, and credentials to your deployment. Since 5.16 the web console and Jolokia bind to 127.0.0.1 by default, so run these locally on each broker host or go through your own secured proxy.

How to diagnose it

1. Confirm the partition, not just a quiet period. A bridge with zero throughput is normal when there is no remote consumer demand. The signature of a real partition is the combination: queue depth growing on the origin broker, consumers present on the remote broker, and bridge counters flat or bridge MBeans absent. Check all three before concluding anything.

2. Enumerate the bridge MBeans. There is no single bridgeCount attribute on the broker MBean. List the network connector MBeans, then their child bridges, and read EnqueueCounter and DequeueCounter on each. A missing child bridge MBean where your topology expects one is the partition, confirmed.

3. Check directionality. If your connectors are not duplex="true", each direction is an independent bridge. Verify both directions on both brokers. A queue that forwards A to B can be fine while B to A is partitioned.

4. Read the logs on both sides. A disconnect with no subsequent reconnect points at network or remote-broker failure. Repeated connect attempts followed by rejection points at authentication or authorization: the network user the connector authenticates as needs credentials valid on the remote broker, and if you run an authorization plugin it also needs permission to advisory destinations, since demand forwarding rides on advisories. Missing advisory permissions silently prevent bridge formation.

5. Verify reachability at the network layer. From the origin host, connect to the remote transport port and resolve the connector URI hostname. Do this from the broker host itself, not your workstation; broker-to-broker network paths often differ from operator access paths.

6. If the bridge is connected but not forwarding, suspect demand forwarding. Check whether advisory support has been disabled anywhere in the path; without advisories, dynamic demand forwarding cannot work. Also note that with conduitSubscriptions="true" (the default), remote consumer demand is condensed into a single subscription, so you cannot see per-remote-consumer demand in the consumer count. A “virtual” consumer on the origin broker representing remote demand is expected, not a real application consumer.

7. Measure the backlog before reconnecting. Note queue depth, MemoryPercentUsage, and StorePercentUsage on the origin broker. This tells you how large the replay burst will be and whether the origin broker is approaching flow control on its own while partitioned.

Metrics and signals to monitor

The cross-broker asymmetry is the composite signal: messages on one broker, idle consumers on the other, both brokers green on local health checks. Alert on “queue has messages but zero reachable consumers” across the pair, not on either broker alone.

Fixes

Network or remote broker fault

Restore connectivity or bring the remote broker back. The network connector retries with backoff, so the bridge should re-establish on its own within minutes. If the remote broker restarted its transport connector and the bridge does not come back, restart the network connector on the origin broker (the NetworkConnector MBean exposes start and stop operations) rather than restarting the whole broker. A full broker restart also works but forces store recovery and client reconnection, so treat it as the fallback, not the first move.

Authentication or authorization failure

Fix the credentials the network connector presents, or grant the network user the permissions it needs on the remote broker, including advisory destinations. Until this is fixed, every reconnect attempt fails; no amount of waiting helps.

Demand forwarding broken while bridge is connected

Re-enable advisory support if it was disabled, and verify advisory traffic is not being filtered between the brokers. If you change conduitSubscriptions or advisory configuration, expect to restart the connector for it to take effect.

Managing the replay burst

When the bridge comes back, the accumulated backlog replays as fast as the bridge and the receiving broker allow. If the backlog is large relative to the receiving broker’s memory headroom, that burst can trigger producer flow control on the receiver. Watch the receiving broker’s MemoryPercentUsage during reconnection. Mitigations: drain or move part of the backlog before reconnecting if the messages allow it, reduce the network connector’s prefetch to slow the replay, or temporarily raise the receiver’s memory limit. Do not reconnect blindly into a receiver that is already near its memory ceiling.

One-way partition on non-duplex connectors

If only one direction failed, fix that direction’s connector. Do not retrofit duplex="true" without checking your topology: duplex bridges between the same broker pair need unique names, and changing bridge or broker names breaks durable topic subscribers across the network, because the durable subscription name incorporates the bridge name.

Prevention

• Monitor bridges as first-class signals. Enumerate bridge MBeans and alert when an expected bridge is absent or its counters go flat while remote demand exists. Broker up/down checks will never catch a partition.
• Build cross-broker dashboards. Queue depth and consumer count for the same logical destination, side by side across the pair. The asymmetry is the alert.
• Alert on origin-broker memory and store growth during partitions. The partition itself is a ticket; the backlog it builds is what turns it into a page.
• Plan for replay. Know your worst-case partition duration and the backlog it implies. Size the receiving broker’s memory headroom so a full replay does not push it into flow control.
• Keep the network user’s credentials and advisory permissions under change control. Credential rotation and authorization plugin changes are a common cause of bridges that silently never come back.
• Expect bridge flapping during rolling restarts. Brief bridge drops during planned broker bounces are normal. Suppress bridge alerts during maintenance windows, but watch for bridges that fail to return afterward, especially the known case where a remote transport connector restart leaves the bridge down until the connector is restarted.

How Netdata helps

• Per-bridge visibility without a single counter. Netdata collects JMX metrics from ActiveMQ, so you can chart each network connector’s bridge MBeans and their EnqueueCounter/DequeueCounter over time instead of taking manual point-in-time readings.
• Cross-broker correlation in one view. Charting origin queue depth next to remote consumer count and dequeue rate makes the partition signature (backlog on one side, idle consumers on the other) visible at a glance rather than requiring two consoles.
• Runway estimation during the partition. Watching MemoryPercentUsage and StorePercentUsage trend on the origin broker tells you how long you have before the backlog triggers producer flow control locally.
• Replay burst detection. Per-second memory metrics on the receiving broker show the store-and-forward replay spike the moment the bridge reconnects, so you can throttle before flow control engages.
• Log-to-metric timeline. Correlating bridge disconnect events in the broker log with the metric timeline separates network faults, auth rejections, and remote restarts without manual log archaeology.

Related guides

• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ broker down: telling a crashed broker from a hung one
• ActiveMQ broker won’t start: port conflicts, store recovery, and lock contention
• ActiveMQ InactivityIOException: Channel was inactive for too long
• ActiveMQ enqueue outpacing dequeue: reading the rate imbalance before the backlog
• ActiveMQ GC pause death spiral: long pauses, heartbeat timeouts, and reconnect storms
• ActiveMQ disk full on the KahaDB partition: write failures and store corruption risk
• ActiveMQ consumers connected but not acknowledging: the zombie consumer