ActiveMQ network bridge connected but not forwarding: broken demand forwarding

The bridge is up. JMX shows the network bridge MBeans, the broker log shows a successful connection to the remote broker, and yet messages produced on broker A sit in the queue while consumers on broker B wait idle. No errors on either side. Each broker’s local dashboard looks healthy.

This is broken demand forwarding in an ActiveMQ Classic Network of Brokers. A network bridge is not a pipe. It forwards messages only when it knows a consumer exists on the remote side, and it learns that through advisory messages. When the bridge is connected but the demand signal is not flowing, you get a silent stall: backlog on one broker, starvation on the other, nothing alarming on a per-broker view.

The tell is asymmetry: a queue with pending messages and zero local consumers on one broker, and connected but idle consumers for the same destination on the other. Diagnosing this requires correlating state across both brokers, not staring at either one in isolation.

What this means

ActiveMQ network bridges are store-and-forward with demand-driven forwarding. The bridge subscribes to advisory topics on the remote broker, learns about remote consumer demand, and creates a local proxy consumer for the destination. Messages flow because that proxy consumer pulls them across.

If advisories are suppressed, filtered, blocked by authorization, or the demand subscription was never established (or was lost and never re-established), the bridge stays connected but forwards nothing. Messages accumulate locally while consumers wait idle remotely.

Three configuration details make this easy to miss:

• Demand forwarding creates “virtual” consumers. A destination’s ConsumerCount can include bridge demand subscriptions, so consumer counts alone do not tell you whether real application consumers or bridge proxies are present.
• With the default conduitSubscriptions="true", all remote consumer demand is condensed into a single subscription on the bridge. Efficient, but you cannot see per-remote-consumer demand, and the local consumer count will not match the number of real consumers on the far side.
• After a network blip, bridge reconnection is not instantaneous and demand subscriptions must reconverge. The bridge can be back “up” for minutes before forwarding resumes, or never resume correctly.

flowchart LR
  P[Producer] --> QA[Queue on broker A: messages piling up]
  QA --> BR[Network bridge A to B: connected]
  BR -. "consumer advisories (broken or missing)" .-> ADV[Advisory topics on broker B]
  ADV -. "no demand signal" .-> BR
  CB[Consumers on broker B: idle]
  CB -- "real demand, not propagated" --> ADV

Common causes

Quick checks

Run these against both brokers. All are read-only.

# 1. Enumerate bridge MBeans on broker A (there is no single bridgeCount attribute)
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,connector=networkConnectors,networkConnectorName=*,networkBridge=*'

# 2. Bridge throughput counters (EnqueueCounter / DequeueCounter per bridge MBean)
#    A connected bridge with flat counters is the smoking gun.

# 3. Queue depth and consumer count for the stuck destination on broker A
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/QueueSize'
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/ConsumerCount'

# 4. Repeat QueueSize and ConsumerCount on broker B for the same destination
#    Asymmetry (messages + 0 local consumers on A, idle consumers on B) confirms the pattern.

# 5. Bridge connect/disconnect history
grep -i "network" /opt/activemq/data/activemq.log | tail -30

# 6. Authorization denials on the remote broker
grep -ci "not authorized" /opt/activemq/data/activemq.log

# 7. Connection count sanity check on both brokers
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/CurrentConnectionsCount'

Bridge connections count as connections on the broker, so check 7 only tells you the transport is up, not that forwarding works. That is exactly the trap.

Verified for the 5.x line: the bridge MBean object name is built from the network connector name plus the remote address (...connector=networkConnectors,networkConnectorName=<name>,networkBridge=<remote address>), and each bridge MBean exposes EnqueueCounter, DequeueCounter, and ReceivedCounter attributes. The Jolokia paths above match that layout.

How to diagnose it

1. Confirm the asymmetry. On broker A: QueueSize > 0 with ConsumerCount of 0 or only bridge proxy consumers. On broker B: real application consumers connected to the same destination, dequeue rate zero. If both brokers look fine alone but the pair shows this split, you are in demand forwarding territory, not a consumer or producer problem.

2. Verify the bridge is actually connected. Enumerate the network bridge MBeans under the network connector. Absent MBeans mean the bridge is down, which is a different problem: partition, remote broker down, DNS, or inter-broker auth failure. Present MBeans with flat EnqueueCounter/DequeueCounter means connected but not forwarding.

3. Check advisory flow. Dynamic demand forwarding depends on advisory messages propagating consumer demand. If advisorySupport="false" is set on either broker, dynamic forwarding cannot work; only statically configured destinations will forward. Also verify the network connector’s credentials are authorized to subscribe to advisory topics on the remote broker. Authorization denials in the remote broker log are a common root cause when security plugins are in use.

4. Account for conduitSubscriptions. With the default conduitSubscriptions="true", remote demand appears as a single condensed subscription. Do not conclude “no demand” just because the local consumer count is 1 while 10 consumers run on the far side. Check whether the bridge’s proxy consumer exists for the destination at all; its absence, not its count, is the problem.

5. Check the timeline. If forwarding stopped after a network interruption or remote broker restart, look at bridge reconnect timestamps in the log. Demand subscriptions must reconverge after reconnection, which normally takes minutes. If the bridge has been connected well past that window with no proxy consumers recreated, the demand subscription was lost and is not recovering. Operators have reported broker versions where a queue goes to 0 consumers after an interruption and the bridge never re-establishes demand; the reported workarounds (delete and recreate the queue, or wrap the bridge URI in a failover transport) are debated and version-specific.

6. Rule out filters and selectors. If some destinations forward and others do not, inspect the network connector’s included/excluded destination configuration and any consumer selectors. A bridge that forwards everything except one queue is a configuration mismatch, not a demand forwarding failure.

7. For multi-hop topologies, check TTL. Default networkTTL is 1, so subscriptions and messages traverse one broker hop. In a mesh deeper than two brokers, demand may not propagate far enough. On 5.9 and later you can tune messageTTL and consumerTTL separately.

Metrics and signals to monitor

The single most valuable derived alert here is cross-broker: “queue has pending messages and zero local consumers on broker A while broker B has consumers for that destination.” Neither broker trips a local alert; the pair does.

Fixes

Re-enable advisory support

If advisorySupport="false" was set (sometimes done to reduce destination count and MBean overhead, since every advisory topic is a real destination), dynamic demand forwarding is broken by design. Re-enable advisory support on brokers that participate in network connectors. If you need to limit advisory overhead, scope it rather than disabling it broker-wide.

Fix inter-broker authorization

If the network connector’s credentials cannot subscribe to advisory topics on the remote broker, demand never propagates. Grant the connector user read access to the advisory topics and the destinations being bridged, per your authorization plugin. Watch the remote broker log after the change to confirm denials stop.

Handle stuck messages with replayWhenNoConsumers

For the case where a queue has messages but no local consumers and the bridge stays stuck, ActiveMQ 5.6+ offers a destination policy using conditionalNetworkBridgeFilterFactory with replayWhenNoConsumers="true", which tells the bridge to replay messages back to the originating broker when a destination has depth but no active consumers. On versions older than 5.9, the official documentation also requires enableAudit="false" so duplicate detection does not suppress the replayed messages. This changes routing semantics and can interact with duplicate delivery expectations; test it on a non-critical destination first.

Recover a dead demand subscription

If the bridge is connected but the proxy consumer for the destination never reappeared after an interruption, the least disruptive recovery is to bounce the demand: stop and restart the affected consumer application on the remote broker so fresh consumer advisories fire, which should trigger the bridge to recreate its proxy subscription. Restarting the broker or deleting and recreating the queue also works but is disruptive; deleting a queue can discard its messages depending on how you do it, so drain or verify the queue first and treat this as a last resort.

Adjust TTL for deeper topologies

If demand must cross more than one broker hop, raise networkTTL (or set messageTTL and consumerTTL on 5.9+) to match your topology depth. TTL also governs how far messages travel; raising it changes loop exposure in mesh topologies, so change it deliberately.

Consider static bridging only as a deliberate design choice

staticBridge="true" forwards unconditionally without demand, which sidesteps this entire failure mode but wastes bandwidth and breaks the store-and-forward economics of a NoB. It is a valid choice for small, fixed topologies, not a patch for broken advisories.

Prevention

• Alert on the cross-broker condition, not per-broker state. “Queue with pending messages and zero local consumers” catches broken demand forwarding. Build it as a joined query across every broker pair.
• Track bridge throughput as a first-class signal. Bridge enqueue/dequeue rates should track cross-broker consumer demand. Flat bridge counters with demand present is a ticket-level alert.
• Baseline demand reconvergence time. Measure how long forwarding takes to resume after a planned bridge restart. If an unplanned event exceeds that baseline, alert.
• Use canary messages across brokers. A synthetic produce-on-A, consume-on-B round trip per bridged destination catches stalls that metric thresholds miss, including selector and filter mismatches.
• Document advisory and authorization dependencies. The network connector user’s advisory subscription rights and the advisorySupport setting belong in your broker configuration review checklist, because both break forwarding silently.
• Watch replay behavior on reconnect. After any bridge recovery, monitor the receiving broker’s memory usage; a store-and-forward replay burst can trigger flow control there, turning a demand problem into a second incident.

How Netdata helps

• Per-bridge enqueue and dequeue counter rates collected from the network bridge MBeans, so “connected but zero throughput” is visible as a chart, not a log-grep exercise.
• Queue depth and consumer count per destination on every broker, making the cross-broker asymmetry (depth on one side, idle consumers on the other) visible on one dashboard instead of two per-broker views.
• Dequeue rate per destination on both brokers, confirming consumer-side starvation rather than assuming it from queue depth.
• Broker log monitoring for bridge connect/disconnect and authorization denial events, correlated on the same timeline as the throughput counters.
• Alerting on composite conditions like “queue depth above zero AND local consumer count zero AND remote consumers present”, the signature of broken demand forwarding and invisible to single-broker alerts.

Related guides

• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ broker down: telling a crashed broker from a hung one
• ActiveMQ InactivityIOException: Channel was inactive for too long
• ActiveMQ enqueue outpacing dequeue: reading the rate imbalance before the backlog
• ActiveMQ consumers connected but not acknowledging: the zombie consumer
• ActiveMQ GC pause death spiral: long pauses, heartbeat timeouts, and reconnect storms