ActiveMQ producer flow control: why send() hangs and producers block silently

Your application is healthy. No exceptions, no error logs, no timeouts. But requests are piling up and every thread that touches the message producer is stuck inside send(). Thread dumps show all your producer threads parked in the JMS client, waiting on a socket write that never completes. Nothing is wrong on the client. Nothing looks wrong on the broker either, until you check one number: MemoryPercentUsage is at 100.

This is ActiveMQ Classic producer flow control, and it is the most common way ActiveMQ takes down upstream services. When a destination memory limit, the broker system memory limit, or the store/temp limit is reached, the broker stops reading from the producer’s socket. TCP backpressure does the rest. The producer’s send() blocks indefinitely. No exception. No log entry on the producer side. No timeout, unless you configured one. The official documentation notes this is frequently misread as a “hung producer” when the producer is actually just waiting for space.

This article covers how the mechanism works, how to confirm it in minutes, and how to make it loud instead of silent.

What this means

Producer flow control is ActiveMQ’s way of protecting itself from running out of memory. Every message the broker accepts is charged against two budgets: the destination’s memory limit and the broker-wide system memory limit (<systemUsage><memoryUsage> in activemq.xml, by default a percentage of JVM heap). For persistent messages the store limit matters too; for non-persistent overflow, the temp store limit.

When any of these limits is hit, the broker does not reject the message. It stops reading bytes off the producer’s TCP connection. The kernel TCP buffers fill, the client’s send buffer fills, and the producer’s send() call blocks in a socket write. From the client’s perspective the broker has gone mute.

Two things make this nasty in production:

• It is silent by default. No exception, no producer-side log, no timeout. Unless you set sendFailIfNoSpaceAfterTimeout or sendFailIfNoSpace, the producer hangs forever.
• It can be broker-wide. If the shared system memory pool hits 100%, every producer on every destination blocks, even producers writing to empty queues. One slow consumer can freeze your entire service mesh.

flowchart TD
  A[Producer calls send] --> B[Transport thread reads socket]
  B --> C[Persist to KahaDB if persistent]
  C --> D[Charge message to destination and broker memory]
  D -->|under limit| E[Cursor and dispatch to consumers]
  D -->|limit reached| F[Broker stops reading producer socket]
  F --> G[TCP receive buffer fills]
  G --> H[send blocks: no exception, no log, no timeout]

One scope note: persistent messages are sent synchronously by default and block in send() as described. Non-persistent messages are sent asynchronously by default and will not block in send() unless you configure a ProducerWindowSize on the connection factory. If your async producers seem unaffected while memory climbs, that is why, and your visibility gap is even bigger.

Common causes

Quick checks

All read-only. Jolokia paths assume the web console on 8161 with default credentials; adjust for your deployment.

# Broker-level memory: 100 means flow control is active right now
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/MemoryPercentUsage'

# Find which destination is full (wildcard across all queues)
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=*/MemoryPercentUsage'

# Store and temp store: persistent and non-persistent ceilings
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/StorePercentUsage'
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/TempPercentUsage'

# For the suspect queue: consumers, backlog, inflight
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/ConsumerCount,QueueSize,InFlightCount,EnqueueCount,DequeueCount'

# Broker log: flow control leaves traces on the broker side
grep -i "memory limit" /opt/activemq/data/activemq.log | tail -20
grep -i "prevent flooding" /opt/activemq/data/activemq.log | tail -20

# Client side: confirm producers are parked in socket write
jstack <producer_pid> | grep -A5 "SocketOutputStream"

Two notes on these checks. First, the log path varies by install ($ACTIVEMQ_HOME/data/activemq.log is typical; service wrappers may redirect elsewhere). Second, the broker-side log line is your fastest confirmation that flow control engaged, since the producer logs nothing at all.

How to diagnose it

1. Confirm flow control is the problem. Check broker MemoryPercentUsage. If it is 100, flow control is active. If it is not, also check StorePercentUsage and TempPercentUsage; a full store blocks persistent producers the same way.
2. Localize to a destination. Wildcard-query per-destination MemoryPercentUsage. One destination at 100 with a per-destination policy limit means only that destination’s producers are blocked. Nothing at 100 individually but the broker at 100 means the shared pool is exhausted and everyone is blocked.
3. Identify the failing consumer. For the full destination, compare ConsumerCount, InFlightCount, and dequeue rate. Inflight equal to consumer_count x prefetch_size (default prefetch is 1000 for queues) with a flat dequeue rate means consumers have the messages but are not acking: stuck, deadlocked, or waiting on a slow downstream dependency.
4. Check the silent-growth suspects. Look at ActiveMQ.DLQ depth and durable subscriber PendingQueueSize. A DLQ that grows forever (no TTL by default) or an orphaned durable subscription accumulates store and memory pressure until the whole broker tips over. This is the classic slow-burn version of the incident.
5. Rule out look-alikes. Long GC pauses also freeze producers, but they show as connection drops and a sawtooth connection count, not a memory gauge pinned at 100. A hung broker process shows a dead JMX endpoint, not readable MBeans reporting 100%.
6. In a Network of Brokers, check downstream. Flow control propagates upstream through network bridges. If your local broker looks fine, check memory on the broker that actually has the consumers.

Metrics and signals to monitor

Fixes

Unblock producers right now

Flow control releases the moment space frees up, so the fastest fix is restoring consumption.

• Restart or disconnect the stuck consumer. If inflight is pinned and the consumer application is wedged, restarting it returns its inflight messages to the queue for redelivery to healthy consumers. This is disruptive to in-flight work on that consumer, so confirm it is actually stuck first.
• Drain the backlog. If consumers are alive but slow (downstream dependency recovering), scaling consumer instances out is the clean fix.
• Purge or move dead weight. Purging a queue or clearing the DLQ frees memory and store immediately. This destroys messages, so export or browse them first if they have business value.

Raising memoryUsage in activemq.xml is a real option but requires a broker restart, which makes it a mitigation for next time rather than an in-incident fix. Keep the broker memory limit around 60-70% of JVM max heap; the JVM needs headroom for non-message objects.

Make blocking visible instead of silent

This is the permanent fix for the “nobody knew” problem. Configure one of the escape hatches:

<systemUsage>
  <systemUsage sendFailIfNoSpaceAfterTimeout="5000">
    <memoryUsage>
      <memoryUsage percentOfJvmHeap="70"/>
    </memoryUsage>
  </systemUsage>
</systemUsage>

• sendFailIfNoSpaceAfterTimeout="5000": send() blocks up to the timeout, then throws javax.jms.ResourceAllocationException. Usually the right choice: transient backpressure is absorbed, genuine exhaustion becomes a loud, catchable exception with a message your logging will actually capture. Set the timeout honestly. Values in the tens of minutes are indistinguishable from the default silent hang.
• sendFailIfNoSpace="true": fail immediately with the same exception. Right for latency-sensitive paths where any blocking is unacceptable, but every transient spike becomes an error your producer must handle.
• Both can also be set per destination via <policyEntry>, so you can fail fast on critical queues while letting batch queues block.

Producers must catch and log ResourceAllocationException, and ideally route to a fallback (retry with backoff, spill to a local buffer, shed load). An exception nobody handles is only marginally better than a hang.

The alternative mode: producerFlowControl=false

Setting producerFlowControl="false" on a destination policy stops the broker from blocking producers. Understand what you are trading:

• Persistent messages keep flowing to the store via cursors until disk fills. You have moved the cliff from “producers block” to “broker disk full”, which is worse to recover from.
• For non-persistent messages, once memory and temp store are exhausted, messages can be silently dropped if no fail-if-no-space option is set. This is the silent message loss pattern: producers see nothing, consumers just see less traffic.

Use it deliberately, with sendFailIfNoSpace configured and tight store/disk monitoring, not as a way to make the alerts stop.

Fix the root cause on the consumer side

Flow control is a symptom. The common roots: consumers blocked on a slow database or downstream API, prefetch too large for the processing rate (1000 messages buffered per queue consumer by default), consumer thread pool exhaustion, and DLQ or durable-subscription accumulation slowly eating the shared pool. Reducing prefetch to match real processing throughput is often the single most effective change for slow-consumer cascades.

Prevention

• Alert below the cliff. Ticket at 80% broker MemoryPercentUsage sustained, page at 100% with active producers. Also alert on the growth rate: memory climbing a few percent per minute means minutes of runway, not hours.
• Set per-destination memory limits via <policyEntry> so one runaway queue cannot drain the shared broker pool.
• Configure sendFailIfNoSpaceAfterTimeout on every production broker, and make producer code handle the exception. Silent blocking should be an opt-in, not the default you inherited.
• Put a TTL on DLQ messages and alert on any non-zero DLQ depth. DLQ growth is the most common slow path to store and memory exhaustion.
• Reap orphaned durable subscriptions. Any offline durable subscriber with a growing pending count is a permanent leak.
• Monitor store and disk independently. The configured store limit and the physical partition are different ceilings; watch both.
• Run a canary. A synthetic produce/consume round trip on a dedicated queue catches “metrics look fine but nothing flows”, which metric-only monitoring misses.
• For async (non-persistent) producers, set ProducerWindowSize on the connection factory so flow control can actually apply to them.

How Netdata helps

• Netdata collects the ActiveMQ broker’s JMX/Jolokia metrics, so MemoryPercentUsage, StorePercentUsage, and TempPercentUsage are graphed continuously, letting you see the climb toward the flow control cliff instead of discovering it at 100%.
• Per-destination queue depth, enqueue/dequeue rates, and inflight counts on one dashboard make the slow-consumer cascade visible as a shape: dequeue flattening while depth and memory rise together.
• Alerting on memory usage thresholds and on the enqueue/dequeue imbalance gives you the leading indicator (consumption falling behind) rather than the lagging one (producers already hung).
• Correlating broker metrics with host-level disk usage on the KahaDB partition closes the gap between ActiveMQ’s configured store limit and real disk capacity.
• Historical retention lets you distinguish the slow-burn version (DLQ and store creeping up over weeks) from a sudden burst, which changes the fix.

Related guides

This is currently the only guide in this section. For the full signal catalog, failure archetypes, and monitoring maturity model for the broker, see the ActiveMQ operations guide hub.