$ guides / activemq / activemq-inflight-count-high ▌

Operations Guides

ActiveMQ InFlightCount high: prefetch full, acks stalled, and zombie consumers

A queue shows QueueSize near zero, consumers are connected, and nothing is being processed. Dequeue rate is flat, message age is rising upstream, and the only number that looks wrong is InFlightCount, sitting at a suspiciously round value. This is the zombie consumer pattern: the broker dispatched messages into consumer prefetch buffers, and those messages are never coming back as acknowledgments.

Operators misread this state constantly. They watch QueueSize, see an “empty” queue, and conclude the broker is fine while hundreds or thousands of messages sit in consumer limbo: charged against memory, invisible to other consumers, and pinning journal files. This article covers how to read InFlightCount correctly, how to tell a genuinely stuck consumer from one that is high by design, and how to fix it without guessing.

For the broader broker mental model, see how ActiveMQ Classic actually works in production.

What this means

InFlightCount is the number of messages dispatched to consumers but not yet acknowledged. Dispatch and acknowledgment are separate events. The broker pushes messages to each consumer up to that consumer’s prefetch window (default 1000 for queues, 32766 for topics). DequeueCount for a destination only increments when an acknowledgment comes back, not when the message is dispatched.

That gap between dispatch and ack is where this symptom lives:

QueueSize includes inflight messages. QueueSize=1000 with InFlightCount=1000 means every message is sitting in consumer prefetch. The queue is drained from the broker’s perspective, and the web console can show a queue that looks empty while InFlightCount is high.
High inflight with low dequeue means consumers received messages but are not acking. They are stuck, overloaded, deadlocked, or waiting on a slow downstream dependency.
InFlightCount should not sustainably equal consumer_count x prefetch_size. If it does, every prefetch buffer is full and no acks are flowing. That is the smoking gun for the zombie consumer pattern.

One exception before you page anyone: CLIENT_ACKNOWLEDGE and transacted (SESSION_TRANSACTED) sessions run high inflight by design. Messages stay inflight until the application explicitly acks or commits. With AUTO_ACKNOWLEDGE, inflight is typically very low. Know your consumers’ ack mode before declaring an incident.

flowchart TD
  A[InFlightCount high] --> B{Dequeue rate normal?}
  B -- Yes --> C[Normal: ack latency or CLIENT_ACK / transacted design]
  B -- No --> D{Consumer count as expected?}
  D -- No --> E[Consumers down or partitioned - different incident]
  D -- Yes --> F{Inflight equals consumer_count x prefetch?}
  F -- Yes --> G[Zombie consumers: prefetch full, acks stalled]
  F -- Partial --> H[Slow or degraded consumer - find per-consumer saturation]
  G --> I[Identify client, thread dump, evict]
  H --> I

Common causes

Cause	What it looks like	First thing to check
Stuck or deadlocked consumer	Inflight pinned at prefetch per consumer, dequeue zero, consumer count unchanged	Thread dump the consumer JVM, look for blocked processing threads
Slow downstream dependency	Inflight high, dequeue trickling, consumer count normal	Consumer app logs and dependency latency (database, HTTP calls)
Zombie consumer (connected but dead inside)	Inflight == consumer_count x prefetch, sustained	Per-consumer DispatchedQueueSize and client IDs/IPs via JMX
CLIENT_ACKNOWLEDGE or transacted by design	Inflight high but dequeue keeps pace, age acceptable	Confirm ack mode; this is expected, not a fault
Prefetch too large for processing time	Each consumer holds far more than it can process in a reasonable window	Compare prefetch size to per-message processing time
Pooled consumers holding prefetched messages	Messages stuck with consumers that are “closed” but pooled	Check connection/session pooling config on the consumer side

Quick checks

All read-only. The Jolokia examples assume the default web console on 8161 with default credentials; adjust for your deployment.

# Per-destination inflight, queue size, dequeue, and consumer count
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/InFlightCount,QueueSize,DequeueCount,ConsumerCount'

Compute the saturation ratio: if InFlightCount equals ConsumerCount multiplied by your prefetch size (default 1000 for queues), every buffer is full.

# Take two dequeue readings 30s apart to get a real dequeue rate
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/DequeueCount'
sleep 30
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/DequeueCount'

# Memory pressure: inflight messages still charge against memory accounting
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/MemoryPercentUsage'

# Who is connected: established OpenWire connections by client IP
ss -tn state established '( sport = :61616 )'

# Journal impact: unacked inflight messages can pin journal files
# (path varies by installation)
ls /opt/activemq/data/kahadb/db-*.log | wc -l

How to diagnose it

Confirm the bundle, not the single number. InFlightCount alone is not pageable. The actionable pattern is: inflight pinned near total prefetch AND dequeue collapsed AND queue age rising AND sustained for minutes. Transacted and delayed-ack consumers will false-page on inflight alone.
Check the ack mode. Ask the consumer team (or read the connection factory config): AUTO_ACKNOWLEDGE, CLIENT_ACKNOWLEDGE, or SESSION_TRANSACTED? With AUTO_ACK, sustained high inflight is almost always a fault. With the other two, high inflight can be the design, and you judge by dequeue rate and message age instead.
Drill from destination to consumer. InFlightCount lives on the destination MBean. Per-consumer state is under the subscription MBeans (endpoint=Consumer), where DispatchedQueueSize shows which specific consumer is holding messages. A single consumer with dispatched == prefetch for more than a couple of minutes is your suspect.
Identify the client. Subscription MBeans carry clientId. Map that to the consumer host, or correlate with ss output on the broker for the transport port. Consumer client IPs also appear in transport thread names in a broker thread dump.
Inspect the consumer, not the broker. The broker did its job; it dispatched. Take a thread dump of the consumer JVM (jstack <pid>) and look for processing threads blocked on a database call, an HTTP client without a timeout, a lock, or a deadlock. If the consumer process is alive but all processing threads are parked, you have a zombie consumer.
Check for pooled-consumer leakage. If the consumer framework pools connections or sessions (for example, a caching connection factory), “closed” consumers can retain prefetched messages that never get processed. The symptoms match a stuck consumer but the fix is in the pooling configuration, not the application logic.
Rule out redelivery churn. If consumers are rolling back or failing mid-batch, inflight can stay high while dequeue flickers. Check the redelivery rate and DLQ depth; a poison-message loop is a different incident from a stuck consumer.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
InFlightCount (per destination)	Messages dispatched but unacked; the core signal here	Sustainably equals consumer_count x prefetch
Inflight/prefetch ratio	Saturation of the dispatch window	Ratio near 1.0 sustained >2 minutes
Dequeue rate	Acks arriving; the proof consumers are finishing work	Collapses while inflight is high and consumers connected
ConsumerCount	Distinguishes “no consumers” from “dead consumers”	Normal count with zero dequeue
Per-consumer DispatchedQueueSize	Identifies the specific stuck consumer	One consumer pinned at prefetch
MemoryPercentUsage	Inflight messages still charge against memory accounting	Climbing alongside inflight; flow control cliff at 100%
QueueSize + InFlightCount together	QueueSize zero with high inflight = messages in consumer limbo	The misleading “empty queue” reading
Redelivery rate	Distinguishes stuck consumers from rollback loops	Rising redeliveries with flickering dequeue

Fixes

Evict the stuck consumer

Disconnecting a stuck consumer forces the broker to redeliver its inflight messages to the remaining healthy consumers. This is the fastest recovery when you have multiple consumers and one is wedged. Drop the connection via JMX or restart the consumer process. Warning: any work the consumer did before acking is lost and will be reprocessed, so your consumers must tolerate redelivery (they already must, for the general case). Do not restart the broker for this; it is a consumer-side fault.

Fix the downstream stall

If the consumer is alive but waiting on a slow database or API, evicting it only moves the problem. Add timeouts to the consumer’s external calls, fix the dependency, and let the prefetch drain naturally. Watch dequeue rate recover.

Tune prefetch to match processing reality

A prefetch of 1000 against a consumer that processes 10 messages per second means up to 100 seconds of work held hostage per consumer, invisible to the rest of the pool. Reducing prefetch lowers the blast radius of a stuck consumer, improves load balancing across the pool, and shortens detection time. Tradeoff: very small prefetch adds dispatch round trips and can hurt throughput for fast consumers. Tune per destination with the policy entry’s prefetch settings or per connection.

Address pooled consumers

If consumer pooling is retaining prefetched messages, disable consumer caching in the pooling configuration or set prefetch to 0 for pooled consumers so messages are pulled on demand rather than pushed into a buffer nobody drains.

Automate eviction of dead consumers

ActiveMQ’s slow consumer strategies can abort consumers that stop acknowledging. The acknowledgment-based strategy (AbortSlowAckConsumerStrategy, available since 5.9) checks time since last ack and works even for small prefetch values, which the older buffer-fullness strategy does not. Tradeoff: aggressive aborting can kick consumers during legitimate long processing or GC pauses, so set the ack timeout well above your p99 processing time and test against transacted consumers.

Watch for producer flow control as a second incident

If inflight buildup has pushed memory to the flow-control cliff and producers are now blocking silently, you have a second incident layered on the first. See ActiveMQ producer flow control and ActiveMQ memory limit reached.

Prevention

Alert on the bundle, not the number. Inflight pinned near total prefetch AND dequeue collapsed AND sustained >5 minutes on critical queues. Raw InFlightCount alerts will false-page every transacted consumer.
Monitor InFlightCount alongside QueueSize on every critical queue. Watching QueueSize alone is exactly how this pattern hides.
Track the inflight/prefetch ratio per consumer as a saturation signal.
Right-size prefetch per destination based on measured processing time, not the default.
Cap consumer-side wait times. Every external call from a message listener needs a timeout; a consumer that can block forever will eventually block forever.
Audit connection pooling config on consumers so pooled sessions cannot retain prefetched messages.
Keep redelivery and DLQ signals separate so rollback loops are not misread as stuck consumers.

How Netdata helps

Correlates InFlightCount, DequeueCount, ConsumerCount, and QueueSize per destination on one timeline, so the “empty queue, full prefetch” contradiction is visible at a glance instead of across two console tabs.
Tracks dequeue rate as a derived rate from cumulative counters, making the collapse-in-acks pattern obvious against the inflight plateau.
Surfaces MemoryPercentUsage next to inflight so you can see a stuck consumer walking the broker toward the flow-control cliff before producers block.
Retains per-second history, which lets you distinguish a transient dispatch burst from a sustained zombie-consumer pin after the fact.
Lets you alert on composite conditions (inflight near prefetch plus low dequeue plus consumer count unchanged) rather than a single noisy threshold.