ActiveMQ queue with zero consumers: ConsumerCount at zero and a growing backlog

A production queue shows ConsumerCount=0, QueueSize is climbing, and EnqueueCount keeps ticking up while DequeueCount is flat. Nobody is draining the queue. Every message that arrives stays in the broker, charged against destination memory and, for persistent messages, written into the KahaDB journal. Left alone, this ends in one of two places: broker memory hits 100% and producer flow control silently blocks every producer, or the store fills and persistent messaging halts entirely.

Broker-level consumer counts aggregate across all destinations, so a single dead queue is invisible in global dashboards while every other queue looks healthy. The detection has to be per-destination.

This article covers ActiveMQ Classic 5.x. Artemis has a different address/queue model and different MBean names; the failure logic is similar but the commands below are Classic-specific.

What this means

ConsumerCount is a per-destination JMX attribute reporting the number of active consumers subscribed to that queue. When it reads zero on a queue that is supposed to have consumers, messages have nowhere to go. The broker keeps accepting them (producers are fine), they accumulate in the pending cursor and the journal, and the queue’s memory footprint grows with every enqueue.

The escalation path is mechanical:

flowchart TD
  A[ConsumerCount drops to 0] --> B[QueueSize grows, dequeue flat]
  B --> C[Destination MemoryPercentUsage climbs]
  C --> D{Which limit first?}
  D --> E[Broker memory 100%: producer flow control, send blocks silently]
  D --> F[StorePercentUsage or disk fills: persistent messaging halts]
  E --> G[Upstream services hang on send]
  F --> H[Store corruption risk on disk full]

Two nuances matter before you treat this as a simple “restart the consumer” incident:

• Connected is not the same as processing. This article is about ConsumerCount=0. If the count is non-zero but dequeue is flat, you have a different problem: stuck or saturated consumers. Check InFlightCount against prefetch before concluding consumers are absent. See ActiveMQ producer flow control: why send() hangs and producers block silently for the stuck-consumer variant.
• In a Network of Brokers, demand-forwarding creates virtual consumers. A bridge subscription can keep ConsumerCount above zero on the origin broker while no real application consumer exists anywhere. If the queue has messages and remote consumers are idle, suspect a demand-forwarding break, not a local consumer outage.

Common causes

Quick checks

All read-only. The Jolokia examples assume the web console on 8161 with default credentials; adjust for your deployment.

# Confirm ConsumerCount is really zero on the affected queue
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/ConsumerCount'

# Full picture in one shot: depth, enqueues, dequeues, inflight, expired
for ATTR in QueueSize EnqueueCount DequeueCount InFlightCount ExpiredCount MemoryPercentUsage; do
  echo -n "$ATTR: "
  curl -s -u admin:admin \
    "http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=MY.QUEUE/$ATTR"
  echo
done

# Check whether broker-level or store limits are approaching
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/MemoryPercentUsage'
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/StorePercentUsage'

# Check disk on the KahaDB partition (independent of StorePercentUsage)
df -h /opt/activemq/data/kahadb/

# Look for auth failures from consumer IPs after a credential change
grep -i "authentication failed\|invalid credentials" /opt/activemq/data/activemq.log | tail -20

# Count current connections (sudden drop from a segment = partition)
ss -tn state established '( sport = :61616 )' | wc -l

What you are looking for:

• QueueSize growing, DequeueCount flat, InFlightCount at or near zero: consistent with genuinely absent consumers.
• QueueSize stable but ExpiredCount rising: messages are arriving and expiring unconsumed. The backlog is being hidden by TTL. This is silent data loss, not a healthy queue.
• MemoryPercentUsage or StorePercentUsage climbing: you are on the escalation clock. The rate of climb tells you your runway.

How to diagnose it

1. Confirm the outage is real and scoped. Verify ConsumerCount=0 on the specific queue, not just a gap in your metrics pipeline. Check whether sibling queues on the same consumer application also lost consumers (whole app down) or just this one (subscription-level problem, wrong destination name after a config change, selector mismatch).

2. Establish when it happened. The ConsumerCount drop time is your anchor. Line it up against deploy events, credential or certificate rotations, network changes, and broker GC events. Most zero-consumer incidents trace to a change within the preceding hour.

3. Classify the cause. Work the table above: consumer process state on its hosts, broker auth logs, connection count by client segment, bridge status if NoB. If many destinations lost consumers simultaneously and partially recovered, check broker GC pauses against the 30s default inactivity timeout.

4. Assess the backlog damage. Read QueueSize, EnqueueCount delta over the outage window, and per-destination MemoryPercentUsage. Estimate time to the nearest limit: remaining memory divided by (enqueue rate times average message size). If the queue has a TTL, check ExpiredCount: part of your “backlog” may already be silently expiring into the DLQ.

5. Check downstream limits. Broker-level MemoryPercentUsage, StorePercentUsage, and actual disk free on the KahaDB partition. Store percent and disk percent are different things; whichever fills first wins. One undrained queue can pin journal files and fill the partition even while other queues stay healthy. See ActiveMQ KahaDB journal files not deleted: one unacked message pinning a 32MB log.

6. Only then touch anything. Do not restart the broker as a first move. The broker is doing its job; the consumers are gone. Restarting the broker does not bring consumers back and adds KahaDB recovery time to your outage.

Metrics and signals to monitor

The page condition worth encoding: PAGE when a critical queue drops to zero consumers AND (enqueue rate > 0 OR QueueSize > 0 OR oldest message age rising) AND broker uptime > 600s. The uptime guard suppresses restart flaps; the traffic guard suppresses legitimately idle queues and temp/admin destinations. Raw ConsumerCount alone is not page-safe.

Message age has no direct JMX attribute. Deriving it requires browse() on the destination MBean and reading JMSTimestamp of the first message, which is expensive on a deep backlog. Do it once during diagnosis, not on a tight polling loop.

Fixes

Consumer application crash

Restart the consumer application after you know why it died. Check for OOM kills, unhandled exceptions in message processing, and downstream dependency failures that made it crash-loop. If the consumer was killed by its own heap exhaustion while processing large messages, expect it to die again the moment it reconnects and pulls the backlog. Consider throttling its prefetch before reconnecting so it drains in smaller bites.

Tradeoff: reconnecting consumers against a large backlog creates a drain burst. The dequeue spike is fine for the broker, but if consumers write to a database, the burst lands there. Rate-limit on the consumer side if the downstream is fragile.

Deployment failure

Roll back or roll forward to a consumer build that actually connects. Verify the new version’s destination names, selectors, and credentials match the broker. A consumer that connects but subscribes to MY.QUEUE.V2 while producers send to MY.QUEUE shows up in global consumer counts and drains nothing; per-destination counting is what catches this.

Authentication failure after credential rotation

Fix the credentials on the consumer side (or roll the rotation back) and confirm reconnection in the broker’s accept path. Broker log auth failures from consumer IPs are the confirmation. This cause is increasingly common in environments with automated secret rotation; the rotation pipeline and the consumer’s credential refresh have to be atomic, or every rotation is a consumer outage.

Network partition

Restore connectivity between the consumer segment and the broker. While partitioned, decide whether producers should keep sending: every message enqueued during the partition is backlog you must drain later. If the queue feeds a time-sensitive workflow and messages have TTLs, some of the backlog will expire rather than drain, which may be the correct business outcome but should be a decision, not a surprise.

Broker-side mass disconnection (GC pause)

If broker GC pauses exceeded the client inactivity timeout and knocked consumers off, the fix is on the broker: heap sizing and GC tuning. Consumers with failover transport usually reconnect on their own; those without it may need a restart. See ActiveMQ memory limit reached: MemoryPercentUsage at 100% and the flow-control cliff and ActiveMQ MemoryPercentUsage climbing: reading the flow-control leading indicator for the memory side.

Containing the damage while consumers are down

If restoration will take a while and limits are approaching:

• Watch broker memory, not just the queue. A zero-consumer queue with store-based cursors pages messages to disk, so memory may climb slower than QueueSize suggests. Non-persistent messages on VM cursors go straight to memory and are the fast path to flow control.
• If broker memory is the binding constraint, the options are raising the memory limit (requires broker restart) or slowing/stopping producers to the dead queue. Producer flow control will eventually do this for you, silently, by blocking send(). Configuring sendFailIfNoSpaceAfterTimeout converts the silent block into an exception producers can handle.
• If disk is the binding constraint, you are racing the KahaDB partition. Every unacked message pins journal files. If the partition fills, persistent messaging stops and store corruption becomes possible. Free space or expand the filesystem before that happens. See ActiveMQ disk full on the KahaDB partition: write failures and store corruption risk.

Prevention

• Per-destination consumer alerting. Alert on ConsumerCount per critical queue, never on broker-global consumer count. The global number hides exactly this incident.
• Expected consumer counts as configuration. Define the minimum expected consumer count per queue and alert on any deviation below it, not just zero. A queue that should have four consumers running on one is a degraded state that becomes zero at the next deploy.
• Deploy coordination. Consumer deploys should verify reconnection (ConsumerCount returns to expected within a window) as a rollout gate.
• Credential rotation testing. Rotate credentials in staging against real consumer builds before production. Auth failure after rotation is one of the most common causes and one of the most preventable.
• TTL and DLQ hygiene. Know which queues have TTLs. During a consumer outage, TTL turns backlog growth into silent expiry into the DLQ. Monitor ExpiredCount so this is visible.
• Canary flow. A synthetic produce/consume round-trip on a canary queue catches broken consumption paths that metric gaps miss.
• Headroom planning. Size memory and KahaDB disk for the worst-case consumer outage you are willing to ride out: enqueue rate times that duration times average message size. See ActiveMQ monitoring checklist: the signals every production broker needs.

How Netdata helps

• Per-destination ConsumerCount, QueueSize, and enqueue/dequeue rates collected together make the zero-consumer pattern visible as a single correlated view instead of three separate JMX scrapes you have to line up by hand.
• The composite page condition (ConsumerCount zero plus backlog growing plus uptime guard) maps directly onto alert logic over per-queue metrics, avoiding both the global-count blind spot and flap pages during broker restarts.
• Broker MemoryPercentUsage and StorePercentUsage trending alongside queue depth shows which limit the backlog will hit first and how fast, so you know whether you have minutes or hours.
• ExpiredCount and DLQ depth tracking surfaces the silent version of this incident: backlog that is expiring rather than accumulating.
• Connection count and JVM GC metrics on the broker help distinguish “consumer app died” from “broker GC knocked every consumer off at once” without log archaeology.

Related guides

• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ QueueSize growing: reading the broker’s backlog gauge correctly
• ActiveMQ producer flow control: why send() hangs and producers block silently
• ActiveMQ memory limit reached: MemoryPercentUsage at 100% and the flow-control cliff
• ActiveMQ MemoryPercentUsage climbing: reading the flow-control leading indicator
• ActiveMQ per-destination memory usage: one noisy queue blocking every producer
• ActiveMQ KahaDB journal files not deleted: one unacked message pinning a 32MB log
• ActiveMQ disk full on the KahaDB partition: write failures and store corruption risk
• ActiveMQ KahaDB corruption: the broker won’t start after an unclean shutdown
• ActiveMQ KahaDB db.data index bloat: slow lookups and slow startup recovery
• ActiveMQ monitoring checklist: the signals every production broker needs
• ActiveMQ monitoring maturity model: from survival to expert