ActiveMQ store usage climbing: the store exhaustion spiral

StorePercentUsage was 62% last month. Last week it was 74%. This morning it crossed 80% and your alert fired. Queue depths look normal, consumers are connected, dequeue rates look healthy. Nothing is obviously on fire, but the persistent store keeps growing and nobody knows why.

This is the store exhaustion spiral, one of the most common slow-burn failure patterns in ActiveMQ Classic. Unlike the memory-pressure cascade, which develops in minutes, the store spiral develops over days or weeks. Messages accumulate in the KahaDB journal faster than they are acknowledged, journal files pile up on disk, and the store limit creeps closer. When StorePercentUsage reaches 100%, the broker stops accepting persistent messages entirely and every persistent producer blocks. The warning signs were visible for weeks.

The most frequent root cause is not your application queues at all. It is the Dead Letter Queue. DLQ messages have no TTL by default, they pin KahaDB journal files, and most teams never look at the DLQ until it has eaten the store.

What this means

KahaDB stores persistent messages in sequential journal files (db-*.log, 32MB each by default) plus a B-tree index (db.data). A journal file is only eligible for deletion when every message in it has been acknowledged. A single unacknowledged message pins the entire 32MB file. KahaDB runs cleanup periodically (every 30 seconds by default, with a checkpoint every 5 seconds), but cleanup can only delete files with no outstanding references.

The spiral works like this:

1. Messages are produced faster than they are acknowledged somewhere in the system. Often that “somewhere” is the DLQ, an offline durable topic subscriber, or a queue with a subtle consumer problem.
2. Each pending message holds a reference into a journal file. Enough scattered pending messages and no journal file becomes fully reclaimable.
3. Journal files accumulate on disk. StorePercentUsage (ActiveMQ’s accounting against the configured storeUsage limit) climbs steadily.
4. At 100%, the broker blocks all persistent producers via flow control. Persistent messaging halts.

flowchart TD
  A[Consumption lags production] --> D[Pending messages accumulate]
  B[DLQ grows with no TTL] --> D
  C[Offline durable subscriber] --> D
  D --> E[Journal files pinned by unacked messages]
  E --> F[StorePercentUsage climbs over days or weeks]
  F --> G[Store hits 100 percent]
  G --> H[All persistent producers block]

Two things make this pattern treacherous. First, your application queues can look completely healthy while the store fills, because the growth is happening in the DLQ or an orphaned subscription. Second, dequeue counters mislead: messages moved to the DLQ count as dequeued from the source queue, so your dequeue rate looks fine while messages are being discarded into the store’s biggest leak.

Common causes

Quick checks

These are all read-only. The Jolokia examples assume the web console on localhost:8161 with default credentials; adjust for your environment.

# Broker store usage percent
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/StorePercentUsage'

# DLQ depth
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ/QueueSize'

# All queue depths at once: look for anything growing that should not be
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=*/QueueSize'

# Journal file count and disk usage of the KahaDB directory
ls /opt/activemq/data/kahadb/db-*.log | wc -l
du -sh /opt/activemq/data/kahadb/
df -h /opt/activemq/data/

# Index size: large index means large pending backlog and slow recovery
ls -lh /opt/activemq/data/kahadb/db.data

# Open file descriptors: large journal backlogs raise the count
BROKER_PID=$(pgrep -f activemq)
echo "Open: $(ls /proc/$BROKER_PID/fd | wc -l)"
grep 'Max open files' /proc/$BROKER_PID/limits

Two readings matter here that teams routinely confuse: StorePercentUsage is measured against the configured storeUsage limit in activemq.xml, not against physical disk. If the limit is set higher than the partition, the disk fills first and the broker fails before StorePercentUsage ever reaches 100%. Monitor both independently.

How to diagnose it

1. Confirm the trend. Look at StorePercentUsage over days, not minutes. A slow monotonic climb is the spiral. A flat line that jumped once is a backlog event. Plot actual disk usage on the KahaDB partition alongside it.

2. Check the DLQ first. It is the most common root cause. If ActiveMQ.DLQ depth is large or growing, browse a sample of messages and look at the JMSDestination property to see which source queues they came from. That tells you which application flow is producing failures. Also check per-destination ExpiredCount, since expired messages route to the DLQ by default and feed it silently.

3. Check for orphaned durable subscribers. Enumerate durable subscription MBeans and look at PendingQueueSize. Any subscription with pending messages and no active consumer is a permanent storage leak. Dev and test subscriptions that were never unsubscribed are the classic offender.

4. Compare enqueue and dequeue rates per destination. A sustained ratio above 1.0 on any queue means accumulation. Remember that DLQ transfers inflate dequeue counts on the source queue, so cross-check against DLQ enqueue growth before declaring a queue healthy.

5. Check journal file growth against queue depths. If the db-*.log count grows while all queue depths look small, you have the pinning problem: a few unacknowledged messages scattered across many files prevent any file from being reclaimed. To see exactly which destinations and transactions hold references into journal files, enable TRACE logging on org.apache.activemq.store.kahadb.MessageDatabase temporarily. This is verbose, so turn it off after diagnosis.

6. Verify the limit against reality. Read the storeUsage limit from activemq.xml and compare it to partition size and current usage. Note that KahaDB preallocates journal files, so raw disk consumption of the directory can run higher than the logical message data size.

Metrics and signals to monitor

Alert on StorePercentUsage above 80% as a ticket, and treat anything above 70% with a positive trend as worth investigating the same day. The page-level condition is 100% on the active broker role, but if your monitoring only fires at 100% you have no runway at all: producers are already blocked.

Fixes

Drain or purge the DLQ

This is usually the fastest way to reclaim store. First investigate: browse messages, group by JMSDestination, and understand what is failing. Then export anything you need for replay and purge the rest. Purging the DLQ frees the message references that pin journal files; cleanup reclaims the files on a later pass, so disk reclamation lags the purge by roughly the cleanup interval (30 seconds by default).

Tradeoff: purged DLQ messages are gone. If they represent business transactions you must replay, export them first. Purging without investigating guarantees the spiral resumes.

Remove orphaned durable subscriptions

Unsubscribe durable subscriptions whose owning application no longer exists. This stops the accumulation and frees the pinned journal references. Verify the subscription is genuinely abandoned before removing it; an offline-but-legitimate subscriber will lose its backlog.

Tradeoff: deletion is permanent. If there is any doubt, note the client ID and subscription name and confirm with the owning team.

Fix the consumption lag

If a real application queue is growing, the fix is on the consumer side: scale consumers, fix the downstream dependency slowing them, or resolve whatever is causing redelivery loops. See ActiveMQ producer flow control: why send() hangs and producers block silently for the producer-side symptoms when this bites.

Do not just raise the store limit

Increasing storeUsage buys time but does nothing about the underlying accumulation, and it makes things worse if the new limit exceeds physical disk. If you raise it as a stopgap, keep it below partition capacity with at least 20-30% headroom for journal rotation and filesystem reserve, and treat it strictly as runway while you fix the root cause.

Reclaim space after a backlog drain

After consumers catch up, StorePercentUsage drops but files on disk are removed only as cleanup finds them fully unreferenced. If usage stays high after queues drain, suspect the pinning problem and use the TRACE logging from step 5 of diagnosis to find the holdout references. A broker restart forces a clean checkpoint but is disruptive and triggers journal recovery; it is not a first-line fix.

Prevention

• Alert on DLQ depth. Any non-zero depth is a processing failure. Sustained growth is an active incident. This single alert prevents most store spirals.
• Set a TTL on DLQ messages so dead letters expire instead of accumulating forever. Pair this with a DLQ consumer or export process if you need them retained for a window.
• Use per-destination DLQs via IndividualDeadLetterStrategy so failures are attributable to their source flow and one noisy queue does not hide others.
• Audit durable subscriptions regularly. Alert on any subscription with zero active consumers and growing pending count.
• Trend StorePercentUsage and journal file count on a weekly dashboard. The spiral is always visible weeks in advance if anyone is looking.
• Size the store limit against the partition, not the other way around, and monitor disk free independently of StorePercentUsage.
• Set sendFailIfNoSpaceAfterTimeout on the destination policy in activemq.xml so that if the store ever fills, producers get exceptions instead of silently blocking forever, and your upstream services fail loudly rather than hanging.

How Netdata helps

• StorePercentUsage trended per-second over weeks. The spiral is a trend problem, and long-retention per-second metrics make a 1% per day climb visible long before the 80% threshold.
• DLQ depth correlated with journal file count. Seeing DLQ growth and db-*.log accumulation on the same timeline is the fastest confirmation of the classic root cause.
• Store usage vs actual disk usage side by side. Netdata collects both the JMX store accounting and host filesystem metrics from the same node, so the dangerous case where the limit exceeds the partition is visible in one view.
• Enqueue vs dequeue rates per destination. Sustained ratios above 1.0 show which queue is accumulating, and correlating dequeue with DLQ growth exposes the “healthy dequeue that is actually DLQ transfers” trap.
• Anomaly detection on slow drifts. Gradual monotonic growth is exactly what static thresholds catch late and ML-based anomaly scoring catches early, which is where the runway to fix this lives.

For the full signal set and maturity progression, see the ActiveMQ monitoring checklist and the monitoring maturity model.

Related guides

• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ memory limit reached: MemoryPercentUsage at 100% and the flow-control cliff
• ActiveMQ MemoryPercentUsage climbing: reading the flow-control leading indicator
• ActiveMQ monitoring checklist: the signals every production broker needs
• ActiveMQ monitoring maturity model: from survival to expert
• ActiveMQ producer flow control: why send() hangs and producers block silently
• ActiveMQ memoryUsage vs JVM heap: the two memory budgets teams confuse