ActiveMQ store is full: StorePercentUsage at 100% and persistent messaging halted

StorePercentUsage has hit 100% on the broker MBean and persistent producers have stopped. Their send() calls are either blocked in flow control or being rejected, depending on your destination policy. Non-persistent traffic may still flow, which makes the outage look partial from the outside.

This is the disk-side equivalent of the memory wall. Where MemoryPercentUsage at 100% blocks producers against the configured memory limit, StorePercentUsage at 100% blocks persistent producers against the configured storeUsage limit in activemq.xml. The broker log shows a “Persistent store is Full” message, and producers stall silently unless you configured sendFailIfNoSpaceAfterTimeout.

This failure is almost always a slow burn that built up over days or weeks, and the fix is usually finding what is pinning journal files rather than adding disk. If you only add disk without fixing the root cause, you will be back here with a bigger store.

What this means

ActiveMQ Classic persists messages to KahaDB, a write-ahead journal plus a B-tree index. Messages are appended to sequential journal files (db-*.log, 32MB each by default). A journal file can only be deleted when every message in it has been acknowledged. A single unacknowledged message anywhere in the file pins the entire 32MB.

StorePercentUsage is not disk usage. It is KahaDB’s accounted size measured against the <storeUsage> limit in <systemUsage> in activemq.xml. Two consequences:

• If the limit is set lower than the physical disk, the broker halts persistent messaging with free disk still available.
• If the limit is set higher than the physical disk, the OS partition fills first and you get disk-full failures before the broker’s own limit trips. Monitor both independently.

At 100%, persistent messaging is halted. Producers block in flow control by default, with no exception and no timeout, so upstream services hang rather than fail. The cascade typically looks like this:

flowchart TD
  A[Consumers lag or DLQ accumulates] --> B[Unacked messages pin journal files]
  B --> C[KahaDB cleanup cannot reclaim files]
  C --> D[StorePercentUsage climbs to 100%]
  D --> E[Persistent producers blocked or rejected]
  E --> F[Upstream services hang on send]
  G[Store limit misconfigured vs workload] --> D
  H[Orphaned durable subscriptions] --> B

Common causes

DLQ accumulation deserves emphasis. DLQ messages have no TTL by default, the default SharedDeadLetterStrategy funnels every destination’s failures into one queue, and nothing consumes it. Each DLQ message pins whatever journal file it lives in. This is the most common cause of the store exhaustion spiral.

Quick checks

All read-only. Substitute your broker name, host, and credentials.

# Store usage percent (the confirming signal)
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/StorePercentUsage'

# Memory usage, to see if you are hitting both walls
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/MemoryPercentUsage'

# DLQ depth
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ/QueueSize'

# All queue depths at once, for the biggest backlogs
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=*/QueueSize'

# Journal file count and total store size on disk
ls /opt/activemq/data/kahadb/db-*.log | wc -l
du -sh /opt/activemq/data/kahadb/

# Physical disk headroom on the KahaDB partition
df -h /opt/activemq/data/

# Index size (large index means large pending backlog and slow recovery)
ls -lh /opt/activemq/data/kahadb/db.data

# Recent store-full events in the broker log
grep -i "store is full\|store usage" /opt/activemq/data/activemq.log | tail -20

How to read the results:

• Journal file count high but all queue depths low: suspect DLQ, orphaned durable subscriptions, or pending transactions pinning files.
• StorePercentUsage at 100 but df shows plenty free: your storeUsage limit is the constraint, not the disk.
• StorePercentUsage above 100: the metric can overshoot because usage is accounted against the limit while journal files are allocated in whole units. The broker still blocks at 100.

How to diagnose it

1. Confirm the wall. StorePercentUsage at 100 on the broker MBean. Also check MemoryPercentUsage and TempPercentUsage; hitting multiple limits at once changes the recovery order.
2. Check physical disk. df -h on the KahaDB partition. If the partition itself is above 90%, you have a disk-full problem layered on top, and freeing space is the first move regardless of the configured limit.
3. Find the space consumer. Compare per-queue QueueSize values against du -sh on the store. The DLQ is the usual suspect. Also enumerate durable topic subscribers and look for PendingQueueSize growing with zero connected consumers.
4. Identify what pins journal files. If journal file count is high relative to visible backlog, enable TRACE logging on org.apache.activemq.store.kahadb.MessageDatabase to see which destination holds references into old journal files. This is the authoritative way to find the pinning destination. It is verbose; turn it off after diagnosis.
5. Trace DLQ messages to their source. Browse DLQ messages and inspect the JMSDestination property and exception headers. That tells you which queue and which failure is feeding the DLQ.
6. Check consumption health. For the queues feeding the backlog: ConsumerCount, DequeueCount rate, and InFlightCount. Consumers connected but not acking (inflight pinned at prefetch) means the consumer application is stuck, not the broker.

Metrics and signals to monitor

Fixes

DLQ accumulation

Export or browse the DLQ messages first if you need them for root-cause analysis, then purge the DLQ. Purging is destructive: those messages are gone. After purging, journal file reclamation is not instant; KahaDB cleanup runs periodically (every 30 seconds by default) and files disappear as their last references clear.

Then fix the feeder. Inspect the JMSDestination property on DLQ messages to find the source queue, and fix the consumer bug or message format mismatch that is dead-lettering traffic. If you do not, the DLQ refills.

Consumer lag

If a real backlog on application queues is the cause, get consumers draining: restart stuck consumer instances, scale consumers out, or fix the downstream dependency slowing them. Once acks flow, cleanup reclaims journal files and StorePercentUsage drops. Watch the dequeue rate and the journal file count to confirm reclamation is actually happening, not just consumption.

Orphaned durable subscriptions

Unsubscribe durable subscriptions whose applications no longer exist. This requires JMX or the web console; there is no automatic expiration. Each orphaned subscription accumulates every message published to its topic, permanently.

Store limit misconfiguration

If the limit itself is the problem (disk has headroom, backlog is legitimate), raise <storeUsage> in activemq.xml. The limit is read at broker startup, so applying a new value requires a restart or a runtime update of the SystemUsage MBean via JMX. A restart with a large store means KahaDB recovery time; plan for it.

Sizing guidance: keep journalMaxFileLength (default 32MB) well below the store limit, since usage is checked at whole-file granularity and a single file can overshoot the limit. Leaving roughly 30% headroom between the store limit and the physical partition gives room for overshoot, the temp store, and broker logs on the same partition. Both storeUsage and tempUsage support a percentLimit attribute expressed as a percentage of total available space; since 5.15.x a related total attribute can set the absolute available size explicitly (useful when only part of a partition is available to the broker, or the filesystem reports more than Long.MAX_VALUE capacity), which keeps the limit honest relative to the actual filesystem.

Unreclaimable journal files after everything is consumed

If messages are drained but files remain, check for ack-only journal files. Since 5.14.0, KahaDB ack compaction (enableAckCompaction, default true) rewrites files containing only acknowledgements so the originals can be deleted. On older versions, or after unclean shutdowns, files can linger; the TRACE logging on MessageDatabase shows exactly which destination is holding each file.

Buying time during the incident

If producers are hung and you need traffic flowing while you clean up, the options are unpalatable but real: purge the DLQ (fastest store relief in the common case), or temporarily raise the store limit via JMX. Do not disable producer flow control to “fix” this. With producerFlowControl="false" you remove the backpressure signal and the broker still cannot persist to a full store, so you trade a visible block for unpredictable send behaviour.

Prevention

• Alert on the leading indicators, not the cliff. Ticket at StorePercentUsage above 70-80% with a growing trend, page at 100% on the active broker. The spiral takes days; being surprised at 100% means you had no trend visibility.
• Alert on any non-zero DLQ growth and on the DLQ-to-enqueue rate ratio. Put a TTL on DLQ messages via the dead letter strategy so the DLQ cannot grow forever.
• Audit durable subscriptions regularly. Alert on subscriptions with pending messages and no connected consumer.
• Track journal file count and db.data size alongside StorePercentUsage. They expose pinning before the limit trips.
• Monitor disk free on the KahaDB partition independently of StorePercentUsage, and size the store limit below the partition with headroom.
• Configure sendFailIfNoSpaceAfterTimeout so producers get an exception instead of an infinite silent block. On 5.16.0 and later it can be set per destination via destination policy (sendFailIfNoSpace too); before that both were broker-wide <systemUsage> options only.

How Netdata helps

• StorePercentUsage, MemoryPercentUsage, and TempPercentUsage collected together from the broker MBean, so you can see which limit wall you actually hit and whether others are approaching.
• Per-destination queue depth, enqueue/dequeue rates, and consumer counts let you separate “DLQ leak” from “consumer lag” in one view instead of correlating Jolokia calls by hand.
• DLQ depth as a first-class metric with alerting on any sustained growth, catching the storage leak weeks before the store fills.
• Host-level disk usage and I/O latency on the KahaDB partition collected alongside broker metrics, so the “store limit vs physical disk” confusion resolves visually.
• Durable subscriber pending counts and expired message counts surface the silent store consumers that queue depth alone misses.
• Historical retention shows the slow-burn trend: a store climbing 2% a day is obvious on a week-long graph and invisible in a point-in-time check.

Related guides

• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ memory limit reached: MemoryPercentUsage at 100% and the flow-control cliff
• ActiveMQ MemoryPercentUsage climbing: reading the flow-control leading indicator
• ActiveMQ monitoring checklist: the signals every production broker needs
• ActiveMQ monitoring maturity model: from survival to expert
• ActiveMQ producer flow control: why send() hangs and producers block silently