ActiveMQ StorePercentUsage vs actual disk space: the limit that fires after the disk is already full

Your broker just stopped accepting persistent messages. Producers are blocked or erroring, the KahaDB partition is at 100% disk usage, and yet your dashboard shows StorePercentUsage at 62%. The alert you set on store usage never fired. This is one of the most common ActiveMQ monitoring traps: StorePercentUsage and actual disk usage are two different counters measured against two different denominators, and if you have not reconciled them, one of them is lying to you.

The inverse failure is quieter. StorePercentUsage hits 100%, the broker logs “Persistent store is Full” and blocks every persistent producer, and the disk still has 40% free. Teams expand the volume, restart the broker, and nothing changes, because the disk was never the binding constraint. The configured storeUsage limit was.

This article covers how to detect which side of the mismatch you are on, how to fix the limit, and how to alert on both signals independently so neither failure mode surprises you again. For the broader mental model of how KahaDB, the store limit, and flow control fit together, see How ActiveMQ Classic actually works in production.

What this means

StorePercentUsage is the ratio of ActiveMQ’s internal store usage counter to the configured storeUsage limit in activemq.xml, not to the physical disk. It answers “how much of my configured store budget is consumed by persistent message data?” It does not answer “how full is the disk?”

Disk usage on the KahaDB partition answers a different question: “how much space is left for journal files, the index, the temp store, broker logs, and anything else sharing the filesystem?” The broker does not control everything on that partition, and its own on-disk footprint can exceed its logical store accounting because KahaDB journal files are only reclaimed when every message in a file has been acknowledged. One unacked message pins an entire 32MB journal file.

This gives you four possible states, only one of which is healthy:

flowchart TD
  A[storeUsage limit configured] --> B{Limit vs disk capacity?}
  B -->|limit smaller than disk| C[Broker blocks at 100% StorePercentUsage with disk to spare]
  B -->|limit larger than disk| D[OS disk fills first, StorePercentUsage never reaches 100%]
  B -->|limit matches disk minus headroom| E[Healthy: both signals fire in the right order]
  C --> F[False full: expand limit or shrink data]
  D --> G[False safe: writes fail before alerts fire]

Two more behaviors make this worse. First, at startup the broker validates the configured limit against the space actually available on the KahaDB directory. If the limit exceeds available space, it logs a WARN and resets the limit down to the available space, with a message like “Store limit is 102400 mb, whilst the data directory … only has 21729 mb of usable space - resetting to maximum available disk space: 21729 mb”. That WARN is easy to miss, and it means your effective limit may be nothing like what activemq.xml says. Second, on versions before 5.12.0 the limit is evaluated once at startup and never re-checked, so a partition that fills up from other consumers after startup is invisible to the broker’s accounting.

Common causes

Quick checks

All of these are read-only.

# 1. Read the broker's store accounting via Jolokia
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/StorePercentUsage'

# 2. Check the physical reality on the KahaDB partition
df -h /opt/activemq/data/kahadb/

# 3. Compare on-disk footprint with logical store usage
du -sh /opt/activemq/data/kahadb/

# 4. Count journal files (each pinned file is up to 32MB held hostage)
ls /opt/activemq/data/kahadb/db-*.log | wc -l

# 5. Look for the silent startup clamp
grep -i "resetting to maximum available" /opt/activemq/data/activemq.log

# 6. Look for store-full flow control events
grep -i "Persistent store is Full" /opt/activemq/data/activemq.log | tail -20

# 7. Confirm the configured limit
grep -A3 "storeUsage" /opt/activemq/conf/activemq.xml

Adjust the data and conf paths for your installation. The key comparisons are check 1 against check 2, and check 3 against the limit from check 7. If du -sh shows more data than the configured limit, journal files are being pinned by unacknowledged messages. If df shows the partition nearly full while StorePercentUsage is low, the limit is above the disk or something else is consuming the partition.

How to diagnose it

1. Establish both denominators. Pull StorePercentUsage from JMX and df -h from the partition. Convert both to absolute numbers: StorePercentUsage times the configured limit gives logical store bytes; df gives physical used and available. Write the two numbers down before reasoning further.

2. Determine which limit actually bound. If writes are failing or producers are blocked and disk is full while StorePercentUsage is under 100%, the OS hit its wall first: the limit is larger than the disk, or the limit was never clamped and another consumer filled the partition. If producers are blocked with “Persistent store is Full” in the log and disk has room, the configured (or clamped) limit bound first.

3. Check for startup clamping. Search the broker log for the “resetting to maximum available disk space” WARN. If present, the effective limit is the clamped value, not the configured one. This also tells you the partition was already crowded when the broker started.

4. Check whether on-disk usage exceeds logical store usage. Compare du -sh on the KahaDB directory against the configured limit. A large gap means journal file pinning: some messages in old db-*.log files are unacknowledged, and KahaDB GC cannot reclaim those files even though the broker’s logical accounting has moved on. The usual suspects are DLQ messages with no TTL, offline durable subscribers, and stuck inflight messages. See ActiveMQ KahaDB journal files not deleted for that specific failure mode.

5. Check what else shares the partition. Broker logs and the temp store (data/tmp_storage) typically live on the same filesystem. Non-broker processes writing to the same mount are invisible to every JMX counter. Use du -sh on the top-level directories of the mount to find them.

6. Check the timeline. Did the disk fill gradually over weeks (classic store exhaustion spiral, often DLQ-driven) or jump suddenly (external consumer, log rotation failure, a burst of large persistent messages)? The store exhaustion spiral is slow and steady; an external consumer usually shows up as a step change in df that does not correlate with enqueue rate.

Metrics and signals to monitor

Alert on StorePercentUsage and on disk usage independently, with thresholds set so the tighter of the two constraints pages first. If your limit is 50 GB on a 60 GB partition, the store alert should fire first. If your limit is 100 GB on a 60 GB partition, the disk alert should fire first. If you cannot say which fires first, your alerting has the same bug as your configuration.

Fixes

Reconcile the configured limit against the real partition

Set storeUsage in activemq.xml below the partition size, with headroom for the index, journal rotation, the temp store if it shares the partition, and broker logs. Reserve at least 20 to 30% of the partition. The shipped default of 100 GB is a placeholder, not a recommendation; treat any broker still running it as unconfigured.

<systemUsage>
  <systemUsage>
    <storeUsage>
      <storeUsage limit="40 gb"/>
    </storeUsage>
  </systemUsage>
</systemUsage>

Changing the limit requires a broker restart on versions before 5.12.0, since the limit is evaluated at startup. Plan the restart; do not kill -9 a broker with writes in flight, because unclean shutdown risks journal or index corruption and a long recovery.

Give KahaDB a dedicated partition

Most limit-versus-disk confusion disappears when nothing else can consume the filesystem. With a dedicated mount for the KahaDB directory, df on that mount and the broker’s footprint describe the same thing, and clamping becomes predictable. If the temp store must share it, size the limit to account for both.

Use the periodic disk usage check (5.12.0 and later)

The broker attribute schedulePeriodForDiskUsageCheck (milliseconds, default 0, disabled) makes the broker periodically re-evaluate store and temp limits against actual available space. Enable it when the partition is genuinely shared or when capacity changes without restarts (resizable cloud volumes). Leave it disabled on dedicated partitions where the startup evaluation is sufficient. It protects against shrinking space; whether it raises the limit again when space grows is not clearly documented, so do not rely on it for expansion.

Consider percentLimit and the total attribute (version-dependent)

Instead of an absolute limit, storeUsage supports a percentLimit attribute that sizes the limit as a percentage of available disk space, which adapts to the partition automatically. From 5.15.x, a total attribute lets you declare the total capacity explicitly so the filesystem is not queried, which is useful for filesystems that report misleading sizes (the documented example is EFS) or when the broker should only use part of a partition. Verify both against your broker version before relying on them.

Drain what is pinning the store

If the immediate cause is journal pinning rather than a mis-sized limit: investigate and purge or export DLQ contents, unsubscribe abandoned durable subscriptions, and resolve stuck consumers. After the backlog drains, store usage drops immediately but disk reclamation lags, since journal cleanup runs periodically (default every 30 seconds). Do not conclude the fix failed because df has not moved yet.

Prevention

• Alert on both signals independently. StorePercentUsage above 80% is a ticket, 100% is a page. Disk usage above 80% on the KahaDB partition is a ticket, above 90% is a page. Whichever constraint is tighter should page first, by design.
• Alert on divergence. If the ratio of on-disk KahaDB size to logical store usage grows over weeks, journal pinning is building. Catch it before the partition fills.
• Grep for the clamp WARN after every restart. Make it part of the deploy checklist. A broker that started against a crowded partition is running with a limit you did not choose.
• Document the effective limit. Record configured limit, clamped limit (if any), partition size, and what else shares the partition, in the runbook. During an incident you want this lookup to take zero seconds.
• Size for the worst consumer outage. Headroom should cover the backlog that accumulates during your maximum tolerable consumer downtime, plus the 20 to 30% filesystem reserve.
• Re-verify after any volume resize. On pre-5.12.0 brokers the limit is fixed at startup; a resized partition changes nothing until a restart.

How Netdata helps

• Netdata charts disk usage per mount point at per-second granularity, so the KahaDB partition’s fill rate and step changes from external consumers are visible alongside broker metrics rather than in a separate tool.
• The ActiveMQ collector surfaces StorePercentUsage from JMX on the same dashboards as the host’s disk metrics, which makes the divergence between the two signals a visual comparison instead of a mental calculation.
• Journal file count and KahaDB directory size can be tracked as host-level metrics, exposing the pinning pattern (disk footprint exceeding the configured limit) as a trend before it becomes a full disk.
• Correlating DLQ depth, dequeue rate, and store usage on one timeline shows whether a climbing store is a live backlog or a dead-letter leak, which determines the fix.
• Alerts can be set on both the broker accounting and the physical partition independently, so the tighter constraint pages first regardless of which one it is.

Related guides

• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ KahaDB journal files not deleted: one unacked message pinning a 32MB log
• ActiveMQ memory limit reached: MemoryPercentUsage at 100% and the flow-control cliff
• ActiveMQ MemoryPercentUsage climbing: reading the flow-control leading indicator
• ActiveMQ monitoring checklist: the signals every production broker needs
• ActiveMQ monitoring maturity model: from survival to expert
• ActiveMQ per-destination memory usage: one noisy queue blocking every producer
• ActiveMQ producer flow control: why send() hangs and producers block silently
• ActiveMQ store is full: StorePercentUsage at 100% and persistent messaging halted
• ActiveMQ store usage climbing: the store exhaustion spiral
• ActiveMQ memoryUsage vs JVM heap: the two memory budgets teams confuse