ActiveMQ disk full on the KahaDB partition: write failures and store corruption risk

The KahaDB partition is at 100%. The broker log shows journal write failures, persistent producers have stopped, and you are in the worst ActiveMQ failure mode: the one where freeing space may not be enough, because an in-flight write at the moment the disk filled can leave the journal or index corrupt.

This is not the same incident as StorePercentUsage hitting 100%. StorePercentUsage measures KahaDB against the configured storeUsage limit in activemq.xml. Disk full measures the partition against physical capacity with df. They are independent limits, and either can fire first. If storeUsage is larger than the partition, the OS runs out of space while the broker still thinks it has headroom, and the failure arrives with no warning from any ActiveMQ metric. This guide covers the OS-level case; for the configured-limit case, see ActiveMQ store is full.

The corruption risk is specific: the broker acks persistent producers only after the journal fsync completes. A disk that fills mid-write can leave a partially written journal record or a torn index update. On the next restart, KahaDB recovery may fail, take hours replaying journals, or require a forced index rebuild.

What this means

KahaDB keeps three things on this partition, typically under data/kahadb/ (or activemq-data/ depending on packaging):

• db-*.log: sequential journal files, 32MB each by default. This is the write-ahead log every persistent message hits first.
• db.data: the B-tree index mapping message IDs to journal locations. Page-file I/O, and it grows with pending message count.
• db.redo: the redo log used during recovery.

The same partition usually also holds the temp store (data/tmp_storage) for non-persistent overflow and often the broker log files. They all compete for the same bytes.

When the partition fills, the sequence is:

flowchart TD
  A[Partition reaches 100%] --> B[Journal write fails: ENOSPC]
  B --> C[Persistent sends cannot be journaled]
  C --> D[Persistent messaging halts]
  B --> E[Write interrupted mid-record]
  E --> F[Journal or index corruption risk]
  F --> G[Next restart: recovery fails or runs long]
  D --> H[Producers block or error]

A journal file is only reclaimable when every message in it has been acknowledged, so disk consumption is pinned by the oldest unacked messages, not by average throughput. That is why this incident usually has a slow, visible runway (days of journal growth) followed by a cliff.

Common causes

Quick checks

# 1. Confirm the partition and its usage
df -h /opt/activemq/data/kahadb/

# 2. See what is consuming space (stay on this filesystem with -x)
du -xh --max-depth=1 /opt/activemq/data/ | sort -rh | head -20

# 3. Count and size the journal files
ls /opt/activemq/data/kahadb/db-*.log | wc -l
du -sh /opt/activemq/data/kahadb/

# 4. Check the index size (large index also means slow recovery later)
ls -lh /opt/activemq/data/kahadb/db.data

# 5. Compare broker-side store accounting vs physical disk
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/StorePercentUsage'

# 6. Check DLQ depth, the most common silent space consumer
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ/QueueSize'

# 7. Find what else is on this mount (non-ActiveMQ consumers)
du -xh --max-depth=1 "$(df --output=target /opt/activemq/data/kahadb/ | tail -1)" | sort -rh | head -20

# 8. Look for write failures in the broker log
grep -i "no space\|IOException\|store.*error" /opt/activemq/data/activemq.log | tail -20

All read-only. Adjust paths to your layout; the KahaDB directory default is activemq-data or data/kahadb under the install, but packaging varies.

How to diagnose it

1. Confirm which limit fired. Compare df output with StorePercentUsage from JMX. If df is at 100% and StorePercentUsage is at 60%, the configured storeUsage limit exceeds the physical partition and the OS filled first. If both are at 100%, you have both problems; read the store is full guide alongside this one.

2. Identify the space consumer. The du output from check 2 names the responsible subdirectory. Journal files dominating means message accumulation. tmp_storage dominating means non-persistent overflow. Log directory dominating means log retention, a much easier fix.

3. If journal files dominate, find what pins them. Journal files are deleted only when every message they contain is acked. Check DLQ depth first (no TTL by default, pins files forever), then queue depths and offline durable subscribers. A queue with a handful of ancient unacked messages can pin many journal files.

4. Assess corruption exposure before restarting anything. If the broker is still running, check the broker log for journal write errors. If the broker already crashed or was killed during the full-disk window, assume the index may be inconsistent and plan recovery time into the incident. Index recovery time scales with index size and journal count; a multi-GB db.data can mean 30+ minutes of startup.

5. Check the ext4 reserved blocks factor. ext4 reserves 5% of blocks for root by default. The broker typically runs as a non-root user, so it can hit ENOSPC while df still shows a few percent “free”. On a 1TB partition that hidden reserve is about 50GB. Confirm with tune2fs -l /dev/<device> | grep -i reserved.

Metrics and signals to monitor

Keep at least 20% of the partition free for journal rotation, cleanup, and filesystem overhead; 30% if you need to survive a worst-case consumer outage.

Fixes

Free space immediately (broker still running)

If the partition is full but the broker is alive, free space from something that is not KahaDB first: rotate or delete old broker log files, clear application logs or dumps on the same mount, and remove any non-ActiveMQ files identified in diagnosis. This buys room without touching the store.

Do not manually delete db-*.log files. Journal files are the store. Deleting them by hand is data loss and near-certain corruption.

Drain the backlog

If journal growth is from a live backlog, the clean fix is consumption: restart or scale consumers, then let KahaDB’s cleanup task reclaim journal files. Reclamation is not instant; the checkpoint/cleanup cycle runs periodically (default every 30s), so disk frees in steps after the backlog drains.

Purge or export the DLQ

If the DLQ is the pinner, export messages you need for forensics (they carry JMSDestination and failure context), then purge. Purging is destructive: purged messages are gone, so if your business requires reprocessing, export first. After purging, set a TTL on DLQ messages so this cannot silently recur.

Expand the partition

If the workload legitimately needs more store, grow the filesystem or move KahaDB to a larger volume. When you do, reconcile the configured storeUsage limit with physical capacity: the limit should sit below partition size with at least 20% headroom, so the broker’s own accounting trips before the OS does.

Reclaim the ext4 reserve (with care)

tune2fs -m reduces the reserved block percentage. On large dedicated data partitions, lowering it from 5% to 1% recovers meaningful space. It is a live-safe operation on mounted ext4, but treat it as a one-time capacity correction, not a substitute for headroom, and never do it on the root filesystem.

Recover from actual corruption

If the broker crashed during the full-disk window and fails to start, the recovery path is a forced index rebuild: stop the broker, delete db.data and db.redo (never the journal files), and restart. KahaDB rebuilds the index by replaying all journal files, which takes time proportional to store size. Two cautions: startup options such as ignoreMissingJournalfiles and checkForCorruptJournalFiles can get a broker past corrupt journal entries but may lose messages, and if cleanup already deleted journal files that held state for inactive durable subscribers, those subscriptions may not survive the rebuild. Test the procedure in staging before you need it in production.

Prevention

• Monitor disk free independently of StorePercentUsage. The two limits are decoupled; alert on both. Page at >90% disk used on the active broker role only, ticket at >80%.
• Size storeUsage below physical capacity. Leave at least 20-30% of the partition for rotation, cleanup, and the filesystem itself. If storeUsage exceeds the partition, you have configured this incident to happen.
• Give KahaDB a dedicated partition or volume. Sharing with broker logs, temp store, and OS files turns unrelated growth into a store-corruption event.
• Bound the DLQ. Per-destination DLQs plus a TTL, plus an alert on any non-zero depth. Unbounded DLQ is the most common root cause of slow store exhaustion.
• Enable KahaDB integrity checks proactively. checksumJournalFiles (default true since 5.9.0) and checkForCorruptJournalFiles let the broker detect, rather than silently propagate, journal damage.
• Account for the ext4 reserve in capacity math. Either lower it on the dedicated data volume or subtract it from usable capacity in your alerting thresholds.
• On 5.12+, consider schedulePeriodForDiskUsageCheck. The broker can periodically recheck actual disk space and shrink store/temp limits when other processes consume the disk. It mitigates the “external consumer fills the partition” case; it does not fix a store limit that was oversized from the start.
• Alert on journal file count growth. It is the earliest leading indicator, days before disk full. Pair it with DLQ depth and store usage trend for the full picture of the store exhaustion spiral.

How Netdata helps

• Per-mount disk usage at per-second granularity, so you see the KahaDB partition filling as a trend, not a surprise, with alerting thresholds you can set below the ext4 reserve boundary.
• Correlation between OS disk metrics and broker JMX metrics (StorePercentUsage, TempPercentUsage, queue depth, DLQ depth) on one dashboard: the comparison step in this guide, done continuously instead of at 3 a.m.
• Journal file count and directory sizes via file/directory collection, catching the pinning pattern (files accumulating while queues look drained) before disk full.
• Disk I/O latency on the store device alongside enqueue rate, distinguishing “disk full” from “disk slow” when persistent throughput degrades.
• Anomaly detection on disk growth rate, which flags the slow exhaustion spiral days ahead of the cliff even when absolute usage is still below static thresholds.

Related guides

• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ memory limit reached: MemoryPercentUsage at 100% and the flow-control cliff
• ActiveMQ MemoryPercentUsage climbing: reading the flow-control leading indicator
• ActiveMQ monitoring checklist: the signals every production broker needs
• ActiveMQ monitoring maturity model: from survival to expert
• ActiveMQ per-destination memory usage: one noisy queue blocking every producer
• ActiveMQ producer flow control: why send() hangs and producers block silently
• ActiveMQ store is full: StorePercentUsage at 100% and persistent messaging halted
• ActiveMQ store usage climbing: the store exhaustion spiral
• ActiveMQ memoryUsage vs JVM heap: the two memory budgets teams confuse