$ guides / activemq / activemq-kahadb-journal-files-growing ▌

Operations Guides

ActiveMQ KahaDB journal files not deleted: one unacked message pinning a 32MB log

Your application queues are empty or nearly empty. Consumers are connected and processing. Yet the KahaDB directory keeps growing, db-*.log files pile up, StorePercentUsage creeps upward, and disk free space trends toward zero. This is the classic KahaDB journal pinning problem, and it confuses operators precisely because the visible queue state looks healthy.

The mechanism is simple and unforgiving: a KahaDB journal file (default 32MB) is only reclaimable when every message stored in that file has been acknowledged. One unacknowledged message anywhere in the file pins the entire file on disk. If that one message is sitting in the DLQ, held by an offline durable subscriber, or stuck in a dead consumer’s prefetch buffer, the file stays, and new files keep being written behind it.

This article covers how to confirm journal pinning, find the destination holding the messages, and clear the backlog without losing data you still need.

What this means

KahaDB stores persistent messages in sequential append-only journal files named db-*.log, alongside a B-tree index file db.data that maps message IDs to journal locations. Messages are never updated or deleted in place. When a message is acknowledged, an ack record is appended to the current journal. A periodic cleanup cycle (journal GC, roughly every 30 seconds, with checkpoint/index flush roughly every 5 seconds) walks the journal files and deletes any file whose messages are all acknowledged and no longer referenced by the index.

The catch: reclamation is all-or-nothing per file. A 32MB file containing 10,000 acknowledged messages and one unacknowledged message is not reclaimable. That single message costs you 32MB. Multiply this by a DLQ that accumulates messages scattered across months of journal files, or an offline durable subscriber whose messages are interleaved with live traffic, and disk usage grows without bound while every queue dashboard looks fine.

Two more properties matter operationally:

Reclamation lags consumption. Cleanup runs periodically, not immediately. After you drain a backlog, files disappear on the next GC cycles, not instantly. Do not conclude the fix failed because files are still there 30 seconds later.
Store usage and disk usage are different numbers. StorePercentUsage is measured against the configured store limit in activemq.xml. If that limit is larger than the physical partition, the disk fills before ActiveMQ’s own accounting reaches 100%. Monitor both independently.

flowchart LR
  P[Producer send] --> J[Journal write db-N.log]
  J --> C[Consumer dispatch]
  C --> A{All messages in file acked?}
  A -->|yes| GC[Cleanup cycle deletes file]
  A -->|no - one unacked| PIN[File pinned on disk]
  PIN --> GROW[File count and disk usage grow]
  DLQ[DLQ message] -. pins .-> PIN
  DUR[Offline durable subscriber] -. pins .-> PIN
  STUCK[Stuck consumer prefetch] -. pins .-> PIN

Common causes

Cause	What it looks like	First thing to check
DLQ accumulation	Application queues empty, DLQ depth non-zero or slowly growing, journal count climbs for weeks	`QueueSize` on `ActiveMQ.DLQ` (and any per-destination DLQs)
Offline durable subscriber	Topic store grows, one subscription shows pending messages with zero active consumers	`PendingQueueSize` on durable subscription MBeans
Slow or stuck consumer	Queue depth low but `InFlightCount` pinned near prefetch, dequeue rate near zero	`InFlightCount` vs consumer count times prefetch
General consumption lag	Enqueue rate persistently above dequeue rate across real queues	Enqueue/dequeue rate ratio per destination
Expired messages routed to DLQ	`ExpiredCount` rising on source queues, DLQ growing in step	`ExpiredCount` per destination plus DLQ depth

The DLQ is the most common hidden culprit. Messages moved to the DLQ count as dequeued from the source queue, so source queues look drained while the DLQ quietly pins journal files. DLQ messages accumulate until purged; nothing removes them automatically.

Quick checks

All of these are read-only. Adjust host, port, and credentials for your environment.

# Count journal files (no JMX metric exists for this; filesystem only)
ls /opt/activemq/data/kahadb/db-*.log | wc -l

# Total KahaDB footprint and index size
du -sh /opt/activemq/data/kahadb/
ls -lh /opt/activemq/data/kahadb/db.data

# Physical disk on the KahaDB partition
df -h /opt/activemq/data/

# Broker store accounting vs physical disk
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/StorePercentUsage'

# DLQ depth - the usual suspect
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ/QueueSize'

# Every queue's depth at once - look for anything non-zero you forgot about
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=*/QueueSize'

# Inflight counts - messages stuck in consumer prefetch still pin journal files
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=*/InFlightCount'

# Find durable subscription MBeans - PendingQueueSize lives on the
# subscription MBean, not on the topic destination MBean
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/search/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Topic,destinationName=*,endpoint=Consumer,clientId=*,consumerId=*'
# Then read PendingQueueSize and Active on each MBean the search returns.
# <!-- TODO: verify the subscription MBean object name pattern (endpoint=Consumer and the offline/InactiveDurableSubscription naming) against your ActiveMQ version -->

# Expired messages (they land in the DLQ by default)
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=*/ExpiredCount'

The diagnostic pattern: if journal file count is high and growing, but every application queue shows QueueSize near zero, the pinning messages are almost certainly in the DLQ, in an offline durable subscription, or inflight to a consumer that is not acking.

How to diagnose it

Establish the growth rate. Record the journal file count and the KahaDB directory size twice, an hour apart. ls /opt/activemq/data/kahadb/db-*.log | wc -l and du -sh. A growing count with flat application queue depths confirms pinning rather than a live backlog.
Check the DLQ first. Get QueueSize on ActiveMQ.DLQ and on any per-destination DLQs if you use IndividualDeadLetterStrategy. A non-zero DLQ on a broker with otherwise empty queues is almost always your answer. Each of those messages is also a processing failure worth investigating on its own.
Check durable subscribers. Enumerate the subscription MBeans from the search above and look for PendingQueueSize greater than zero with no active consumer. An abandoned dev/test subscription accumulates every message published to its topic, scattered across journal files.
Check inflight messages. If a destination shows QueueSize of zero but a high InFlightCount, messages are sitting in consumer prefetch buffers unacknowledged. With CLIENT_ACKNOWLEDGE or transacted sessions this can be by design; with a stuck consumer it is a leak. Correlate with dequeue rate: inflight high plus dequeue near zero means stuck consumers.
Check expiry. Rising ExpiredCount on any queue means messages are being silently moved to the DLQ (default behavior), feeding cause number one.
If JMX shows nothing pending anywhere, go deeper. Enable TRACE logging on the KahaDB message database to see which destinations still reference each journal file during GC cycles. TRACE logging on a busy broker is verbose; enable it briefly and disable it after you capture a few GC cycles.
Confirm reclamation lag before concluding failure. After you clear the pinning messages, files disappear on the periodic cleanup cycle, not instantly. Give it a few minutes and re-check the file count before escalating.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Journal file count (filesystem)	Direct measure of unreclaimed storage; no JMX equivalent	Count above 2x baseline or steadily growing
`StorePercentUsage`	Broker’s own store accounting; 100% halts persistent messaging	Above 70% and climbing
Disk free on KahaDB partition	Physical limit, independent of configured store limit	Above 80% used
DLQ `QueueSize`	Each message is a failure and a journal pin	Any non-zero, or any sustained growth
Durable subscriber `PendingQueueSize`	Orphaned subscriptions are permanent leaks	Offline subscriber with pending messages over 1 hour
Per-queue `InFlightCount`	Unacked prefetched messages pin files	Inflight pinned near prefetch with dequeue near zero
Per-queue `ExpiredCount`	Expired messages feed the DLQ silently	Any unexpected sustained increase
`db.data` index size	Grows with pending message count; slows recovery	Above 500MB

Fixes

Clear the DLQ

Investigate first, then drain. Browse DLQ messages and check the JMSDestination property and redelivery headers to learn which source queue produced them and why they failed. Fix the consumer bug or message format problem before purging, or the DLQ refills. After investigation, purge or export the DLQ. Purging is destructive: those messages are gone. If there is any chance you need to replay them, export first.

Remove orphaned durable subscriptions

For each offline durable subscriber with pending messages, confirm the owning application is genuinely decommissioned, then unsubscribe it via JMX or the web console. This is destructive: the pending messages are discarded. If the subscriber should be running, the fix is to bring it back, not to delete the subscription.

Unstick consumers

If inflight messages are pinning files, restart the stuck consumer application. Its inflight messages are redelivered to other consumers on disconnect and get acknowledged, releasing the journal references. Then find out why it stopped acking: downstream dependency, thread pool exhaustion, or an ack-mode bug.

Wait for cleanup, then verify

After the pinning messages are acked or removed, journal GC reclaims the files on its periodic cycle. Watch the file count drop over the next several minutes. If the count does not drop after the backlog is gone, that is when deeper investigation (TRACE logging, possible known GC bugs in older 5.x versions) is warranted.

Do not do these

Do not delete db-*.log files by hand. The index references them. You will corrupt the store.
Do not restart the broker as a first fix. Restart does not unpin journal files; the unacked messages are still in the store after recovery, and a large store makes restart recovery slow.

Prevention

Set a TTL on DLQ messages so dead-lettered traffic cannot accumulate forever. The DLQ has no automatic expiry by default.
Use per-destination DLQs (IndividualDeadLetterStrategy) so poison messages are attributable to their source queue and one bad flow does not hide among others.
Alert on any non-zero DLQ depth, not just growth. Every DLQ message is a failed business transaction.
Audit durable subscriptions regularly. Alert on any subscription with zero active consumers and a growing pending count. Delete dev/test subscriptions as part of decommissioning.
Monitor journal file count as a first-class signal. It is a filesystem count with no JMX equivalent, so your monitoring agent must collect it from the OS.
Monitor StorePercentUsage and disk free independently. If the configured store limit exceeds physical disk, the OS fills first.
For mixed-workload brokers, consider isolating destinations so one slow consumer or DLQ cannot pin journal files shared with unrelated queues.

How Netdata helps

Netdata’s ActiveMQ collector pulls StorePercentUsage, per-destination queue depth, inflight count, and enqueue/dequeue counters from JMX, so you can see store growth alongside the destinations driving it on one dashboard.
DLQ depth is charted like any other queue, which turns the most common pinning cause from a hidden leak into a visible line.
The disk plugin tracks free space and inode usage on the KahaDB partition independently of the broker’s own accounting, catching the case where the configured store limit exceeds physical disk.
Per-second system metrics let you correlate a journal file count spike (via a filesystem check) with the dequeue collapse or consumer drop that caused it, narrowing the timeline of the pinning event.
Durable subscriber pending counts and expired message counts are available per destination, so the two silent accumulation paths show up before the disk alarm fires.