ActiveMQ KahaDB db.data index bloat: slow lookups and slow startup recovery

You restart the broker after an unclean shutdown and it sits there for 20, 30, 40 minutes before it accepts a single client connection. Or the broker is running, but persistent message dispatch feels sluggish and disk reads on the KahaDB partition are elevated for no obvious reason. You look at the KahaDB directory and the journal files are not the problem. The problem is db.data, the B-tree index, sitting at multiple gigabytes.

db.data maps message IDs to their locations in the journal files. It grows with the number of pending (unconsumed, unacknowledged) messages the broker is tracking. A small index is fast to traverse and fast to recover. A multi-GB index means every lookup walks a deeper tree with more page-file I/O, and every unclean restart means a long recovery pass before the broker is usable.

This article covers how to confirm index bloat, what is feeding it, and how to shrink the index without losing messages or durable subscriptions.

What this means

KahaDB keeps two kinds of files in its data directory:

• db-*.log: sequential journal files (default 32MB each) holding the actual messages.
• db.data: the B-tree index mapping message IDs to journal offsets, plus destination and subscription metadata.

The index scales with pending message count, not with total throughput. A broker that has pushed a billion messages but consumes everything promptly can have a tiny db.data. A broker with a modest message rate but a large standing backlog (slow consumers, a neglected DLQ, an orphaned durable subscription) will grow a large index.

Two operational consequences:

1. Slower lookups at runtime. A larger index means more pages to traverse on every store operation, more page-file I/O, and more disk pressure on the KahaDB partition. The mechanics are commonly misstated: db.data uses page-file I/O, not a fully memory-mapped file as is sometimes claimed. Either way, a bigger index means more I/O work per operation.
2. Slower startup recovery. After an unclean shutdown, the broker replays journal files and rebuilds index state before accepting client connections. Recovery time scales with index size and journal file count. A multi-GB index can mean 30+ minutes of recovery. During recovery the process is alive but clients cannot use the broker, which looks exactly like an outage to everything upstream.

There is no clean JMX metric for db.data size. You measure it at the filesystem.

Rough size bands from operational experience:

Treat these as rules of thumb, not hard limits. The trend matters more than any single reading.

Common causes

The index itself is rarely the root cause. It is a symptom of pending messages. Fix what is keeping messages pending and the index problem becomes manageable.

Quick checks

All of these are read-only and safe to run during an incident.

# 1. Index file size (the signal itself)
ls -lh /opt/activemq/data/kahadb/db.data

# 2. Journal file count and total store footprint
ls /opt/activemq/data/kahadb/db-*.log | wc -l
du -sh /opt/activemq/data/kahadb/

# 3. Disk headroom on the KahaDB partition
df -h /opt/activemq/data/kahadb/

# 4. Queue depths across all queues (find the backlog)
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=*/QueueSize'

# 5. DLQ depth
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ/QueueSize'

# 6. Durable subscribers with pending messages (orphan detection)
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Topic,destinationName=*/PendingQueueSize'

# 7. Store usage as the broker accounts it
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/StorePercentUsage'

Note on check 6: destinationName=* PendingQueueSize returns per-topic pending, not per-subscription. Durable subscribers (active and inactive) are exposed as separate subscription MBeans under endpoint=Consumer with consumerId=Durable(<clientId>:<subscriptionName>); query destinationType=Topic,destinationName=*,endpoint=Consumer,clientId=*,consumerId=Durable*/PendingQueueSize for per-subscription pending counts.

Interpretation shortcuts:

• Big db.data + big DLQ: the DLQ is your index driver.
• Big db.data + an offline durable subscription with a huge pending count: orphaned subscription.
• Big db.data + high application queue depth: genuine consumer lag.
• Big db.data + low pending everywhere + many journal files: pinning or stale index entries from crash cycles.

How to diagnose it

Work through this in order. Stop when you find the driver.

flowchart TD
  A[db.data large and growing] --> B{Pending messages high somewhere?}
  B -->|Yes, app queues| C[Consumer lag: fix consumption first]
  B -->|Yes, DLQ| D[Drain or expire DLQ after investigation]
  B -->|Yes, durable sub| E[Unsubscribe orphaned durable subscribers]
  B -->|No| F{Journal files pinned?}
  F -->|Yes| G[Find the unacked message pinning files]
  F -->|No| H[Stale index from crash cycles: plan rebuild]
  C --> I[Index shrinks as backlog drains]
  D --> I
  E --> I
  G --> I
  H --> J[Stop broker, delete db.data, restart to rebuild from journal]

1. Baseline the index. Record db.data size, journal file count, and StorePercentUsage. Take a second reading after an interval. A stable large index with no backlog points at stale entries; a growing index points at active accumulation.
2. Find the pending messages. Sort queue depths. Check the DLQ. Check durable subscription pending counts. The index tracks pending messages, so the largest pending set is almost always your answer.
3. Check for pinning. If journal file count is high but total pending is low, one unacked message per old file can pin it. KahaDB GC can only delete a journal file when every message in it has been acknowledged.
4. Check crash history. Repeated unclean shutdowns (kill -9, power loss, OOM killer) leave the index in a worse state each cycle and lengthen the next recovery. If your last few restarts were unclean and recovery time keeps growing, a planned rebuild during a maintenance window is cheaper than the next unplanned one.
5. Quantify recovery risk. If the index is already multi-GB, your next unclean restart is a 30+ minute outage. That fact alone usually justifies scheduling the fix now rather than after the incident.

Metrics and signals to monitor

Index size and journal file count have no built-in JMX metrics. Both need filesystem-level collection, which is exactly the kind of gap a host-level agent fills.

Fixes

Fixes are grouped by cause. The ordering matters: always drain the pending messages before considering an index rebuild, otherwise the rebuild just recreates the bloat.

Drain the actual backlog

If application queues hold the backlog, fix consumption first: scale consumers, unblock their downstream dependency, or temporarily pause producers. As pending messages are consumed and acknowledged, the index has less to track. The index file itself does not shrink on demand, but it stops growing and recovery time improves once the journal set shrinks.

Deal with the DLQ

Investigate DLQ messages (check JMSDestination, redelivery count, exception properties), then purge or export them. Configure a TTL on DLQ messages so they stop accumulating forever, and consider per-destination DLQs so one noisy flow does not hide behind a shared queue. See the store exhaustion guide for the full spiral: ActiveMQ store usage climbing.

Remove orphaned durable subscriptions

Enumerate durable subscriptions and unsubscribe any whose owning application no longer exists. This is a permanent storage leak otherwise, and it feeds the index on every published message.

Force an index rebuild (planned maintenance)

If the index is bloated with stale entries after crash cycles, or it stays huge after the backlog is drained, the definitive fix is:

1. Drain or confirm acceptable message state first.
2. Stop the broker cleanly.
3. Delete db.data (keep the journal files).
4. Start the broker. It rebuilds the index from the journal.

Two serious warnings:

• Rebuild time scales with journal data. The broker replays journal files to reconstruct the index. With a large journal set this can take a long time; plan the window accordingly and do not assume it is minutes.
• Durable topic subscription state can be lost. Subscription metadata lives in the journal files; once KahaDB’s cleanup task has reclaimed journal files that carried that metadata, a rebuild from the remaining journal cannot recover the subscriptions. If you rely on durable topics, verify subscription state after the rebuild and be prepared to recreate subscriptions.

Do not do this reactively at 3 a.m. unless the alternative is worse. A broker with a bloated index is still serving; a broker mid-rebuild is not.

If the index is actually corrupted

Corruption (for example after an unclean shutdown with writes in flight) is a different situation from bloat. The broker may refuse to start or throw index errors. KahaDB provides recovery options such as ignoreMissingJournalfiles=true and checkForCorruptJournalFiles=true, but these can lose messages. Treat corruption recovery as a data-loss-risk operation: take a copy of the KahaDB directory first, and prefer restoring from a known-good state where you have HA or backups.

Prevention

• Alert on the index trend, not just the size. A weekly-growing db.data is a backlog problem announcing itself months early.
• Monitor pending-message drivers directly. Queue depth, DLQ depth, durable subscriber pending count, and journal file count are all leading indicators of index bloat. The index is the trailing indicator.
• Put TTLs on DLQ messages and audit durable subscriptions on a schedule.
• Avoid unclean shutdowns. Use clean stop procedures, give the broker time to checkpoint, and make sure the OOM killer or an orchestrator is not hard-killing the JVM (container memory limits vs. JVM heap is a common cause).
• Measure startup recovery time after every restart. If it is creeping up, your index and journal set are creeping up. Schedule a rebuild on your terms before an unplanned crash schedules it on theirs.
• Size the KahaDB partition with headroom. Recovery and rebuild operations need working room; keep at least 30% free for worst-case backlog growth.

How Netdata helps

• Filesystem-level collection of db.data size and journal file count, the two KahaDB signals that have no JMX metric, so the index trend is visible without custom cron jobs.
• Correlation of index growth against queue depth, DLQ depth, and StorePercentUsage on one dashboard, so you can see which backlog is feeding the index instead of guessing.
• Disk space and disk I/O latency on the KahaDB partition alongside broker metrics, catching the I/O degradation a large index causes before users feel it.
• Broker availability and recovery-window visibility: per-second process and port monitoring shows exactly how long a restart takes to return to service, turning “startup felt slow” into a measured recovery duration you can trend.
• JVM and GC metrics to distinguish index-driven slowness from the GC pause death spiral, which presents similarly (slow broker, unhappy clients) but needs a completely different fix.

Related guides

• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ KahaDB journal files not deleted: one unacked message pinning a 32MB log
• ActiveMQ memory limit reached: MemoryPercentUsage at 100% and the flow-control cliff
• ActiveMQ MemoryPercentUsage climbing: reading the flow-control leading indicator
• ActiveMQ monitoring checklist: the signals every production broker needs
• ActiveMQ monitoring maturity model: from survival to expert
• ActiveMQ per-destination memory usage: one noisy queue blocking every producer
• ActiveMQ producer flow control: why send() hangs and producers block silently
• ActiveMQ store is full: StorePercentUsage at 100% and persistent messaging halted
• ActiveMQ store usage climbing: the store exhaustion spiral
• ActiveMQ memoryUsage vs JVM heap: the two memory budgets teams confuse