ActiveMQ KahaDB corruption: the broker won't start after an unclean shutdown

The broker was killed (kill -9, power loss, OOM killer, container limit hit) and now it will not come back. Either the process exits during startup, or it sits there with the log stuck partway through KahaDB recovery and port 61616 never accepts a client. The log mentions db.data, a db-*.log journal file, or an IOException about a missing data file.

This failure became rarer after 5.14, but it still happens, and it carries the highest stakes of the classic ActiveMQ failures: every recovery path trades downtime against message loss. The wrong move at 3 a.m. is deleting the store to get the port back, then discovering you wiped the backlog you were supposed to preserve.

What this means

KahaDB has two halves on disk, in the KahaDB directory (typically /opt/activemq/data/kahadb/):

• db-*.log: sequential journal files, 32MB each by default. Every persistent message is appended here first (write-ahead log). The broker acks a persistent send only after the journal fsync completes.
• db.data: the B-tree index that maps message IDs to journal locations.
• db.redo: a summary of pending metadata changes not yet applied to the index.
• lock: the broker lock file, also used for shared-storage HA.

A clean shutdown flushes the index and closes journal files consistently. An unclean shutdown with writes in flight can leave the journal ahead of the index, the index referencing journal ranges that were never fully written, or a journal file with a torn final batch. On the next start, KahaDB reconciles all of this: it reads stored metadata, then replays journal events not yet reflected in the index, using db.redo as a guide.

Two consequences follow:

1. Recovery is normal and can be slow. After any unclean shutdown, journal replay and index work happen before the broker serves clients. On a large store this takes minutes to hours. During this window the process is alive and the transport port may even be open, but the broker is not accepting client connections. A TCP port check is not proof the broker is ready.
2. Real corruption blocks recovery. If the index or a journal file is damaged beyond what replay can reconcile, startup fails outright or loops. Now you are choosing between rebuilding the index from the journal (slow, usually lossless), starting with recovery flags that skip damaged data (fast, lossy), or discarding the store (instant, total loss).

Common causes

The last row is the trap: in shared-storage HA, a standby broker looks exactly like “broker up but port not serving.” That is expected behavior, not corruption. Do not “fix” it by deleting files in a shared store while the active broker holds the lock. That is how you get real corruption.

Quick checks

All of these are read-only and safe against a stopped or starting broker.

# Is the broker process actually alive, and for how long?
ps -o pid,etime,rss,cmd -C java | grep -i activemq

# What is the broker doing right now? Watch the log in real time.
tail -f /opt/activemq/data/activemq.log

# Is the transport port listening? (Listening does not mean serving clients.)
ss -tlnp | grep 61616

# What is in the KahaDB directory? Sizes and timestamps matter.
ls -lh /opt/activemq/data/kahadb/

# How many journal files, and how big is the index?
ls /opt/activemq/data/kahadb/db-*.log | wc -l
ls -lh /opt/activemq/data/kahadb/db.data

# Leftover temp files from an interrupted recovery?
ls -lh /opt/activemq/data/kahadb/*.tmp 2>/dev/null

# Was the JVM OOM-killed during a previous startup attempt?
dmesg -T | grep -i -E "killed process|oom" | tail

# Is the partition full? A full disk during recovery makes everything worse.
df -h /opt/activemq/data/kahadb/

Two numbers to take before you touch anything:

• Index size. Healthy is under 100MB. 100MB to 1GB means a significant backlog and slow startup. Multi-GB means recovery measured in tens of minutes or more; plan for that instead of assuming a hang.
• Journal file count. This bounds replay time and tells you how much message data is at stake if you end up discarding the store.

How to diagnose it

1. Confirm the shutdown was unclean. Check dmesg for OOM kills, container runtime events for limit kills, the tail of the previous broker log for a clean shutdown marker or an abrupt stop, and your supervisor’s records. This tells you corruption is plausible and which files were being written at the time.
2. Decide: recovering or stuck? Tail the log for 2 to 5 minutes. If recovery messages advance and iostat shows sustained reads on the KahaDB device, the broker is replaying the journal. On a large store, let it run. If the log has not moved in minutes, disk is idle, or the same exception repeats, you have genuine corruption or an interrupted recovery loop.
3. Identify which half is damaged. The startup exception usually names its target. Errors about the index, metadata pages, or db.data point at the index (fix: rebuild it from the journal). Errors naming a specific db-<n>.log or checksum failures point at a journal file (fix: recovery flags, with possible message loss). Could not locate data file means the index references journal files that are not on disk.
4. Rule out an interrupted previous recovery. If a prior startup attempt was killed mid-recovery, large *.tmp files can accumulate (operators have reported these growing to fill a disk). If the partition is now full, recovery cannot complete. Stop the broker, remove the large *.tmp files and db.redo, then restart and let recovery run from the beginning.
5. Size the job before choosing a fix. Index size plus journal count tells you whether a full rebuild is a 5-minute or a 3-hour operation. That number drives the decision in the Fixes section, and it is the number your stakeholders will ask for.

flowchart TD
  A[Broker will not start after unclean shutdown] --> B{Log advancing and disk busy?}
  B -->|yes| C[Journal replay in progress: wait and monitor]
  B -->|no| D{What does the startup error name?}
  D -->|index or db.data| E[Rebuild index: move db.data aside, restart]
  D -->|a db-N.log file| F[Enable recovery flags, accept possible message loss]
  D -->|missing data file| G[Index references deleted journals: rebuild index]
  E --> H[Verify canary send/receive, then open traffic]
  F --> H
  G --> H

Metrics and signals to monitor

Fixes

Work down this list. Each step trades more downtime for less data loss, in the direction you want: try the lossless option first.

Let recovery finish

If the log and disk I/O show progress, the correct fix is patience plus communication. Kill-and-retry resets recovery to the beginning and can leave *.tmp debris behind, making the next attempt slower and risking a full disk. If you must restart recovery, do it cleanly: stop the broker, remove stale *.tmp files, confirm free disk space, then start once.

Rebuild the index from the journal (lossless in the common case)

If db.data is corrupt but the journal files are intact, delete the index and let the broker rebuild it by replaying the journal. This is the standard recovery for index corruption and for “missing journal file” errors caused by an index that references files already gone.

# DESTRUCTIVE to the index (journal data is preserved). Take a backup first.
systemctl stop activemq        # or your supervisor equivalent
cp -a /opt/activemq/data/kahadb /opt/activemq/data/kahadb.bak   # if disk allows
mv /opt/activemq/data/kahadb/db.data /opt/activemq/data/kahadb/db.data.corrupt
mv /opt/activemq/data/kahadb/db.redo /opt/activemq/data/kahadb/db.redo.corrupt 2>/dev/null
systemctl start activemq

The broker replays every journal file to rebuild the index. On a multi-GB store this takes minutes to hours, during which the port may be open but clients are not served. Do not open producer traffic until a canary send/receive succeeds. One caveat: durable topic subscribers whose state lived only in the damaged portion of the index can be affected; verify durable subscriptions after recovery.

Start with recovery flags (lossy, use deliberately)

KahaDB exposes startup flags on the persistence adapter in activemq.xml. All default to false:

<persistenceAdapter>
    <kahaDB directory="${activemq.data}/kahadb"
            checkForCorruptJournalFiles="true"
            ignoreMissingJournalfiles="true"/>
</persistenceAdapter>

• checkForCorruptJournalFiles="true": validates journal files on startup and attempts to recover them, so a torn file does not abort startup.
• ignoreMissingJournalfiles="true": missing journal files are reported and skipped instead of failing startup.

The tradeoff is real: whatever lived in the skipped or corrupt portion is gone, and the broker will start without telling you which messages were sacrificed. There is also a long-standing known issue (AMQ-6938) where ignoreMissingJournalfiles does not cover every code path: if journal files were manually deleted while the index still references them, startup can still fail with Could not locate data file. The reliable fix for that case is the index rebuild above, not the flag. A related flag, checksumJournalFiles, enables checksums used to detect journal corruption and has defaulted to true since 5.9.0; if you are on something older, enable it before you need it.

Discard the store (total message loss)

If the store is damaged beyond repair, or the business decision is that the backlog is worth less than the downtime:

# DESTRUCTIVE: permanently deletes all persisted messages, durable subscriptions, and scheduler state.
systemctl stop activemq
mv /opt/activemq/data/kahadb /opt/activemq/data/kahadb.lost
systemctl start activemq

The broker comes up immediately with an empty store. Every persistent message, durable subscription, and scheduled message is gone. Make this call explicitly, with the journal file count and backlog size in front of you, and keep the renamed directory until you are certain nothing needs to be salvaged.

Prevention

• Stop killing the broker. Most KahaDB corruption starts with kill -9, a power event, or an OOM kill. Give your service manager a generous stop timeout so shutdowns flush cleanly, and size the container or cgroup memory limit above JVM max heap plus overhead so the OOM killer is not your shutdown mechanism.
• Bound your recovery time. Recovery duration is a function of index size and journal count. If you monitor db.data size and journal file count continuously, you catch the backlog growth that turns a 2-minute recovery into a 2-hour one. Journal files are pinned by single unacknowledged messages, so DLQ hygiene and stuck-consumer detection are corruption-recovery controls too. See journal files not deleted and store usage climbing.
• Keep free disk headroom on the KahaDB partition. Recovery needs working space, and a disk that fills mid-recovery can create the corruption you were trying to fix. Reserve at least 20%.
• Enable journal checksums (checksumJournalFiles="true", default since 5.9.0) so corruption is detected cleanly at startup instead of surfacing as bizarre runtime failures later.
• Stay on a supported release. KahaDB corruption after hard crashes was far more common on old versions (5.4.0/5.4.1 were notorious; 5.4.2 delivered the KahaDB corruption fixes but was itself retracted, so 5.4.3 is the usable fix line), and recent releases carry KahaDB fixes. If you are on a deprecated series, that is part of your incident.
• Be careful with KahaDB on NFS or CIFS. Unclean shutdowns on network filesystems have produced unrecoverable index corruption in the field. Local or SAN block storage is the safer target; if you must use NFS for shared-storage HA, treat NFS health as a broker-critical signal.
• Health-check with a canary, not a port. A TCP check passes while the broker is mid-recovery. A synthetic send/receive on a canary queue tells you when the broker is actually serving, and prevents load balancers and operators from treating “port open” as “ready.”

How Netdata helps

• Process uptime and restart detection distinguish a broker that is slowly recovering from one crash-looping on a corrupt store, and correlate the failure with the unclean shutdown that caused it.
• Disk space and I/O metrics on the KahaDB partition show whether recovery is actively reading (healthy replay) or stalled (genuine hang), and whether free space is about to become the next problem.
• Journal file count and db.data size collected over time tell you, before any incident, how long your worst-case recovery will take. That turns a 3 a.m. guess into a known number.
• Port reachability plus a canary check closes the “port open but not serving” gap that makes KahaDB recovery look like a ready broker to shallow health checks.
• Store and backlog signals (StorePercentUsage, queue depth, DLQ depth) explain why the store grew large enough to make recovery painful in the first place, which is the prevention loop.

Related guides

• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ KahaDB journal files not deleted: one unacked message pinning a 32MB log
• ActiveMQ memory limit reached: MemoryPercentUsage at 100% and the flow-control cliff
• ActiveMQ MemoryPercentUsage climbing: reading the flow-control leading indicator
• ActiveMQ monitoring checklist: the signals every production broker needs
• ActiveMQ monitoring maturity model: from survival to expert
• ActiveMQ per-destination memory usage: one noisy queue blocking every producer
• ActiveMQ producer flow control: why send() hangs and producers block silently
• ActiveMQ store is full: StorePercentUsage at 100% and persistent messaging halted
• ActiveMQ store usage climbing: the store exhaustion spiral
• ActiveMQ memoryUsage vs JVM heap: the two memory budgets teams confuse