ActiveMQ broker won't start: port conflicts, store recovery, and lock contention

The broker process is running, or maybe it is not, and clients cannot connect. You restart the service, and it still does not come up. Or worse: the process starts, the log goes quiet, and ten minutes later you are still staring at a broker that has not opened its transport connectors.

“Won’t start” in ActiveMQ Classic covers four distinct failure modes that look identical from the outside: something else is holding port 61616 or 8161, KahaDB is replaying a large journal and simply is not done yet, the store is corrupted after an unclean shutdown and the broker is refusing to start, or a store lock in a shared-storage HA pair is held by the wrong broker and this one is waiting forever. Each has a different fix, and applying the wrong one (especially deleting store files or force-clearing locks) can turn a slow start into permanent message loss.

What this means

ActiveMQ Classic startup is sequential: the JVM loads the configuration, initializes the persistence adapter (KahaDB by default), acquires the store lock, recovers the journal and index, and only then starts transport connectors and the Jetty web console. A failure or stall at any earlier step means no listeners ever come up.

Two consequences matter operationally:

• The port can be open before the broker accepts clients. During KahaDB recovery the process is alive and may even have a listening socket, but it is not dispatching messages. A naive TCP health check can report “up” while the broker is still replaying the journal.
• A waiting broker looks like a dead broker. In shared-storage HA, a standby that cannot acquire the lock file in the KahaDB directory just keeps waiting. Its process runs, its transport connectors never start. That is correct behavior for a standby and a failure if it is supposed to be the active node.

So the first diagnostic question is never “why did it crash” but “which startup phase is it stuck in.”

Common causes

Quick checks

Run these read-only checks before changing anything.

# 1. Is the broker process actually running, and how long?
pgrep -af activemq

# 2. What is the process doing? Recovery burns CPU reading journals; a lock wait is idle.
top -p $(pgrep -f activemq) -bn1 | tail -2

# 3. Who, if anyone, is listening on the transport and console ports?
ss -tlnp | grep -E ':(61616|8161)'

# 4. How big is the store? This determines recovery time.
ls /opt/activemq/data/kahadb/db-*.log | wc -l
ls -lh /opt/activemq/data/kahadb/db.data

# 5. Is there free disk on the KahaDB partition?
df -h /opt/activemq/data/kahadb/

# 6. What does the startup log say so far?
tail -100 /opt/activemq/data/activemq.log

# 7. Is the lock file present, and who holds it?
ls -l /opt/activemq/data/kahadb/lock

Adjust paths to your deployment; data/kahadb and data/activemq.log are the conventional defaults. If pgrep -f activemq matches more than one JVM, pass a single PID to top and jstack.

Interpreting check 2 is the fastest triage split: high CPU on the broker process with no listeners almost always means journal recovery is in progress. Near-zero CPU with no listeners means it is waiting on something, usually the store lock.

How to diagnose it

flowchart TD
    A[Broker not accepting clients] --> B{Process running?}
    B -- No --> C[Read log: crash, bind error, or store exception]
    B -- Yes --> D{CPU busy?}
    D -- Yes --> E[KahaDB recovery in progress: wait, watch I/O]
    D -- No --> F{Listening on 61616?}
    F -- Yes --> G[GC stall or store stall: thread dump]
    F -- No --> H{Lock file held by another broker?}
    H -- Yes --> I[HA lock contention: verify active node]
    H -- No --> J[Stale lock or port conflict: check log and ss]

Step 1: Read the startup log as a timeline

The startup log is the primary artifact. The sequence to establish: persistence adapter initialization begins, then either recovery completes and connectors start, or something fails in between.

During KahaDB recovery there is a characteristic quiet window: the log announces the persistence adapter and then emits nothing while the journal files are read and the index is rebuilt, until recovery completes. Operators routinely mistake this silence for a hang and kill the broker mid-recovery, which makes the next startup worse. On 5.x the replay announces itself with INFO Recovering from the journal @<position> and ends with INFO Recovery replayed N operations from the journal in X seconds.; a corrupted index logs WARN Index corrupted. Recovering the index through journal replay.

A hard failure looks different: an exception and stack trace during adapter init (corrupt journal or index), a bind error naming the port, or the JVM exiting entirely.

Step 2: Rule out port conflicts

If the log shows a bind failure, or the broker dies shortly after connector startup, find the holder:

# Identify the process holding the port
ss -tlnp 'sport = :61616'
ss -tlnp 'sport = :8161'

Common holders: a second ActiveMQ instance from a package install you forgot about, an old broker JVM that did not die on the previous stop, a Docker container publishing 61616, or any other Java web app on 8161. The default activemq.xml ships protocol-specific transport connectors: OpenWire on 61616, plus AMQP (5672), STOMP (61613), and MQTT (1883) that are active in 5.15/5.18 and commented out in 5.19 and later. A port conflict on 61616 therefore blocks OpenWire, and one on 5672/61613/1883 blocks that protocol. The web console port is configured in conf/jetty.xml if it was changed from 8161.

Step 3: Confirm recovery versus corruption

For a CPU-busy broker with no listeners, confirm it is doing useful recovery work:

# Watch read I/O on the store device
iostat -xd 2 5

# Thread dump: recovery shows journal/index reader threads doing real work
jstack $(pgrep -f activemq) > /tmp/broker-threads.txt

Recovery time scales with the number of db-*.log journal files and the size of the db.data index. A multi-GB index means startup recovery of 30 minutes or more is expected, not a bug. Two settings can stretch this dramatically: checkForCorruptJournalFiles="true" forces verification of every journal file at startup, and checksumJournalFiles (default true since 5.9.0; false before that) adds checksum reads. Both trade startup speed for safety.

If the log instead shows exceptions while reading journal or index files, you are in the corruption path. The usual precursor is an unclean shutdown (kill -9, OOM kill, power loss) with writes in flight.

Step 4: Diagnose lock contention (shared-storage HA)

With KahaDB on shared storage, the default Shared File Locker takes a file lock on the lock file in the KahaDB directory. Exactly one broker holds it and becomes active; every other broker polls for it. With failIfLocked="false" (the default), a broker that cannot get the lock waits forever, polling every lockAcquireSleepInterval (default 10000 ms), with no connectors started and no loud error.

So for an idle broker with no listeners:

1. Determine which host is supposed to be active.
2. On the peer host, check whether a broker is running and holding the lock.
3. If the previous active broker crashed: the lock file may be stale. On local or SAN storage the OS releases the file lock when the process dies, so a leftover lock file on disk is usually harmless in itself; the real risk is clustered filesystems and NFS where lock semantics differ. On NFSv3 a lock held by a dead broker is never released by the server, and recovery typically requires clearing the lock state (in practice, restarting NFS lock services or the affected clients). NFSv4 releases locks after a lease timeout, on the order of tens of seconds, and is the required choice for shared-storage HA. OCFS2 is a known bad choice: it does not honor POSIX file locks from Java correctly, and two brokers can both believe they are active, which corrupts the store.

Before deleting a lock file or forcing acquisition, be certain the other broker is genuinely dead. Two active brokers on one store is the worst outcome in this entire article.

Metrics and signals to monitor

Fixes

Port conflict

Stop or reconfigure the process holding the port. If the holder is a zombie broker JVM from a previous run, kill it and confirm the port frees before restarting. If two services legitimately need the host, move the ActiveMQ connector or the web console (conf/jetty.xml) to a different port. Do not just restart ActiveMQ repeatedly hoping to win the bind race.

Long KahaDB recovery

The fix is patience, plus verifying progress via I/O and thread dumps. Killing the broker mid-recovery does not skip recovery; it reruns it, possibly from an earlier point. Prevention (below) is where you actually reduce this: keep the store small.

If startups are consistently slow because of checkForCorruptJournalFiles="true", consider disabling it once you have confidence in clean shutdowns, accepting the integrity-check tradeoff consciously rather than by accident.

Store corruption

If journal or index files are genuinely corrupt and the broker refuses to start, KahaDB provides recovery switches: ignoreMissingJournalfiles="true" to tolerate absent journal files, and checkForCorruptJournalFiles="true" to detect and isolate corrupt ones. Treat these as emergency surgery, not configuration. Skipping or discarding journal data can lose messages. Take a full copy of the KahaDB directory before starting with recovery flags, start the broker, verify queue depths and message availability, then drain or export what matters and rebuild the store cleanly if possible. The official documentation describes both flags as startup recovery options and gives no recommendation to keep them enabled permanently; revert them once the broker is back up.

Lock contention

If the peer broker is dead and the lock is genuinely stale, stop all broker JVMs, confirm nothing holds the lock file, then start the intended active broker. If failover is routinely slow or stuck, revisit the locker configuration: verify the shared filesystem’s lock semantics (NFSv4, not v3; never OCFS2), and consider lockKeepAlivePeriod so the active broker periodically proves it holds the lock, protecting against a standby acquiring it after external lock-file tampering (the default of 0 disables that keepalive and opens a dual-master window). If you want a standby to fail loudly instead of waiting forever, failIfLocked="true" changes that behavior, at the cost of needing external automation to restart it.

Prevention

• Shut down cleanly. Most store corruption and ugly recoveries trace back to kill -9, OOM kills, or power loss during writes. Give the broker a real stop with a generous timeout.
• Keep the store small. Journal file count and index size are your startup-time budget. Backlogged queues, unbounded DLQs, and orphaned durable subscribers all grow the store and therefore your next recovery. See the related guides on DLQ growth and durable subscriber leaks.
• Monitor disk and store usage independently. Store percent usage tracks a configured limit, not physical disk; either one can hit 100% first and both are startup risks.
• Alert on restart behavior. Track broker uptime and time-from-process-start to port-listening. If recovery time is climbing across restarts, your store is growing and an incident is being scheduled.
• Test failover. In shared-storage HA, periodically verify the standby actually acquires the lock and starts connectors when the active dies. A failover that has never been tested is a failover that does not work.
• Pin health checks to real service. During recovery a TCP connect can succeed before the broker serves clients. A canary send/receive on a test queue is the only check that proves the broker is truly up.

How Netdata helps

• Process liveness and port reachability together separate “broker down” from “broker starting,” and uptime tracking filters restart flaps out of alerts.
• Per-process CPU and disk I/O during startup make recovery visible: high CPU and read throughput on the KahaDB device means journal replay, not a hang.
• Filesystem metrics on the KahaDB partition (free space, plus journal file count and db.data size via simple collectors) let you trend the inputs to recovery time and alert before the store gets large enough to make restarts scary.
• JVM metrics via JMX (heap, GC pauses, thread count) catch the case where the broker finishes recovery and then immediately stalls in GC, which presents identically to a startup hang.
• Correlating uptime, port state, CPU, and I/O on one dashboard turns “won’t start” from a guessing game into reading off which phase the broker is stuck in.

Related guides

• ActiveMQ InactivityIOException: Channel was inactive for too long
• ActiveMQ consumers connected but not acknowledging: the zombie consumer
• ActiveMQ disk full on the KahaDB partition: write failures and store corruption risk
• ActiveMQ.DLQ growing: dead letter queue accumulation and poison messages
• ActiveMQ DLQ never expires: setting TTL so the dead-letter queue stops leaking storage
• ActiveMQ offline durable subscriber pending messages: the silent storage leak
• ActiveMQ enqueue outpacing dequeue: reading the rate imbalance before the backlog
• ActiveMQ expired message count climbing: TTL expiry and silent correctness loss
• ActiveMQ GC pause death spiral: long pauses, heartbeat timeouts, and reconnect storms
• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ InFlightCount high: prefetch full, acks stalled, and zombie consumers
• ActiveMQ JVM heap exhaustion: OutOfMemoryError and the OOM kill