ActiveMQ broker down: telling a crashed broker from a hung one

Your pager says the ActiveMQ broker is down. The first question is not “how do I restart it” but “what kind of down is this.” A crashed broker (process gone, port closed) and a hung broker (process alive, port possibly open, nothing moving) have different causes, different evidence, and different safe responses. Restarting a hung broker without capturing diagnostics destroys the evidence. Assuming a crashed broker will come straight back up ignores the KahaDB recovery window, where the process runs and the port may even bind, but no client gets served for minutes to hours.

There is a third case that generates most of the false pages: the broker is running, the TCP port accepts connections, and yet no messages flow. The JVM can be frozen in a long GC pause or stalled on a store write while the kernel’s accept queue keeps answering SYN packets. A listening port does not prove useful service. The only check that proves a broker is doing its job is a canary send and receive on a test queue.

The triage order: process, port, message flow, then root cause.

What this means

“Broker down” is at least four distinct states:

1. Crashed. The JVM process is gone. Common causes: OOM killer (container or cgroup limit exceeded, or system-level), an unhandled JVM fatal error, or an operator kill. The port is closed because nothing owns it.
2. Failed to start. The process exists briefly or runs but never opens the transport connector. Common causes: KahaDB journal or index corruption after an unclean shutdown, or a port conflict.
3. Recovering. After an unclean shutdown, KahaDB replays journal files and rebuilds the index before serving clients. The process is alive. Depending on timing, the port can be open while the broker is not yet accepting or serving client connections. For a large store this window is minutes to tens of minutes; a multi-GB index can mean 30+ minutes of recovery.
4. Hung. The process is alive and the port listens, but the broker moves no messages. The usual suspects: a GC pause death spiral, a store stall (disk I/O or lock), file descriptor exhaustion, or thread exhaustion. To a TCP health check, this looks healthy. To your producers and consumers, the broker is dead.

The diagnostic flow:

flowchart TD
  A[Broker down alert fires] --> B{JVM process alive?}
  B -->|no| C[Crashed: check logs, OOM killer, exit reason]
  B -->|yes| D{Transport port listening?}
  D -->|no| E{Recent unclean shutdown?}
  E -->|yes| F[KahaDB recovery in progress - watch journal replay]
  E -->|no| G[Startup failure: store corruption, port conflict, FD exhaustion]
  D -->|yes| H{Canary send/receive works?}
  H -->|no| I[Hung: thread dump, GC pauses, store stall]
  H -->|yes| J[Not down - check HA standby role or alert logic]

Common causes

Quick checks

Run these read-only checks in order. They take under two minutes and classify the failure before you touch anything.

# 1. Is the broker JVM process alive?
pgrep -af activemq

# 2. Is the OpenWire port listening? (default 61616; adjust for your connectors)
ss -tlnp | grep -E '61616|5672|61613'

# 3. Count established client connections on the OpenWire port
ss -Htn state established '( sport = :61616 )' | wc -l

# 4. If the process is gone: did the kernel OOM killer take it?
dmesg -T | grep -i -E 'out of memory|killed process' | tail -5

# 5. Tail the broker log for crash, startup, or recovery evidence
tail -100 /opt/activemq/data/activemq.log

# 6. If it just restarted after a crash: how much store must it replay?
ls /opt/activemq/data/kahadb/db-*.log | wc -l
ls -lh /opt/activemq/data/kahadb/db.data

# 7. If alive and listening: are the broker counters moving? Take two readings 30s apart.
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/TotalEnqueueCount,TotalDequeueCount'

# 8. If counters are flat: check GC activity
BROKER_PID=$(pgrep -f activemq | head -1)
jstat -gcutil $BROKER_PID 1000 5

# 9. Check file descriptor pressure for the broker process
echo "Open: $(ls /proc/$BROKER_PID/fd | wc -l)"
grep 'Max open files' /proc/$BROKER_PID/limits

# 10. Check store device latency and space
df -h /opt/activemq/data/kahadb/
iostat -xd 1 3

One trap to know before you script any of this: the broker’s Health MBean (service=Health) exposes a CurrentStatus attribute that reads “Good” but is not refreshed automatically. It is only recomputed when you invoke the health() or healthList() JMX operation. A monitor that just reads the attribute will report “Good” on a hung broker. Treat it as useless unless you invoke the operation first.

How to diagnose it

1. Classify: process, port, flow. Work through checks 1-3 and 7 above. You now know which of the four states you are in: crashed, failed to start, recovering, or hung.
2. If crashed, capture the exit reason before restarting. Read the broker log tail and the kernel log. An OOM kill points to heap exhaustion or a container memory limit set below the JVM’s real footprint. A fatal JVM error leaves an hs_err_pid file in the working directory. Restarting is fine operationally, but capture the evidence first or it is gone.
3. If it will not start, suspect the store. An unclean shutdown (kill -9, power loss, OOM kill) with writes in flight can corrupt the KahaDB journal or index, and the broker refuses to start. See ActiveMQ KahaDB corruption. Do not delete journal files to force a start; that loses messages.
4. If it is recovering, wait with eyes open. During journal replay and index rebuild the broker process runs and the port can be open without the broker accepting or serving clients. Estimate the window from journal file count and index size: a multi-GB db.data means a long recovery. Paging the on-call again during a known recovery window is noise; measure the actual recovery duration once and set expectations from it.
5. If it is hung, capture a thread dump before anything else. jstack <pid>, or kill -3 <pid> (SIGQUIT writes the dump to the JVM’s stdout log; it is non-disruptive). A thread dump is the single most valuable artifact in a hang: it shows whether transport threads are parked in GC, blocked on store writes, or deadlocked. Take two dumps 10-15 seconds apart so you can tell stuck threads from slow ones.
6. Prove the hang with counters, not vibes. Two readings of TotalEnqueueCount and TotalDequeueCount 30 seconds apart. Both flat while producers are connected and queues have depth means the broker is not processing. Flat enqueue with blocked producers points at flow control; check MemoryPercentUsage for 100%.
7. Run a canary before declaring recovery. Send a test message to a canary queue and consume it back within a time window. If you use the failover transport for the canary client, set timeout (for example failover:(tcp://broker:61616)?timeout=3000); the default is -1, which blocks a send indefinitely when the broker is unavailable, and your health check itself hangs.
8. Rule out the standby. In shared-storage HA, the standby process runs but never starts its transport connectors while it waits for the store lock. A “port down” alert on the standby is a false positive every time. Confirm the broker’s HA role before paging.

Metrics and signals to monitor

Alerting discipline matters as much as the signals. Page when the process is absent or the client port stops listening in production and uptime was over 600s before the failure. The uptime condition excludes restart flaps and crash loops, which deserve a ticket, not a page. Confirm with a failed canary, not just a failed TCP connect. Never page on a standby whose connectors are intentionally down.

Fixes

Crashed: OOM kill

Restore service with a clean start, then fix the sizing. Check whether the kill came from the cgroup/container limit or the system OOM killer. If the JVM heap plus non-heap exceeds the container limit, the kernel kills the process regardless of -Xmx. Size heap so ActiveMQ’s memoryUsage limit is roughly 60-70% of JVM max heap, and the JVM total fits under the container limit. The follow-up work is in ActiveMQ JVM heap exhaustion.

Failed to start: KahaDB corruption

Follow the corruption runbook rather than improvising. The recovery flags (ignoreMissingJournalfiles, checkForCorruptJournalFiles) can get a broker started but may lose messages; decide that tradeoff explicitly, with a backup of the KahaDB directory first.

Recovering: KahaDB journal replay

Usually the fix is patience plus communication. If recovery windows after crashes are routinely long, shrink them by keeping the backlog and index small (drain the DLQ, fix consumer lag) so there is less to replay. Track actual recovery duration so your alerts can suppress re-pages during a known window.

Hung: GC pause death spiral

With thread dumps and GC evidence captured, restart the broker. A hung JVM in a death spiral does not recover on its own. In a shared-storage HA pair this is a failover; on a standalone broker it is a full outage plus a KahaDB recovery window on restart, so do not skip the diagnostics to save thirty seconds. Then address the cause: heap too small for the connection and destination count, or a leak from destination explosion. For heaps over 4GB, G1GC gives shorter pauses than older collectors. Detail in ActiveMQ GC pause death spiral.

Hung: store stall

Check disk free space first (a full KahaDB partition halts persistent writes and risks corruption; see disk full on the KahaDB partition), then write latency on the store device. In shared-storage HA, NFS latency or lock daemon problems stall the active broker the same way. If the stall is a failing device or an NFS server, failing over to the standby may be the right move, but confirm the standby can actually acquire the lock first.

Hung: FD exhaustion

Short term, restart recovers the descriptors. The real fix is the limit: the default Linux ulimit of 1024 is far too low for a production broker. Set at least 65536 and monitor the open-FD trend so a leak shows up as growth before it shows up as an outage.

Prevention

• Run a continuous canary. Produce and consume a test message on a dedicated queue and alert on round-trip failure or latency. This catches every hung-but-listening state that port checks miss.
• Page on conditions, not events. Process absent or port closed AND uptime over 600s AND not an HA standby, ideally confirmed by canary failure. Everything else is a ticket.
• Raise the FD limit to at least 65536 before you need it.
• Enable GC logging permanently. Post-incident, the GC log is the difference between “we think it was GC” and “here is the 12-second pause at 03:14.”
• Size the heap and the ActiveMQ memory limit deliberately, and keep the container memory limit above the JVM’s real footprint.
• Keep the store small. DLQ TTLs, consumer lag fixes, and durable subscriber cleanup all shrink the journal and index, which shrinks recovery time after every future crash.
• Use clean shutdowns. SIGTERM and let KahaDB checkpoint. kill -9 buys you a recovery window and a corruption risk on the next start.

How Netdata helps

• Process and uptime tracking distinguishes a crash from a hang immediately and enforces the uptime-over-600s page condition without custom scripting.
• JMX collection of TotalEnqueueCount and TotalDequeueCount makes “counters flat while port is open” a visible, alertable condition instead of a manual two-readings check at 3 a.m.
• JVM heap and GC pause metrics correlate the sawtooth connection pattern with specific long pauses, confirming the GC death spiral in one view.
• File descriptor and thread count per process catch the slow exhaustion curve days before the accept failures start.
• Disk free space and per-device I/O latency on the KahaDB partition connect store stalls to the hang they cause, separating storage problems from broker problems.
• Connection count over time shows the drop-spike sawtooth of a reconnection storm and gives you the baseline deviation alerting that a static threshold cannot.

Related guides

• How ActiveMQ Classic actually works in production: a mental model for operators
• ActiveMQ GC pause death spiral: long pauses, heartbeat timeouts, and reconnect storms
• ActiveMQ JVM heap exhaustion: OutOfMemoryError and the OOM kill
• ActiveMQ KahaDB corruption: the broker won’t start after an unclean shutdown
• ActiveMQ disk full on the KahaDB partition: write failures and store corruption risk
• ActiveMQ InFlightCount high: prefetch full, acks stalled, and zombie consumers