ActiveMQ HA failover stuck: standby cannot acquire the store lock

The active broker died. The standby broker’s process is running, the JVM looks healthy, and yet no client can connect anywhere. ss shows nothing listening on 61616 on either node. The failover you designed for is not happening.

This is the shared-storage master/slave stall: the standby is blocked waiting to acquire the lock file in the KahaDB directory, and until it gets that lock it will not start its transport connectors. That behavior is by design. What is not by design is the lock never becoming available. The usual suspects are a stale lock file left behind by a crashed active, an unhealthy NFS lock daemon, or a SAN mount that is unavailable on the standby.

The dangerous twin of this failure is split-brain: two brokers both believing they hold the lock. Everything you do during this incident must avoid converting “nobody is serving” into “two brokers are writing to one store”, because the second state corrupts KahaDB.

What this means

In a shared-file-system master/slave pair, both brokers point at the same KahaDB directory on shared storage. The first broker to take an exclusive file lock on the lock file in that directory becomes the master, opens its transport connectors, and serves clients. Every other broker sits in a loop polling for that lock (with the default Shared File Locker, the poll interval is lockAcquireSleepInterval, default 10000 ms). A waiting broker’s process runs, but it does not register the full broker MBeans and does not open its ports.

So “standby process up, connectors down” is the healthy waiting state. The incident state is: the active is gone, the lock is still not acquirable, and the standby waits forever. Failover has stalled.

flowchart TD
  A[Active broker fails] --> B[Standby polls KahaDB lock file]
  B --> C{Lock acquired?}
  C -->|Yes| D[Standby starts transport connectors]
  D --> E[Clients reconnect: failover complete]
  C -->|No| F{Why is the lock unavailable?}
  F --> G[Stale lock after crash]
  F --> H[NFS lock daemon unhealthy]
  F --> I[SAN mount unavailable]
  F --> J[Old active still alive and holding lock]
  G --> B
  H --> B
  I --> B
  J --> B

Common causes

Quick checks

All read-only unless noted. Run these on the standby host first, then on whatever remains of the old active. Paths assume the broker data directory is /opt/activemq/data; adjust for your layout.

# Is anything listening on the client ports, on either node?
ss -tlnp | grep -E '61616|5672|61613'

# Is the broker process alive on the standby?
pgrep -af activemq

# Inspect the lock file in the KahaDB directory
ls -la /opt/activemq/data/kahadb/
stat /opt/activemq/data/kahadb/lock

# What is the standby doing about the lock?
grep -i "lock" /opt/activemq/data/activemq.log | tail -30

# Is the shared volume mounted and how is it mounted?
mount | grep -i -E 'nfs|kahadb|activemq'
df -h /opt/activemq/data/kahadb/

# Which block device backs the store, and how is its latency?
iostat -x 1 5

Two checks that write, both safe:

• Write test on the shared mount from the standby: create and remove a small file in the KahaDB directory (for example touch /opt/activemq/data/kahadb/.writetest && rm /opt/activemq/data/kahadb/.writetest). If this hangs or fails, the mount is your problem, not the lock. Do not touch the lock file itself.
• Client-side view: from a client host, attempt a TCP connect to both brokers’ transport ports. Confirming “nothing is serving anywhere” frames the incident correctly.

How to diagnose it

1. Confirm the old active is truly dead. Check the old host directly: process gone, JVM gone, ideally the whole host down. If the old broker is alive in any form, it may still hold the lock, and the standby is behaving correctly. Do not force anything while this is ambiguous. An old master that comes back to life after you force a takeover is how you get two writers on one store.

2. Confirm the standby is alive and in the lock-wait state. Process running, no transport connectors listening, and broker log lines showing it attempting or waiting on the store lock. The standby typically does not register the full broker MBeans, so JMX going quiet on the standby is expected, not a second failure.

3. Check the mount before the lock. Run the write test above. If basic I/O on the KahaDB path hangs or errors, the lock is a symptom. Diagnose the storage layer (SAN path state, NFS server health, network to the filer) instead of the broker.

4. Check NFS lock infrastructure if the store is on NFS. The Shared File Locker relies on OS-level file locking. On NFS that means the lock manager (rpc.statd and the NFS lock daemon) must be healthy on both client and server. Apache’s documentation calls out that on NFSv3, if the client holding a lock terminates abnormally, the server may never release that lock. Verify the lock manager state and whether the NFS server still lists the crashed client as the lock holder.

5. Check storage latency. iostat -x on the device backing the store. Sustained high await/w_await (tens of milliseconds or worse) means lock operations and journal fsyncs are crawling. On NFS this includes network round-trips. High latency can make failover look stuck when it is merely glacial.

6. Decide between wait, restart, or manual lock removal. If the mount is healthy, the old active is verifiably dead, and the lock file is stale, restart the standby broker process first (least invasive). Only if that does not clear it consider removing the stale lock file manually. That is a destructive step: if any broker anywhere still holds or can reacquire that lock, deleting the file opens the door to dual-master and KahaDB corruption. Triple-check step 1 first.

7. After takeover, verify the full chain. Standby acquires the lock, transport connectors open, clients reconnect, and store recovery (journal replay after an unclean shutdown) completes. Large stores can take many minutes to recover before the broker accepts traffic; the port may open before the broker is ready to serve.

Metrics and signals to monitor

Do not alert on a healthy standby’s down connectors; that is the expected waiting state. Do alert when no broker holds the lock after an active failure, and page immediately if two brokers ever appear active at once.

Fixes

Stale lock file after a crash

Verify beyond doubt that the old active cannot write to the store (process dead, host down, or the host fenced from the shared volume). Restart the standby broker process; re-entering the acquisition path often clears the wait. If the lock file itself is stale and blocks acquisition, remove it manually, then let the standby acquire. Destructive: deleting the lock file while any other broker holds or can grab it creates a dual-master scenario and risks store corruption. If you cannot guarantee exclusivity, fence the old host at the storage or network layer first.

NFS lock daemon or NFSv3 stale lock

Restart or recover the lock manager on the client, and check the NFS server for a lock still registered to the crashed client. On NFSv3, such a lock may not be released on abnormal client termination, so recovery can require intervention on the NFS server side rather than anything on the broker hosts. If NFS lock reliability is a recurring theme in your incident reviews, change the design (see Prevention) rather than getting faster at manual recovery.

SAN mount unavailable

Restore the mount on the standby: fix the path, rescan, remount per your storage vendor’s runbook. The broker does not need to change; once the KahaDB path is healthy and the lock is free, the standby proceeds on its own poll cycle (default every 10 seconds) or after a restart.

Standby stuck with a healthy mount and a free lock

Restart the standby broker. Watch the log for lock acquisition and for KahaDB recovery progress. If the standby still cannot acquire the lock with the old active verifiably dead and the mount healthy, suspect the filesystem’s locking semantics (clustered filesystems such as OCFS2 are documented to not support the locking mechanism Java uses) and move to a locker that does not depend on OS file locking.

Prevention

• Monitor the lock state, not just the ports. Track HA role per node and alert on “no lock holder after active failure” and “two actives”, never on standby connectors being down.
• Set a keep-alive on the Shared File Locker. With lockKeepAlivePeriod at its default of 0, the master does not verify it still holds the lock; a value significantly lower than lockAcquireSleepInterval (at most half) lets a master detect a lost lock instead of silently continuing as a second writer.
• Use NFS mount options suited to failover. hard mounts retry indefinitely and can delay failure detection by many minutes. Tune timeo/retrans and understand the tradeoff between soft mounts (faster failure, corruption risk) and hard mounts (safer writes, slower failover) with your storage vendor.
• Consider the Lease Database Locker. It replaces OS-level file locking with a short-lived JDBC lease, sidestepping NFS lock daemon failure modes entirely. It requires NTP-synchronized clocks across brokers, because lease expiry decisions are time-based.
• Avoid storage that does not support Java file locking semantics. OCFS2 is a documented problem case in the shared-storage documentation (it does not honor Java file locking, so two brokers can both believe they are master), and Windows clients using a UNC path over the Microsoft NFS client are a known field problem (the lock operation is not supported on that combination). Validate locking behavior before you build HA on a shared filesystem.
• Rehearse failover. Periodically kill the active in a controlled window and time how long until the standby serves. This surfaces stale-lock and lock-daemon problems before a real 3 a.m. crash does.
• Plan for split-brain containment. Fencing (STONITH, storage-level fencing, or network isolation of the old active) turns the dangerous ambiguity of “is the old master really gone” into a guarantee.

How Netdata helps

• Per-node process and port visibility distinguishes “standby waiting for the lock” from “standby dead” without logging into either host, and shows the moment the new active’s connectors open.
• Block device latency and mount-level disk metrics on the shared volume surface the storage-side causes (NFS latency spikes, SAN degradation) that make lock acquisition stall.
• Disk free on the KahaDB partition catches the adjacent failure where the shared volume fills and breaks both brokers.
• Correlating role, connector accept rate, and enqueue/dequeue on one dashboard shortens the key incident question: is any broker actually serving clients right now?
• Alert conditions can encode the correct HA semantics: alert when zero brokers serve after a failure, stay quiet about a healthy standby’s closed connectors.

Related guides

• ActiveMQ broker down: telling a crashed broker from a hung one
• ActiveMQ broker won’t start: port conflicts, store recovery, and lock contention
• ActiveMQ InactivityIOException: Channel was inactive for too long
• ActiveMQ connection and session leak: clients that never close
• ActiveMQ consumers connected but not acknowledging: the zombie consumer
• ActiveMQ disk full on the KahaDB partition: write failures and store corruption risk
• ActiveMQ.DLQ growing: dead letter queue accumulation and poison messages
• ActiveMQ DLQ never expires: setting TTL so the dead-letter queue stops leaking storage
• ActiveMQ offline durable subscriber pending messages: the silent storage leak
• ActiveMQ enqueue outpacing dequeue: reading the rate imbalance before the backlog
• ActiveMQ expired message count climbing: TTL expiry and silent correctness loss
• ActiveMQ GC pause death spiral: long pauses, heartbeat timeouts, and reconnect storms