ActiveMQ shared-storage HA split-brain: two brokers holding the store lock

In a shared-storage HA pair, exactly one ActiveMQ Classic broker is supposed to hold the KahaDB store lock and run active. The second broker sits in a polling loop, waiting for the lock, with its transport connectors down. Split-brain is the state where that invariant breaks: both brokers believe they are active, both accept client connections, and both write to the same KahaDB store on shared storage.

Two writers on one store means message duplication, ordering violations, and real risk of journal and index corruption. Clients connected to the “wrong” broker get messages the other broker also delivers. Recovery is not just “restart one node”: you have to decide which broker’s view of the store is authoritative, and the store itself may already be damaged.

The trigger is almost always the storage layer, not ActiveMQ. NFS lock-manager edge cases, NFSv4 lease and grace-period races, a filesystem that does not implement POSIX locking correctly (OCFS2 is the classic offender), or a misconfigured SAN can all let the standby acquire a lock the active broker still believes it holds. NFS-based locking in particular is notoriously unreliable for this purpose.

This article covers how to confirm split-brain fast, how to contain it without making the corruption worse, the root causes worth checking, and how to keep it from recurring.

What this means

The shared file locker works like this: the first broker to start takes an exclusive file lock on the lock file in the KahaDB directory (using Java’s FileLock, which maps to OS-level advisory locking). That broker becomes active, starts its transport connectors, and serves clients. The second broker blocks in a retry loop, periodically attempting to acquire the same lock. When the active broker shuts down cleanly or crashes and the OS releases the lock, the standby acquires it and promotes.

Split-brain happens when the lock mechanism lies. The standby’s lock acquisition succeeds even though the active broker is still running and writing. From that point:

• Both brokers accept producer and consumer connections.
• Both write journal files (db-*.log) and update the index (db.data) in the same directory.
• Producers may be load-balanced or failed over onto either broker, so the same logical message stream is written twice with interleaved journal state.
• KahaDB’s index, which assumes a single writer, can end up pointing at journal entries written by either broker, or at overwritten state.

flowchart TD
  A[Storage disruption: NFS outage, partition, lock daemon failure] --> B[Standby retries lock acquisition]
  B --> C{Does the lock actually exclude the active broker?}
  C -->|Yes: normal failover| D[Standby promotes, old active is down]
  C -->|No: lock mechanism lies| E[Both brokers hold the lock]
  E --> F[Both accept connections and write KahaDB]
  F --> G[Duplicate delivery and ordering violations]
  F --> H[Journal and index corruption]
  H --> I[Broker may fail to restart after containment]

The dangerous property of this failure is that both brokers can look healthy in isolation. Each is accepting connections, enqueuing, and dequeuing. The failure is only visible when you look at the pair, at the lock state, or at client-level symptoms like duplicate deliveries.

Common causes

Quick checks

All of these are read-only. Run them on both brokers before changing anything.

# 1. Which brokers are actually serving client traffic?
#    In a healthy pair, only one should have the transport port listening.
ss -tlnp | grep 61616

# 2. Does the broker MBean exist? Standby brokers typically do not
#    register the full MBean tree while waiting for the lock.
curl -s -u admin:admin \
  'http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/BrokerId'

# 3. Search both broker logs for lock acquisition and release events.
grep -i "lock" /opt/activemq/data/activemq.log | tail -50

# 4. What filesystem is the KahaDB directory on?
df -T /opt/activemq/data/kahadb/
mount | grep -E "nfs|ocfs|gfs|cifs"

# 5. Who owns the lock file right now, and when was it touched?
ls -l /opt/activemq/data/kahadb/lock

# 6. Is the store already showing damage (journal growth from two writers)?
ls -lh /opt/activemq/data/kahadb/

# 7. Storage latency on the store device. For NFS mounts use nfsiostat
#    (ships with nfs-utils); iostat only covers block devices.
nfsiostat 1 5

A few notes on interpretation:

• If both hosts answer on the transport port and both return a BrokerId, you have split-brain (or someone started two independent brokers against one store, which is the same problem).
• df -T telling you the store is on OCFS2 or CIFS is itself a finding: the shared file locker does not work correctly on those filesystems. OCFS2 only supports fcntl-style locking, which is incompatible with Java’s FileLock, so both brokers can believe they hold the lock. CIFS/SMB is not supported for this use.
• Compare lock acquisition timestamps in the two broker logs against your storage or network event timeline. The classic NFSv4 signature is: storage outage, standby promotes during the outage, storage heals, old active resumes writing without realizing it lost the lock.

How to diagnose it

1. Confirm both brokers are active. Use checks 1 and 2 above on both hosts. One active and one waiting is normal; two active is the incident. If only one is active but clients report duplicates, check for a second broker process on the same host or an orphaned process from a failed restart.

2. Establish the timeline. Pull lock acquisition and release events from both broker logs and line them up with NFS server logs, network device logs, or hypervisor events. You are looking for the window where the standby acquired the lock while the active was still running. This tells you the mechanism (outage race vs. bad filesystem vs. bad config) and which broker has been writing longer.

3. Check the locker configuration. In activemq.xml, find the persistence adapter’s locker settings. If lockKeepAlivePeriod is 0 (or unset on versions where that means disabled; it is not applicable before 5.9.0), the active broker never revalidates its lock and will never demote itself. The keep-alive mechanism exists precisely to close the “I still think I am master” window.

4. Verify the filesystem actually enforces exclusive locks. If the store is on NFS, confirm it is NFSv4 (not v3, which has the opposite problem: locks survive abnormal client death and can block failover forever). If it is a cluster filesystem, confirm it is GFS2 or another POSIX-locking-correct option, not OCFS2. If it is a SAN LUN, confirm with your storage team that fencing is enforced.

5. Assess store damage. Look at journal file count and modification times in the KahaDB directory. Interleaved modification times across recent journal files are consistent with two writers. The definitive test comes later: whether the surviving broker starts cleanly and passes KahaDB recovery without errors.

6. Assess client impact. Ask the application teams whether they saw duplicate deliveries or out-of-order processing during the window. This determines whether you need application-level deduplication or replay cleanup after the broker side is fixed.

Metrics and signals to monitor

Both-active detection is a PAGE. Store corruption risk makes this one of the few conditions where waking someone up at 3 a.m. is always right. Conversely, do not page on the standby broker’s transport connectors being down: that is the expected state.

Fixes

Immediate containment: stop one broker

This is disruptive and there is no way around it. While both brokers run, the store keeps getting worse. Pick a winner and stop the loser.

• Prefer keeping the broker that acquired the lock first in the timeline you built in diagnosis, unless it is clearly unhealthy. Its journal history is more likely to be the coherent one.
• Stop the losing broker with a normal shutdown first. Reserve kill -9 for a broker that will not stop; an unclean shutdown adds recovery work on the next start.
• Before restarting anything, take a copy of the entire KahaDB directory if disk allows. If recovery later destroys evidence or loses messages, you will want the pre-incident state.

Do not let the stopped broker start again automatically until the root cause is fixed. Disable the service or systemd auto-restart on that node temporarily, or it may re-acquire the “lock” and recreate the split-brain.

Repair and restart the survivor

Start the surviving broker alone and watch the log through KahaDB recovery. Journal replay and index rebuild after an unclean or contested state can take minutes to hours on a large store, and the transport ports may be open before the broker accepts client connections. Do not declare recovery until a canary send and receive succeeds.

If the broker refuses to start or reports journal corruption, the ignoreMissingJournalfiles=true and checkForCorruptJournalFiles=true options can get it up, but they can lose messages. Treat that as a data-loss decision, not a routine flag.

Fix the locking configuration

If you must stay on shared-file locking, enable lock keep-alive so the active broker periodically revalidates its lock file and demotes itself if the file was lost or modified:

<persistenceAdapter>
  <kahaDB directory="/shared/kahadb">
    <locker>
      <shared-file-locker lockAcquireSleepInterval="10000" lockKeepAlivePeriod="5000"/>
    </locker>
  </kahaDB>
</persistenceAdapter>

Two rules from the AMQ-5549 discussion: lockKeepAlivePeriod of 0 disables the protection entirely, and the keep-alive period must be shorter than the acquire sleep interval (at most half is the recommended relationship), or the detection logic cannot work as intended.

Fix the storage layer

• NFS mount options matter. The AMQ-5549 testing suggests aggressive timeouts so the client notices storage failure quickly: timeo=100,retrans=1,soft,noac. Be aware that soft mounts trade lock reliability for failure detection and can surface I/O errors to the broker mid-write; understand that tradeoff before adopting it.
• Move off NFS if you can. NFS-based locking for this purpose is notoriously unreliable, and the JIRA record shows no mount-option combination that fully eliminated the dual-active window. A SAN LUN with proper fencing, or GFS2 for a cluster filesystem, is the supported-grade alternative. Do not use OCFS2 or CIFS/SMB.
• Fix NFSv3 specifically if that is what you have: locks are not released on abnormal client termination, which produces the mirror-image failure where failover never happens. Recovery from a stuck NFSv3 lock can require restarting the affected ActiveMQ instances.

Consider a different locker or topology

If you are on ActiveMQ Artemis rather than Classic, the internals differ: split-brain fixes such as ARTEMIS-4143 land in specific versions (2.29.0 and later for that issue), Artemis has a built-in network health check that can stop a broker during a partition, and shared storage needs a filesystem with real exclusive-lock support.

Prevention

• Alert on both-active, not just broker-down. The single most important check: periodically verify that exactly one broker in the pair has its transport connectors up and its full MBean tree registered. Two is a page.
• Alert on lock events. Treat lock acquisition on the standby without a preceding clean shutdown of the active as a page-level event. That ordering is the split-brain signature.
• Monitor the storage layer as a first-class dependency. NFS/SAN latency, mount health, and storage error logs belong on the same dashboard as broker health, because the storage event always precedes the lock event.
• Test failover regularly. A planned failover in a maintenance window exercises the lock path and tells you whether your filesystem honors exclusive locking before you find out at 3 a.m.
• Keep clocks synchronized on both broker hosts and the storage server. Your forensic timeline depends on comparable timestamps.
• Know your filesystem’s locking semantics and document them in the runbook. “KahaDB is on NFS” should immediately raise the question “which NFS version, with which mount options, and why not SAN?”
• Plan the application side. Assume that any both-active window produces duplicates. Idempotent consumers or deduplication keys turn a split-brain from a correctness incident into an availability incident.

How Netdata helps

• Pair-level role visibility: tracking process, port, and JMX availability for both brokers on one dashboard makes “two actives” obvious instead of requiring someone to check each node separately.
• Storage correlation: disk latency and I/O error signals on the KahaDB device, graphed next to broker enqueue rates, surface the NFS/SAN event that precedes the lock race.
• Connection and enqueue asymmetry: nonzero client connections or enqueue rate on the node that should be standby is an early both-active indicator, often visible before clients report duplicates.
• Log-based lock events: alerting on lock acquisition and release patterns in the broker log catches the failover-ordering anomaly at the moment it happens rather than after corruption.
• Store growth signals: journal file count and disk usage on the shared partition show the abnormal dual-writer growth rate during and after an incident, which helps size the recovery effort.

Related guides

• ActiveMQ broker down: telling a crashed broker from a hung one
• ActiveMQ broker won’t start: port conflicts, store recovery, and lock contention
• ActiveMQ InactivityIOException: Channel was inactive for too long
• ActiveMQ connection and session leak: clients that never close
• ActiveMQ consumers connected but not acknowledging: the zombie consumer
• ActiveMQ disk full on the KahaDB partition: write failures and store corruption risk
• ActiveMQ.DLQ growing: dead letter queue accumulation and poison messages
• ActiveMQ DLQ never expires: setting TTL so the dead-letter queue stops leaking storage
• ActiveMQ offline durable subscriber pending messages: the silent storage leak
• ActiveMQ enqueue outpacing dequeue: reading the rate imbalance before the backlog
• ActiveMQ expired message count climbing: TTL expiry and silent correctness loss
• ActiveMQ GC pause death spiral: long pauses, heartbeat timeouts, and reconnect storms