ZooKeeper split-brain: two nodes both reporting leader

Your monitoring polls each ZooKeeper node independently and two of them return zk_server_state leader. Two servers in the same ensemble both believe they are the active leader. Treat this as an unconditional page: each side can accept writes that diverge from the other.

ZAB (ZooKeeper Atomic Broadcast) is designed to make sustained split-brain impossible. A leader only becomes durable after a quorum of followers (floor(N/2)+1) has acknowledged the NEW_LEADER proposal. In a clean partition, the minority side cannot reach quorum and must block writes. So when two nodes both report leader for more than a brief convergence window, something has broken the quorum accounting: an asymmetric partition, a disk-full recovery bug, an authentication bypass, or a stale-read protocol corner case.

Polling each node independently is exactly the right way to catch this. ZAB normally resolves a transient double-leader within seconds, but divergent state written during that window can persist. Any sustained occurrence is a data-divergence risk, and reconciliation must happen from the majority side.

What this means

In a healthy ensemble of N voting members, exactly one node reports zk_server_state leader at any time. The rest report follower (or observer for non-voting members). Quorum size is floor(N/2)+1: 2 for a 3-node ensemble, 3 for a 5-node, 4 for a 7-node.

A sustained double-leader means both sides believe they have quorum but are not communicating. In a clean partition this cannot persist: the smaller side cannot reach quorum and stays in LOOKING state or drops to read-only. The cases where it does persist are dangerous because both sides continue to accept writes against divergent state.

The common shape is transient: ZAB converges within seconds, but a monitoring poll catches the overlap. The dangerous shape is sustained, where a bug or misconfiguration lets two leaders coexist long enough for clients to write divergent transactions.

flowchart TD
  A[Two nodes report
zk_server_state leader] --> B{Persists over 60s?}
  B -->|No| C[Brief convergence
or rolling restart]
  B -->|Yes| D[Unconditional PAGE
data divergence risk]
  D --> E[Find majority side
by zxid and synced_followers]
  E --> F[Quarantine minority
rebuild from majority snapshot]

Common causes

Quick checks

Run these read-only. None mutate ensemble state.

# Poll every node for server state. Two leaders = split-brain.
for host in zk1 zk2 zk3 zk4 zk5; do
  echo -n "$host: "
  echo mntr | nc -w 2 $host 2181 | grep '^zk_server_state'
done

# Cross-check with stat. "Mode: leader" should appear exactly once.
for host in zk1 zk2 zk3 zk4 zk5; do
  echo -n "$host: "
  echo stat | nc -w 2 $host 2181 | grep '^Mode'
done

# Check read-only state. A real leader returns rw; a stale leader may return ro.
for host in zk1 zk2 zk3 zk4 zk5; do
  echo -n "$host: "
  echo isro | nc -w 2 $host 2181
done

# Compare zxids across all nodes. Diverged zxids on the two "leaders" confirms split-brain.
# <!-- TODO: verify zk_zxid is available via mntr; some versions may require stat or JMX -->
for host in zk1 zk2 zk3 zk4 zk5; do
  echo -n "$host: "
  echo mntr | nc -w 2 $host 2181 | grep '^zk_zxid'
done

# On each candidate leader, how many followers are synced? Real leader has quorum.
echo mntr | nc zk_candidate 2181 | grep -E 'zk_followers|zk_synced_followers|zk_pending_syncs'

# Confirm mntr is actually allowed. 3.5.3+ blocks it unless whitelisted.
echo mntr | nc zk1 2181 | head -3
# "mntr is not executed because it is not in the whitelist" means the check is broken, not the cluster.

# Check disk space on the transaction log volume of every node.
# Path comes from dataLogDir (or dataDir if dataLogDir is unset) in zoo.cfg.
ssh zk1 'df -h /var/lib/zookeeper/txnlog'

# Check uptime to rule out cold-start noise. Suppress alerts if under 600s.
# <!-- TODO: verify zk_uptime is available via mntr in the ZK versions in use; added in 3.6.0 -->
for host in zk1 zk2 zk3 zk4 zk5; do
  echo -n "$host uptime: "
  echo mntr | nc -w 2 $host 2181 | grep '^zk_uptime'
done

# Check election-time connectivity. Both quorum ports must work.
# Port values come from the server.N lines in zoo.cfg (host:quorum_port:election_port).
nc -zv zk_peer QUORUM_PORT
nc -zv zk_peer ELECTION_PORT

If mntr returns nothing, your 4lw.commands.whitelist does not include mntr. The check is silently broken, not the cluster. Whitelist at least mntr, stat, isro, and ruok before continuing.

How to diagnose it

1. Confirm the split-brain is real, not a polling artifact. Poll every node three times, two seconds apart. If two nodes consistently return zk_server_state leader across all three polls, the condition is sustained. A single-poll overlap that resolves on the next poll is convergence noise.

2. Determine which side has actual quorum. On each candidate leader, read zk_followers and zk_synced_followers. The real leader has at least floor(N/2) synced followers. The minority “leader” either has fewer followers or none, because the other side is not talking to it.

3. Compare zxids across all nodes. The majority side should have the higher zxid (more recent epoch and counter). A lower zxid on a node claiming to be leader means it is operating on stale state. Epoch (upper 32 bits of zxid) takes priority over counter: a higher epoch with a lower counter still wins.

4. Look for the trigger. Check election activity on every node for recent election storms . Cross-reference with host-level signals: network partitions (inter-node ping, security group changes), disk-full events on dataLogDir, GC pause spikes , and any recent deployment or scaling event that added nodes quickly.

5. Check for data divergence. On every node, read these counters. Any non-zero delta is a data integrity violation and confirms the divergence is not just role confusion but actual state corruption.

6. Identify divergent writes. Compare the children and data of critical znodes (Kafka broker registrations, HBase region assignments, distributed locks) between the two sides. Application logs of dependent services usually surface which side accepted writes that the other did not.

Metrics and signals to monitor

Fixes

If it is a monitoring artifact

If the double-leader clears within one or two poll intervals and election activity only spikes once, ZAB converged normally. This is common during rolling restarts and brief network blips. Suppress the alert during planned maintenance and tighten the polling window so you only page on sustained conditions (more than 60 seconds).

Asymmetric partition or firewall

Restore connectivity on both quorum ports. ZooKeeper uses two inter-server ports: one for follower-to-leader communication and one for leader election. A firewall that blocks only the election port is invisible until the next election, then catastrophic. Once connectivity is restored, ZAB re-converges: the side with quorum retains leadership, the minority side drops to follower. Verify by re-polling zk_server_state across all nodes and confirming zk_synced_followers returns to N-1.

Disk-full recovery

When a node fills its transaction log disk and recovers with a stale snapshot, it can diverge from the ensemble after rejoining. The fix is to rebuild the affected node from the majority side.

WARNING: Destructive. The following deletes all local ZooKeeper state on the target node. Back up the data directory first if you need it for forensics.

# Stop ZooKeeper on the divergent node.
systemctl stop zookeeper  # or equivalent for your deployment

# Back up for forensics if needed.
cp -a /var/lib/zookeeper/version-2 /tmp/zk-forensics-$(date +%s)

# Delete the contents of version-2 (log.* and snapshot.* files).
# Path comes from dataDir and dataLogDir in zoo.cfg.
rm -rf /var/lib/zookeeper/version-2/*

# Restart. The node performs a full SNAP sync from the leader.
systemctl start zookeeper

Stale-read protocol corner case

The ZooKeeper documentation notes that “two servers simultaneously think they are the leader” can occur if the TCP connection timeout is smaller than syncLimit x tickTime. In this state, a sync followed by a read can return stale data from the false leader. Mitigation: tune TCP keepalive and retransmit timeouts so dead connections are detected within the sync window, and keep syncLimit x tickTime comfortably above the expected network failure detection time.

SASL quorum auth bypass (CVE-2023-44981)

If you run with quorum.auth.enableSasl=true and an endpoint whose SASL authentication ID lacks a fully qualified domain name has joined the ensemble, the authorization check can be bypassed and the leader can be modified. Patch to a fixed version. Rotate quorum credentials, audit ensemble membership, and rebuild any node that may have accepted unauthorized proposals.

Reconciling divergent state

When divergence is confirmed, the only safe path is to pick the majority side as the source of truth and rebuild the minority side from it.

WARNING: Data loss. Any writes accepted by the minority side are lost. Notify dependent service owners (Kafka, HBase, Solr) before bringing reconciled nodes back, because ephemeral nodes and watch notifications will fire as clients reconnect.

Stop the minority nodes, wipe their data directories (same procedure as disk-full recovery above), and let them SNAP sync from the majority leader. Replay lost writes manually if they are recoverable from application logs.

Prevention

• Poll every node independently for zk_server_state. Aggregated “is there a leader” checks hide split-brain. The check that catches it is the one that compares per-node state.
• Alert on more than one leader unconditionally. No baseline, no rate, no smoothing. The condition is binary.
• Gate the alert on uptime over 600s. Cold-start elections produce transient double-leaders during cluster-wide restarts.
• Monitor both quorum ports. Reachability on the follower-to-leader port does not imply reachability on the election port.
• Treat dataLogDir disk space as a page, not a ticket. Disk-full on the transaction log volume is the root cause of the disk-full recovery split-brain pattern.
• Keep ZooKeeper patched. CVE-2023-44981 and the disk-full recovery fix are both reasons to be on a current 3.8.x or 3.9.x line.
• Test failover and partition recovery. Controlled chaos exercises that split the ensemble are the only way to validate that monitoring catches the condition and that the reconciliation procedure works.
• Whitelist mntr, stat, isro, and ruok in 4lw.commands.whitelist. A silently blocked mntr returns empty output that monitoring systems often interpret as “all zeros,” masking the real state.

Correlating with Netdata

• Per-second polling of zk_server_state on every node, with per-node dashboards, makes a transient double-leader visible even when ZAB converges in seconds. A 15-second poller will miss most real split-brain windows.
• Correlating zk_server_state with zxid, zk_followers, and zk_synced_followers on a single timeline shows which side has quorum and which is stale, without switching between tools.
• Host-level signals alongside ZooKeeper metrics let you confirm root cause in one view: network throughput and TCP retransmits on inter-node links, disk latency and space on dataLogDir, JVM pause time, and file descriptor pressure.
• Cold-start suppression via zk_uptime gating prevents false pages during rolling restarts while still paging on the real condition.

Related guides

• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper quorum loss: no leader elected and every write is failing