ZooKeeper quorum loss: no leader elected and every write is failing

Every write to your ZooKeeper ensemble is timing out. Clients report ConnectionLoss and SessionExpired. Downstream systems that depend on ZK for coordination, such as Kafka controller elections or HBase region assignment, are cascading into failure. On the surviving ZK nodes, ruok still returns imok. The process is alive; the ensemble is not.

Quorum loss is ZooKeeper’s worst-case availability scenario. When fewer than floor(N/2)+1 voting members can communicate, no leader can be elected and every write fails. Surviving nodes sit in LOOKING state, unable to make progress through ZAB.

First response priority: count reachable nodes, check for a network partition, and restore enough members to form quorum. Do not blindly restart survivors. A rolling restart of an already-degraded ensemble can destroy the only in-memory copy of recent transactions.

What this means

Quorum is floor(N/2)+1 voting members. For a 3-node ensemble, quorum is 2; for 5 nodes, 3; for 7 nodes, 4. Observers do not count toward quorum. When the number of reachable, in-sync voting members drops below the threshold, FastLeaderElection cannot complete (FastLeaderElection has been the only election algorithm since ZK 3.6.0). Surviving members stay in LOOKING and ZAB cannot make progress.

User-visible symptoms during confirmed quorum loss:

• Every write request returns ConnectionLoss or times out.
• Reads may or may not work, depending on readonlymode.enabled (default false).
• No node reports leader via mntr or srvr.
• zk_sum_leader_unavailable_time is non-zero and growing.
• zk_looking_count may keep incrementing as members retry election.

Suggested page rule: page when no leader exists for more than 60 seconds AND ensemble_size > 1 AND majority of nodes have uptime greater than 600 seconds (this rules out a cold full-cluster start). Split-brain, more than one node reporting leader simultaneously, pages unconditionally because it means data divergence.

Critical nuance: ruok is shallow. It confirms the JVM is alive and the client port is open. A node in LOOKING with no quorum still returns imok. Do not rely on ruok for quorum health. Pair it with isro and mntr.

isro has its own trap. The command returns ro only when readonlymode.enabled=true and the node is actually serving stale reads. With the default readonlymode.enabled=false, surviving nodes do not enter read-only mode at all during quorum loss. Treat isro as a reliable quorum-loss signal only when you have explicitly enabled read-only mode.

flowchart TD
  A[Healthy ensemble
1 leader, N-1 followers] --> B{floor N/2 +1
reachable?}
  B -- No --> C[Surviving nodes
enter LOOKING]
  C --> D[No leader elected
all writes fail]
  B -- Yes --> E[New leader elected
within seconds]
  D --> F{readonlymode
enabled?}
  F -- true --> G[isro returns ro
stale reads served]
  F -- false --> H[Node refuses
client requests]

Common causes

Quick checks

Run these on each surviving member. All are read-only.

# Check server state on every node - is any 'leader' present?
echo mntr | nc localhost 2181 | grep zk_server_state

# Read-only mode status. 'ro' only when readonlymode.enabled=true
echo isro | nc localhost 2181

# Shallow liveness - returns 'imok' even during quorum loss
echo ruok | nc localhost 2181

# Server mode from srvr (alternative path if mntr is not whitelisted)
echo srvr | nc localhost 2181 | grep Mode

# Leader-only: how many followers are connected and synced?
echo mntr | nc localhost 2181 | grep -E 'zk_(followers|synced_followers|pending_syncs)'

# Compare zxids across members - divergent zxids signal partition or lag
echo mntr | nc localhost 2181 | grep zk_zxid

# Uptime - gates cold-start false positives
echo mntr | nc localhost 2181 | grep zk_uptime

# Leader election history - increments on each LOOKING entry
echo mntr | nc localhost 2181 | grep zk_looking_count

# Cumulative write unavailability
echo mntr | nc localhost 2181 | grep -E 'zk_.*leader_unavailable_time'

If mntr returns nothing, your 4lw.commands.whitelist probably does not include it. Since ZK 3.5.3, only srvr is whitelisted by default. Add mntr and isro to the whitelist in zoo.cfg, otherwise monitoring silently reports zeros.

Network reachability between members:

# Probe both inter-server ports to each peer
nc -zv peer-hostname 2888
nc -zv peer-hostname 3888

Both ports must be reachable in both directions. A firewall that blocks only the election port will not be noticed until the next leader election, at which point it is catastrophic.

How to diagnose it

1. Confirm quorum loss. Run mntr against every member. Count nodes that respond. If fewer than floor(N/2)+1 respond, or if every responding node reports no leader for more than 60 seconds, quorum loss is confirmed.
2. Classify unreachable nodes. For each non-responding member, determine whether the JVM is alive (ps -f QuorumPeerMain), the client port is closed (nc -zv host 2181), or only the election port is unreachable. The classification drives the recovery action.
3. Check for a network partition. From each surviving member, probe both inter-server ports on every peer. Cloud environments are particularly prone to silent security-group changes that block one direction or one port.
4. Check uptime to rule out cold start. If majority uptime is under 600 seconds, the ensemble may still be converging after a coordinated restart. Wait two minutes before declaring quorum loss.
5. Check for split-brain. If more than one node reports leader, treat as a partitioned-brain condition. Page unconditionally and treat as a data integrity risk.
6. Check leader election churn. zk_looking_count incrementing repeatedly means members are stuck in election loops. Common causes: TCP proxy timing out on election traffic, asymmetric connectivity, or members unable to persist votes due to disk problems.
7. Check correlated failures. Did all members of one availability zone, rack, or storage tier fail simultaneously? Correlated failures point to infrastructure, not application, root cause.
8. Check leader-side replication metrics once a leader exists. zk_followers and zk_synced_followers only appear on the leader. If they are absent, no leader has been elected yet.

Metrics and signals to monitor

Leader-only metrics (zk_followers, zk_synced_followers, zk_pending_syncs) are invisible on followers. Monitoring that queries only one node will never see them. Either identify and query the current leader, or query all members and filter for the leader-reported values.

Fixes

Recovery follows one principle: restore enough voting members to form quorum. How you do that depends on the cause.

Network partition

If a partition splits the ensemble, restore connectivity. Once floor(N/2)+1 members can reach each other on both inter-server ports, FastLeaderElection completes within seconds and writes resume. Verify with mntr showing exactly one leader and isro returning rw on the leader.

Cloud action: audit security groups, network ACLs, route tables. On-prem: check switch fabric, firewall rules, and recent change control. The most common cause in cloud environments is a silent security-group change that blocks inter-server traffic.

Multiple node failures

Restart failed members one at a time, in the order they failed if known. Wait for each to rejoin and sync before restarting the next. Do not restart all members simultaneously; you risk losing the in-memory state on the only node with the most recent committed transactions.

If a member has a corrupt transaction log or snapshot, it cannot rejoin cleanly. Inspect the logs for snapshot errors or transaction-log IOException. You may need to rebuild the member from a healthy snapshot plus transaction log replay.

Dynamic reconfiguration will not rescue you

A frequent operator instinct is to shrink the ensemble via dynamic reconfiguration to exclude failed members. This does not work during quorum loss. Dynamic reconfiguration requires a quorum of the old configuration to make progress. You need quorum to change quorum. If three of five members are permanently gone, dynamic reconfig cannot reduce the ensemble to three. You must either restore enough members or rebuild from a known-good snapshot and transaction log with a reconfigured zoo.cfg.

Read-only mode trap

If readonlymode.enabled=true, surviving nodes serve stale reads while writes fail. This is dangerous because dependent services may appear healthy while losing writes. Kafka leader election, HBase region assignment, and any system using ephemeral nodes for liveness will fail silently. Confirm write recovery explicitly before declaring the incident resolved.

# Verify writes work after recovery - run from a client or zkCli.sh
create /quorum-recovery-test "ok"
get /quorum-recovery-test
delete /quorum-recovery-test

Do not blindly restart survivors

A surviving member in LOOKING state still holds its in-memory data tree and the transaction log on disk. If it is the most-recently-updated survivor, its state is your recovery source. Restarting it discards in-memory state and forces recovery from the on-disk snapshot and logs, which may be older than what was in memory. Always exhaust connectivity and configuration causes before restarting any survivor.

Prevention

• Monitor zk_server_state across the whole ensemble. Page when no leader exists for more than 60 seconds outside cold-start windows.
• Alert on zk_looking_count increments outside maintenance. Every unplanned election is an incident worth investigating.
• Track zk_sum_leader_unavailable_time deltas. Any non-zero delta is a direct measurement of write unavailability.
• Watch zk_synced_followers on the leader. Alert at floor(N/2), which means one more failure breaks quorum.
• Probe both inter-server ports in monitoring, not just the client port. Election port reachability is invisible until you need it.
• Run chaos exercises. Kill one member in production regularly. Validate that monitoring catches it and that the ensemble re-elects within seconds.
• Use odd-sized ensembles. Even-sized ensembles waste a member without adding fault tolerance, and a half-split partition produces two minorities.
• Pin the ZK version and track CVEs. Recent releases fix serious quorum-security issues. CVE-2023-44981 (SASL quorum peer auth bypass, fixed in 3.9.1/3.8.3/3.7.2) and CVE-2024-23944 (persistent watcher ACL bypass, fixed in 3.9.2/3.8.4) both affect cluster integrity.
• Document the recovery procedure before you need it. Quorum loss at 3 a.m. is the wrong time to learn that dynamic reconfig cannot help.

How Netdata helps

• Per-second zk_server_state collection across every ensemble member surfaces leader absence within seconds, faster than typical 30 to 60 second scrape intervals.
• zk_sum_leader_unavailable_time deltas directly measure write unavailability, so you alert on impact rather than proxy signals.
• Leader-only metrics (zk_followers, zk_synced_followers, zk_pending_syncs) are collected from whichever node is currently leader, eliminating the gap left by static monitoring configs that query a fixed node.
• Correlation with host-level network, disk, and CPU metrics on the same timeline distinguishes quorum loss caused by a network partition from quorum loss caused by disk stalls or GC pauses.
• ML anomaly detection on zk_looking_count, zk_uptime, and connection counts flags election churn and restart cascades before they become full outages.
• Cold-start suppression via zk_uptime gating prevents false pages during planned rolling restarts while still paging on real quorum loss once majority uptime crosses the threshold.

Related guides

• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper leader election storm: an ensemble that keeps re-electing
• ZooKeeper unexpected leader election: finding why the leader dropped
• ZooKeeper split-brain: two nodes both reporting leader
• ZooKeeper server stuck in LOOKING: a node that never rejoins the quorum
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls
• ZooKeeper write latency high: read zk_updatelatency, not just avg_latency
• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper quorum ack latency high: followers slow to acknowledge proposals