ZooKeeper unrecoverable error: when a node’s integrity is compromised

You got paged because zk_unrecoverable_error_count incremented. This is one of the few mntr counters you never want to see move. It tracks errors ZooKeeper cannot recover from internally: data corruption, invariant violations, or resource exhaustion that puts the node into a state it cannot safely keep serving from.

ZooKeeper is designed to fail fast. When it hits an unrecoverable condition it does not limp along. The critical thread logs a severe error, notifies the supervision listener, and the process exits. By the time the counter increments, the JVM is usually already gone or about to be, and the question is no longer “is this node healthy” but “do I trust its on-disk state at all”.

This guide covers first-five-minute checks, correlated integrity metrics, and the decision between a clean restart and a full rebuild from a trusted snapshot.

What this means

zk_unrecoverable_error_count is exposed by mntr and counts critical internal failures since process start. Any increment pages because the integrity of the node’s data tree may be compromised: the server may have served incorrect data, written a partial transaction, or hit an internal invariant violation.

Three things are true the moment this counter increments:

  1. The ZooKeeper process is either dead or about to exit. Fail-fast is intentional, not a bug.
  2. The node’s on-disk snapshot and transaction log cannot be assumed trustworthy without verification.
  3. The blast radius depends on whether the rest of the ensemble still has quorum. If this was the leader or a quorum-critical follower, dependent services (Kafka controller, HBase master, Solr overseer) are about to react to leadership changes.

The triage question is whether this is an isolated node failure or a systemic integrity problem. Correlate with zk_digest_mismatches_count and zk_snapshot_error_count. Multiple error types incrementing together indicates a systemic problem, not a transient blip.

flowchart TD
  A["Disk full / IO error / perm denied"] --> B["SyncRequestProcessor catches Throwable"]
  B --> C["ZooKeeperCriticalThread.handleException"]
  C --> D["Increment zk_unrecoverable_error_count"]
  C --> E["Log severe unrecoverable error"]
  C --> F["listener.notifyStopping"]
  F --> G["Process exit"]
  G --> H["Ensemble shrinks"]
  H --> I{"Quorum lost?"}
  I -->|Yes| J["Writes unavailable ensemble-wide"]
  I -->|No| K["Followers re-elect a leader"]
  J --> L["Dependent services cascade"]
  K --> M["Single-node rebuild path"]

The recovery path forks at the bottom of that diagram. A quorum-losing event needs immediate ensemble triage; a single-node failure in a healthy ensemble follows the rebuild path described later in this guide.

Common causes

CauseWhat it looks likeFirst thing to check
Transaction log disk fullProcess exits mid-write; dataLogDir partition at 100%df -h on the txnlog partition
Disk I/O or hardware errorProcess exits with IOException in txnlog; kernel logs report medium errordmesg, iostat -x, smartctl -a
Permission denied on data directoryCommon in containerized deploys; fails at startup or snapshot rotationls -la on dataDir and dataLogDir
Digest mismatch during load or synczk_digest_mismatches_count increments alongsideCheck the same counter on every ensemble member
Snapshot write or load failurezk_snapshot_error_count increments before the crashInspect latest snapshot file size and integrity
32-bit JVM hitting file size limitProcess exits when a txnlog segment crosses ~2GBConfirm JVM architecture: java -version

The dominant production trigger is a full or failing dataLogDir. ZooKeeper fsyncs every write synchronously, so any condition that blocks or fails that fsync is a candidate.

Quick checks

Run these read-only. None modify state.

# Confirm the increment is real (run on the affected node)
echo mntr | nc localhost 2181 | grep -E 'zk_unrecoverable_error_count|zk_digest_mismatches_count|zk_snapshot_error_count|zk_restore_error_count'

# Is the process actually still alive?
echo ruok | nc localhost 2181
echo isro | nc localhost 2181
echo mntr | nc localhost 2181 | grep -E 'zk_server_state|zk_uptime'

# Disk space on both persistence partitions
df -h /var/zookeeper/txnlog /var/zookeeper/data

# Disk health and current IO pressure
iostat -x 1 5
dmesg -T | grep -iE 'error|medium|i/o' | tail -30

# Find the actual error in the ZK log
grep -iE 'unrecoverable|severe|IOException|denied|digest' /var/log/zookeeper/zookeeper.log | tail -50

# Compare integrity counters across the whole ensemble
for h in zk1 zk2 zk3; do
  echo "== $h =="
  echo mntr | nc "$h" 2181 | grep -E 'zk_unrecoverable_error_count|zk_digest_mismatches_count|zk_snapshot_error_count'
done

The ensemble-wide sweep matters. A single node incrementing only zk_unrecoverable_error_count is a different incident from three nodes all incrementing zk_digest_mismatches_count at the same zxid.

How to diagnose it

  1. Confirm the counter delta. Compare the current value against your last scrape. Absolute values are meaningless for alerting because they include historical events. Alert on increase(zk_unrecoverable_error_count) > 0.

  2. Verify process state. If ruok does not return imok, the JVM has already exited. Check zk_uptime against your last known value: a reset indicates a restart, consistent with fail-fast exit.

  3. Read the ZooKeeper log around the timestamp. Search for “Severe unrecoverable error” and the surrounding stack trace. The thread name tells you which subsystem hit the error. SyncRequestProcessor failures point at the txnlog disk; snapshot-related failures point at dataDir or memory pressure.

  4. Pull the correlated counters from every ensemble member. This is the single most important step. Three signals to correlate:

    • zk_digest_mismatches_count - data tree divergence from the expected checksum
    • zk_snapshot_error_count - errors during snapshot creation or load
    • zk_restore_error_count - errors during state restore

    If only the unrecoverable counter moved on one node, you have a localized failure. If digest mismatches moved on multiple nodes, you have an ensemble-wide integrity problem and the recovery path is different.

  5. Identify the trigger. Match the log entry to system state at that instant:

    • Disk full: df history or your disk-space metric will show the partition hit 100%.
    • I/O error: dmesg will show SCSI/ATA errors; SMART will show reallocated sectors.
    • Permission denied: recent container image change, user change, or volume mount option change.
    • Digest mismatch during follower sync: usually indicates the follower’s local state diverged, often after an unclean shutdown.

  6. Decide on the recovery path. Covered in the next section. The decision tree is: is the on-disk state trustworthy, and is the rest of the ensemble healthy?

Metrics and signals to monitor

SignalWhy it mattersWarning sign
zk_unrecoverable_error_countPrimary page signal. Any delta is catastrophic.Any increment, ever
zk_digest_mismatches_countConfirms or rules out data corruption. Moving on multiple nodes means systemic.Any increment
zk_snapshot_error_countIndicates the node cannot persist its state safely. Recovery after crash is at risk.Any increment; escalate to PAGE if unrecoverable also moves
zk_uptimeResets on process exit. Confirms the fail-fast exit happened.Sudden reset outside planned restart
zk_server_stateTells you whether the dead node was leader. A leader exit triggers election.Unplanned transition
zk_synced_followers (leader only)Tells you whether the surviving ensemble still has quorum headroom.Drops to floor(N/2)
Disk free on dataLogDirThe dominant root cause. Monitor as a leading indicator.Trend toward zero
zk_fsynctime p99Underlying write-path health. Sustained elevation precedes many crashes.Sustained p99 > 10ms

Fixes

Recovery depends on root cause and whether the on-disk state is trustworthy. Never restart a node and assume it is healthy without first verifying why it crashed.

Disk full on the transaction log partition

Clearing space is necessary but not sufficient. The process exited because fsync failed; it will not resume on its own once space is freed.

  1. Free space. Usually that means triggering autopurge manually or removing old snapshots outside the retain set. Be careful: deleting the wrong files corrupts recovery state.
  2. Restart the ZooKeeper process under your supervisor (systemd, supervisord, kubectl, etc.).
  3. Verify the node rejoins cleanly: zk_server_state should report follower (or leader after election), zk_uptime should be increasing, and zk_synced_followers on the leader should include this node.
  4. Configure autopurge.purgeInterval and autopurge.snapRetainCount if they are not set, so this does not recur.

See the dedicated guide on autopurge misconfiguration and on the txnlog disk full crash.

Disk I/O or hardware errors

If dmesg or SMART report failing storage, the on-disk state cannot be trusted even if the files look intact.

  1. Evacuate the node: remove it from the ensemble membership or stop the process.
  2. Replace or reformat the failing disk.
  3. Wipe dataDir and dataLogDir on the affected node and let it perform a full sync from the leader. Warning: this destroys the node’s local data tree; only do this when the rest of the ensemble has a trusted leader. For a large data tree this will be a SNAP sync; see the SNAP sync blast radius guide.
  4. Watch zk_follower_sync_time and confirm the node rejoins zk_synced_followers before returning it to service.

Permission denied on the data directory

Most common after a container image change, a UID rewrite, or a volume mount option change. The fix is operational, not data-related.

  1. Confirm the ZooKeeper process user owns dataDir and dataLogDir: ls -la /var/zookeeper/data /var/zookeeper/txnlog.
  2. Fix ownership: chown -R zookeeper:zookeeper /var/zookeeper/data /var/zookeeper/txnlog.
  3. Check the container spec or systemd unit for User=, fsGroup, or securityContext drift.
  4. Restart. Existing data is fine; the failure was on a new file write or snapshot rotation.

Digest mismatch or confirmed data corruption

This is the worst case. A digest mismatch means the in-memory tree diverged from the expected checksum, and clients may have been reading incorrect data.

  1. Check zk_digest_mismatches_count on every ensemble node.
  2. Single node only: remove the node from the ensemble, wipe its dataDir and dataLogDir, and let it rebuild from a healthy leader via SNAP sync. Warning: wiping is destructive; confirm you have a trusted leader before deleting local state. Do not trust its local snapshot.
  3. Multiple nodes: escalate immediately. You likely need to restore from an external backup taken before the divergence, or rebuild the ensemble from a known-good snapshot. Do not pick a “winner” among divergent nodes without understanding which copy is authoritative for your downstream services.
  4. After recovery, audit downstream consumers (Kafka, HBase, Solr) for state that may have been written based on stale or corrupt reads.

If snapshot.trust.empty is set to true on this node, revisit that decision. It allows recovery when snapshot files are missing but transaction logs exist, but misusing it during upgrades has caused znodes to vanish. It should be false in steady state.

File too large on a 32-bit JVM

If java -version reports a 32-bit JVM and the txnlog segment is approaching 2GB, migrate to a 64-bit JVM. This is a deploy-time fix, not a runtime one. Wipe the node’s data after the upgrade and let it resync.

Prevention

  • Monitor the integrity counters as a group. Alert on increase(zk_unrecoverable_error_count) > 0 and escalate severity when zk_digest_mismatches_count or zk_snapshot_error_count move in the same window.
  • Treat dataLogDir free space as a leading indicator. Disk full is the dominant root cause. Page on < 10% or < 2GB free, whichever comes first.
  • Keep dataLogDir on dedicated storage. Sharing the txnlog volume with snapshots or other workloads is the single most common cause of fsync stalls that escalate into unrecoverable errors. See the dataLogDir separation guide.
  • Configure autopurge. autopurge.purgeInterval and autopurge.snapRetainCount must be set explicitly; the default in some distributions is disabled.
  • Watch fsync latency trends. Sustained zk_fsynctime p99 growth predicts storage problems before they become crashes.
  • Keep snapshot.trust.empty=false in steady state. Only flip it during a controlled upgrade and revert immediately after.
  • Run a 64-bit JVM. The 2GB txnlog ceiling on 32-bit JVMs is an avoidable failure mode.
  • Test recovery. A node that cannot rejoin via SNAP sync in a reasonable time is a hidden incident. Large data trees make rebuilds slow.

How Netdata helps

  • Per-second scraping of the integrity counters. The increment shows up within a second, with no mntr parsing lag, and the delta is computed for you so historical absolute values do not generate repeat alerts.
  • Same-dashboard correlation of zk_unrecoverable_error_count, zk_digest_mismatches_count, and zk_snapshot_error_count. The systemic-versus-isolated call is visual instead of four separate terminal sessions.
  • Disk space, disk I/O, and filesystem metrics on the same host. The dominant root cause is visible alongside the symptom, with per-device breakdowns for dataLogDir versus dataDir when they are on separate volumes.
  • Leader-aware multi-node views. zk_server_state, zk_synced_followers, and zk_uptime across the ensemble make it immediately clear whether quorum was lost and which node needs the rebuild.
  • Cold-start suppression. Restart storms immediately after a fail-fast exit do not generate noise from non-critical metrics, while the integrity counters themselves still page.