ZooKeeper data tree digest mismatch: detecting corruption before it spreads

When zk_digest_mismatches_count increments on a ZooKeeper node, the in-memory data tree on that node has diverged from the checksum ZooKeeper expects. This is a data-integrity alarm, not a performance signal. Clients reading from that node may be receiving wrong answers, and if the divergence came from a ZAB replication bug rather than local corruption, the same divergence may be propagating to other ensemble members.

The digest feature (enabled by default since ZooKeeper 3.6.0) computes a running checksum of the entire data tree using an incremental adHash scheme. On every write, ZooKeeper updates the hash and compares it against the expected digest attached to the transaction. When the comparison fails, the counter ticks up and ZooKeeper logs a “First digest mismatch on txn” message. A single increment is enough to page.

What this means

ZooKeeper holds its entire data tree in JVM heap. Every znode, its data, its ACL, its children list: all live in memory and replicate across the ensemble via ZAB. There is no per-read checksum validation and no on-disk integrity scan at runtime. The adHash digest is the only continuous in-process integrity check ZooKeeper runs.

The digest is incremental: when a znode changes, ZooKeeper computes current_hash = current_hash + hash(new_data) - hash(old_data). The hash function is CRC-based: fast, not collision-resistant, but adequate for detecting bit flips and dropped transactions. The expected digest for each committed transaction is computed by the leader and attached to the proposal. Followers compare their locally-computed digest against this expected value when they apply the commit. Nodes under /zookeeper (the internal subtree, including quota stats) are excluded from the calculation because their bookkeeping can legitimately diverge.

When the comparison fails:

  1. The DIGEST_MISMATCHES_COUNT counter (exposed via mntr as zk_digest_mismatches_count) increments.
  2. ZooKeeper logs the mismatched zxid and the expected vs actual digest values.

The node keeps running. It does not crash, does not stop serving reads, and does not drop out of the ensemble on its own. Without an explicit alert on the delta, a diverged node can serve stale or corrupted data indefinitely while appearing healthy to shallow liveness checks like ruok.

One logging gotcha: the mismatch log line prints the TxnDigest object as the “expected digest” (for example, “expected digest is 2,10124071835”) while the actual digest is a single number, making the two values hard to compare by eye. This is tracked as ZOOKEEPER-4845 and is still open.

Common causes

CauseWhat it looks likeFirst thing to check
Memory error (bit flip, DIMM fault)Single node, one increment, no corresponding log corruption on other nodes, no recent restart or syncECC error counts from edac-util or IPMI SEL; whether the kernel logged a machine check
ZAB replication bug (DIFF sync with empty committedLog)Single follower after it rejoined or after leader restart; mismatch appears immediately after sync completesZooKeeper version against ZOOKEEPER-4352 (3.6.3, 3.7.0) and ZOOKEEPER-4444 (3.6.3 through 3.8.1); leader’s minCommittedLog state
Disk corruption in snapshot or txnlogMismatch on startup during snapshot or log replay, or after a hard crash; may coincide with zk_snapshot_error_count or zk_restore_error_count incrementsdmesg for disk errors; zk_snapshot_error_count and zk_restore_error_count on the same node
Follower restart with stale on-disk stateFollower fails to synchronize after being killed and restarted twice; “Digests are not matching” in the log, follower may serve deleted znodesWhether the node was restarted multiple times in quick succession; ZooKeeper version against ZOOKEEPER-4444

Quick checks

These are all read-only. Run them on every ensemble member, not just the one that alerted.

# Check the mismatch counter on each node
echo mntr | nc localhost 2181 | grep zk_digest_mismatches_count

# Check for the other integrity signals that travel with digest mismatches
echo mntr | nc localhost 2181 | grep -E 'zk_(unrecoverable_error|snapshot_error|restore_error)_count'

# Confirm which node is the leader (leader-only metrics only appear there)
echo mntr | nc localhost 2181 | grep zk_server_state

# Pull the digest history associated with zxids (3.6.0+)
echo hash | nc localhost 2181

# Look for the mismatch log lines on the affected node
grep -i "digest" /var/log/zookeeper/zookeeper.log | tail -50

# Check for hardware memory errors on the host
edac-util -v     # or: ipmitool sel list | grep -i memory

# Check for disk-level errors that could have corrupted snapshot or txnlog
dmesg -T | grep -iE 'error|medium|failed|ecc|mce' | tail -30

# Compare the current zxid across all ensemble members to see if one is behind
echo mntr | nc localhost 2181 | grep zk_zxid

If hash is not whitelisted you will get “hash is not executed because it is not in the whitelist.” Add it to 4lw.commands.whitelist if you need it, or use the AdminServer /commands/hash endpoint on port 8080 instead. The same whitelist caveat applies to mntr since ZooKeeper 3.5.3.

How to diagnose it

The single most important diagnostic question is: how many nodes incremented?

flowchart TD
    A["zk_digest_mismatches_count incremented"] --> B{"Which nodes incremented?"}
    B -->|"One node"| C{"Did it increment on startup?"}
    B -->|"Multiple nodes"| D["Stop. Escalate immediately."]
    C -->|"Yes, during load or replay"| E["Disk corruption in snapshot or txnlog"]
    C -->|"No, at runtime"| F{"Recent follower sync or restart?"}
    F -->|"Yes"| G["ZAB replication bug or stale on-disk state"]
    F -->|"No"| H["Memory error. Check ECC counts."]
    E --> I["Rebuild node from healthy snapshot"]
    G --> I
    H --> I
    D --> J["Likely ZAB bug. Stop writes, preserve data, file issue with zxids."]

  1. Collect the counter from every node. Do not trust a single-node scrape. A digest mismatch on one follower is a local problem. A digest mismatch on the leader, or on multiple followers simultaneously, is an ensemble problem.

  2. Note whether the increment happened at startup or at runtime. A mismatch during snapshot or txnlog replay points to on-disk corruption: the file was written correctly by ZK but the bytes on disk no longer match. A mismatch at runtime, after the tree was already loaded, points to either a memory error or a ZAB-layer inconsistency introduced during sync.

  3. Correlate with sync events. If the mismatch appeared immediately after a follower rejoined the ensemble or after the leader restarted, check the ZooKeeper version against the known ZAB bugs. ZOOKEEPER-4352 affects 3.6.3 and 3.7.0: when the leader has minCommittedLog=0 (empty in-memory committed log), it sends only a DIFF to a rejoining follower instead of a full snapshot, leaving the follower with an inconsistent tree. ZOOKEEPER-4444 affects 3.6.3 through 3.8.1: a follower killed and restarted twice fails to synchronize and produces “Digests are not matching” while serving stale or deleted znodes.

  4. Check the hardware. If only one node is affected and there is no recent sync or restart, treat it as a suspected memory error until proven otherwise. Check ECC error counts, check dmesg for machine-check exceptions, and check whether the host is bare metal with faulty DIMMs or a VM on a noisy host.

  5. Capture the mismatched zxid from the log before you touch anything. The log line contains the transaction at which the divergence was detected. If you end up filing a ZooKeeper issue or doing forensic comparison of snapshots, you need this number.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
zk_digest_mismatches_countThe primary integrity signal. Any increment means the in-memory tree diverged from the expected checksum.Any increase() greater than 0. Alert on the delta, never the absolute value.
zk_unrecoverable_error_countCo-travels with digest mismatches when the corruption is severe enough that ZK considers the state unrecoverable.Any increment. PAGE unconditionally.
zk_snapshot_error_countIndicates snapshot creation or loading failed. If a digest mismatch appears alongside snapshot errors, disk corruption is the likely root cause.Any increment.
zk_restore_error_countIndicates recovery failed on startup. Pairs with digest mismatches that fire during log or snapshot replay.Any increment.
zk_zxid (per node)Comparing zxids across the ensemble reveals whether the diverged node is behind, ahead, or at the same transaction position as peers.A node whose zxid does not match peers after sync completes.
zk_server_stateConfirms which node is leader, which is critical because leader-side digest mismatches are far more dangerous than follower-side.Unexpected role transitions around the time of the mismatch.

Fixes

There is no in-place repair for a diverged data tree. The correct response is to remove the node from the ensemble and rebuild its state from a known-good source.

Single-node mismatch: eject and rebuild

If exactly one node incremented and the rest of the ensemble is clean:

  1. Stop the ZooKeeper process on the affected node. This prevents it from serving further stale or corrupted reads.
  2. Preserve the data directory for forensics. Copy dataDir and dataLogDir aside before you wipe them. If this turns out to be a ZAB bug rather than local corruption, the maintainers will want the corrupted snapshot and txnlog.
  3. Wipe the data directories. Remove the contents of dataDir/version-2/ and dataLogDir/version-2/ (or wherever your dataDir and dataLogDir point). This is destructive. Confirm you have a forensic copy from step 2 before proceeding.
  4. Restart the node. It will rejoin the ensemble and perform a full sync from the leader (SNAP sync). The leader serializes its complete data tree and sends it. The rebuilt node’s tree will match the leader’s, and the digest will be correct.
  5. Verify. After sync completes, confirm zk_digest_mismatches_count is zero on the rebuilt node and that its zxid matches the leader.

Do not attempt to patch the diverged znodes by hand. The node’s state is already known to be wrong; rebuilding from a healthy peer is the only reliable path.

If the node that mismatched is the leader, force a leader election first (stop the leader, let the ensemble elect a new one from the clean followers), then rebuild the former leader as a follower.

Multi-node mismatch: stop and escalate

If the mismatch appears on the leader, or on more than one follower, do not rebuild individual nodes. The divergence is likely coming from the replication layer itself, and rebuilding nodes from a leader that is producing bad digests will propagate the corruption.

  1. Stop writes if you can. If dependent systems (Kafka, HBase) can be paused or failed over to a secondary coordination path, do so. Every additional transaction on a diverged ensemble risks wider inconsistency.
  2. Preserve all data directories on all nodes. This is your forensic evidence.
  3. Identify the most-trusted node. Usually this is the node with the highest committed zxid that has not logged a digest mismatch. If you cannot identify one with confidence, do not pick arbitrarily.
  4. Escalate. Multi-node digest divergence is a ZAB-level event. File a ZooKeeper issue with the mismatched zxids, the versions, and the sync history. If you have a support relationship with a vendor distribution, engage it.

Version-specific bugs

If your version matches a known ZAB bug, the rebuild procedure above is the workaround, but you should also plan an upgrade:

  • ZOOKEEPER-4352 (3.6.3, 3.7.0): Restarting the leader to rebuild its in-memory committed log before the follower rejoins avoids the empty-committedLog DIFF path.
  • ZOOKEEPER-4444 (3.6.3 through 3.8.1): Deleting the data directory on the affected follower and letting it re-sync is the documented workaround. Upgrading past 3.8.1 is the durable fix.
  • ZOOKEEPER-4845 (open): The confusing log format is not a data-integrity bug, but it makes forensics harder. The “expected digest” field in the log is a TxnDigest object representation, not a single comparable number.

Prevention

  • Alert on the delta of zk_digest_mismatches_count, not the absolute value. The counter is monotonic within a process lifetime and any increment is a PAGE.
  • Monitor zk_unrecoverable_error_count, zk_snapshot_error_count, and zk_restore_error_count alongside the digest counter. A digest mismatch with no other error counters moving suggests memory or ZAB; a digest mismatch with snapshot or restore errors suggests disk.
  • Run ZooKeeper on hosts with ECC memory. The adHash digest catches single-bit flips that non-ECC memory will not report on its own. Without ECC, the digest mismatch may be your first and only indication of a failing DIMM.
  • Keep dataLogDir on dedicated, healthy storage. Disk corruption in the txnlog or snapshot is a root cause most preventable by operational hygiene.
  • Stay current on patch releases. The ZAB sync bugs that produce false-positive digest mismatches (ZOOKEEPER-4352, ZOOKEEPER-4444) are fixed in newer minor lines. Running old 3.6.x or 3.7.x in production means carrying known data-divergence bugs.
  • Whitelist the hash four-letter word, or use the AdminServer equivalent. When a mismatch fires, you want the digest history available immediately, not after a config change and restart.

How Netdata helps

  • Correlation across ensemble members in a single view lets you answer the critical triage question (one node vs multiple nodes) without manually scraping each node and comparing outputs. Per-second collection means a single increment is visible within a second of occurring.
  • Co-display of zk_server_state, zk_zxid, and zk_digest_mismatches_count lets you see whether a mismatch followed a sync event, a leader election, or a restart, which is the fastest path to distinguishing a ZAB bug from local corruption.
  • Integration with host-level signals (ECC errors from kernel logs, disk error counters, dmesg patterns) means the hardware-cause hypothesis can be confirmed or eliminated from the same dashboard, without switching tools mid-incident.