ZooKeeper dataLogDir sharing a disk with snapshots: the #1 fsync-latency footgun

You are chasing intermittent ZooKeeper write-latency spikes that appear to have no cause. Average latency is fine most of the time. Then, every few minutes, p99 update latency jumps by an order of magnitude, zk_outstanding_requests briefly climbs, and clients on tight timeouts see a flicker of connection churn. By the time you SSH in, the cluster looks healthy again.

The disk is not full, iostat averages look reasonable, and the spikes do not line up with any obvious workload change. The transaction log and snapshot directory are both on the same volume, and that is exactly the problem.

If dataLogDir is not explicitly set in zoo.cfg, ZooKeeper writes transaction-log fsyncs and snapshot files to the same directory on the same device. The transaction log is the most latency-sensitive I/O path in ZooKeeper: every write request blocks on an fsync before it can be acknowledged. A snapshot is a large bulk serialization of the entire data tree. When both share a disk, every snapshot run collides with the fsync queue, and you get write stalls that look random because snapshot timing is randomized.

What this means

ZooKeeper has two on-disk artifacts with very different I/O characteristics.

The transaction log (dataLogDir/version-2/log.<zxid>) is a write-ahead log. Every mutation is appended and fsynced before the operation is acknowledged to a quorum. On a healthy dedicated SSD, fsync completes in under 2 milliseconds. This is the path that determines write latency for the entire ensemble.

The snapshot (dataDir/version-2/snapshot.<zxid>) is a fuzzy serialization of the entire in-memory data tree. Snapshot writes are large, sequential, and asynchronous relative to the request pipeline, but they still consume disk bandwidth and write cache. A snapshot of a few-hundred-megabyte data tree can saturate a shared device for seconds.

When dataLogDir is unset, both writes land on the same directory and the same device. ZooKeeper pre-allocates transaction log files in 64MB chunks for sequential append, and a snapshot bursts into the middle of that sequential stream. The result is fsync latency spikes that align with snapshot creation, followed by write-latency propagation through the rest of the pipeline.

The default snapCount is 100,000 transactions, with randomization so ensemble members do not snapshot simultaneously. In ZooKeeper 3.5.x and later, snapSizeLimitInKb adds a log-size-based trigger on top of the transaction-count trigger. Either way, snapshots fire at semi-random intervals, which is why the resulting stalls appear to come from nowhere.

flowchart TD
  A[snapCount or snapSizeLimitInKb hit] --> B[Snapshot thread serializes DataTree]
  B --> C[Bulk sequential write to dataDir]
  C --> D{dataLogDir on same device?}
  D -- yes, the default --> E[txnlog fsync stalls behind snapshot I/O]
  E --> F[zk_fsynctime p99 spikes]
  F --> G[zk_updatelatency p99 follows]
  G --> H[zk_outstanding_requests climbs]
  H --> I[Quorum ACK latency rises]
  I --> J[Possible leader heartbeat miss]

The propagation chain is what makes this footgun hard to spot. The disk is the root cause, but the visible pain is in write latency, request queuing, and (in the worst case) missed heartbeats. Operators chase the symptoms individually and miss the shared disk underneath.

Common causes

CauseWhat it looks likeFirst thing to check
dataLogDir unset in zoo.cfgfsync p99 spikes that line up with snapshot file mtimes; default installgrep dataLogDir zoo.cfg returns nothing
dataLogDir set but on same underlying deviceSame symptom as unset; LVM or RAID presents one device as twolsblk and findmnt for both paths
Cloud burst credits exhausted on shared volumeSpikes also occur outside snapshot windows; cloud IOPS metric at limitCloud provider IOPS or burst-balance metric
Colocated workload writing to same diskSpikes line up with non-ZooKeeper process I/Oiotop or iostat -x during the spike

Quick checks

# Is dataLogDir explicitly set?
grep -E '^dataLogDir' /etc/zookeeper/zoo.cfg

# Where is dataDir?
grep -E '^dataDir' /etc/zookeeper/zoo.cfg

# Are the two paths on the same block device?
findmnt -n -o SOURCE /var/lib/zookeeper/data
findmnt -n -o SOURCE /var/lib/zookeeper/log

# Look at recent fsync warnings
grep "fsync-ing the write ahead log" /var/log/zookeeper/zookeeper.log | tail -20

# Are snapshot file mtimes correlated with the fsync warnings?
ls -lt /var/lib/zookeeper/data/version-2/snapshot.* | head -10

# Latest fsync percentile (ZK 3.6+)
echo mntr | nc localhost 2181 | grep -E 'zk_.*fsynctime'

# Threshold-exceed counter on ZK 3.4.x
echo mntr | nc localhost 2181 | grep zk_fsync_threshold_exceed_count

If dataLogDir is unset and the fsync warning timestamps line up with snapshot file modification times, you have your answer. If dataLogDir is set but findmnt shows both paths on the same block device, you have the same problem with an extra layer of misdirection.

Note: mntr and other four-letter-word commands require whitelisting via 4lw.commands.whitelist in ZooKeeper 3.5+.

How to diagnose it

  1. Capture the spike at per-second resolution. Collect zk_p99_fsynctime, zk_p99_updatelatency, and zk_outstanding_requests at one-second granularity. Minute-level scraping will hide the burst pattern because snapshots complete quickly.

  2. Confirm snapshot alignment. List snapshot files with modification times and overlay them on the fsync p99 chart. The signature of this footgun is a fsync spike within a few seconds of every snapshot file mtime.

  3. Rule out other disk pressure. During a spike, run iostat -x 1 against the underlying device. If %util approaches 100 and await spikes while snapshot I/O is in flight, you have confirmed contention. If %util is low but await is high, suspect cloud burst credit exhaustion or storage throttling instead.

  4. Rule out GC. Check JVM pause metrics. GC pauses produce a similar write-stall signature, but they also produce read-latency spikes. Pure disk contention leaves read latency on followers unaffected.

  5. Check the leader specifically. Because the leader fsyncs before broadcasting proposals, leader fsync contention is what actually stalls the ensemble. Filter mntr output to the node reporting zk_server_state leader and read the fsync percentiles there.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
zk_p99_fsynctime (3.6+)Direct measure of the latency-critical disk pathSustained p99 above 10 ms; spikes above 100 ms
zk_fsync_threshold_exceed_count (3.4.x)Counter of fsyncs exceeding fsync.warningthresholdms (default 1000 ms)Any non-zero increment rate
zk_p99_updatelatencyWhether fsync spikes are propagating to clientsTracks fsynctime upward
zk_outstanding_requests on leaderWhether the write pipeline is queuing behind diskSustained non-zero value
zk_p99_quorum_ack_latencyWhether followers are also blocked on their own fsyncElevated on leader
Snapshot file mtimeAnchors the timing of bulk I/O eventsLines up with fsync spikes
OS-level %util and await on the deviceDistinguishes contention from throttlingSpikes to saturation during snapshot writes

The zk_avg_fsynctime and zk_avg_latency aggregates will under-report the problem. Snapshot contention produces short, sharp bursts that wash out in any average. Use the percentile metrics added in 3.6, or compute deltas against the threshold counter on older versions.

Fixes

Move dataLogDir to a dedicated device

This is the canonical fix and the single highest-leverage configuration change in ZooKeeper. Even a small dedicated SSD is enough. The transaction log is append-only and never read at runtime, so capacity matters less than latency.

Procedure (rolling, one node at a time, leader last):

  1. Provision the new device. Format and mount it. The ZooKeeper documentation is explicit that a dedicated partition is not enough; the goal is a device that does only transaction-log appends and nothing else.

  2. Stop the node.

    # Stop the local ZooKeeper node (distro-specific; example for systemd)
systemctl stop zookeeper

  3. Create the new directory and copy the current transaction log files. ZooKeeper expects the version-2 subdirectory to exist and to contain the current log files.

    mkdir -p /var/lib/zookeeper/log/version-2
cp -a /var/lib/zookeeper/data/version-2/log.* /var/lib/zookeeper/log/version-2/
chown -R zookeeper:zookeeper /var/lib/zookeeper/log

  4. Set dataLogDir in zoo.cfg.

    dataDir=/var/lib/zookeeper/data
dataLogDir=/var/lib/zookeeper/log

  5. Restart the node. On the first restart after this change, expect a warning of the form Snapshot directory has log files. Check if dataLogDir and dataDir configuration is correct. if any log.* files remain under dataDir/version-2. Removing or moving them out of the snapshot directory resolves the warning.

    systemctl start zookeeper
echo ruok | nc localhost 2181     # should return imok
echo mntr | nc localhost 2181 | grep zk_server_state

  6. Repeat per node, leader last, verifying quorum is intact between each restart.

When you cannot add a dedicated physical device

If a separate physical device is genuinely unavailable, the next-best options are, in decreasing order of effectiveness:

  • A separate cloud volume attached to the instance. Even a small provisioned-IOPS volume outperforms a shared gp2/gp3 for this workload.
  • A separate partition on the same device. This is the weakest mitigation. It helps with filesystem journal contention but does not isolate the underlying disk’s write cache or queue, so snapshot writes still interfere with fsync.

The ZooKeeper documentation’s wording on this is direct: a dedicated partition is not enough. The transaction log wants a device that does only sequential appends and nothing else.

Do not set forceSync=no

A common reflex when chasing fsync latency is to set zookeeper.forceSync=no. This skips the fsync call after each transaction-log write, relying on the OS page cache instead. It eliminates the warning and the latency, but it also weakens durability. On a leader crash you can lose acknowledged transactions, which means divergent ensemble state. The ZooKeeper documentation classifies this as an unsafe option. Do not use it to mask the underlying contention.

Prevention

  • Make dataLogDir on a dedicated device a provisioning default. Bake it into the AMI, image, Helm chart, or configuration-management module. The default install leaves it unset because that is the simplest path that works in dev, not because it is safe in production.
  • Monitor zk_p99_fsynctime directly. Most teams monitor disk space but not disk latency. Disk space and disk latency are independent failure modes, and the latency signal is the one that predicts write stalls.
  • Track snapshot file mtimes alongside fsync metrics. When the two correlate, the diagnosis is seconds instead of hours.
  • Enable autopurge. Set autopurge.purgeInterval and autopurge.snapRetainCount so that old logs and snapshots do not accumulate and shift the contention profile over time.
  • For cloud deployments, monitor burst credit balances. Volumes with burstable IOPS can produce sudden cliffs when credits run out. The cliff often lines up with the next snapshot run and produces a compounded spike.

How Netdata helps

  • Per-second collection of zk_fsynctime percentiles and zk_updatelatency percentiles makes the burst pattern visible at the resolution it actually occurs. Minute-level scraping averages the spikes away.
  • ML anomaly detection on zk_p99_fsynctime flags snapshot-correlated spikes as anomalous even when the absolute value is below a static threshold, which is the typical early-warning signature for this footgun.
  • Correlated charts of zk_outstanding_requests, zk_updatelatency, and OS-level disk await on the same timeline let you distinguish disk contention from GC in a single view instead of pivoting across tools.
  • The ZooKeeper collector auto-detects the leader and surfaces leader-specific metrics separately, so leader fsync contention is not hidden by averaging across the ensemble.
  • Filesystem and disk collectors surface the underlying device’s %util, await, and (on cloud) IOPS figures, letting you distinguish shared-disk contention from cloud throttling without a separate monitoring stack.