ZooKeeper “fsync-ing the write ahead log took too long”: the disk warning behind most write stalls

The warning:

fsync-ing the write ahead log in SyncThread:0 took 1234ms which will adversely affect operation latency...

fires when fsync on the transaction log exceeds fsync.warningthresholdms (default 1000ms). The wording is deliberate: every write in ZooKeeper blocks on a quorum of fsyncs. If fsync takes a second, every write takes a second. If fsync takes 10 seconds, you are one missed heartbeat away from a leader election.

This is the canary for the failure pattern the playbook calls “Disk Sync Deadlock”: the single most common cause of ZooKeeper outages, more than GC, more than network. When you see this warning, the bottleneck is almost never ZooKeeper itself. It is the disk under dataLogDir. The fix is usually storage, not config.

This article covers what the warning means, what to check first, how to confirm the root cause, and what to change so it stops.

What this means

When ZooKeeper processes a write (create, setData, delete, setACL), the leader assigns a zxid, appends the transaction to a write-ahead log under dataLogDir, and broadcasts the proposal to followers. Both leader and followers must fsync their txnlog before the proposal can be ACKed; the leader commits only once a quorum of ACKs (each backed by a completed fsync) arrive. Every fsync is on the critical write path, on every quorum member, for every write.

This is why fsync latency is the most operationally important signal in ZooKeeper. The 1000ms default threshold is already 200 to 1000 times slower than what a dedicated SSD should deliver. If the warning fires, you are already past the point where downstream systems (Kafka controller, HBase region assignment, anything holding a distributed lock) start timing out.

Two severity bands worth memorizing:

• The warning itself (above 1000ms by default): writes are slow and dependent systems are degraded, but the ensemble is still up.
• Sustained fsync above roughly 200ms: the playbook calls this the heartbeat-miss risk band. With default tickTime=2000ms, followers tolerate syncLimit x tickTime (default 10s) before giving up on the leader. Sustained 200ms+ fsync on a busy leader eats enough of that heartbeat budget that the next network blip or GC pause pushes you over.

Version notes that bite people:

• Older ZooKeeper (3.4.13 / 3.5.4 and earlier) emits “adversely effect operation latency” with “effect” spelled with an E, a typo in the source. From 3.4.14, 3.5.5, 3.6.0 onward (ZOOKEEPER-3062), the message also appends fsync.warningthresholdms=<value> so you can see the configured threshold at a glance. Grepping for “adversely affect” with an A on an old cluster will miss every line.
• In 3.4.8 and earlier, fsync.warningthresholdms in zoo.cfg was silently ignored (ZOOKEEPER-2195, fixed in 3.4.9). On those versions, set it as -Dzookeeper.fsync.warningthresholdms=... on the JVM. If you “tuned” the threshold and the warnings still show 1000ms, that is why.

Common causes

The single most common, by a wide margin, is the first. dataLogDir either not set at all (so txnlog lands in dataDir next to the snapshots) or set but on the same underlying disk as snapshots. This is the configuration mistake the ZooKeeper admin guide explicitly calls out under “Things to Avoid”, and it has the cheapest, most reliable fix.

Quick checks

All read-only and safe on a production leader. None touch the write path.

# Find recent fsync warnings. Note the typo on older versions ("effect", not "affect").
grep -E "fsync-ing the write ahead log" /var/log/zookeeper/zookeeper.log | tail -20

# Confirm whether the configured threshold is what you think it is.
grep -E "fsync.warningthresholdms" /path/to/zoo.cfg /path/to/zkEnv.sh 2>/dev/null

# Pull fsync latency metrics. Percentiles need ZK 3.6+.
echo mntr | nc localhost 2181 | grep -E "zk_.*fsynctime|zk_fsync_threshold_exceed_count"

# Confirm where txnlog actually lives. The dirs 4lw reports datadir and logdir sizes.
echo dirs | nc localhost 2181

# OS-level disk latency on the txnlog device. Look at await and %util.
iostat -x 1 5

# Check whether txnlog and snapshots share a filesystem.
df -h <dataDir> <dataLogDir>

# SMART for hardware degradation on local disks.
smartctl -a /dev/<txnlog-device> | grep -E "Reallocated|Pending|Offline_Uncorrectable"

If mntr returns nothing on a 3.5+ cluster, four-letter commands are not whitelisted. Add 4lw.commands.whitelist=mntr,stat,srvr,dirs to zoo.cfg, or query the AdminServer at http://localhost:8080/commands/monitor instead.

How to diagnose it

Goal: confirm the disk under dataLogDir is the bottleneck, identify which cause from the table applies, and rule out GC and network as primary causes.

1. Pull the recent warning lines. Sort them by timestamp. Isolated spikes or a sustained trend? A sustained trend that began at a specific time points to a deployment, a config change, a burst credit cliff, or a disk starting to fail.

2. Compare fsync latency to update latency. If zk_p99_fsynctime and zk_p99_updatelatency move together, the write path is disk-bound. If update latency is high but fsync is normal, the bottleneck is quorum ACK (network or followers), not local disk.

3. Confirm with iostat -x 1 on the txnlog device. Signals to look for: await climbing into tens or hundreds of milliseconds, %util near 100, or w/s collapsing while await spikes (the cloud throttling signature).

4. Distinguish “slow disk” from “busy disk”. A dedicated, healthy SSD should give fsync p99 under 2ms. If you see that during quiet periods but spikes during snapshot creation, the cause is contention with snapshots. If fsync is bad even when nothing else is happening, the disk itself is degraded or the storage layer is throttling.

5. Rule out GC. GC stalls produce the same downstream symptom (everything stalls, sessions expire), but the diagnostic signature is different. GC affects reads and writes equally, and zk_jvm_pause_time_ms p99 will be elevated. Fsync stalls hit writes only.

6. Rule out a missing dataLogDir. This is the most common single cause. echo dirs | nc localhost 2181 plus df on both directories will tell you in one step whether txnlog and snapshots share a filesystem.

7. In cloud environments, check the storage layer directly. On AWS, look at BurstBalance (gp2) or volume IOPS exhaustion metrics (gp3). On GCP persistent disk, look at sustained IOPS vs. the provisioned limit. The “fine for hours, then cliff” pattern is almost always burst credit exhaustion.

flowchart TD
    A["fsync warning in log"] --> B{"Sustained or spike?"}
    B -->|"spike every snapCount"| C["Snapshots share disk with txnlog"]
    B -->|"sustained trend"| D{"Read latency also high?"}
    D -->|"yes"| E["GC stall: check jvm_pause_time_ms"]
    D -->|"no, writes only"| F{"iostat await high?"}
    F -->|"no"| G["Quorum ACK: check quorum_ack_latency"]
    F -->|"yes"| H{"dataLogDir on own device?"}
    H -->|"no"| I["Set dataLogDir, separate disk"]
    H -->|"yes"| J["Disk degraded or cloud throttling"]

Metrics and signals to monitor

Two reminder gotchas. First, zk_avg_latency aggregates reads and writes; on a read-heavy cluster it hides a bad write stall. Use zk_updatelatency and zk_readlatency separately (3.6+). Second, zk_avg_fsynctime and friends are not sliding windows in 3.4.x. They are cumulative since srst or restart. For alerting, use the 3.6+ percentiles, or compute deltas externally.

Fixes

Fixes group by cause. None of them are “restart ZooKeeper”. The warning is a symptom of storage, and a restart does not change storage.

Shared disk: txnlog and snapshots on the same volume

This is the highest-leverage fix in this article. Set dataLogDir to point at a separate device, ideally a small dedicated SSD, and rolling-restart the ensemble. Even a modest dedicated disk beats a large shared one, because the txnlog write pattern (small, synchronous, fsync on every operation) is the worst possible neighbour for snapshot I/O (large, sequential, bursty).

Tradeoff: requires a rolling restart, which means one planned leader election per member. Schedule it. Do not skip this on a cluster that has any production write load.

Cloud storage throttling

If iostat shows await spiking while w/s collapses, you are being throttled at the storage layer. Options, roughly in increasing cost:

• Move the txnlog to provisioned IOPS storage (io1/io2 on AWS, premium SSD on Azure, pd-ssd on GCP). The transaction log is the worst possible workload for burst-credit volumes.
• Increase provisioned IOPS on an existing gp3 volume.
• Put dataLogDir on instance-store NVMe if your workload tolerates rebuilding the log on instance loss. A single member losing its disk is recoverable (it resyncs from leader); multiple members losing disks at once is not.

Do not “solve” this by raising fsync.warningthresholdms. That suppresses the warning without changing the latency, and you lose the only early signal you have.

Co-located noisy I/O

Identify the neighbour. iostat -x names the device; iotop or pidstat -d names the process. Common culprits on shared ZooKeeper hosts: log shipping agents writing to the same disk, backup jobs, monitoring agents doing local caching, and (in embedded deployments like old Kafka) the parent application’s own I/O. Move the neighbour, throttle the neighbour, or move dataLogDir.

Disk hardware degradation

A gradual upward trend in fsync times over days or weeks, with SMART showing reallocated or pending sectors, is a failing disk. Replace it. This is the one case where the fix is not a ZooKeeper config change at all. It is a hardware ticket. Do not wait for the disk to fail completely; degraded ZooKeeper fsync is often the first user-visible symptom of a disk going bad.

Filesystem journal contention

The txnlog partition on ext4 with the default data=ordered journaling mode pays a metadata journal write alongside every transaction. For the txnlog specifically, data=writeback is acceptable because ZooKeeper has its own crash recovery via log plus snapshot replay. XFS is also a common operator choice for this workload.

Autopurge stall

If warning timestamps line up with autopurge runs, the purge is competing with the write path for the same disk. Confirm autopurge.purgeInterval and autopurge.snapRetainCount are set (a purgeInterval of 0 disables autopurge entirely), and that the purge is not running against the txnlog volume at a bad moment.

Prevention

• Always set dataLogDir to a dedicated device. This is the single most impactful ZooKeeper config decision for write latency. Without it, snapshot I/O and fsync contend for the same spindle.
• Monitor fsync directly, not just zk_avg_latency. Track zk_p99_fsynctime (3.6+) or zk_fsync_threshold_exceed_count. The aggregated latency metric hides this entire failure mode on read-heavy clusters.
• Alert on sustained fsync above 200ms. That is the heartbeat-miss risk band, not just a performance issue.
• Never put dataLogDir on a burst-credit volume in the cloud. Provisioned IOPS or instance-store NVMe only.
• Enable autopurge explicitly. The default disables it in some distributions, and a slowly filling txnlog partition is a silent killer.
• Trend fsync latency, do not just alert on it. A slow upward drift over weeks is the early signal of disk degradation.
• Tune the warning threshold down, not up. 250ms is reasonable. The default 1000ms is already a “you are broken” threshold, not an early warning.

How Netdata helps

• Per-second collection of zk_fsynctime percentiles and zk_fsync_threshold_exceed_count lets you see the shape of the stall (isolated spike, sustained plateau, daily cliff) in the same dashboard as disk-level iowait, await, and %util from the underlying device.
• Anomaly detection on fsync latency flags the slow upward drift that precedes disk failure, before it crosses a static threshold.
• Correlating fsync with zk_updatelatency, zk_outstanding_requests, and zk_jvm_pause_time_ms in one view resolves the “disk, GC, or quorum ACK?” question in seconds rather than minutes.
• Per-disk I/O metrics on the dataLogDir device let you confirm the storage layer is the cause without a separate terminal session.
• Cloud integration (AWS CloudWatch, GCP Monitoring) brings BurstBalance and IOPS exhaustion into the same dashboard as the ZooKeeper metrics, which is where the root cause of “fine for hours, then cliff” usually lives.

Related guides

• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• How ZooKeeper actually works in production: a mental model for operators
• ZooKeeper leader election storm: an ensemble that keeps re-electing
• ZooKeeper monitoring checklist: the signals every production ensemble needs
• ZooKeeper monitoring maturity model: from survival to expert
• ZooKeeper quorum loss: no leader elected and every write is failing
• ZooKeeper unexpected leader election: finding why the leader dropped