ZFS synchronous write latency high: fsync, NFS, and database commits stalling

Your database commit latency just jumped from 2ms to 40ms. Or NFS clients are reporting sluggish writes while the pool itself looks healthy: zpool status -x says all pools are healthy, read latency is fine, and throughput has not collapsed. Applications that call fsync(), open files with O_SYNC, or run over NFS are stalling, while everything else seems normal.

This is the ZFS synchronous write latency problem. The bottleneck is almost never the data path. It is the ZIL (ZFS Intent Log) path: the mechanism ZFS uses to guarantee that synchronous writes survive a crash. When the ZIL backs up, every fsync, every NFS COMMIT, and every database transaction commit stalls behind it.

The failure mode has a small set of causes and a fast diagnostic path. The one tempting “fix” (sync=disabled) trades correctness for speed and must never touch a database dataset.

What this means

Every synchronous write on ZFS goes through the ZIL before the application gets an acknowledgment. ZFS writes the intent record to the ZIL, flushes it to stable storage, and only then returns success to the caller. The actual data lands in the pool later, during the next transaction group (TXG) sync, which by default happens every 5 seconds (zfs_txg_timeout).

Where the ZIL lives determines your commit latency:

• With a SLOG (Separate Intent Log device): the ZIL lives on a dedicated fast device. A good NVMe SLOG delivers sub-millisecond commit latency.
• Without a SLOG: the ZIL lives on the pool’s data vdevs. On a spinning pool, synchronous commit latency is typically 10-50ms, because each commit competes with normal data I/O on rotating media.

So when sync latency is high, one of three things is true: there is no SLOG and the pool itself is the commit device, the SLOG is slow or failed and ZFS has fallen back to the pool ZIL, or something is stalling the ZIL pipeline itself.

flowchart TD
    A[App calls fsync / O_SYNC write / NFS COMMIT] --> B[ZIL intent record]
    B --> C{SLOG present and healthy?}
    C -->|Yes| D[Write to SLOG device]
    C -->|No| E[Write to pool data vdevs]
    D --> F[Ack to application]
    E --> F
    F --> G[Data lands in pool at next TXG sync]
    D -.SLOG fails.- H[Silent fallback to pool ZIL - pool still ONLINE]

The critical operational detail: high syncq_wait in zpool iostat -l points at the ZIL/SLOG path, not the data path. That one column is the fastest way to confirm you are in the right article.

Common causes

Quick checks

All of these are read-only and safe to run during an incident.

# 1. Per-queue latency: syncq_wait is the key column
zpool iostat -l 1

# 2. Latency histograms including the log vdev (tail latency, not averages)
zpool iostat -wl <pool> 5

# 3. Is there a SLOG, and is it healthy?
zpool status -v

# 4. ZIL commit counters: stalls and errors
cat /proc/spl/kstat/zfs/zil

# 5. TXG sync times: is the write pipeline behind?
tail -20 /proc/spl/kstat/zfs/<pool>/txgs

# 6. Per-vdev throughput: find a single slow device
zpool iostat -v 1

# 7. Sync property per dataset
zfs get -r sync <pool>

# 8. Queue depths: pending vs active per queue type
zpool iostat -q -v 1

Two things to know about these outputs. First, zpool iostat -l shows averages, which smooth out bursty stalls; use -w histograms to see p95/p99 behavior. Second, there is no zil_commit_latency kstat. ZIL latency must be inferred from syncq_wait, the log vdev latency in zpool iostat -wl, and application-level fsync timing.

How to diagnose it

1. Confirm the symptom is sync-specific. Run zpool iostat -l 1 for 30 seconds during the application slowdown. If write total_wait is elevated and syncq_wait is the dominant component while read latency stays normal, you are looking at a ZIL/SLOG problem. If asyncq_wait is also high, suspect the broader write pipeline (TXG pressure, saturation) instead.

2. Check for a SLOG and its state. zpool status -v shows the logs section. If there is no log vdev, the pool’s data disks are serving every commit. If there is one and it shows errors, FAULTED, or REMOVED, ZFS has silently fallen back to the pool ZIL. This fallback does not put the pool into DEGRADED state, so zpool status -x will report healthy while your database commits slow down 10x-100x.

3. Check ZIL counters. cat /proc/spl/kstat/zfs/zil and look at zil_commit_stall_count and zil_commit_error_count. Either one incrementing is a ticket-level signal that commits are stalling or failing at the ZIL layer. zil_commit_count minus zil_commit_writer_count tells you how often commits were satisfied without an actual ZIL write; the raw ratio is less important than whether stalls are climbing. For per-object attribution, inspect /proc/spl/kstat/zfs/<pool>/objset-*; the /proc/spl/kstat/zfs/zil summary is global.

4. Rule out TXG sync pressure. tail -20 /proc/spl/kstat/zfs/<pool>/txgs and read the stime column (sync duration, nanoseconds). If stime consistently exceeds zfs_txg_timeout (default 5s), the pool cannot flush dirty data fast enough, and on a pool without a SLOG the ZIL is competing with that backlog on the same disks. Also check ndirty against zfs_dirty_data_max; sustained dirty data above 80% of the limit means the write throttle is close to engaging.

5. Isolate a slow device. zpool iostat -v 1 and zpool iostat -wl <pool> 5. In a mirror or RAIDZ vdev, writes wait on the slowest member. One device with 3x+ the latency of its peers explains both the sync stalls and any async write degradation. Cross-reference with zpool status error counters and device SMART data.

6. Check the datasets, not just the pool. zfs get -r sync <pool>. A dataset set to sync=always forces every write through the ZIL, even writes the application never asked to be synchronous. On a busy file-serving dataset, that alone can saturate a SLOG that was sized for database commits.

7. Check capacity and fragmentation. If the ZIL lives on the pool, allocation difficulty becomes commit latency. zpool list -o name,cap,frag: above 85% capacity with rising fragmentation, metaslab allocation gets expensive and everything on the write path, including ZIL commits on pool vdevs, slows down.

Metrics and signals to monitor

Note the instrumentation gap: the /proc/spl/kstat/zfs/zil summary is global, while per-object ZIL counters live under /proc/spl/kstat/zfs/<pool>/objset-*; there is no direct ZIL latency kstat. Pair these counters with application-side fsync timing (database commit latency, NFS server RPC latency) to close the loop.

Fixes

Add a SLOG device

For any sync-heavy workload on spinning disks (NFS, databases, mail servers, VM storage), this is the structural fix. Add a dedicated log device:

# Add a log device (SLOG) to the pool
zpool add <pool> log <device>

Choose the device for latency and power-loss safety, not throughput. The SLOG absorbs every synchronous commit, so it needs low write latency and the ability to survive a power failure without losing acknowledged writes. Enterprise NVMe with power-loss protection is the standard choice; a consumer SSD with a volatile write cache can acknowledge writes it has not actually persisted, which defeats the purpose. Mirror the SLOG if the workload justifies it: an unmirrored SLOG failure combined with a crash before the next TXG commits can lose recently acknowledged sync writes.

Replace or re-add a failed SLOG

If zpool status -v shows the log vdev FAULTED:

# Replace a failed SLOG device
zpool replace <pool> <old-log-device> <new-device>

The pool continues serving sync writes from the main pool ZIL in the meantime, so this is a performance emergency for sync workloads, not a data-loss emergency. Still treat it as urgent: you are running without the commit latency your applications were sized for.

Fix the underlying write pipeline

If TXG stime is elevated and dirty data is climbing, the ZIL problem is downstream of a pool that cannot flush. Identify the slow vdev with zpool iostat -v 1, check dmesg for SATA/SAS resets or timeouts, and check whether a scrub or resilver is competing for bandwidth. See ZFS ARC hit ratio low: cache misses, cold caches, and working sets that outgrew RAM for the read-path side of pool pressure; for the capacity angle, see ZFS capacity planning: runway estimation before the pool fills.

Revisit per-dataset sync settings

If a dataset has sync=always but the application on it does not need every write forced synchronous, set it back to sync=standard (the default) so only explicit sync requests hit the ZIL. This is a safe change: sync=standard still honors every fsync and O_SYNC write.

About sync=disabled

sync=disabled makes sync latency vanish because ZFS stops doing sync writes: the ZIL is bypassed and the application is told its data is committed when it is not. On a crash or power loss, everything written in the last TXG window (up to several seconds) is gone, even though the application believes it was safely committed.

ZFS itself stays crash-consistent; the filesystem will not corrupt. But a database that was told its WAL flush succeeded and then loses that data can be left in a state far worse than a clean crash. Never use sync=disabled for databases, and treat it as unacceptable anywhere acknowledged writes matter. It is a profiling tool for proving the ZIL is your bottleneck, not a production setting.

Prevention

• Monitor the log vdev explicitly. SLOG failure does not degrade pool state, so zpool status -x misses it. Alert on log vdev state and on zil_commit_stall_count increments.
• Baseline sync latency per workload. Sync-heavy pools should hold total_wait under 10-50ms. Alert on sustained 2x deviation from your rolling baseline, using -w histograms for tail latency rather than averages.
• Track SLOG wear. SLOG devices absorb concentrated sync writes and have finite endurance. Monitor device-level SMART/NVMe wear indicators; ZFS SLOG counters such as zil_itx_metaslab_slog_write show activity, not remaining endurance. Replace at 70-80% wear for production.
• Keep capacity and fragmentation in the safe zone. If the ZIL lives on the pool, the capacity-fragmentation cliff becomes a commit-latency cliff. Act at 85% capacity, not at the wall.
• Audit sync properties. sync=always where it is not needed burns SLOG capacity; sync=disabled anywhere near a database is a latent data-loss incident. Include zfs get sync in dataset property drift checks.
• Size the SLOG for the workload’s commit rate and mirror it if losing acknowledged writes on a crash-plus-SLOG-failure would hurt.

How Netdata helps

• Netdata collects ZFS pool latency, throughput, and per-vdev I/O at per-second resolution, so syncq_wait-driven stalls show up as they happen instead of being smoothed away by interval sampling.
• Per-vdev charts make the one-slow-disk pattern visible immediately: one vdev diverging from its peers while sync latency climbs.
• ZIL commit counters (commits, stalls, errors) are charted over time, so you can correlate a zil_commit_stall_count ramp with the exact minute application commit latency degraded.
• TXG sync duration and dirty data trends let you distinguish a ZIL/SLOG bottleneck from broader write-pipeline saturation without digging through kstats by hand during an incident.
• Log vdev state changes surface alongside device health metrics, closing the “pool ONLINE but SLOG dead” blind spot that zpool status -x leaves open.
• Correlating storage signals with application-level latency (database commit time, NFS server response time) on one dashboard shortens the path from “commits are slow” to “the SLOG faulted at 03:12”.

Related guides

• ZFS ARC hit ratio low: cache misses, cold caches, and working sets that outgrew RAM
• ZFS zfs_arc_max: capping the ARC without starving read performance
• ZFS ARC and the OOM killer: applications killed while the cache will not shrink fast enough
• ZFS ARC shrinking below c_max: reading memory pressure before latency hits
• ZFS ARC using all memory: the Linux default that eats your RAM
• ZFS capacity planning: runway estimation before the pool fills
• ZFS checksum errors (CKSUM): the definitive signal of silent corruption
• ZFS checksum errors on multiple devices: suspect RAM or the controller, not the disks
• ZFS device UNAVAIL or REMOVED: a disk that fell off the bus
• How ZFS actually works in production: a mental model for operators
• ZFS monitoring checklist: the signals every production pool needs
• ZFS monitoring maturity model: from survival to expert