ZFS pool I/O latency high: reading zpool iostat -l before blaming the disks

Applications are stalling on storage calls, zpool iostat -l shows ugly latency numbers, and someone has already said “the disks are dying.” Before you open a hardware ticket, look at which latency column is actually elevated. ZFS reports total wait, disk wait, and queue wait separately, and the split between them tells you whether the problem is the devices or something inside ZFS itself.

The most common misdiagnosis is treating high total_wait as a disk problem. total_wait includes queueing time inside ZFS: ZIL commits, TXG sync pressure, dirty data throttling, fragmentation-driven allocation cost, and competition from scrub or resilver. If disk_wait is low while total_wait is high, your disks are fine and the bottleneck is above them.

This guide covers reading the latency output correctly, correlating it with queue depth and TXG state, and fixing the actual cause. It assumes a working knowledge of TXGs, the ARC, and the ZIL. If you need that background, start with How ZFS actually works in production.

What this means

zpool iostat -l (available since OpenZFS 0.7.0) reports average latency per pool or per vdev, broken into separate columns:

• total_wait: average total I/O time, queueing plus disk service.
• disk_wait: average time the I/O spent at the disk only.
• syncq_wait: time spent in the synchronous priority queues. This is the ZIL/SLOG path.
• asyncq_wait: time spent in the asynchronous priority queues. This is the TXG commit path.
• scrub / trim / rebuild: time I/O spent queued behind the scrub, TRIM, or rebuild (sequential resilver) queues.

Each column has read and write sub-columns. The single most useful move is total_wait minus disk_wait: the difference is time the I/O spent inside ZFS before it ever reached a device.

flowchart TD
  A[total_wait high on zpool iostat -l] --> B{disk_wait also high?}
  B -- Yes --> C[Backend problem: slow vdev, saturation, or dying device]
  B -- No --> D{Which queue wait dominates?}
  D -- syncq_wait --> E[ZIL / SLOG path: sync writes stalled]
  D -- asyncq_wait --> F[TXG commit path: sync pressure or dirty data throttling]
  D -- scrub / rebuild --> G[Scrub or resilver inflating latency - expected]
  D -- none clearly --> H[Check -q queue depth and txgs stime]

Two instrumentation facts shape how you read this output:

• -l reports averages. Bursty latency spikes get smoothed out. Use zpool iostat -w for latency histograms so you can see the tail buckets, which are what applications actually feel.
• Scrubs and resilvers legitimately inflate latency. Scrub I/O queues behind application I/O at lower priority, so the scrub column and overall wait times climb during a scan window without anything being wrong.

Common causes

Quick checks

All read-only, safe to run during an incident:

# 1. Average latency per vdev, 1-second samples
zpool iostat -l -v 1

# 2. Latency histograms: the tail latency that averages hide
zpool iostat -w 5

# 3. Queue depths per vdev: pend growing means backend saturation
zpool iostat -q -v 1

# 4. Is a scrub or resilver running right now?
zpool status | grep -A5 scan

# 5. Recent TXG sync durations (stime column, nanoseconds)
cat /proc/spl/kstat/zfs/<pool>/txgs | tail -20

# 6. Pool capacity and fragmentation together
zpool list -o name,size,alloc,free,cap,frag

# 7. Memory pressure on the ARC (read-latency cause)
grep -E "^(size|c_max|hits|misses|memory_throttle_count)" /proc/spl/kstat/zfs/arcstats

# 8. Kernel-side device problems
dmesg | grep -i -E "ata|sas|reset|timeout" | tail -20

Reading notes:

• On the -q output, sustained pending much greater than active on the same vdev means that device is saturated. Pending near zero while latency stays high means the delay is inside ZFS, not the device.
• On the txgs kstat, stime is the sync phase duration. Consistently above the TXG timeout (default 5 seconds via zfs_txg_timeout) means the pool cannot flush dirty data as fast as it accumulates.
• SLOG health is easy to miss. A slow (not yet faulted) SLOG device is invisible to pool redundancy and zpool status -x, while sync write latency jumps. Check the logs section of zpool status -v explicitly.

How to diagnose it

Work through these in order. Each step narrows the layer.

1. Rule out the expected explanation first. Run zpool status and check the scan line. If a scrub or resilver is in progress, elevated total_wait and a large scrub or rebuild wait column are expected behavior, not an incident. Scrubs are I/O intensive and compete with production I/O by design. Note the window and re-measure after it completes.

2. Split total_wait from disk_wait. Run zpool iostat -l 1 for 30-60 seconds. If disk_wait tracks total_wait, the devices are slow: go to step 5. If disk_wait stays low while total_wait is high, the contention is inside ZFS. Continue.

3. Check the histogram, not just the average. Run zpool iostat -w 5. Averages smooth out bursts; the histogram shows whether you have a healthy unimodal distribution or a tail of I/Os landing in the high-latency buckets. A small fraction of I/Os in the worst buckets is often what the application is actually complaining about.

4. Read the queue wait split. High syncq_wait on writes points at the ZIL/SLOG path: synchronous writes (fsync, O_SYNC, NFS) are stalling before they reach disk. High asyncq_wait on writes points at the TXG commit path. For the syncq case, check SLOG device state in zpool status -v and confirm which datasets actually need sync semantics (zfs get sync <dataset>). For the asyncq case, continue to step 6.

5. If disk_wait is genuinely high, find the slow device. Run zpool iostat -l -v 1 and compare vdevs. One vdev consistently 3x or more slower than its peers is a dying or degraded device, even if zpool status -x reports healthy. Check per-device READ/WRITE/CKSUM counters in zpool status, device SMART data, and dmesg for link resets. In a RAIDZ vdev, the slowest disk sets the write latency for the whole stripe.

6. Confirm TXG pressure. Read /proc/spl/kstat/zfs/<pool>/txgs and watch stime and ndirty over a few minutes. stime regularly exceeding 2x zfs_txg_timeout (over ~10 seconds with defaults) means the write pipeline is saturated. Cross-check dirty data against zfs_dirty_data_max (/sys/module/zfs/parameters/zfs_dirty_data_max); ZFS begins throttling writers at zfs_delay_min_dirty_percent (default 60%) of that limit, and throttling shows up as write latency that looks exactly like slow disks. See ZFS dirty data throttling for the full mechanism.

7. Check the capacity-fragmentation axis. If zpool list -o name,cap,frag shows capacity above ~85% and fragmentation climbing, metaslab allocation is getting expensive and every write pays for it. This is a slow-creep cause, not a sudden one, and it amplifies everything above.

8. For read latency, check the ARC before the disks. If read total_wait is high but the workload used to be fast, check ARC hit rate and memory_throttle_count in /proc/spl/kstat/zfs/arcstats. A shrinking ARC forces reads to disk and produces exactly the “suddenly slow storage” symptom. See ZFS ARC hit ratio low and ZFS ARC shrinking below c_max.

Metrics and signals to monitor

Latency severity thresholds from field experience: sync-heavy workloads (databases, NFS) should see total_wait under roughly 10-50ms; async workloads tolerate up to ~100ms; anything sustained above 1 second indicates a serious problem. Treat these as baseline-relative: alert on divergence from your own rolling baseline and on histogram tail growth, not on a single absolute number, because scrub windows will otherwise false-fire every week.

Fixes

ZIL/SLOG bottleneck

• Confirm the SLOG device is healthy in zpool status -v. A failed SLOG falls back to the pool vdevs for the ZIL, which is a large latency jump for sync workloads but no data loss. Replace a failed SLOG with zpool replace.
• Check zfs get sync per dataset. Do not set sync=disabled on databases or NFS exports to chase latency; that trades crash consistency for speed.
• If there is no SLOG and the workload is sync-heavy (NFS, databases), adding a low-latency, power-loss-protected SLOG device is the structural fix. Mirror it: an unmirrored SLOG that fails across a crash can lose recently synced data.
• Tradeoff: a SLOG only helps synchronous writes. If syncq_wait is low and asyncq_wait is high, a SLOG does nothing.

TXG pressure and dirty data throttling

• If a scrub or resilver is consuming I/O bandwidth during a write burst, pausing the scrub (zpool scrub -p <pool>, where supported) relieves the pressure at the cost of delaying integrity verification.
• Identify and throttle the write source if possible (bulk loads, rsync, backup jobs).
• zfs_dirty_data_max can be raised on systems with RAM headroom to absorb larger bursts. Tradeoff: bigger bursts mean bigger TXG syncs, which can extend stime further if the backend is the real constraint. Raising the limit when the disks genuinely cannot keep up just moves the stall.
• Do not reach for hardware replacement here. This pattern (high asyncq_wait, low disk_wait, periodic write freezes) is ZFS working as designed against a flush-rate problem.

Backend saturation or single slow vdev

• If one vdev is consistently much slower than its peers and its error counters are incrementing, plan proactive replacement before it faults. In a RAIDZ1 vdev this is urgent: you are one failure from data loss.
• If all vdevs are saturated (pend » active across the pool, disk_wait high), the pool needs more IOPS: add vdevs, or move hot datasets. ZFS stripes across top-level vdevs, so adding vdevs helps new writes immediately; existing data is not restriped.
• Check dmesg for SATA/SAS link resets and controller timeouts before assuming media failure.

Capacity and fragmentation

• Prune snapshots holding space: zfs list -t snapshot -o name,used,refer -s used -r <pool>. Deleting files does not free space that snapshots still reference.
• There is no in-place defragmentation in ZFS. Scrub does not defragment. The only real fix for severe fragmentation is zfs send | zfs recv into a fresh pool. Plan for this; it is not an incident-time action.
• Keep write-heavy pools below ~85% capacity. See ZFS capacity planning.

ARC-driven read latency

• If memory_throttle_count is incrementing and ARC size is falling under application pressure, set an explicit zfs_arc_max so the ARC and applications stop fighting. See ZFS zfs_arc_max tuning and ZFS ARC and the OOM killer.
• If the working set simply outgrew RAM, the fix is more RAM, an L2ARC that actually fits the workload, or reducing the working set.

Prevention

• Baseline the histogram, not the average. Collect zpool iostat -w distributions continuously so tail baselines exist before the incident. Alerting on averages alone misses the tail and fires late.
• Alert on the split, not the total. Track total_wait minus disk_wait per pool. A widening gap is an early warning of TXG or ZIL trouble while users are still happy.
• Track TXG stime and dirty data pressure as leading indicators. They move before latency does.
• Trend capacity and fragmentation together. The latency they cause arrives gradually and is expensive to reverse.
• Monitor SLOG device health and wear via device-level SMART/NVMe indicators. There is no ZFS-level kstat for SLOG write volume.
• Schedule scrubs deliberately and annotate their windows, so latency inflation during scans is recognized as expected instead of paged on.
• Cover the baseline signals in ZFS monitoring checklist: the signals every production pool needs so latency alerts arrive with the context (queue depth, TXG state, ARC pressure) already attached.

How Netdata helps

• Netdata’s ZFS collector charts pool I/O latency per pool, so the total-vs-disk split is visible as a time series rather than a one-off command during the incident.

• ARC statistics from /proc/spl/kstat/zfs/arcstats (size, hit rate, memory_throttle_count) are charted alongside pool latency, which makes the “read latency up because ARC is shrinking” correlation immediate.
• TXG sync duration from the per-pool txgs kstat can be correlated with write latency spikes to confirm TXG pressure instead of guessing.
• Per-vdev throughput and queue depth views let you spot the single slow device in a mirror or RAIDZ group without sampling zpool iostat -v by hand.
• Because scrub windows, capacity, and fragmentation are charted on the same host, legitimate latency inflation from scans is visually separable from a real regression.

Related guides

• ZFS ARC hit ratio low: cache misses, cold caches, and working sets that outgrew RAM
• ZFS zfs_arc_max: capping the ARC without starving read performance
• ZFS ARC and the OOM killer: applications killed while the cache will not shrink fast enough
• ZFS ARC shrinking below c_max: reading memory pressure before latency hits
• ZFS ARC using all memory: the Linux default that eats your RAM
• ZFS capacity planning: runway estimation before the pool fills
• ZFS checksum errors (CKSUM): the definitive signal of silent corruption
• ZFS checksum errors on multiple devices: suspect RAM or the controller, not the disks
• ZFS device UNAVAIL or REMOVED: a disk that fell off the bus
• ZFS dirty data throttling: the write delay that masquerades as slow disks
• How ZFS actually works in production: a mental model for operators
• ZFS monitoring checklist: the signals every production pool needs