ZFS I/O queue depth: telling backend saturation apart from a hang

The pool is slow. Applications are timing out on writes, or reads that used to take milliseconds now take seconds. The first question that decides everything else is this: is the storage backend working hard and falling behind, or has something actually stopped moving? Both look identical from the application side. Both show up as “high latency.” The fix for one (more IOPS, better devices, workload shaping) is completely different from the fix for the other (a dead disk, a stuck controller, a hung I/O that needs intervention).

zpool iostat -q answers that question directly. It shows, per vdev and per queue type, how many I/O operations are pending (waiting to be issued to the device) and how many are active (issued and waiting for completion). The relationship between those two counters, sampled over time, separates “latency is high because the backend is saturated” from “latency is high because something is hung.”

What this means

ZFS submits I/O to devices through per-vdev queues, split by type: synchronous reads and writes, asynchronous reads and writes, plus queues for scrub, TRIM, and rebuild work. Each queue holds requests in one of two states:

• pend: the request is queued in ZFS and has not yet been issued to the device. ZFS is deliberately holding it, either because the queue’s concurrency limit is reached or because the device is being throttled.
• activ: the request has been issued to the device and is waiting for completion. At this point the timing belongs to the hardware, the HBA, and the kernel block layer, not to ZFS scheduling.

That split gives you the diagnostic lever. Saturation and hangs live on opposite sides of it:

• Saturation: I/O arrives faster than the device can service it. ZFS fills the active slots up to the queue’s concurrency cap, and the overflow accumulates in pend. You see pend growing while activ stays pinned at a stable ceiling. I/O is completing, just not fast enough.
• Hang: requests get issued and never come back. Activ grows or stays stuck high while completions stall and pend keeps climbing behind them. The device (or the bus, or the controller) is not answering.

One measurement subtlety matters before you start interpreting: all queue statistics in zpool iostat -q are instantaneous measurements sampled at the end of the interval, not averages. A single sample can miss a burst or catch one. Never draw a conclusion from one row of output. Watch the counters move across several samples.

flowchart TD
    A[High storage latency] --> B[zpool iostat -q -v 1, several samples]
    B --> C{pend growing?}
    C -- "no, queues shallow" --> D[Problem inside ZFS: TXG, ARC, fragmentation]
    C -- "yes" --> E{activ at stable ceiling, I/O completing?}
    E -- "yes" --> F[Backend saturation]
    E -- "activ stuck high, no completions, disk_wait climbing" --> G[Hung device or bus]
    C -- "syncq pend high, asyncq low" --> H[ZIL/SLOG bottleneck]

Common causes

Quick checks

All of these are read-only and safe to run during an incident.

# Watch queue depth per vdev, per queue type, 1-second samples
zpool iostat -q -v 1

# Add average latency columns (total_wait, disk_wait, syncq_wait, asyncq_wait)
zpool iostat -l -v 1

# Latency histograms for tail behavior
zpool iostat -w -v 5

# Pool and vdev state, plus error counters
zpool status -v

# Check for a scrub or resilver that explains queue activity
zpool status | grep -A5 scan

# Kernel messages: link resets, task aborts, device timeouts
dmesg | grep -i -E "ata|sas|reset|timeout"

# Deadman events: ZFS's own hung-I/O detector
zpool events -v | grep -i deadman

Notes on interpretation:

• zpool iostat -q without -v shows only top-level vdevs, which can hide the one member device or log vdev that is actually the problem. Always add -v when hunting a bottleneck.
• The latency columns (-l) and histograms (-w) are not available on very old releases; if they are missing, you have -q and plain throughput output only.
• The queue types shown are syncq_read, syncq_write, asyncq_read, asyncq_write, scrubq_read, and where supported trimq_write and rebuildq_write.

How to diagnose it

1. Establish the shape of the problem. Run zpool iostat -q -v 1 and watch at least 30 seconds of samples. Because samples are instantaneous, one screenshot proves nothing. You are looking for trends: which counters move, and in which direction.

2. Check whether queues are deep at all. If pend and activ are near zero across all vdevs while applications report high storage latency, the backend is not the bottleneck. The problem is inside ZFS itself: TXG sync pressure, ARC starvation, or fragmentation. Look at TXG stime and ARC pressure, not at disks.

3. If pend is growing, check activ. Activ pinned at a stable ceiling with completions flowing is saturation. The device is doing everything it is allowed to do; demand exceeds supply. This is the “busy, not broken” case.

4. If activ is stuck high, check whether anything completes. This is the hang signature. Confirm it with zpool iostat -l: on a saturated-but-working device, disk_wait is elevated but bounded and roughly stable. On a hung device, disk_wait climbs without bound as outstanding requests age. Cross-check dmesg for SATA/SAS link resets, task aborts, and timeouts, and check zpool events -v for deadman events. The deadman subsystem exists precisely for this case: it fires when an individual I/O has been stuck for about 5 minutes (zfs_deadman_ziotime_ms, default 300000 ms) or a pool sync for about 10 minutes (zfs_deadman_synctime_ms, default 600000 ms).

5. Split by queue type. If syncq_write pend is high while the async queues are quiet, the bottleneck is the synchronous write path: the ZIL, or the SLOG device if you have one. Check the log vdev’s state and queues specifically in the -v output. This is the signature of databases and NFS workloads stalling while the rest of the pool looks fine.

6. Split by vdev. In a multi-vdev pool, the slowest vdev gates the whole stripe for many operations. If one vdev’s queues are deep and its latency is 3x or more above its peers, you have a device-level problem even if nothing has errored yet. A dying disk often shows up here first, as a slow device with zero error counts, before SMART or ZFS counters move.

7. Rule out dirty data throttling for async writes. The async write queue’s effective concurrency is dynamic: ZFS throttles writers as dirty data builds (starting at zfs_delay_min_dirty_percent, default 60% of zfs_dirty_data_max). A growing asyncq_write pend during a write burst can mean ZFS is deliberately slowing writers to protect memory, not that the disk is saturated. Check TXG stime in /proc/spl/kstat/zfs/<pool>/txgs and the dirty data level before blaming hardware. See ZFS dirty data throttling for that failure mode in detail.

8. Rule out background work. A scrub or resilver legitimately fills scrubq_read or rebuildq_write and competes with user I/O. Check the scan line in zpool status before concluding anything about the devices.

Metrics and signals to monitor

Fixes

If it is saturation

The devices are healthy but undersized for the workload, or the workload changed. Options, in order of increasing commitment:

• Throttle or reschedule the heaviest writer. Bulk jobs, backups, and scrubs compete with foreground I/O. A scrub running during peak hours is the most common self-inflicted saturation.
• Add vdevs. Pool IOPS scales with top-level vdev count, not with disks per RAIDZ vdev. Widening the pool is the structural fix.
• Move hot data to faster media, or add a special vdev if metadata and small blocks are the bottleneck.
• Tune queue concurrency cautiously. The ZIO scheduler exposes per-queue min_active/max_active tunables (for example zfs_vdev_async_read_max_active) plus the aggregate zfs_vdev_max_active. Raising them can help fast NVMe devices that the spinning-disk defaults underutilize, but raising them on a genuinely saturated device just moves the queue deeper into the hardware and raises latency. Treat this as a last resort and change one value at a time.

If it is a hang

A hung I/O is a hardware or driver event, and the response is containment, not tuning:

1. Confirm with disk_wait, dmesg, and zpool events before acting. Deadman events are ZFS telling you it agrees.
2. If the pool has redundancy, offline the offending device (zpool offline <pool> <device>) to stop I/O from queueing behind it. This stops new I/O to that device but leaves pool data safe as long as redundancy is intact. Do not offline anything in a degraded or non-redundant vdev: that can take the pool down or lose data.
3. Investigate the physical path: cable, backplane, controller, firmware. Hung devices with link resets in dmesg are often cabling or power, not the disk itself.
4. Note that zfs_deadman_failmode defaults to wait, meaning ZFS will sit on the hung I/O indefinitely rather than fail it. That is why the pool can look frozen while still reporting ONLINE.

If it is the ZIL/SLOG

• Check the log vdev state in zpool status -v. A failed SLOG falls back to the in-pool ZIL: sync writes keep committing, but latency jumps by orders of magnitude, and nothing in the pool name tells you why. Replace the device.
• If the SLOG is healthy but its queue is deep, the sync write rate exceeds what the device can commit. That is a sizing problem: faster SLOG, or reduce unnecessary sync semantics at the application layer. Do not set sync=disabled on datasets that need durability.

If it is dirty data throttling

The queue is a symptom; the write pipeline is the cause. Confirm with TXG stime and dirty data level, then follow the throttling runbook rather than touching disks.

Prevention

• Trend queue depth, do not just spot-check it. Sustained pend growth over weeks is a saturation runway warning long before applications notice. A common headroom rule: keep sustained device utilization under about 70% of rated maximum.
• Baseline per-vdev behavior. Queue depth thresholds are relative. Know what normal looks like per vdev per queue type so a drift stands out.
• Alert on deadman events through ZED. zpool events is in-memory only and lost on reboot. ZED persistence is the difference between catching a hung device at minute five and finding it in the morning.
• Track the composite, not the single counter. Saturation tickets fire on sustained high pend relative to baseline; hangs escalate through the deadman signal. Latency alone should not page: scrubs and resilvers legitimately inflate it.
• Schedule scrubs and resilvers into known windows and annotate them in your monitoring, so their queue activity is never mistaken for a device problem.

How Netdata helps

• Continuous per-disk latency, throughput, and utilization charts fill the gap between manual zpool iostat samples: a pend buildup that you would miss between end-of-interval snapshots shows up as rising device latency over time.
• Per-device breakdowns make queue asymmetry visible without live sampling during an incident, which is how you catch the single slow member in a mirror or RAIDZ group before it errors.
• Correlating disk latency against throughput on the same dashboard mirrors the saturation-versus-hang decision: rising latency with steady throughput is saturation; stalled throughput with climbing latency is a hang.
• Alerting on device latency anomalies and on pool state changes closes the loop: queue-drift trends warn early, and deadman-style signals confirm when an I/O has actually stopped.

Related guides

• ZFS ARC hit ratio low: cache misses, cold caches, and working sets that outgrew RAM
• ZFS zfs_arc_max: capping the ARC without starving read performance
• ZFS ARC and the OOM killer: applications killed while the cache will not shrink fast enough
• ZFS ARC shrinking below c_max: reading memory pressure before latency hits
• ZFS ARC using all memory: the Linux default that eats your RAM
• ZFS capacity planning: runway estimation before the pool fills
• ZFS checksum errors (CKSUM): the definitive signal of silent corruption
• ZFS checksum errors on multiple devices: suspect RAM or the controller, not the disks
• ZFS device UNAVAIL or REMOVED: a disk that fell off the bus
• ZFS dirty data throttling: the write delay that masquerades as slow disks
• How ZFS actually works in production: a mental model for operators
• ZFS monitoring checklist: the signals every production pool needs