ZFS one slow disk in a vdev: the dying drive that drags the whole pool

The pool is ONLINE. zpool status -x says all pools are healthy. Error counters are zero or near zero. And yet write latency has doubled, TXG syncs are stretching, and applications are complaining about storage. When one disk in a vdev starts dying, it usually does not fail cleanly. It develops bad sectors, and the drive firmware starts retrying reads and remapping blocks internally. Each retry adds milliseconds to individual I/Os. The disk never errors out, so ZFS never faults it.

In a RAIDZ vdev this is especially punishing: a stripe write is not complete until every child disk has written its part, so the entire vdev moves at the speed of its slowest disk. One drive running at a third of its peers’ speed gates every write through that vdev. Mirrors are more forgiving on reads (ZFS can read from the faster leg) but a slow mirror leg still drags writes.

This guide walks through confirming that a single slow disk is the cause, identifying which disk it is, and replacing it before it finishes dying. If the pool is RAIDZ1, treat this as urgent: redundancy is intact today, but the drive is actively failing, and one more failure in that vdev is data loss.

What this means

A drive with developing media defects handles them internally first: ECC retries, head re-seeks, spare-sector remapping. These operations are invisible to the OS as errors because the drive eventually returns good data. What the OS sees is latency. The drive that used to answer in 8 ms now takes 30 ms, then 100 ms, with a long tail of multi-second outliers when it hits a bad region.

Because the drive still answers, ZFS keeps it ONLINE. There is no state transition to alert on. The only signals are performance signals: per-device latency divergence, stretched TXG sync times, and eventually slow-I/O counters. This is why the pattern catches teams off guard. Every health check they have is state-based, and the state never changes until the drive falls off the bus completely, which is the worst possible time to find out.

flowchart TD
  A[Disk develops bad sectors] --> B[Firmware retries and remaps internally]
  B --> C[Per-I/O latency rises, no errors returned]
  C --> D[Stripe write waits for slowest child]
  D --> E[Whole vdev slows to that disk's speed]
  E --> F[TXG sync time extends]
  F --> G[Dirty data accumulates, writes throttle]
  G --> H[Application write latency spikes]
  C -.->|pool still shows| I[ONLINE, zero or few errors]

Two follow-on effects matter. First, extended TXG sync times mean the open transaction group accumulates more dirty data, which can push the pool toward the write throttling described in ZFS dirty data throttling: the write delay that masquerades as slow disks. The slow disk and the throttle look similar from the application side; the per-device latency check below is how you tell them apart. Second, if the drive degrades to the point of returning corrupt data, you start seeing checksum errors. At that point you are in ZFS checksum errors (CKSUM): the definitive signal of silent corruption territory.

Common causes

The distinguishing question is always the same: is the slowness localized to one device, or distributed? One device means hardware at that device or its path. All devices means look at ZFS internals, the controller, or the workload.

Quick checks

All read-only and safe to run during an incident.

# Per-vdev and per-device latency, 1-second samples, 5 iterations
# Look for one device whose total_wait is 3x or more above its vdev peers
zpool iostat -l -v <pool> 1 5

# Latency histograms: averages hide outliers, this shows the tail
zpool iostat -w -v <pool> 5

# Queue depths: pending >> active on one device points at that device
zpool iostat -q -v <pool> 1 5

# Current pool state and cumulative error counters per device
zpool status -v <pool>

# Kernel messages for link resets, timeouts, task aborts
dmesg | grep -i -E "ata|sas|reset|timeout" | tail -50

# Recent TXG sync times: is write latency explained by slow syncs?
cat /proc/spl/kstat/zfs/<pool>/txgs | tail -20

# Slow I/O counters per device (OpenZFS 2.0+)
zpool status -s <pool>

Notes on interpreting these:

• In zpool iostat -l output, total_wait is the number to compare first (queue time plus disk time). disk_wait isolates the device itself. If one disk’s disk_wait is several times its peers’, the drive or its path is the problem, not ZFS queueing.
• Averages lie. A disk serving most I/Os at 10 ms but stalling for seconds on a bad region can show a deceptively reasonable average. Use -w histograms to see the tail before concluding anything.
• zpool status -s counts slow I/Os per leaf vdev, where “slow” means the I/O took longer than zio_slow_io_ms (default 30000 ms). That default is very forgiving: a disk taking 2 to 10 seconds per I/O will cripple the pool without ever incrementing the counter. A zero slow-I/O count does not clear the disk.
• Error counters in zpool status are cumulative since the last zpool clear. A dying disk in the latency phase often shows zero errors. That is expected and does not contradict the latency evidence.

How to diagnose it

1. Confirm the symptom is storage. From the application side you see write stalls or elevated fsync/commit latency. Check zpool iostat -l -v <pool> 1 5. If pool-level latency is elevated, keep going. If pool latency is normal, the problem is not this pool.

2. Localize to a vdev. With per-vdev output, find which top-level vdev carries the latency. In a multi-vdev pool, one slow vdev shows clearly against the others.

3. Localize to a device. Within that vdev, compare child devices. The signature of a dying disk is one device with total_wait and disk_wait persistently 3x or more above its siblings, across several sample intervals, under steady load. A single sample is not enough; sample for at least a minute under representative load.

4. Rule out ZFS-internal causes. Check TXG sync times (/proc/spl/kstat/zfs/<pool>/txgs, stime column) and queue depths (zpool iostat -q). If all devices show equally high latency with deep pending queues, you are looking at saturation or dirty-data throttling, not one bad disk. If queue depths are flat and one device is slow, that device is the problem.

5. Check for transport problems. Run the dmesg grep above. SATA link resets, task aborts, and device timeouts on the same disk point at the cable, backplane slot, or controller port rather than the media. Note which slot the disk is in.

6. Confirm with SMART. Pull the disk’s serial from zpool status -v (or by matching the device path), then:

   # SMART attributes for the suspect disk
   smartctl -A /dev/sdX
   
   # Kick off a short self-test, then read results after it completes
   smartctl -t short /dev/sdX
   smartctl -l selftest /dev/sdX
   

   On spinning disks, watch Reallocated_Sector_Ct, Current_Pending_Sector, and Offline_Uncorrectable. Any non-zero pending or uncorrectable count, or a rising reallocation count, confirms the diagnosis. Be aware of the trap: a disk in the early retry phase can show completely clean SMART data while still being the slow device in the vdev. Clean SMART does not exonerate a disk that latency data has already convicted.

7. Decide. One device, persistently slow, clean transport logs, SMART degrading or even neutral: replace it proactively. The drive is telling you it is dying. Believe it before it finishes.

Metrics and signals to monitor

Fixes

Replace the disk proactively

This is the real fix, and it is worth doing before total failure rather than after. A planned replacement of a slow-but-working disk is faster and safer than an emergency replacement of a dead one, because ZFS can read from the old disk during the resilver instead of reconstructing everything from parity.

# Replace in place; the pool keeps redundancy from the old disk where possible
zpool replace <pool> <old-device> <new-device>

# Watch progress
zpool status <pool>

Expect the resilver to add significant I/O load for hours to days on large HDD vdevs. During that window the pool is exposed: on RAIDZ1, a second failure in the same vdev loses the pool. Do not defer this on RAIDZ1. On RAIDZ2/Z3 you have more slack, but the slow disk is still degrading, and “replace at the earliest convenience” means days, not quarters. Resilver mechanics and the redundancy window are covered in more depth in ZFS device UNAVAIL or REMOVED: a disk that fell off the bus.

Temporary mitigation: take the slow disk offline

If the slow disk is crushing production latency and you cannot replace it immediately, offlining it removes it from the I/O path:

# Disruptive: reduces redundancy to zero on RAIDZ1. Have the replacement in hand first.
zpool offline <pool> <device>

The vdev continues from parity (RAIDZ) or the other leg (mirror), at full remaining-device speed. The tradeoff is explicit: you trade redundancy for latency. On RAIDZ1 this leaves you with zero fault tolerance, so treat it strictly as a bridge to immediate replacement, not a workaround.

OpenZFS 2.4.0 added vdev properties (autosit, sit_out) for RAIDZ and dRAID. When autosit=on, slow-outlier detection can make a child sit out normal reads and reconstruct it from parity instead of waiting on that disk; autosit defaults to off and outlier detection requires at least five children. It does nothing for writes, which still go to every child to maintain redundancy, and it does not fix the disk. Replacement is still the fix.

If the evidence points at the path, not the media

If dmesg shows link resets and SMART is clean, swap the cable or move the disk to a different slot or port before replacing it. If the latency follows the disk, it is the disk. If it stays on the slot, it is the cable, backplane, or controller port. Multiple slow disks on one controller means the controller or its expander: do not start replacing disks in that case.

If SMART is clean but latency says the disk is guilty

This happens. A drive in the retry phase can burn enormous time on internal recovery without logging a single reallocated sector yet. When per-device latency data consistently identifies one disk, the latency data wins. Replace the disk and let the vendor’s diagnostics argue about it afterward.

Prevention

You cannot prevent disks from dying. You can prevent the dying phase from being invisible.

• Alert on per-device latency divergence, not just pool state. Pool state will not change until the disk is gone. A relative alert (one device’s p95 latency more than 3x its vdev peers for several minutes) catches the pattern in its cheap phase.
• Trend per-device latency over weeks. A disk that gets 20% slower month over month is on the way out. Point-in-time zpool status checks miss this entirely because the counters that matter are performance counters, not error counters.
• Scrub on schedule and read the results. Scrubs surface bad regions early by reading everything. A scrub that suddenly takes much longer on the same data volume is itself a device-degradation signal.
• Baseline resilver time. Know roughly how long a resilver takes on your pools so you can size the risk window when you replace a disk, especially on RAIDZ1.
• Avoid drive-managed SMR disks in RAIDZ. Their internal garbage collection produces exactly this slow-device signature under write load, with no actual defect to replace.
• Keep spares on the shelf. The difference between “urgent” and “routine” for this incident is often whether a compatible replacement disk is already in the building.

How Netdata helps

• Netdata collects per-device ZFS latency, throughput, and queue metrics continuously, so the “one disk 3x slower than its peers” pattern shows up as a visible divergence in a time series rather than something you have to catch with a lucky zpool iostat during the incident.
• Correlating per-device latency with TXG sync time on the same dashboard tells you immediately whether a write stall is one slow disk or a pool-wide pipeline problem like dirty-data throttling.
• Pool error counters (READ/WRITE/CKSUM per device) are tracked over time, so the transition from “slow but clean” to “slow and erroring” triggers attention before the disk faults.
• Scrub duration trended over time exposes the slow scrub that often precedes a confirmed disk failure.
• Retained high-resolution history lets you look back and see when the disk started degrading, which is useful both for confirming the diagnosis and for vendor warranty claims.

Related guides

• ZFS dirty data throttling: the write delay that masquerades as slow disks
• ZFS device UNAVAIL or REMOVED: a disk that fell off the bus
• ZFS checksum errors (CKSUM): the definitive signal of silent corruption
• ZFS checksum errors on multiple devices: suspect RAM or the controller, not the disks
• ZFS monitoring checklist: the signals every production pool needs
• How ZFS actually works in production: a mental model for operators
• ZFS capacity planning: runway estimation before the pool fills