ZFS resilver slow or stalled: multi-day rebuilds and the second-failure race

A disk was replaced, zpool status shows resilver in progress, and the ETA says three days. Or worse: the scanned byte count has not moved in twenty minutes. Either way, the pool is running with reduced redundancy, and every hour the resilver takes is an hour where the next disk failure becomes a data-loss event.

Two facts frame everything below. First, ZFS deliberately throttles resilver I/O so production traffic wins. A slow resilver is often the system working as designed; the throttle-vs-risk trade-off is a decision you make, not an accident. Second, on large RAIDZ pools of spinning disks, a multi-day resilver is normal arithmetic, and a resilver that is decelerating frequently means a second device in the same vdev is also failing. That second case is the one that kills pools.

This article covers how to tell “slow by design” from “slow because something is wrong”, how to baseline the expected rebuild time, and what to do when the progress counter stops moving.

What this means

Resilver is ZFS reconstructing data onto a replaced or faulted device. The mechanics differ by topology, and the difference drives rebuild time:

• Mirror vdevs resilver by copying the device, mostly sequential I/O, proportional to device size. OpenZFS 2.0 also offers sequential resilver (zpool replace -s), which works on mirrors and dRAID but not RAIDZ, and automatically runs a scrub afterward to verify checksums.
• RAIDZ resilver walks the block pointer tree and reconstructs only allocated blocks. That is proportional to used space rather than device size, but the I/O pattern is driven by metadata layout and can be effectively random on a fragmented pool. This is why a 60%-full RAIDZ2 of 18 TB HDDs can still take days.

During the entire window, the pool is DEGRADED. On RAIDZ1, one more device failure in that vdev is total data loss. RAIDZ2 buys you one more failure, but the window is still the highest-risk state the pool will be in. The goal is not “make resilver fast at all costs”. It is “make the vulnerability window as short as possible without taking production down”.

A stall is a different event. ZFS operators define it roughly as zero bytes of progress for 10+ minutes. Slow is a trade-off; stalled is a bug, a hung device, or a dying second disk, and it needs active diagnosis.

Common causes

Quick checks

All of these are read-only and safe on a production pool.

# Overall pool and scan state
zpool status -x
zpool status | grep -A5 scan

# Take two progress samples to distinguish slow from stalled
zpool status | grep -A5 scan
sleep 600
zpool status | grep -A5 scan

# Per-vdev throughput: which device is limiting the rebuild
zpool iostat -v 1

# Per-vdev latency: is one surviving disk much slower than its peers
zpool iostat -l -v 1
# TODO: verify -l flag availability by OpenZFS version; latency columns were added in later 2.x releases

# Queue depths: saturation (pend >> activ) vs a hang
zpool iostat -q -v 1

# Error counters on every device; zero is the only acceptable value
zpool status -v

# Kernel-level device distress
dmesg | grep -i -E "ata|sas|reset|timeout" | tail -50

# Hung I/O detection (in-memory only; check ZED logs for history)
zpool events -v | grep -i deadman

# SMART on the surviving members of the degraded vdev
smartctl -A /dev/sdX

# Current resilver/scan controls (module parameters, live view)
cat /sys/module/zfs/parameters/zfs_resilver_min_time_ms
cat /sys/module/zfs/parameters/zfs_scan_vdev_limit
ls /sys/module/zfs/parameters/ | grep -i -E "resilver|scan"

How to diagnose it

The core question: is the rebuild progressing at the rate physics allows, slower than physics allows, or not progressing at all?

1. Confirm the scan is running and note the numbers. zpool status shows bytes scanned, current rate, percentage, and ETA on the scan: line. Write down the scanned-bytes value. Treat the ETA with suspicion: it is computed from the average rate since start, so it is routinely inaccurate, especially early.

2. Baseline the expected time. Estimate from used capacity and realistic device throughput, not from the ETA. For RAIDZ, the driver is used space: a pool with 40 TB allocated, rebuilt through surviving HDDs that can each sustain roughly 150-200 MB/s sequential under shared load, lands in the tens-of-hours range before you subtract the throttle and production contention. For mirrors, the driver is full device size but the I/O is sequential. If your observed rate is within a plausible factor of that estimate, you are looking at a long-but-normal rebuild. If it is 10x off, keep digging.

3. Distinguish slow from stalled. Take two zpool status samples 10 minutes apart. If the scanned-bytes counter has not moved, that is a stall. If it moved, compute the effective rate yourself ((bytes2 - bytes1) / 600) and compare against step 2.

4. Check for a second failing device. This is the highest-stakes branch. In zpool status, look at the READ, WRITE, and CKSUM columns on the surviving members of the degraded vdev. Any non-zero value, especially one that increments between samples, is a dying disk. Cross-check with zpool iostat -v 1 (one device markedly slower than its peers) and smartctl -A on that device. A decelerating resilver plus climbing error counters means you are losing the second-failure race in real time.

5. Separate throttling from hardware. During a quiet window, watch zpool iostat -q -v 1. If pending queues on the data vdevs are near-empty while the resilver rate stays low, the scan engine is yielding to (nonexistent) production I/O and the throttle is the limiter. If pending queues are deep and latency is up, the backend is saturated and the resilver is competing for real bandwidth.

6. If stalled, look for hung I/O. Check zpool events -v | grep -i deadman. Deadman events mean an individual I/O has been stuck for minutes, which points at a device, cable, or controller problem rather than ZFS scheduling. Correlate with dmesg for link resets or timeouts. Sequential scans checkpoint progress (by default every two hours) so a reboot resumes from the last checkpoint rather than always from zero. A replacement that starts while resilver progress is below zfs_resilver_defer_percent (default 10%) can restart from scratch; check zpool history for repeated scan starts.

flowchart TD
  A[Resilver running] --> B{Progress moving over 10 min?}
  B -- No --> C[Stalled scan]
  C --> D[Check deadman events and dmesg]
  D --> E[Device, cable, or controller fault]
  B -- Yes --> F{Rate near physics estimate?}
  F -- Yes --> G[Normal long rebuild: monitor and wait]
  F -- No, slower --> H{Error counters climbing on survivors?}
  H -- Yes --> I[Second device failing: escalate now]
  H -- No --> J{Queues deep, latency high?}
  J -- Yes --> K[Production contention or saturated backend]
  J -- No --> L[Throttle-limited: consider tuning trade-off]

Metrics and signals to monitor

Fixes

If the rebuild is normal but long

Do less. The default throttling exists to keep production latency acceptable, and on large HDD RAIDZ pools a multi-day rebuild is expected behavior. Reduce discretionary load (backups, scrubs on other pools sharing the enclosure, bulk jobs), keep watching the error counters on survivors, and let it finish. Resilver and scrub are mutually exclusive per pool, and a resilver preempts a running scrub, so do not expect scrub results while this is ongoing.

If the throttle is the limiter and you accept the trade-off

You can raise resilver priority with current scan controls such as zfs_resilver_min_time_ms (default 1500 ms) and zfs_scan_vdev_limit (default 16 MiB per leaf device) under /sys/module/zfs/parameters/. This is a live, reversible change, but understand what you are buying: every extra I/O the scan issues is an I/O production traffic does not get, and latency will rise. Change one parameter at a time, measure the effect on both the scan rate and application latency, and record the original values so you can revert. A reasonable policy is to raise resilver priority during off-peak hours and restore defaults during peak.

If production is suffering and you need relief now

On supported versions, zpool scrub -p <pool> pauses an active scan, including a resilver. This is the correct lever when the alternative is an application incident, but be explicit about the cost: pausing extends the reduced-redundancy window. Pause during the peak, resume immediately after.

If a second device is failing

The resilver is no longer the event; the second disk is.

• Stop all non-essential write load to reduce stress on the degraded vdev.
• Pull SMART data on the suspect device and prepare its replacement immediately.
• If the suspect device is erroring hard, offlining it may reduce error-storm contention, but only if the remaining redundancy still covers the vdev. On RAIDZ1 already down one disk, offlining a second device takes the pool to FAULTED. Know your topology before you touch anything.
• If errors are appearing on multiple unrelated devices simultaneously, suspect RAM or the controller rather than the disks and follow the multi-device checksum failure path instead.

If the scan is stalled with no progress

Work the hardware path, not the ZFS path: deadman events, dmesg link resets, per-device latency, cabling and controller health. A resilver frozen at zero bytes is almost always a device that is physically present but not servicing I/O. Resolve the underlying fault, then confirm the scan resumes. Avoid repeated reboots as a diagnostic: they do not fix hung devices and they cost you whatever progress was not yet durable.

For mirror pools: use sequential resilver next time

On supported OpenZFS versions, zpool replace -s performs a sequential resilver in a mirror vdev that is substantially faster than the default healing resilver, followed by an automatic scrub to verify integrity. It is not available for RAIDZ. For RAIDZ pools, the levers are topology planning and capacity headroom, covered below.

Prevention

• Baseline before you need it. Record actual resilver times per pool after every replacement, alongside used capacity. A stored baseline turns “is this slow?” into a comparison instead of a guess.
• Treat DEGRADED as a same-day ticket. The pool stays online after a disk failure, which makes it easy to defer. Every day deferred is a day in the single-failure-away state. Replacements and spares should be stocked before the failure, not ordered after.
• Prefer RAIDZ2 or RAIDZ3 for large spinning disks. The larger the disks, the longer the rebuild, and the more a single parity device is worth. RAIDZ1 on multi-TB HDDs is where multi-day resilvers and single-failure tolerance combine into the worst risk profile.
• Scrub on schedule and alert on results. Scrubs surface slowly dying disks (rising repaired-error counts) weeks before they fail outright, which lets you replace proactively instead of resilvering under pressure.
• Configure hot spares and autoreplace where appropriate, and verify ZED is alerting on pool events so DEGRADED and deadman events page someone rather than sitting in zpool status.
• Monitor fragmentation and capacity. A heavily fragmented, near-full pool resilvers slower and scrubs slower. Both trends are visible long before a disk fails.

How Netdata helps

• Resilver progress as a time series: Netdata tracks pool scan state continuously, so you can see the rebuild rate trend and spot deceleration or a flatline instead of relying on point-in-time zpool status checks.
• Per-vdev error counters alongside the rebuild: READ, WRITE, and CKSUM counts per device are charted next to scan progress, which is exactly the correlation that exposes a second failing disk mid-resilver.
• Per-vdev latency and queue depth: correlating device latency with scan rate separates “throttled by design” from “backend saturated” from “one disk dying” without manual iostat sessions.
• Pool state transitions and ZED events: DEGRADED, additional vdev failures, and deadman events become alerts with history, not lines you had to be watching to catch.
• Latency impact of the throttle trade-off: application-visible storage latency during the rebuild lets you tune resilver priority against a measured cost instead of guessing.

Related guides

• ZFS device UNAVAIL or REMOVED: a disk that fell off the bus
• ZFS checksum errors (CKSUM): the definitive signal of silent corruption
• ZFS checksum errors on multiple devices: suspect RAM or the controller, not the disks
• ZFS I/O queue depth: telling backend saturation apart from a hang
• ZFS dirty data throttling: the write delay that masquerades as slow disks
• ZFS capacity planning: runway estimation before the pool fills
• How ZFS actually works in production: a mental model for operators