ZFS resilver in progress: the reduced-redundancy window after a disk replace

You replaced a failed disk, zpool status now shows scan: resilver in progress, and the pool is DEGRADED. This is normal recovery, but it is not a safe state. Until the resilver completes, the affected vdev is running with reduced redundancy, and every hour in this window is an hour where one more failure can take the pool down.

The playbook rule: a resilver in progress is a TICKET, not a page, but it is a ticket you actively manage. You do not close it because the replacement is physically seated. You close it when the scan finishes, the vdev is back at full redundancy, and the error counters on the surviving disks are still zero.

This article covers how to read the progress output, why the ETA is unreliable, what the real risk is for your topology, and what to watch on the surviving disks while the rebuild runs.

What this means

A resilver reconstructs the data that belongs on the replaced device by reading the surviving devices in the same vdev. Two properties matter operationally:

• Redundancy is reduced for the entire duration. On a mirror, you are down one side. On RAIDZ1, you have zero parity left: a second failure in the same vdev during the resilver is pool loss. On RAIDZ2 you drop to single-parity tolerance; RAIDZ3 drops to double-parity. The pool stays online and serves I/O, but the safety margin you sized the pool for is partially or fully gone.
• The rebuild hammers the surviving disks. The resilver reads from exactly the devices you are now depending on. This is the worst time to discover that a second disk in the group was also dying.

flowchart TD
  A[Disk fails or is replaced] --> B[Pool DEGRADED]
  B --> C[zpool replace triggers resilver]
  C --> D[Resilver in progress: reduced redundancy]
  D --> E{Resilver completes?}
  E -->|yes| F[Full redundancy restored]
  E -->|second failure in same vdev| G[RAIDZ1: data loss
RAIDZ2/Z3: tolerance consumed]
  D -->|sequential resilver| H[Auto scrub verifies checksums]
  F --> H
  H --> I[Pool back to healthy baseline]

RAIDZ and mirror resilvers are different workloads

The workload difference changes how you plan:

• RAIDZ resilver (healing resilver) walks the block tree. It reconstructs only allocated blocks, but the walk produces random read I/O across the surviving disks. On HDDs this is IOPS-bound, not bandwidth-bound, and it gets slower as the pool fills and fragments. Multi-day resilvers on large RAIDZ pools with spinning disks are common.
• Mirror resilver is sequential. It copies the device, with sequential I/O proportional to disk size. Much faster and much more predictable.
• Sequential resilver (zpool replace -s) speeds up mirrors and dRAID; the flag exists in OpenZFS 2.0, while dRAID itself was introduced in OpenZFS 2.1. It is not supported for RAIDZ. When a sequential resilver finishes, ZFS automatically starts a scrub to verify checksums, so expect continued background I/O after the “resilver done” line appears.

The -s flag is not the default; if you want sequential resilver on a mirror you must pass it explicitly at replace time.

Why the ETA lies

The scan: line shows bytes done, rate, percentage, and an ETA. Treat the ETA as a rough guess, especially in the first hour:

• The rate is an average since the resilver started. Early numbers include warmup and metadata-heavy phases and do not reflect steady-state throughput.
• RAIDZ healing resilvers are random I/O. The rate varies with where the block tree walk currently is, how fragmented the pool is, and how much production I/O is competing.
• ZFS throttles scan I/O to protect production traffic by default, so the rate drops when your workload is busy and recovers when it is idle. An ETA computed during a quiet night will be wildly optimistic about a busy day.

Use zpool iostat -v for actual per-device throughput instead of trusting the scan line’s rate column.

Quick checks

# Current resilver status and progress
zpool status <pool> | grep -A5 "scan:"

# Watch progress once a minute
watch -n 60 'zpool status <pool> | grep -A5 "scan:"'

# Real per-vdev throughput: is the rebuild actually moving?
zpool iostat -v <pool> 5

# Latency per vdev: are surviving disks struggling?
zpool iostat -l <pool> 5

# Queue depths: is production I/O piling up behind resilver I/O?
zpool iostat -q -v <pool> 5

# Error counters on the SURVIVING devices: this is the check that matters most
zpool status -v <pool>

# Kernel log for transport-level trouble on surviving disks
dmesg | grep -i -E "ata|sas|reset|timeout"

All of these are read-only and safe to run at any interval.

What to watch on the surviving disks

The most important monitoring target during a resilver is not the new disk. It is the error counters and latency of the disks the resilver is reading from. A resilver is effectively a full-surface read test of every surviving device in the vdev, and it routinely surfaces latent failures.

• READ/WRITE/CKSUM counters on surviving devices. Zero is the only acceptable value. A counter that starts incrementing during the resilver means the rebuild is reading a disk that is also failing. On RAIDZ1, that is the beginning of the worst-case scenario: stop non-essential I/O and escalate immediately.
• Per-vdev latency divergence. If one surviving disk is consistently slower than its peers in zpool iostat -l, it is a replacement candidate. A slow disk both extends the window and signals distress.
• Scrub interaction. Scrubs and resilvers are mutually exclusive per pool; the resilver preempts any scrub that was running. Do not try to force a scrub mid-resilver. If you used sequential resilver, the verification scrub starts automatically afterward.

Metrics and signals to monitor

Operating during the window

These are judgment calls, but they are the decisions a resilver forces on you:

• Do not defer it. DEGRADED with no resilver running is worse than resilver in progress. If the pool is DEGRADED and no scan is active, find out why: hot spare did not activate, replacement was never issued, or the replace failed.
• Reduce non-essential I/O if the topology is thin. On RAIDZ1 or a two-way mirror, the marginal value of shortening the window is high. Postponing backups, scrubs on other pools sharing the enclosure, and bulk jobs is a reasonable trade.
• Do not stack maintenance. This is not the time for firmware updates, controller work, or replacing a second disk in the same vdev. One change at a time, and the resilver finishes first.
• Expect elevated latency and plan around it. Elevated latency during resilver is expected behavior, not a new incident. Alert thresholds that fire on latency alone will false-fire for the entire window; correlate with queue depth and error counters instead.
• Throttle tradeoff. Default scan throttling prioritizes production I/O over resilver speed. Raising resilver priority shortens the risk window but taxes production latency. The tunables and their defaults vary by OpenZFS version, so check the module parameter documentation for the version you run before changing anything.
• If it stalls. A resilver that makes zero progress for more than 10 minutes is abnormal. Check dmesg for device resets, check per-vdev latency for a disk that stopped responding, and check that the replacement device itself is healthy.

Prevention

You cannot prevent resilvers, but you can control how long and how dangerous they are:

• Baseline your resilver time. Record how long the first resilver on a large pool takes at your current fill level so you can size the risk window before the next disk fails.
• Prefer topology with margin. RAIDZ1 on large HDDs produces multi-day windows with zero redundancy. RAIDZ2/Z3 or mirrors cost capacity but buy you tolerance during the rebuild. This decision is made at pool creation; there is no retrofit.
• Keep pools below the performance cliff. Healing resilvers on RAIDZ slow down as pools fill and fragment. Capacity discipline (see the capacity planning guide below) shortens future resilver windows.
• Scrub on schedule. Regular scrubs surface dying disks before they fail outright, which means replacements happen on your schedule instead of during an incident, and the surviving disks are in known-good shape when the rebuild starts.
• Use sequential resilver where supported. On mirrors (OpenZFS 2.0+) and dRAID pools (OpenZFS 2.1+), zpool replace -s meaningfully shortens the window. RAIDZ has no sequential path; plan accordingly.

How Netdata helps

• Pool and vdev state over time. Netdata tracks the transition into DEGRADED and back, so the ticket opens when the disk fails and closes only when redundancy is actually restored, not when someone racks the replacement.
• Scan progress as a trend. Resilver rate and completion percentage as time series make stalls obvious in a way that repeated zpool status runs do not.
• Per-vdev latency and throughput during the window. Correlating zpool iostat-style latency and IOPS per device surfaces the slow surviving disk that is both extending the rebuild and signaling the next failure.
• Error counter alerting. Incrementing READ/WRITE/CKSUM on surviving devices during an active resilver is exactly the escalation condition, and it is easy to miss in point-in-time zpool status output.
• Production impact correlation. Overlaying application-facing latency with resilver progress tells you whether the rebuild is hurting users or just running quietly in the background.

Related guides

• How ZFS actually works in production: a mental model for operators
• ZFS monitoring checklist: the signals every production pool needs
• ZFS device UNAVAIL or REMOVED: a disk that fell off the bus
• ZFS checksum errors (CKSUM): the definitive signal of silent corruption
• ZFS checksum errors on multiple devices: suspect RAM or the controller, not the disks
• ZFS capacity planning: runway estimation before the pool fills
• ZFS dirty data throttling: the write delay that masquerades as slow disks
• ZFS ARC hit ratio low: cache misses, cold caches, and working sets that outgrew RAM
• ZFS zfs_arc_max: capping the ARC without starving read performance
• ZFS ARC and the OOM killer: applications killed while the cache will not shrink fast enough
• ZFS ARC shrinking below c_max: reading memory pressure before latency hits
• ZFS ARC using all memory: the Linux default that eats your RAM