ZFS replacing a failed disk: zpool replace, autoreplace, and hot spares

A disk in your pool has failed or is failing. The pool shows DEGRADED in zpool status, which means redundancy is gone in that vdev group and the next failure in the same group is data loss. The pool keeps serving I/O, but you are on the clock.

This runbook covers getting a replacement disk in and resilvered: identifying the physical device correctly, choosing between manual zpool replace, autoreplace, and hot spares, what sequential resilver changes, and the extra steps a root pool needs before the new disk can boot the machine.

It assumes you understand vdev topology and pool health states. If not, see the ZFS mental model first.

Before you touch anything

Three checks determine which replacement path you take and whether it is safe to proceed.

# Full pool state, vdev tree, error counters, and scan status
zpool status -v <pool>

# Is a resilver or scrub already running?
zpool status <pool> | grep -A5 "scan:"

# Kernel view of what happened to the device
dmesg | grep -i -E "ata|sas|nvme|reset|timeout" | tail -30

Key facts to extract from zpool status -v:

• Which device is degraded or faulted, and whether the pool still has redundancy for that vdev group. On RAIDZ1 or a two-way mirror, a second failure before the resilver completes is data loss. Treat this as same-day work.
• The scan line. A resilver preempts a running scrub, and the two are mutually exclusive per pool. If a resilver is already running (for example, a hot spare activated), do not start a second replace against the same vdev.
• Error counters on the remaining devices. If READ/WRITE/CKSUM counters are climbing on the surviving mirror side or other RAIDZ members, your risk during resilver is much higher than normal. Consider quiescing the workload.

Identify the physical disk by serial or GUID, not /dev name

Do not trust the /dev/sdX name in zpool status output to tell you which physical disk to pull. Device letters shift across reboots, controller rescans, and enclosure events. The letter you saw yesterday may belong to a different disk today.

Identify the device by something burned into the hardware:

# Machine-parseable status with full device paths
zpool status -p <pool>

# Map the pool device to a stable identity
ls -l /dev/disk/by-id/ | grep <device>

If the pool was created with /dev/disk/by-id/ paths (WWN or ATA serial), the path itself is the identity: match it to the serial number on the disk label or reported by smartctl -i. If the pool was created with raw /dev/sdX names, resolve the current letter to a by-id path before pulling anything, and consider converting the pool: export and reimport with zpool import -d /dev/disk/by-id so future replacements are unambiguous. See ZFS device UNAVAIL or REMOVED for the case where the disk has already fallen off the bus entirely.

For locating the disk in the chassis, use your platform’s enclosure or HBA tooling (SES slot locate LEDs, controller CLI). ZFS itself does not blink drive LEDs portably.

Decision: manual replace, autoreplace, or hot spare

flowchart TD
  A[Disk DEGRADED or FAULTED] --> B{Hot spare configured?}
  B -- yes --> C[ZED attaches spare automatically]
  C --> D[Resilver to spare runs]
  D --> E[Later: install permanent disk, zpool replace, detach spare]
  B -- no --> F{autoreplace=on and same-slot swap?}
  F -- yes --> G[ZED formats and replaces on insertion]
  F -- no --> H[Manual: zpool offline, swap disk, zpool replace]
  E --> I[zpool clear old counters]
  G --> I
  H --> I
  I --> J[Verify ONLINE, scrub]

All three paths converge on the same end state: a resilver onto the new device, then cleanup. They differ in who issues the commands and when.

Path 1: manual zpool replace

The default and the most controllable. The safe sequence for a disk that is still visible but failing:

# 1. Take the failing device offline first (stops ZFS writing to it)
zpool offline <pool> <old-device>

# 2. Physically swap the disk, then confirm the OS sees the new one
ls -l /dev/disk/by-id/

# 3. Replace, addressing the new disk by its stable by-id path
zpool replace <pool> <old-device> /dev/disk/by-id/<new-disk-id>

# 4. Watch the resilver
zpool status <pool> | grep -A5 "scan:"

Notes on the mechanics:

• If the disk is already FAULTED or REMOVED, skip the zpool offline step. Offlining a healthy-but-suspect disk first is good practice: it stops new writes to media you no longer trust while you arrange the swap.
• If new-device is omitted, it defaults to device, which is useful when the disk was swapped in the same slot and the device path is identical.
• zpool replace -w waits for the resilver to complete before returning, which is handy in scripts.
• The resilver starts immediately. During it, the pool runs at reduced redundancy and resilver I/O competes with production I/O. On large HDD pools this runs for hours to days; that window is the real risk, not the command.

Path 2: autoreplace

The autoreplace pool property (default off) tells the ZFS Event Daemon (ZED) to automatically format and replace any new device that appears in the same physical location as a device that previously belonged to the pool. Enable it deliberately:

zpool set autoreplace=on <pool>

This only works if ZED is running and healthy, and it only triggers for a same-slot swap. It does not help you replace a disk into a different slot or enclosure position.

Two caveats from the field:

• autoreplace has had version-specific breakage. OpenZFS 0.8.x had a race where ZED brought the vdev online before the partition device existed, leaving the new disk UNAVAIL. OpenZFS 2.1.7 exposed the separate /dev/disk/by-id/ failure because ZED looked for the old device’s WWN path after the disk was swapped. That by-id bug was fixed in OpenZFS 2.2.1. For stable behavior across swaps and hosts, /dev/disk/by-vdev/ paths configured via /etc/zfs/vdev_id.conf remain the recommended layout; /dev/disk/by-path/ can also encode physical location.
• autoreplace and hot spares are independent features. You do not need autoreplace=on for hot spares to activate.

Treat autoreplace as a convenience you verify after every drive swap, not a guarantee. After any automatic replacement, run zpool status and confirm a resilver actually started.

Path 3: hot spares

A hot spare is a disk already installed, added to the pool as a spare:

# Add a dedicated spare
zpool add <pool> spare /dev/disk/by-id/<spare-id>

When a data disk faults, ZED automatically attaches the spare to the degraded vdev (effectively a zpool attach), and a resilver onto the spare begins. No operator action is required for the pool to regain redundancy, which is the entire point: the dangerous window between failure and resilver shrinks from “whenever someone gets to the datacenter” to minutes.

Two operational facts about spares:

• The spare is a temporary replacement. The vdev now has an extra member. To make the layout permanent, install a new disk, run zpool replace <pool> <failed-disk> <new-disk>, let that resilver finish, then zpool detach <pool> <spare-device>. Order matters: detaching the spare first fails with “resource busy” while it is an active member.
• autoreplace is not involved in spare activation. ZED handles spares regardless of the autoreplace property.

On dRAID pools (OpenZFS 2.1+), distributed spares are built into the vdev rather than added as separate devices, and rebuilds to distributed spares use sequential resilver by default. After the permanent disk goes in, run zpool replace without -s (a healing resilver) to rebalance the distributed spare space back to available.

Sequential resilver: the -s flag

zpool replace -s requests a sequential resilver, supported for mirrors and dRAID (not raidz). Instead of walking the block pointer tree, it copies in LBA order without checksum verification, which is substantially faster on spinning disks. A scrub starts automatically after the sequential resilver completes to verify the copied data.

Use it when resilver time is your primary risk (large HDD mirrors) and you accept that verification is deferred to the post-resilver scrub. For raidz, resilver is metadata-driven and reconstructs only allocated blocks; the -s flag does not apply.

After the resilver: cleanup and verification

The resilver completing is not the end of the procedure.

• Clear old counters. Error counters are cumulative since the last zpool clear. The old device’s READ/WRITE/CKSUM counts persist in history and will confuse the next person reading zpool status. After confirming the resilver finished clean and the root cause (failed disk, cable, controller) is addressed:

# Reset error counters after a confirmed-good replacement
zpool clear <pool>

Only clear after you have confirmed the replacement device is healthy. Clearing counters on a pool that is still accumulating errors hides an active problem.

• Verify final state. Every vdev ONLINE, the scan line shows the resilver completed, zero new errors:

zpool status -v <pool>

• Let the post-resilver scrub finish (automatic after sequential resilver; consider starting one manually after a healing resilver if the pool was degraded a long time). The scrub is what proves the reconstructed data is checksummed-clean.

Root pools: the disk is not just ZFS data

If the failed disk belongs to the pool the system boots from, zpool replace only handles the ZFS data partition. The new disk also needs whatever the bootloader requires:

• The EFI System Partition (or BIOS boot partition on legacy systems) must be created on the new disk and populated. On mirrored root pools, the ESP should be mirrored or at least copied to every boot-capable disk.
• The bootloader must be reinstalled or refreshed against the new disk. There is no universal refresh command: use the boot loader/installer runbook for the installed distribution (for example, Proxmox documents proxmox-boot-tool refresh).
• Test bootability before you call the incident closed. A root pool that is ONLINE but has no valid bootloader on the surviving disk is a delayed outage waiting for the next reboot.

Signals to monitor during and after replacement

How Netdata helps

Disk replacement is where point-in-time zpool status checks fail operators: the dangerous part is the multi-hour resilver window, not the command itself.

• Netdata tracks pool and per-vdev state continuously, so you see the DEGRADED transition the moment it happens rather than at the next manual check.
• Per-device READ/WRITE/CKSUM counters are graphed over time, which distinguishes a one-time SATA reset from a counter that climbs steadily (the dying-disk pattern that should trigger proactive replacement before total failure).
• During the resilver, correlating pool latency and per-vdev I/O against resilver progress tells you whether production impact is within expectation or the resilver is starving foreground I/O.
• Alerting on DEGRADED state with no active resilver catches the failure mode where autoreplace or spare activation silently did not happen.
• Scrub completion and repair counts after the replacement close the loop on data verification.

Related guides

• ZFS device UNAVAIL or REMOVED: a disk that fell off the bus
• ZFS checksum errors (CKSUM): the definitive signal of silent corruption
• ZFS checksum errors on multiple devices: suspect RAM or the controller, not the disks
• ZFS deadman events: hung I/O and a stalled pool sync
• ZFS capacity planning: runway estimation before the pool fills
• How ZFS actually works in production: a mental model for operators