$ guides / zfs / zfs-device-unavail-removed ▌

Operations Guides

ZFS device UNAVAIL or REMOVED: a disk that fell off the bus

You ran zpool status and one of your devices is no longer ONLINE. It shows UNAVAIL or REMOVED, and the pool has flipped to DEGRADED or worse. This is not a ZFS software problem. Something between the kernel and the disk broke: the drive was pulled, the cable or backplane dropped it, the controller lost it, or a hot-plug event went badly.

The good news is that this failure mode is loud and binary. ZFS did not corrupt anything by itself. If the pool is DEGRADED, redundancy is still serving every read and write. Your job is to find out why the device disappeared, decide whether it is coming back, and either re-online it or replace it before a second device in the same vdev fails.

The bad news: a DEGRADED pool is one failure away from data loss, and the clock started when the disk dropped. Treat this as same-shift work, not next-week work.

What this means

ZFS tracks a state per vdev. The two states that matter here:

UNAVAIL: the device could not be opened. ZFS tried the path it knows for the device and the kernel said no. Typical reasons: the disk is dead, the cable or backplane link is down, the controller or HBA dropped the target, or the device node never appeared at boot.
REMOVED: the device was physically removed while the system was running and the platform detected the removal. Hot-removal detection is hardware dependent, so on some platforms a pulled disk shows UNAVAIL instead.

Contrast both with OFFLINE, an administrative state set by zpool offline. If a device shows OFFLINE, someone (or some script) took it offline on purpose. Do not confuse the two: the recovery path and the post-incident questions are completely different.

How the pool reacts depends on the topology of the affected vdev:

If the failed device is part of a mirror, RAIDZ, or dRAID vdev with surviving redundancy, the pool goes DEGRADED. It keeps serving I/O through the remaining copies or parity.
If the failure exhausts the redundancy of a top-level vdev (single-disk pool, stripe, or a mirror that lost its last member), the pool goes FAULTED and cannot serve I/O. See ZFS pool FAULTED: when the pool can no longer serve I/O.
In some configurations ZFS will instead SUSPEND the pool and block I/O while it waits for device connectivity to return. See ZFS pool I/O is currently suspended: a hung pool and blocked I/O.

A pool can also look healthy at the top level while an individual vdev is in trouble. A spare that activated, for example, can leave the pool ONLINE with a failed device in the tree. Always read the full vdev tree, not just the pool state line.

flowchart TD
  A[Device shows UNAVAIL or REMOVED] --> B{Redundancy left in this vdev?}
  B -- yes --> C[Pool DEGRADED, I/O continues]
  B -- no --> D[Pool FAULTED or SUSPENDED, I/O blocked]
  C --> E{Device recoverable?}
  E -- yes, reattached --> F[zpool online, verify state]
  E -- no, dead or unreliable --> G[zpool replace with new disk, resilver]
  D --> H[Restore connectivity, then zpool clear or import]

Common causes

Cause	What it looks like	First thing to check
Disk physically pulled or hot-unplugged	Device REMOVED; enclosure slot empty; coincides with maintenance or someone in the rack	Enclosure logs, change tickets, physical inspection
Cable, connector, or backplane failure	Device UNAVAIL; kernel link errors on one PHY; sometimes multiple disks on the same expander drop together	`dmesg` for SATA/SAS link reset and PHY errors
HBA or controller failure or reset	Several devices on the same controller go UNAVAIL at once	`dmesg` for controller errors; errors on multiple devices sharing a controller
Enclosure or power event	Whole enclosure shelf disappears; REMOVED on many devices	Enclosure power and slot status, PSU state
Dead disk	Device UNAVAIL and never responds again, even to the OS	Whether the kernel still sees the block device at all
Driver or module problem after kernel update	Devices missing after reboot; pool imports with devices UNAVAIL by GUID	Whether the HBA driver loaded and the devices exist in `/dev`

Quick checks

These are all read-only and safe to run during an incident.

# 1. Full pool state with vdev tree and error columns
zpool status -v

# 2. Machine-parseable version for scripting
zpool status -p

# 3. Only pools with problems
zpool status -x

# 4. Kernel evidence of the drop: link resets, task aborts, device removal
dmesg | grep -i -E "ata|sas|scsi|reset|timeout|abort|remov|offline" | tail -50

# 5. What block devices does the kernel see right now?
lsblk -o NAME,SIZE,MODEL,SERIAL,STATE

# 6. ZFS event log: ereports for device failures, probes, deadman
zpool events -v | tail -40

# 7. Is the missing device still visible to the kernel at its expected path?
ls -l /dev/disk/by-id/ | grep <serial-or-wwn>

What to look for:

In zpool status, note the exact device state (UNAVAIL vs REMOVED vs OFFLINE), the READ/WRITE/CKSUM error columns, and whether a resilver or scrub is listed on the scan: line.
In dmesg, find the timestamp of the drop. Link resets and task aborts before the disappearance point at cable, backplane, or controller. Clean “device removed” messages point at hot-unplug.
If the kernel no longer sees the block device at all, this is below ZFS. ZFS cannot online a device the OS cannot see.
Check whether error counts are climbing on the surviving devices in the same vdev. Errors on the remaining mirror member or RAIDZ peers change this from urgent to critical.

How to diagnose it

Confirm the state and the blast radius. Run zpool status -v. Identify which vdev the failed device belongs to, the pool state (DEGRADED vs FAULTED vs SUSPENDED), and whether other devices on the same controller or enclosure are also missing. One missing disk is a disk or cable problem. A shelf or controller worth of missing disks is an infrastructure problem.
Rule out the administrative case. If the device shows OFFLINE, it was taken offline deliberately with zpool offline. Check zpool history for the offline command and who ran it. Do not “fix” an intentional offline until you know why it happened.
Correlate with the kernel log. Match the ZFS state change time to dmesg. SATA link resets (ataX: link is slow to respond, hard resetting link), SAS PHY errors, task aborts, or a clean device removal event each point at a different layer. Multiple devices dropping on one HBA means controller or driver, not individual disks.
Check whether the device is still visible to the OS. If lsblk no longer lists it, ZFS is out of the picture until the hardware path is restored. Reseat the drive or cable, check enclosure slot status, and confirm the HBA sees the target.
Decide: recoverable or not. If the disk came back after a reseat or bus rescan and has no error history, it is a candidate for zpool online. If it had been accumulating READ/WRITE/CKSUM errors before it dropped, or it is not reliably detected, treat it as dead and plan a replacement. A disk that fell off the bus once under suspicious circumstances is not a disk you want to trust with your last copy of parity.
Check the error counters on survivors. Before and during recovery, watch the remaining devices in the affected vdev. Non-zero or growing READ/WRITE/CKSUM on the survivors means the redundancy you are relying on is itself suspect. Escalate accordingly.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Pool health state (`zpool status -x`, `/proc/spl/kstat/zfs/<pool>/state`)	Binary indicator that something is wrong; UNAVAIL/REMOVED devices flip the pool to DEGRADED, FAULTED, or SUSPENDED	Any state other than ONLINE
Per-vdev state and READ/WRITE/CKSUM counters	Pinpoints which device failed and whether survivors are degrading	Non-zero counters on any device; counters growing over time
Resilver status (`scan:` line)	Tells you the vulnerability window after a replacement starts	DEGRADED with no resilver in progress, or stalled resilver
ZFS events (`zpool events`)	ereports for I/O failure and probe failure give the timeline of the drop	Repeated probe failures or deadman events
Kernel log (`dmesg`)	The hardware-level evidence: link resets, aborts, removal events	Reset storms or repeated link flapping on one PHY
Per-vdev latency (`zpool iostat -l -v`)	A device about to drop often goes slow first	One vdev consistently slower than its peers before the drop

Fixes

The device came back and is trustworthy

If the disk reappeared after a reseat, enclosure power cycle, or bus rescan, and it has no error history:

# Bring the device back into the pool
zpool online <pool> <device>

# Verify
zpool status -v

If the pool is SUSPENDED rather than DEGRADED, restore connectivity first, then run zpool clear <pool>, which clears error counters and resumes the pool. zpool online targets a specific device that ZFS could not open; zpool clear clears errors and can unsuspend the pool. Pick the one that matches the failure. If the device was reconnected but the pool still references a stale device node, zpool reopen <pool> reopens all vdevs against current device paths.

After re-onlining, watch the error counters. If READ/WRITE/CKSUM start climbing on the re-attached device, the underlying problem (cable, backplane, the disk itself) is still there. Pull it and replace.

The device is dead or untrustworthy

Replace it. If a new disk is in the same physical slot and shows up at the same device path, a single-argument replace works; otherwise give the new device explicitly:

# Same slot, same path
zpool replace <pool> <old-device>

# New device at a different path
zpool replace <pool> <old-device> <new-device>

# Watch the resilver
zpool status

During the resilver the pool runs at reduced redundancy and rebuild I/O competes with production traffic. Do not start other heavy maintenance (scrubs, large sends) until it completes. If the resilver stalls or runs far below expected device throughput, investigate the rebuild path, not just the new disk.

The device is administratively OFFLINE

Find out why first via zpool history, then zpool online <pool> <device> to bring it back. Onlining an intentionally offlined device without understanding the reason can mask the original problem.

Multiple devices or a whole enclosure dropped

Fix the shared layer first: controller, expander, cable, enclosure power. Do not start replacing individual disks until the common infrastructure is stable, or you will resilver onto a still-flapping path. Once connectivity is stable, re-online or clear as above.

Prevention

Monitor per-vdev state continuously, not ad hoc. zpool status is a snapshot with clearable counters. Export vdev state and READ/WRITE/CKSUM counters to a time series so you can see the slow lead-up (a device going latent, error counts ticking up) before the drop. See ZFS monitoring checklist: the signals every production pool needs.
Run ZED. The ZFS Event Daemon reacts to ereports (device failures, probe failures, deadman events) as they happen instead of waiting for someone to run zpool status. zpool events is in-memory only and is lost on reboot.
Use stable device paths. Build and import pools with /dev/disk/by-id/ paths so recovery commands and replacements reference stable identifiers, not /dev/sdX names that shift across reboots and hot-plug events.
Baseline resilver time. Know how long a rebuild takes on your pool and topology. That number defines your real exposure window and drives spare strategy.
Treat DEGRADED as same-day work. The pool will run DEGRADED indefinitely and quietly. The next failure in that vdev is data loss. This is the single most common operational mistake around device drops. See ZFS pool DEGRADED: redundancy lost and one failure from data loss.
Track enclosure and cabling as failure domains. Correlate devices to controllers, expanders, and shelves in your asset data so a multi-disk drop is immediately recognizable as a shared-infrastructure event.

How Netdata helps

Pool and per-vdev state over time: Netdata tracks pool health and per-vdev READ/WRITE/CKSUM counters continuously, so you can see the error accumulation that preceded the drop instead of discovering it after the fact.
State transition alerting: the transition from ONLINE to DEGRADED, FAULTED, or SUSPENDED is a binary page-level signal that fires regardless of workload, idle state, or topology.
Correlation with hardware events: pairing ZFS vdev errors with per-disk I/O latency lets you spot the classic “one vdev goes slow, then falls off the bus” sequence and act before the disconnect.
Resilver visibility: rebuild progress, pool state, and surviving-device error counters on one dashboard answers the two questions that matter during recovery: is the rebuild moving, and is the remaining redundancy holding.
Event timeline: correlating the ZFS state change with system-level signals around the same timestamp helps separate a cable/controller event from a genuinely dead disk.