You ran zpool status and one of your devices is no longer ONLINE. It shows UNAVAIL or REMOVED, and the pool has flipped to DEGRADED or worse. This is not a ZFS software problem. Something between the kernel and the disk broke: the drive was pulled, the cable or backplane dropped it, the controller lost it, or a hot-plug event went badly.
The good news is that this failure mode is loud and binary. ZFS did not corrupt anything by itself. If the pool is DEGRADED, redundancy is still serving every read and write. Your job is to find out why the device disappeared, decide whether it is coming back, and either re-online it or replace it before a second device in the same vdev fails.
The bad news: a DEGRADED pool is one failure away from data loss, and the clock started when the disk dropped. Treat this as same-shift work, not next-week work.
What this means
ZFS tracks a state per vdev. The two states that matter here:
- UNAVAIL: the device could not be opened. ZFS tried the path it knows for the device and the kernel said no. Typical reasons: the disk is dead, the cable or backplane link is down, the controller or HBA dropped the target, or the device node never appeared at boot.
- REMOVED: the device was physically removed while the system was running and the platform detected the removal. Hot-removal detection is hardware dependent, so on some platforms a pulled disk shows UNAVAIL instead.
Contrast both with OFFLINE, an administrative state set by zpool offline. If a device shows OFFLINE, someone (or some script) took it offline on purpose. Do not confuse the two: the recovery path and the post-incident questions are completely different.
How the pool reacts depends on the topology of the affected vdev:
- If the failed device is part of a mirror, RAIDZ, or dRAID vdev with surviving redundancy, the pool goes DEGRADED. It keeps serving I/O through the remaining copies or parity.
- If the failure exhausts the redundancy of a top-level vdev (single-disk pool, stripe, or a mirror that lost its last member), the pool goes FAULTED and cannot serve I/O. See ZFS pool FAULTED: when the pool can no longer serve I/O.
- In some configurations ZFS will instead SUSPEND the pool and block I/O while it waits for device connectivity to return. See ZFS pool I/O is currently suspended: a hung pool and blocked I/O.
A pool can also look healthy at the top level while an individual vdev is in trouble. A spare that activated, for example, can leave the pool ONLINE with a failed device in the tree. Always read the full vdev tree, not just the pool state line.
flowchart TD
A[Device shows UNAVAIL or REMOVED] --> B{Redundancy left in this vdev?}
B -- yes --> C[Pool DEGRADED, I/O continues]
B -- no --> D[Pool FAULTED or SUSPENDED, I/O blocked]
C --> E{Device recoverable?}
E -- yes, reattached --> F[zpool online, verify state]
E -- no, dead or unreliable --> G[zpool replace with new disk, resilver]
D --> H[Restore connectivity, then zpool clear or import]Common causes
| Cause | What it looks like | First thing to check |
|---|---|---|
| Disk physically pulled or hot-unplugged | Device REMOVED; enclosure slot empty; coincides with maintenance or someone in the rack | Enclosure logs, change tickets, physical inspection |
| Cable, connector, or backplane failure | Device UNAVAIL; kernel link errors on one PHY; sometimes multiple disks on the same expander drop together | dmesg for SATA/SAS link reset and PHY errors |
| HBA or controller failure or reset | Several devices on the same controller go UNAVAIL at once | dmesg for controller errors; errors on multiple devices sharing a controller |
| Enclosure or power event | Whole enclosure shelf disappears; REMOVED on many devices | Enclosure power and slot status, PSU state |
| Dead disk | Device UNAVAIL and never responds again, even to the OS | Whether the kernel still sees the block device at all |
| Driver or module problem after kernel update | Devices missing after reboot; pool imports with devices UNAVAIL by GUID | Whether the HBA driver loaded and the devices exist in /dev |
Quick checks
These are all read-only and safe to run during an incident.
# 1. Full pool state with vdev tree and error columns
zpool status -v
# 2. Machine-parseable version for scripting
zpool status -p
# 3. Only pools with problems
zpool status -x
# 4. Kernel evidence of the drop: link resets, task aborts, device removal
dmesg | grep -i -E "ata|sas|scsi|reset|timeout|abort|remov|offline" | tail -50
# 5. What block devices does the kernel see right now?
lsblk -o NAME,SIZE,MODEL,SERIAL,STATE
# 6. ZFS event log: ereports for device failures, probes, deadman
zpool events -v | tail -40
# 7. Is the missing device still visible to the kernel at its expected path?
ls -l /dev/disk/by-id/ | grep <serial-or-wwn>
What to look for:
- In
zpool status, note the exact device state (UNAVAIL vs REMOVED vs OFFLINE), the READ/WRITE/CKSUM error columns, and whether a resilver or scrub is listed on thescan:line. - In
dmesg, find the timestamp of the drop. Link resets and task aborts before the disappearance point at cable, backplane, or controller. Clean “device removed” messages point at hot-unplug. - If the kernel no longer sees the block device at all, this is below ZFS. ZFS cannot online a device the OS cannot see.
- Check whether error counts are climbing on the surviving devices in the same vdev. Errors on the remaining mirror member or RAIDZ peers change this from urgent to critical.
How to diagnose it
Confirm the state and the blast radius. Run
zpool status -v. Identify which vdev the failed device belongs to, the pool state (DEGRADED vs FAULTED vs SUSPENDED), and whether other devices on the same controller or enclosure are also missing. One missing disk is a disk or cable problem. A shelf or controller worth of missing disks is an infrastructure problem.Rule out the administrative case. If the device shows OFFLINE, it was taken offline deliberately with
zpool offline. Checkzpool historyfor the offline command and who ran it. Do not “fix” an intentional offline until you know why it happened.Correlate with the kernel log. Match the ZFS state change time to
dmesg. SATA link resets (ataX: link is slow to respond,hard resetting link), SAS PHY errors, task aborts, or a clean device removal event each point at a different layer. Multiple devices dropping on one HBA means controller or driver, not individual disks.Check whether the device is still visible to the OS. If
lsblkno longer lists it, ZFS is out of the picture until the hardware path is restored. Reseat the drive or cable, check enclosure slot status, and confirm the HBA sees the target.Decide: recoverable or not. If the disk came back after a reseat or bus rescan and has no error history, it is a candidate for
zpool online. If it had been accumulating READ/WRITE/CKSUM errors before it dropped, or it is not reliably detected, treat it as dead and plan a replacement. A disk that fell off the bus once under suspicious circumstances is not a disk you want to trust with your last copy of parity.Check the error counters on survivors. Before and during recovery, watch the remaining devices in the affected vdev. Non-zero or growing READ/WRITE/CKSUM on the survivors means the redundancy you are relying on is itself suspect. Escalate accordingly.
Metrics and signals to monitor
| Signal | Why it matters | Warning sign |
|---|---|---|
Pool health state (zpool status -x, /proc/spl/kstat/zfs/<pool>/state) | Binary indicator that something is wrong; UNAVAIL/REMOVED devices flip the pool to DEGRADED, FAULTED, or SUSPENDED | Any state other than ONLINE |
| Per-vdev state and READ/WRITE/CKSUM counters | Pinpoints which device failed and whether survivors are degrading | Non-zero counters on any device; counters growing over time |
Resilver status (scan: line) | Tells you the vulnerability window after a replacement starts | DEGRADED with no resilver in progress, or stalled resilver |
ZFS events (zpool events) | ereports for I/O failure and probe failure give the timeline of the drop | Repeated probe failures or deadman events |
Kernel log (dmesg) | The hardware-level evidence: link resets, aborts, removal events | Reset storms or repeated link flapping on one PHY |
Per-vdev latency (zpool iostat -l -v) | A device about to drop often goes slow first | One vdev consistently slower than its peers before the drop |
Fixes
The device came back and is trustworthy
If the disk reappeared after a reseat, enclosure power cycle, or bus rescan, and it has no error history:
# Bring the device back into the pool
zpool online <pool> <device>
# Verify
zpool status -v
If the pool is SUSPENDED rather than DEGRADED, restore connectivity first, then run zpool clear <pool>, which clears error counters and resumes the pool. zpool online targets a specific device that ZFS could not open; zpool clear clears errors and can unsuspend the pool. Pick the one that matches the failure. If the device was reconnected but the pool still references a stale device node, zpool reopen <pool> reopens all vdevs against current device paths.
After re-onlining, watch the error counters. If READ/WRITE/CKSUM start climbing on the re-attached device, the underlying problem (cable, backplane, the disk itself) is still there. Pull it and replace.
The device is dead or untrustworthy
Replace it. If a new disk is in the same physical slot and shows up at the same device path, a single-argument replace works; otherwise give the new device explicitly:
# Same slot, same path
zpool replace <pool> <old-device>
# New device at a different path
zpool replace <pool> <old-device> <new-device>
# Watch the resilver
zpool status
During the resilver the pool runs at reduced redundancy and rebuild I/O competes with production traffic. Do not start other heavy maintenance (scrubs, large sends) until it completes. If the resilver stalls or runs far below expected device throughput, investigate the rebuild path, not just the new disk.
The device is administratively OFFLINE
Find out why first via zpool history, then zpool online <pool> <device> to bring it back. Onlining an intentionally offlined device without understanding the reason can mask the original problem.
Multiple devices or a whole enclosure dropped
Fix the shared layer first: controller, expander, cable, enclosure power. Do not start replacing individual disks until the common infrastructure is stable, or you will resilver onto a still-flapping path. Once connectivity is stable, re-online or clear as above.
Prevention
- Monitor per-vdev state continuously, not ad hoc.
zpool statusis a snapshot with clearable counters. Export vdev state and READ/WRITE/CKSUM counters to a time series so you can see the slow lead-up (a device going latent, error counts ticking up) before the drop. See ZFS monitoring checklist: the signals every production pool needs. - Run ZED. The ZFS Event Daemon reacts to ereports (device failures, probe failures, deadman events) as they happen instead of waiting for someone to run
zpool status.zpool eventsis in-memory only and is lost on reboot. - Use stable device paths. Build and import pools with
/dev/disk/by-id/paths so recovery commands and replacements reference stable identifiers, not/dev/sdXnames that shift across reboots and hot-plug events. - Baseline resilver time. Know how long a rebuild takes on your pool and topology. That number defines your real exposure window and drives spare strategy.
- Treat DEGRADED as same-day work. The pool will run DEGRADED indefinitely and quietly. The next failure in that vdev is data loss. This is the single most common operational mistake around device drops. See ZFS pool DEGRADED: redundancy lost and one failure from data loss.
- Track enclosure and cabling as failure domains. Correlate devices to controllers, expanders, and shelves in your asset data so a multi-disk drop is immediately recognizable as a shared-infrastructure event.
How Netdata helps
- Pool and per-vdev state over time: Netdata tracks pool health and per-vdev READ/WRITE/CKSUM counters continuously, so you can see the error accumulation that preceded the drop instead of discovering it after the fact.
- State transition alerting: the transition from ONLINE to DEGRADED, FAULTED, or SUSPENDED is a binary page-level signal that fires regardless of workload, idle state, or topology.
- Correlation with hardware events: pairing ZFS vdev errors with per-disk I/O latency lets you spot the classic “one vdev goes slow, then falls off the bus” sequence and act before the disconnect.
- Resilver visibility: rebuild progress, pool state, and surviving-device error counters on one dashboard answers the two questions that matter during recovery: is the rebuild moving, and is the remaining redundancy holding.
- Event timeline: correlating the ZFS state change with system-level signals around the same timestamp helps separate a cable/controller event from a genuinely dead disk.
Related guides
- How ZFS actually works in production: a mental model for operators
- ZFS monitoring checklist: the signals every production pool needs
- ZFS monitoring maturity model: from survival to expert
- ZFS pool DEGRADED: redundancy lost and one failure from data loss
- ZFS pool FAULTED: when the pool can no longer serve I/O
- ZFS pool I/O is currently suspended: a hung pool and blocked I/O