zpool status shows the pool state as FAULTED, the vdev tree shows devices FAULTED or UNAVAIL, and the status message says “insufficient replicas for the pool to continue functioning.” Applications cannot read or write anything on the pool. Unlike DEGRADED, where redundancy is still covering for a failed device, FAULTED means ZFS has lost more devices than the vdev topology can tolerate, or cannot open enough devices to guarantee data integrity.
Before touching anything: FAULTED is a statement about ZFS’s ability to serve I/O safely, not a statement that the data is gone. The blocks are usually still on the drives. Whether you get them back depends on what failed, how it failed, and what you do next. Some obvious-looking recovery actions (notably zpool clear on physically failing hardware) make things worse.
This article covers what FAULTED means, how to tell it apart from the neighboring states, how to triage the failure, and the recovery paths in order of safety.
What this means
ZFS tracks health at two levels: each device and vdev has its own state, and the pool aggregates those into a single pool health state visible via zpool list -H -o name,health or /proc/spl/kstat/zfs/<pool>/state. A pool goes FAULTED when one or more top-level vdevs are faulted and insufficient replicas exist to continue functioning. In a mirror, that means both sides are gone. In RAIDZ1, two devices. In a stripe, any single device.
Three distinctions matter before you pick a recovery path:
- FAULTED vs DEGRADED. DEGRADED means a device failed but redundancy is still serving every block. The pool is fully accessible and one failure away from FAULTED. See ZFS pool DEGRADED: redundancy lost and one failure from data loss.
- FAULTED vs SUSPENDED. SUSPENDED means the pool is waiting for device connectivity to be restored; I/O is blocked but the pool has not given up on the devices. A SUSPENDED pool can wait indefinitely and requires manual intervention. FAULTED means ZFS has concluded it cannot continue.
- FAULTED vs UNAVAIL at the device level. UNAVAIL means ZFS cannot open the device at all: disconnected, powered off, path changed. FAULTED at the device level means the device was opened but returned too many I/O errors or corrupt data. This drives triage: UNAVAIL devices point at cabling, controllers, enclosures, or device naming; FAULTED devices point at media or transport errors.
One special case: if the FAULTED pool is the root pool, the system will not boot. Recovery happens from rescue media or an installer environment, and pool import time (minutes on large or dirty pools) can look like a boot hang before it looks like a failure.
stateDiagram-v2 ONLINE --> DEGRADED: device faulted, redundancy intact DEGRADED --> ONLINE: replace + resilver DEGRADED --> FAULTED: another failure in same vdev ONLINE --> SUSPENDED: device connectivity lost SUSPENDED --> ONLINE: device returns SUSPENDED --> FAULTED: insufficient replicas FAULTED --> ONLINE: devices fixed + zpool clear FAULTED --> Imported: import -F / -T / readonly
Common causes
| Cause | What it looks like | First thing to check |
|---|---|---|
| Multiple device failures exceeding redundancy | Two+ devices FAULTED in one RAIDZ1 vdev, or both mirror legs gone | zpool status -v vdev tree |
| Controller or HBA failure | Many devices on the same controller error simultaneously | dmesg for controller resets |
| Cable, backplane, or enclosure fault | Devices UNAVAIL rather than FAULTED, often several at once | Physical connections, enclosure status |
| Device path or naming change | Devices UNAVAIL after reboot or enclosure swap | Whether the disks are visible to the OS at all |
| Corrupted pool metadata | Pool refuses import; errors reference metadata, not a specific device | zpool import output |
| Kernel/ZFS module mismatch after upgrade | Pool cannot be imported at all on boot | Module load errors in dmesg |
Quick checks
All read-only and safe.
# Pool health state and problem summary
zpool status -x
zpool list -H -o name,health
# Full vdev tree: which devices are FAULTED vs UNAVAIL, error counts
zpool status -v
# Machine-parseable vdev states and error columns
zpool status -p
# Kernel view of pool state
cat /proc/spl/kstat/zfs/<pool>/state
# Hardware-level evidence: SATA/SAS resets, timeouts, controller errors
dmesg | grep -i -E "ata|sas|reset|timeout|error"
# Recent ZFS events (in-memory only, lost on reboot)
zpool events -v | tail -40
# What the system can see for import
zpool import
Reading notes. Error counters in zpool status are cumulative since the last zpool clear, and may reset on pool export/import, so a zero counter does not prove a clean history. If CKSUM errors appear on many unrelated devices at once, suspect RAM or the controller rather than individual disks. zpool events is in-memory only; if the system already rebooted, that evidence is gone.
Also note a known OpenZFS issue: zpool events does not always record a statechange event when a device faults, and on OpenZFS 2.2.x and later there is an open defect where a device can be marked FAULTED while the pool state does not immediately transition. If your monitoring keys only on pool state, per-vdev states can be ahead of it.
How to diagnose it
Map the failure scope with
zpool status -v. One device points at the disk. Several devices on one controller point at the controller, cable, or backplane. Devices UNAVAIL point at connectivity, not media.Classify UNAVAIL vs FAULTED per device. For UNAVAIL devices, check physical connections, enclosure power, and whether the OS sees the disks at all before assuming the drives are dead.
Check
dmesgfor transport errors. SATA link resets, SAS aborts, and controller timeouts correlate directly with the READ/WRITE error columns inzpool status. If errors cluster on one controller, that is your suspect.Decide whether the underlying cause is fixed. This is the gate for everything below. If a cable was loose and is reseated, or a controller was replaced, recovery is straightforward. If drives are physically dying, do not let ZFS start hammering them; image first (step 6).
If the pool is not imported at all, try a plain
zpool import <pool>and read the error. The import path has its own recovery options covered below.If any surviving drive looks physically marginal (SMART errors, growing latency, unusual noise), image it with a tool like ddrescue before initiating any resilver or heavy recovery I/O. ZFS resilvers follow the block pointer tree, generating random seeks that accelerate head failure on unstable drives. Image all marginal members first, then recover from the images.
Metrics and signals to monitor
| Signal | Why it matters | Warning sign |
|---|---|---|
Pool health state (zpool status -x, kstat state) | Binary availability signal; FAULTED/SUSPENDED/UNAVAIL is page-worthy and cannot false-fire | Any state other than ONLINE |
| Per-vdev state and READ/WRITE/CKSUM counters | Tells you which device is failing before the pool tips from DEGRADED to FAULTED | Any non-zero counter, or counts growing over time |
Scrub results and permanent error list (zpool status -v) | Uncorrectable errors mean data loss has already occurred; the permanent error list is your recovery checklist | Non-empty permanent error list |
ZFS deadman events (zpool events) | Sustained hung I/O, often the precursor to a faulted device | Any deadman event |
| SMART data on member devices | Reallocated sectors rising plus ZFS CKSUM errors means imminent disk failure | Rising reallocated/pending sector counts |
| Resilver status | During resilver the pool is at reduced redundancy; a second fault here is how FAULTED happens | Resilver stalled or drastically below device throughput |
FAULTED is almost never the first signal. It is the endpoint of a chain of per-vdev errors, DEGRADED state, and often a resilver that did not finish. If you see FAULTED with no prior warning, the gap is in per-vdev monitoring, not in the pool state check. See the ZFS monitoring checklist for the full signal set.
Fixes
Work down this list in order. Stop at the first rung that restores access.
Device issue fixed: zpool clear
If the root cause is genuinely resolved (cable reseated, controller replaced, enclosure power restored) and the pool is imported but FAULTED, zpool clear <pool> resets the error counters and retries I/O on the faulted vdevs. If the devices respond, the pool returns to ONLINE or DEGRADED and any needed resilver begins.
Warning: do not run zpool clear on hardware that is still failing. Clearing errors masks the problem and lets ZFS issue new writes to a faulty drive; those writes may be lost, and the retry I/O itself can finish off a marginal disk. Only clear after the root cause is fixed.
Pool not imported: rewind the last transactions
If the pool fails to import, or imports and immediately faults on metadata errors, zpool import -F <pool> attempts recovery by discarding the last few transaction groups. You lose the most recent seconds of writes but keep the pool. zpool import -T <txg> <pool> rolls back to a specific transaction group if you know a known-good one. The heavier import -FX variant exists for cases where -F is insufficient; treat it as a later resort because it discards more.
Data rescue: read-only import
zpool import -o readonly=on <pool> imports without writing any metadata to the pool. This is the safest way to copy data off a damaged pool: ZFS does not attempt repairs, does not write, and cannot make the damage worse. Mount datasets and copy what matters to other storage.
OpenZFS also provides the zfs_max_missing_tvds tunable, which allows importing a pool read-only even when top-level vdevs are missing or faulted. Set it via /sys/module/zfs/parameters/zfs_max_missing_tvds before import.
Checkpoint rewind
If the pool was checkpointed before the failure, zpool import --rewind-to-checkpoint <pool> rewinds the entire pool to the checkpointed state. This permanently discards the checkpoint and everything written after it.
Failing hardware: image, then recover offline
If the drives themselves are dying, the recovery model changes. Image every surviving member with ddrescue behind a write-blocker, then work against the images. The data is on the drives; the job is to extract it before the drives finish dying. Do not let ZFS resilver or scrub against physically unstable media.
Root pool FAULTED
Boot from rescue media, import the root pool there (read-only if you only need data off it), repair or replace devices, then zpool clear and reboot. Pool import on large or dirty pools can take minutes; what looks like a boot hang may be a slow import. If the import genuinely fails, use the rewind and read-only options above from the rescue environment.
Prevention
- Alert on per-vdev state and error counters, not just pool state. FAULTED is the end of a chain; the chain is visible days or weeks earlier in READ/WRITE/CKSUM counters and DEGRADED transitions.
- Treat DEGRADED as a same-day ticket. The window between DEGRADED and FAULTED is exactly one more device failure, and it is shortest during resilver.
- Baseline resilver duration for your pool size so you know your exposure window when a replacement happens.
- Scrub on a regular schedule and alert on results, not just execution. Scrubs surface weak devices while redundancy can still repair them.
- For root-on-ZFS systems, test boot-from-rescue-media and pool import before you need it.
- Verify ZED is handling device-fault events so spares engage and notifications fire; do not assume automation covered the transition.
How Netdata helps
- Netdata collects ZFS pool health, per-vdev error counters, and capacity, so the DEGRADED-to-FAULTED progression is a visible trend rather than a surprise state change.
- Per-device READ/WRITE/CKSUM charts let you correlate a specific failing disk against SMART data and pool state on one dashboard.
- Scrub and resilver status are tracked alongside pool I/O, so you can see when a resilver stalled and left the pool exposed.
- Deadman and other ZFS events can be alerted on as they happen, catching hung-I/O precursors before a device faults out completely.
- Because pool state transitions are binary and cannot false-fire, a FAULTED/SUSPENDED/UNAVAIL alert is safe to page on directly.