$ guides / zfs / zfs-pool-degraded-state ▌

Operations Guides

ZFS pool DEGRADED: redundancy lost and one failure from data loss

Your monitoring fired, or a routine zpool status run shows the pool in DEGRADED state. One or more devices have failed or gone unavailable, but the pool is still serving I/O because a mirror partner or RAIDZ parity is covering the gap. Applications see no errors.

That is what makes DEGRADED dangerous. The pool is stable in this state and will run there indefinitely, so teams sit on it. But redundancy in the affected vdev group is gone or reduced: on a two-way mirror or RAIDZ1, the next failure in that group is unrecoverable data loss. On RAIDZ2 you have one fault of margin left, not two.

Severity for this state: TICKET with same-day response. Not a page, because data is still intact and a resilver or hot spare may already be handling the fault. Not something to defer, because you are racing the remaining devices in the group.

This guide covers confirming what actually failed, checking whether recovery is already in progress, and deciding between clearing, replacing, and escalating.

What this means

ZFS reports health at two levels: the pool and each individual vdev/device. The pool state is the aggregate of its vdev states.

ONLINE: all vdevs functioning normally.
DEGRADED: one or more vdevs failed, removed, or unavailable, but enough replicas or parity remain to serve every block. The pool is accessible. Redundancy is reduced or exhausted in the affected group.
FAULTED: insufficient replicas remain to continue. The pool cannot serve I/O.
SUSPENDED: waiting for device connectivity to be restored; I/O is blocked.
UNAVAIL: the pool cannot be opened, for example too many devices missing at import.

stateDiagram-v2
    ONLINE --> DEGRADED: device faults, redundancy covers it
    DEGRADED --> ONLINE: device replaced, resilver completes
    DEGRADED --> DEGRADED: resilver running (redundancy not yet restored)
    DEGRADED --> FAULTED: next failure in same vdev group
    FAULTED --> DEGRADED: missing device returns / pool reopened

Two details in that diagram matter operationally. First, a pool actively resilvering is still DEGRADED, because redundancy is not restored until the resilver finishes. DEGRADED plus an in-progress resilver is expected recovery, not a new problem. Second, DEGRADED with no resilver running and no spare engaged means nothing is fixing the fault. That is the configuration that quietly turns into a FAULTED pool six weeks later.

One caveat on detection: in OpenZFS 2.2.x there is a reported regression (issue #16245) where a device can be marked FAULTED while the pool state stays ONLINE for minutes. If you monitor only the pool-level state, you can miss the degraded condition entirely. Check per-device states, not just the summary line.

Common causes

Cause	What it looks like	First thing to check
Physical disk failure	Device FAULTED or UNAVAIL, READ/WRITE/CKSUM counters non-zero and climbing before the fault	`zpool status -v` per-device columns, SMART data on the device
Cable, connector, or backplane issue	Device UNAVAIL or intermittent errors on one device; possibly recovers after reseat	`dmesg` for SATA/SAS resets and timeouts on that device
Controller or HBA failure	Multiple devices on the same controller erroring or dropping simultaneously	`dmesg`, check whether the faulted devices share a controller
Administrator action	Device shows OFFLINE or REMOVED	`zpool history` for recent `offline`/`remove`/`detach` commands
Resilver in progress after replacement	New device present, `scan:` line shows resilver with progress and ETA	`zpool status` scan line
Transient transport glitch	Small one-time error increment, device back ONLINE after clear	Confirm with SMART before clearing; see the fixes section

Quick checks

All read-only. Run these before touching anything.

# Pool-level state and the problem summary only
zpool status -x

# Machine-parseable pool health
zpool list -H -o name,health

# Pool state straight from the kstat (single word: ONLINE, DEGRADED, ...)
cat /proc/spl/kstat/zfs/<pool>/state

# Full vdev tree: per-device state plus READ/WRITE/CKSUM columns,
# and the permanent error list at the bottom if one exists
zpool status -v <pool>

# Is a resilver or scrub already handling this?
zpool status <pool> | grep -A5 -E "scan:|resilver"

# Which vdev is slow or erroring relative to its peers
zpool iostat -v <pool> 1

# Kernel view of the underlying transport
dmesg | grep -i -E "ata|sas|reset|timeout" | tail -30

# Was this an operator action?
zpool history <pool> | tail -20

# Hung I/O events, if any
zpool events -v | grep -i deadman

Things to note while reading the output:

Error counters are cumulative since the last zpool clear, and can also reset on pool export/import. A counter of zero proves nothing about history.
There is a reported OpenZFS bug (#11545) where scrub-repaired checksum errors do not always increment the per-vdev CKSUM counter. Do not treat CKSUM=0 as proof of no corruption.
zpool status -x only reports pools with problems, but an ONLINE pool with non-zero per-device error counters still prints “all pools are healthy”. Always look at the per-device columns in zpool status -v, not just the exit state.

How to diagnose it

Identify which vdev group lost redundancy. In zpool status -v, find the faulted, unavailable, or removed device and note which top-level vdev (mirror-N, raidz1-N, raidz2-N) it belongs to. A DEGRADED spare or L2ARC device is a different severity class: L2ARC loss costs read cache, not data. A failed SLOG costs sync write latency, not data. A failed data, special, or dedup vdev member is the real emergency.
Check whether recovery is already running. Look at the scan: line. If a resilver is in progress with bytes scanned, a rate, and an ETA, the system is healing itself. Your job shifts to monitoring progress and protecting the resilver from competing load. The printed ETA is computed from the average rate since start and is often inaccurate, especially early.
If no resilver is running, find out why. If you have hot spares configured, check whether one engaged: a spare that activated will appear in the vdev tree as a replacement, and the spare itself will no longer show AVAIL. Automatic replacement requires spares to be configured and the pool’s autoreplace property enabled. If the spare is still sitting AVAIL while a device is FAULTED, the automation did not fire and you replace manually. ZED-based spare activation has had reliability bugs historically (issue #8967) ; do not assume it happened.
Assess the remaining devices in the affected group. This is the escalation check. Look at READ/WRITE/CKSUM counters on the surviving members of the same mirror or RAIDZ vdev, and compare their latency in zpool iostat -v 1. If counters on the remaining devices are zero and stable, you have time to do a clean replacement. If errors are climbing on a second device in the same group, escalate immediately: you are watching the failure that takes the data.
Characterize the failed device before deciding clear vs replace. Check SMART data (smartctl -A /dev/sdX, and run smartctl -t short if attributes look marginal) and dmesg for the transport errors that preceded the fault. A device that dropped off the bus once during a power event with clean SMART is a different case from a device with rising reallocated sectors and checksum errors.
Rule out shared infrastructure. If multiple unrelated devices show checksum errors simultaneously, suspect RAM or the controller, not the disks. Correlate with ECC/memory errors before replacing half the chassis.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Pool state (`/proc/spl/kstat/zfs/<pool>/state`, `zpool list -o health`)	The binary availability signal	Anything other than ONLINE; DEGRADED = same-day ticket
Per-vdev READ/WRITE/CKSUM counters	Shows which device is failing and whether the rest of the group is clean	Non-zero on any data-bearing device; climbing counters on surviving group members
Resilver progress (`scan:` line)	Tells you the vulnerability window is closing	Resilver stalled (no progress over tens of minutes) or speed far below device capability
Per-vdev latency (`zpool iostat -l` / `-w`)	A dying disk usually slows down before it faults	One vdev consistently much slower than its peers
Scrub status and permanent errors (`zpool status -v`)	Confirms whether redundancy is actually repairing corruption	Uncorrectable or permanent errors: page, data loss has occurred
Deadman events (`zpool events`)	Detects hung I/O during the degraded window	Any deadman event is a page

Escalation rule: DEGRADED alone is a ticket. DEGRADED plus error counters climbing on the remaining devices in the same group is a page. DEGRADED plus a scrub reporting uncorrectable errors means the loss has already happened.

Fixes

Resilver already in progress: let it finish

Do not restart the resilver, do not reboot, and do not kick off a scrub. Scrubs and resilvers are mutually exclusive per pool; a resilver preempts a running scrub, and the scrub restarts afterward. Resilver is interruptible and safe: if the system reboots or loses power mid-resilver, it resumes where it left off.

Your levers during the window:

Reduce non-essential I/O against the pool. Resilver competes with production I/O, and on large HDD-backed RAIDZ vdevs the window can run for days.
Watch per-vdev latency on the surviving members. The resilver reads every surviving device in the group hard, which is exactly when a marginal second disk tends to fail.
Mirror resilvers copy sequentially; traditional RAIDZ resilvers walk the block tree, which is random I/O proportional to used space. OpenZFS 2.0+ added sequential resilver for mirrors and dRAID, which is faster and queues a scrub automatically afterward.

No resilver running and the device is truly failed: replace it

# Replace a failed device with a new one (physical swap first, or by-id path)
zpool replace <pool> <old-device> <new-device>

If a hot spare exists but did not engage, run zpool replace <pool> <failed-device> <spare-device> manually. The replacement triggers a resilver onto the new device; the pool stays DEGRADED until it completes.

Tradeoff: on RAIDZ1 or a two-way mirror you have zero redundancy margin during the resilver, so schedule the work promptly but do not also schedule a scrub, a big zfs send, or a migration job into the same window.

Transient fault on healthy hardware: clear, carefully

The action message in zpool status offers zpool clear to mark a device repaired. This clears the error counters and returns the device to service. It does not fix hardware. Use it only when you have evidence the fault was transient: clean SMART data, a plausible cause in dmesg (a single SATA reset, a hot-plug event, a power blip), and no counter growth afterward. Clearing a dying disk just resets the scoreboard; it will fault again, possibly at a worse time.

Pool state stuck ONLINE with a faulted device

If you are on OpenZFS 2.2.x and zpool status -v shows a FAULTED device while the pool state still reads ONLINE, you may be hitting the state-transition regression (issue #16245) . zpool reopen <pool> forces the state update. Treat the pool as DEGRADED operationally regardless of what the summary line says.

Prevention

Alert on per-device state, not only pool state. The pool-level summary lags and can mask faults. Collect per-vdev state and error counters continuously; zpool status is point-in-time with clearable counters, so export to a time series and alert on transitions.
Use ZED for event-driven alerting. zpool events is in-memory only and lost on reboot. Be aware that some statechange events are not reliably generated for missing-drive degradation (issue #12629) , so poll state as well as consume events.
Configure and test hot spares. Spares plus autoreplace=on shrink the vulnerable window from days to the length of a resilver. Verify engagement actually works; do not assume ZED fired.
Scrub on schedule. Production pools with redundancy should complete a scrub every 7 to 14 days. A pool that has not scrubbed in months has unknown integrity, and the first scrub after a disk failure is a terrible time to discover that.
Baseline resilver time. Know roughly how long a resilver takes on your pool at current fill. That number is your expected vulnerability window and drives the RAIDZ1-vs-RAIDZ2-vs-mirror conversation for the next pool.
Prefer RAIDZ2 or three-way mirrors where the window is long. If a resilver takes days on large spinning disks, single-parity redundancy means days of zero-margin operation every time a disk fails.

How Netdata helps

Netdata collects ZFS pool health, per-vdev error counters (READ/WRITE/CKSUM), and capacity automatically, so a DEGRADED transition shows up as an alert rather than a surprise during the next manual zpool status.
Per-vdev I/O and latency charts let you spot the dying disk before it faults: the one device trending slower and erroring while its mirror partner stays clean.
Correlating pool state with resilver progress and throughput answers the first incident question (is recovery already running?) from one dashboard instead of three SSH sessions.
Historical counter trends distinguish a one-time transport blip from sustained error growth, which is exactly the clear-vs-replace decision.
Alerting on DEGRADED as a ticket-level condition, with escalation when error counters on surviving group members climb, matches the severity model above without paging on a self-healing resilver.