ZFS device FAULTED - too many errors: a disk ejected from the pool

You ran zpool status and found a device FAULTED with “too many errors”, and the pool has dropped to DEGRADED. ZFS did this on purpose: the device’s READ, WRITE, or CKSUM error counters crossed a threshold, and ZFS took the disk out of service to stop it from corrupting or stalling the pool further.

FAULTED is a verdict, not a glitch. ZFS only faults a device after repeated I/O or checksum failures, so the counters you see are the tail end of a problem that has been building. Your job is to find out whether the disk is dying or something between ZFS and the disk (cable, backplane, controller, power) is at fault, then replace or repair before the redundancy you have left disappears.

Do not reach for zpool clear first. Clearing counters without fixing the hardware is the most common way operators turn a one-disk incident into a repeat incident, and on a RAIDZ1 or mirror vdev with no redundancy left, into data loss.

What this means

ZFS tracks three cumulative error counters per device: READ (the device failed to return data), WRITE (the device failed to complete a write), and CKSUM (the device returned data that did not match the stored checksum). READ and WRITE are transport-level failures. CKSUM means the device returned bits that were silently wrong, which is worse: the drive’s own ECC already tried and failed to correct them before ZFS saw the data.

Errors are reported to the ZFS Event Daemon (ZED) as they happen, and the device is faulted once errors accumulate past a threshold within a short time window. Once faulted, the pool drops to DEGRADED if redundancy remains, or FAULTED if it does not.

flowchart TD
  A[Device ONLINE] -->|I/O or checksum errors| B[Error counters grow]
  B -->|below threshold| A
  B -->|errors cross fault threshold| C[Device FAULTED - too many errors]
  C -->|redundancy remains| D[Pool DEGRADED]
  C -->|no redundancy left| E[Pool FAULTED - data unavailable]
  D -->|replace device + resilver| F[Resilver completes]
  F -->|zpool clear after hardware fixed| A
  C -->|zpool clear without fixing hardware| B

Key properties of this state:

• The pool keeps serving I/O through redundancy. DEGRADED is stable, but you are one failure from data loss in the affected vdev.
• The faulted device is out of service. ZFS will not retry it on its own.
• Error counters are cumulative since the last zpool clear and may also reset on pool export/import. A device showing FAULTED with modest counters may have a much longer error history than the numbers suggest.

Common causes

The cabling and backplane case deserves emphasis: the most common root cause of a FAULTED device with clean SMART data is the physical path, not the drive. If you replace the disk and the fault returns on the new disk in the same slot, the slot is the problem.

Quick checks

All read-only and safe to run during an incident.

# 1. Confirm pool state and which vdev faulted
zpool status

# 2. Full detail: per-vdev state, error counters, permanent errors
zpool status -v

# 3. Machine-readable pool state, for scripting or alerting
zpool list -H -o name,health

# 4. See whether the remaining devices are also accumulating errors
zpool status | grep -E "ONLINE|DEGRADED|FAULTED|UNAVAIL"

# 5. Per-vdev throughput: are peers slow too?
zpool iostat -v 1

# 6. Kernel view: link resets, task aborts, timeouts on the storage path
dmesg | grep -i -E "ata|sas|reset|timeout|abort"

# 7. SMART health of the faulted disk
smartctl -A /dev/sdX
smartctl -t short /dev/sdX    # short self-test, safe to run online

# 8. Recent ZFS events: fault, io_failure, probe_failure history
zpool events -v | tail -50

# 9. Is a resilver or scrub already running or needed?
zpool status | grep -A5 "scan:"

Three things to look for:

• Which counter tripped. READ/WRITE errors point at transport (cable, controller, power, drive electronics). CKSUM points at returned-wrong-data (media, or RAM if it shows on multiple devices).
• Whether other devices have non-zero counters. One device with errors is a device problem. Several is a path, controller, or RAM problem.
• The scan: line. If a hot spare kicked in, a resilver may be in progress. Do not start a second replace on top of it.

How to diagnose it

1. Identify the exact device. From zpool status -v, map the vdev name to a physical disk (serial, slot, /dev/disk/by-id/ path). If the pool uses by-id paths, the path usually encodes model and serial.
2. Classify the error type. READ/WRITE growth means the device or its path failed to complete I/O. CKSUM growth means data came back wrong. Re-run zpool status a few minutes apart to see which counters are moving and how fast.
3. Check the kernel log. SATA link resets, SAS task aborts, or I/O timeouts against that device confirm the transport path is involved. Timeouts against several devices at once point at the controller or power.
4. Check SMART. smartctl -A /dev/sdX: reallocated sector count, current pending sectors, offline uncorrectable, and wear/percentage-used for SSDs. Rising reallocated or pending sectors alongside CKSUM errors means the disk is dying; replace it.
5. Decide device versus path. If SMART is clean and dmesg shows link resets, suspect cable, backplane, or power before condemning the disk. If SMART shows media errors, the disk is the problem regardless of the path.
6. Verify remaining redundancy before touching anything. Confirm the rest of the vdev group is ONLINE with zero error growth. On RAIDZ1 or a two-way mirror with one device faulted, you have no redundancy: treat every further command as a data-loss risk and back up critical datasets before replacing.
7. Check whether a spare engaged. If a hot spare activated, the pool may already be resilvering onto it. Confirm with the scan: line before issuing your own zpool replace.

Metrics and signals to monitor

Fixes

Replace the device (the standard fix)

If SMART or the error pattern condemns the disk, swap the drive, then:

# Replace the faulted device with the new disk
zpool replace <pool> <old-device> <new-device>

# Watch the resilver
zpool status <pool>

Tradeoffs and gotchas:

• The pool stays at reduced redundancy for the whole resilver. RAIDZ resilver walks the block tree (proportional to used space, potentially random I/O); mirror resilver copies the device (sequential). On large HDD pools this runs for hours to days, and that entire window is your exposure. Do not schedule scrubs or major migrations during it.
• ashift mismatch can block the replace. If the pool was created with a different ashift than the new disk reports, the replace can fail. Check what the pool expects and force with -o ashift=N if needed. Verify ashift before ordering replacement media.
• SMR drives are a trap for replacement media. Shingled drives can fault again under sustained resilver write load, putting you right back at “too many errors” on the new disk. Use CMR drives for pool members.
• If the faulted device is still partially responsive, zpool offline <pool> <device> before removal is a clean way to take it out of service, provided redundancy remains.

Fix the path, not the disk

If SMART is clean and dmesg implicates the link: reseat or replace the cable, move the disk to a different slot or port, or fix the controller or power issue. Only after the path is verified stable should you return the device to service and clear errors.

zpool clear: use it last, not first

zpool clear <pool> [device] resets the error counters and returns a FAULTED device to service. It is the correct final step after the hardware is fixed and verified, or after a one-time transient event you have positively identified.

As a first response it is a trap: it resets the visible counters without fixing anything, and if errors keep arriving the device will fault again. Each clear-and-refault cycle also erases the error history you need to see the pattern. If you are running zpool clear more than once on the same device or slot, stop clearing and fix the hardware.

# Only after the root cause is fixed and verified
zpool clear <pool> <device>

If the pool is FAULTED, not just the device

If the pool itself went FAULTED or UNAVAIL, you are past this article’s scope and into import/recovery territory, where every write to the remaining devices carries risk. Stop, capture zpool status -v, zpool events -v, and dmesg, and follow recovery procedures before experimenting with imports.

Prevention

• Alert on error counters, not just pool state. A device usually accumulates non-zero READ/WRITE/CKSUM errors for days before it faults. Catching the growth early lets you schedule the replacement instead of eating a 3 a.m. DEGRADED.
• Trend SMART alongside ZFS counters. Reallocated and pending sectors rising on the same device accruing CKSUM errors is your replacement trigger, well before the fault.
• Scrub on a schedule and read the results. Scrubs surface latent corruption on devices normal reads never touch. A scrub repairing errors from one specific device is an early fault warning.
• Configure ZED to act, not just log. Ensure ZED is running and review /etc/zfs/zed.d/zed.rc: hot spare activation on I/O and checksum errors (ZED_SPARE_ON_IO_ERRORS, ZED_SPARE_ON_CHECKSUM_ERRORS) is commented out by default. If you keep hot spares in the chassis, enable these deliberately and test that a fault actually triggers a spare rebuild.
• Stock the right replacement media. CMR spares matched to the pool’s ashift, so a replacement never becomes a science project mid-incident.
• Treat DEGRADED as same-day work. The pool is stable but one failure from data loss. See ZFS pool DEGRADED: redundancy lost and one failure from data loss.

How Netdata helps

• Per-vdev error counters over time. Netdata collects ZFS pool and vdev metrics continuously, so instead of a point-in-time zpool status you see when READ/WRITE/CKSUM counters started growing, and on which device, before the fault fired.
• Pool state as an alertable signal. The transition from ONLINE to DEGRADED when a device faults is exactly the kind of binary event that should page, not wait for the next manual check.
• Correlation with disk-level health. ZFS error counters next to SMART attributes and per-disk latency on one dashboard separates “dying disk” from “bad cable” faster than bouncing between zpool status, smartctl, and dmesg.
• Resilver tracking. Resilver progress and per-vdev throughput during the rebuild show whether recovery is healthy or stalling, and what it is costing production I/O.
• Event history across reboots. zpool events history does not survive reboots; persisting pool state and error trends means the clear-and-refault cycle stays visible even when someone has been resetting counters.

Related guides

• How ZFS actually works in production: a mental model for operators
• ZFS monitoring checklist: the signals every production pool needs
• ZFS monitoring maturity model: from survival to expert
• ZFS pool DEGRADED: redundancy lost and one failure from data loss