ZFS READ and WRITE errors: transport-level device failures in zpool status

You ran zpool status and the READ or WRITE column on one or more devices is not zero. The pool may still show ONLINE. No application has complained yet. This is where ZFS lulls operators into inaction: redundancy is absorbing the failures, so nothing is visibly broken, but the counters are telling you a device, cable, controller, or power path is misbehaving.

The READ and WRITE columns count transport-level I/O failures: operations where the underlying device did not respond or returned an error status. The CKSUM column is different: it counts blocks where the device returned data but the data did not match the stored checksum. READ and WRITE errors mean the I/O itself failed. CKSUM errors mean the I/O succeeded and the payload was wrong. The distinction matters because it points at different hardware.

The counters are cumulative since the last zpool clear, and they live in memory. They can reset on pool export/import or module reload, so a counter of zero after a reboot tells you nothing about what happened before the reboot. Zero is the only acceptable value in production, and the rate of growth, not the absolute number, sets your urgency.

What this means

When ZFS issues a read or write to a leaf vdev and the device fails to complete it, ZFS increments the READ or WRITE counter for that device and retries or reconstructs the data from redundancy. On a mirror, a write error on one side fails over to the other side. On RAIDZ, write errors can fault the device and trigger a resilver. The pool stays ONLINE because parity or mirror copies are doing their job.

That means a non-zero READ/WRITE count with an ONLINE pool is a device in the process of failing, not a device that has failed. A single one-time increment can be a transient glitch, such as one SATA reset during a hot-plug or a brief power sag. Errors accumulating over minutes are a dying disk, a failing controller, a bad cable or backplane, or unstable power. The rate is the signal.

flowchart TD
  A[Non-zero READ or WRITE in zpool status] --> B{Growing over minutes?}
  B -- No, single increment --> C[Transient glitch likely: note it, watch it]
  B -- Yes --> D{Errors on one device or many?}
  D -- One device --> E[Dying disk: check SMART, plan replacement]
  D -- Multiple on same controller --> F[Controller, cable, backplane, or power]
  D -- READ/WRITE plus CKSUM on same device --> E
  E --> G[Replace device, then zpool clear]
  F --> H[Check dmesg, reseat or replace cabling, verify power]

Common causes

Quick checks

All of these are read-only and safe to run on a production pool.

# Full vdev tree with per-device state and error columns
zpool status -v

# Machine-parseable output, useful for diffing over time
zpool status -p

# Only pools with problems (note: a pool with error counters can still report healthy)
zpool status -x

# Kernel-level view of what the storage stack is seeing
dmesg | grep -i -E "ata|sas|reset|timeout" | tail -50

# SMART attributes on the suspect device
smartctl -A /dev/sdX

# Per-vdev throughput to spot the slow device relative to peers
zpool iostat -v 1 5

# Recent ZFS events (in-memory only, lost on reboot)
zpool events -v | tail -50

Two cautions on interpretation. First, zpool status -x only reports pool state. A pool with thousands of cumulative READ errors is still ONLINE and still reports healthy. Second, dmesg is where the kernel records the underlying transport events: SATA resets, task aborts, device timeouts. ZFS counters without matching dmesg noise are unusual; ZFS counters with a storm of link resets point at cabling, controller, or power rather than media.

How to diagnose it

1. Snapshot the current counters. Run zpool status -p and save the output. You need a baseline to measure growth.

2. Wait and re-measure. Wait 10 to 15 minutes of normal load, run zpool status -p again, and diff the READ/WRITE columns. Flat counters after a one-time increment: transient, document it. Counters climbing over minutes: active failure, treat as urgent. Growth over days: slow degradation, schedule replacement but do not panic.

3. Map the error distribution. One device with errors: suspect the disk. Multiple devices on the same controller or expander: suspect the controller, backplane, or cabling. The correlation rules are direct: I/O errors plus CKSUM errors on the same device means the disk is dying; I/O errors on multiple devices on one controller means the controller.

4. Correlate with the kernel log. Match the ZFS counter increments against dmesg timestamps. SATA link resets and task aborts on one port point at that device or its cable. Resets across many ports simultaneously point at the controller or power.

5. Check SMART on the suspect device. Run smartctl -A /dev/sdX and look at Reallocated_Sector_Ct, Current_Pending_Sector, and Offline_Uncorrectable. Rising reallocated or pending sectors alongside ZFS I/O errors means imminent disk failure. Optionally run a short self-test with smartctl -t short /dev/sdX and re-check attributes after it completes.

6. Compare latency against peers. zpool iostat -v 1 shows per-vdev throughput. A failing device frequently retries internally and shows up slower than its mirror or RAIDZ peers before it faults outright. This is the dying-disk pattern: error counts climbing on one device, that device slower than its peers, pool still ONLINE.

7. Check vdev state, not just pool state. Confirm whether any individual device has gone DEGRADED, FAULTED, or UNAVAIL. Redundancy can keep the pool ONLINE while one vdev is already gone. A reported OpenZFS 2.2.x regression (issue #16245, 2.2.4 through 2.2.7) can leave a FAULTED device without immediately transitioning the pool to DEGRADED, so operators watching only pool-level state miss it. zpool reopen forces a state update.

8. Decide the urgency. Accumulating over minutes with redundancy degraded: act now. Slow growth with full redundancy: replace within the maintenance window. Single static increment: monitor.

Metrics and signals to monitor

Because the error counters live in memory and reset on zpool clear and on export/import, point-in-time checks are not enough. Export the counters to a time-series system so a reboot or a well-meaning zpool clear does not erase your failure history, and run ZED (the ZFS Event Daemon) so error events generate alerts as they happen rather than waiting for someone to run zpool status.

Fixes

Dying disk

Replace the device proactively with zpool replace before it faults. A proactive replacement on a still-functioning disk is faster and safer than a resilver from parity after total failure, because ZFS can read the remaining good data from the old device directly. After the replacement completes and the new device is verified clean, run zpool clear to reset the counters.

Do not clear counters on a failing disk and call it done. zpool clear resets the counters; it repairs nothing. Clearing on genuinely failing hardware masks the problem and lets the pool keep running on a device that is actively dying. If the pool is RAIDZ1 or a two-way mirror already missing redundancy, treat this as urgent: the next failure in the same vdev group is data loss. See ZFS pool DEGRADED: redundancy lost and one failure from data loss.

Controller, cable, backplane, or power

Errors on multiple devices attached to one controller almost never mean multiple disks failed at once. Work the physical layer:

• Reseat or replace the cable to the affected device or backplane slot.
• If errors follow the slot rather than the disk, the backplane or cabling is the problem.
• If errors hit everything behind one HBA simultaneously, suspect the controller or its firmware.
• Correlate with power events: intermittent disconnects across unrelated devices suggest PSU or power distribution instability.

After the physical fix, verify counters stay flat for a sustained period before clearing.

Device offlined or faulted, now recovered

If a device was taken offline or faulted and you have resolved the underlying issue, the recovery sequence is: bring the device back with zpool online, then reset the counters with zpool clear. The two commands do different things. zpool online changes device state; zpool clear resets error counters and can re-online a device whose errors were transient. If the errors were not transient, the device will fault again, which is itself diagnostic.

Transient single increment

A one-time increment that does not grow, with clean SMART data and no dmesg recurrence, does not require hardware replacement. Record it with a timestamp so the next person can tell old history from new growth. This is why the cumulative-since-clear semantics bite: without records, an old “3” and a new “3” look identical.

Prevention

• Alert on any non-zero counter, not on state. zpool status -x returns healthy while error counters climb. Scrape the per-vdev READ/WRITE/CKSUM columns continuously and alert on any non-zero value, with severity set by growth rate.
• Keep counter history outside ZFS. Counters reset on zpool clear and on export/import. Push them to a time-series store so history survives reboots and clears.
• Run ZED. Event-driven alerting on ZFS error events catches failures in seconds instead of at the next manual zpool status.
• Monitor SMART alongside ZFS counters. Reallocated sectors rising plus ZFS errors is the highest-confidence predictor of imminent disk failure.
• Baseline scrub cadence. Scrubs surface latent errors in bursts. Knowing your scrub schedule explains periodic counter jumps that are detection of old damage rather than new damage.
• Track your OpenZFS version. Known regressions affect how device faults surface at the pool level; check per-vdev state, not just the pool state line, especially on 2.2.x.

How Netdata helps

• Netdata collects the per-vdev READ, WRITE, and CKSUM error counters from zpool status continuously, so the growth rate over minutes is visible as a trend instead of a manual diff.
• Pool and vdev state is tracked alongside the counters, which catches the case where the pool stays ONLINE while an individual device faults.
• Disk-level metrics and SMART attributes sit next to the ZFS counters on the same host, so you can correlate reallocated sectors and device latency against error increments without switching tools.
• ZFS event and latency signals in the same dashboard let you separate a dying single disk from controller-wide resets by comparing per-vdev behavior across the pool.
• Counter history persists across reboots and zpool clear, closing the gap where ZFS itself forgets.

Related guides

• How ZFS actually works in production: a mental model for operators
• ZFS monitoring checklist: the signals every production pool needs
• ZFS monitoring maturity model: from survival to expert
• ZFS pool DEGRADED: redundancy lost and one failure from data loss