$ guides / zfs / zfs-online-with-errors ▌

Operations Guides

ZFS pool ONLINE with non-zero errors: why zpool status -x lies

Your monitoring script runs zpool status -x, gets back “all pools are healthy”, and moves on. Meanwhile, one disk in a mirror has 4,000 checksum errors, the pool is silently correcting every bad block from the good side, and the failing disk is weeks from dropping off the bus. You find out when the second disk in the vdev fails and the resilver uncovers corruption it can no longer reconstruct.

zpool status -x is a pool-state check, not a health check. It only surfaces pools whose aggregate state is something other than ONLINE: DEGRADED, FAULTED, SUSPENDED, UNAVAIL. Cumulative per-device READ, WRITE, and CKSUM counters do not change pool state. A pool with ten thousand checksum errors on one device is still ONLINE, still serves I/O, and is still invisible to -x.

The problem gets worse on ZFS 2.2.x. A reported regression (OpenZFS issue #16245, still open as of 2026-09-01) shows a pool remaining ONLINE for minutes even after a device is FAULTED, until a zpool reopen forces a state refresh. If your only signal is -x, you are relying on a state that may itself lag reality.

This article covers what ONLINE-with-errors actually means, how to find the devices that are quietly failing, and what to do about them.

What this means

ZFS reports two independent things in zpool status:

Pool and vdev state: ONLINE, DEGRADED, FAULTED, and friends. This answers “can the pool serve I/O right now.”
Per-device error counters: the READ, WRITE, and CKSUM columns. Cumulative counts of transport failures (READ/WRITE) and checksum mismatches (CKSUM) per device, since the last zpool clear, export/import, or module reload.

-x only looks at the first. A device can rack up errors while remaining ONLINE because redundancy masks every failure: the mirror or RAIDZ parity supplies the correct data, ZFS repairs the bad block, the CKSUM counter increments, and nothing else changes. The pool is functionally DEGRADED-in-waiting: the failing device has not been ejected yet, but every error is evidence it is getting closer.

One more trap: zpool status exits 0 regardless of pool state, including for DEGRADED pools. The exit code means the command ran, not that the pool is healthy. Scripts that gate on $? instead of parsing output are checking nothing at all.

Common causes

Cause	What it looks like	First thing to check
Dying disk, slow degradation	CKSUM or READ errors climbing on exactly one device, that device slower than peers in `zpool iostat -v`	`smartctl -A` on that disk: reallocated sectors, pending sectors
Cable, connector, or backplane issue	READ/WRITE errors with few or no CKSUM errors; SATA resets in `dmesg`	`dmesg` for ata link resets, task aborts, device timeouts
Controller failure	Errors on multiple devices attached to the same controller, simultaneously	Which devices share a controller; correlate `dmesg` errors by HBA
Bad RAM	CKSUM errors on multiple unrelated devices at once; ZFS is writing corrupted data to otherwise healthy disks	ECC error counters; `memtest86` on next maintenance window
One-time transient	A single counter increment (e.g., one SATA reset during heavy backup load) that never repeats	Whether counters move again over days; a static non-zero count is historical, not active
Failing SLOG or L2ARC device	Errors on a log or cache vdev; pool ONLINE because these are not data devices	`zpool status -v` log/cache section; sync write latency for SLOG, read latency for L2ARC

If CKSUM errors appear on several devices with no shared controller, suspect RAM before you suspect five simultaneous disk failures. Disks die one at a time; bad memory corrupts everything it touches.

Quick checks

# Pool state only - this is what lies to you
zpool status -x

# Full status with per-device error counters and permanent error list
zpool status -v

# Machine-parseable status for scripting
zpool status -p

# Scrub history: repairs, errors found, timestamps
zpool status | grep -A3 "scan:"

# Per-vdev I/O rates - is one device slower than its peers?
zpool iostat -v 1 5

# Kernel view of transport problems
dmesg | grep -iE "ata|sas|reset|timeout|error" | tail -30

# SMART data on a suspect device
smartctl -A /dev/sdX

Two things -x will never show you. First, the permanent error list at the bottom of zpool status -v: files and objects with uncorrectable damage. If that list is non-empty, data loss has already happened regardless of what the state line says. Second, scrub repair counts: a scrub that “repaired 128K with 0 errors” found and fixed corruption from redundancy. That is a hardware warning, not a clean bill of health.

Since OpenZFS 2.2.3, zpool status -e filters output to unhealthy vdevs: non-ONLINE devices or devices with non-zero error counters. That is the vdev-level view -x does not give you.

How to diagnose it

flowchart TD
  A[Non-zero READ/WRITE/CKSUM in zpool status] --> B{Errors on one device or many?}
  B -->|One device| C{CKSUM or READ/WRITE?}
  B -->|Many devices| D{Same controller?}
  D -->|Yes| E[Suspect controller, HBA, backplane]
  D -->|No| F[Suspect RAM: run memtest86, check ECC]
  C -->|CKSUM rising| G[smartctl -A, short then long SMART test, schedule replacement]
  C -->|Transport only| H[Check dmesg for link resets, reseat or replace cable]
  G --> I[Scrub, watch repair counts, replace disk if errors return]
  H --> I

Read the actual counters. zpool status -v. Note which device has errors, which counter type, and the magnitude. A device with 3 historical CKSUM errors is a different problem from one at 30,000 and climbing.
Establish whether the errors are active. Record the counters now, record them again in 24 to 72 hours, or run a scrub and compare. A static count is history. A growing count is a failing device.
Correlate with SMART. Pull smartctl -A on the suspect disk. CKSUM errors plus rising Reallocated_Sector_Ct or non-zero Current_Pending_Sector means the drive is dying; plan replacement. Run a short SMART test, then a long one.
Correlate with dmesg. Transport errors (READ/WRITE) should have matching kernel messages: link resets, task aborts, timeouts. No kernel messages with pure CKSUM errors points at silent corruption: the drive returned wrong data and claimed success.
Check the blast radius. Errors confined to one device point at that device or its cable. Errors on multiple devices on one HBA point at the controller. Errors scattered across unrelated devices point at RAM.
Scrub and watch repairs. Start a scrub during a low-load window (it is I/O intensive and will raise latency). When it completes, the scan: line tells you what was repaired. Repairs mean redundancy saved you and the source device needs replacing. Uncorrectable errors mean data loss: the permanent error list in zpool status -v is your recovery checklist.
Only then clear. zpool clear resets the counters and clears device error state. It fixes nothing. If you clear before diagnosing, you destroy the evidence and the errors will quietly come back. Clear only after the root cause is confirmed and resolved, or as a baseline reset after replacing hardware.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Per-device READ/WRITE/CKSUM counters	The actual early-warning system for failing hardware; pool state ignores them	Any non-zero value in production; any growth over time
Counter rate of change	Distinguishes a one-time transient from active failure	Incrementing across consecutive checks or scrubs
Scrub repair count per run	Repairs mean corruption was found and fixed from redundancy; the source device is degrading	Repairs > 0 on any scrub
Permanent error list	Non-empty means confirmed data loss	Any entry at all
Time since last completed scrub	Zero errors means zero errors detected; without scrubs, integrity is unknown	No completed scrub in 30+ days
SMART reallocated and pending sectors	Drive-level confirmation that ZFS-side errors are media failure	Rising week over week on the same disk ZFS blames
Pool state (DEGRADED transitions)	Late-stage signal; by the time this fires, redundancy is already gone	Any state other than ONLINE

Zero is the only acceptable production value for error counters. Everything else is a ticket. Escalate to a page when error growth combines with DEGRADED state, uncorrectable scrub errors, or a non-empty permanent error list, because those combinations mean redundancy has run out.

One known instrumentation gap: OpenZFS bug #11545 means scrub-repaired checksum errors may not always increment the per-vdev CKSUM counter. Do not treat CKSUM=0 as proof of no corruption; the scrub repair count is the more reliable signal.

Fixes

One device with growing errors: replace it

If SMART agrees the disk is dying, replace proactively instead of waiting for ZFS to fault it. Use zpool replace with the old device still attached if possible; the resilver then reads from both the old device and its mirror or parity, which is faster and safer than reconstructing from parity alone. On mirrors and dRAID with OpenZFS 2.0 or later, sequential resilver (zpool replace -s) is faster and auto-starts a scrub afterward. While the resilver runs you are at reduced redundancy: a second failure in that vdev group during the window is data loss on RAIDZ1.

Transient single increment: investigate, then baseline

A single SATA reset during a heavy backup window can increment a counter once and never again. Confirm it is static over several days and one scrub, check cables and seating if you are in the chassis anyway, then zpool clear to reset the baseline so future growth is visible against zero.

Multiple devices on one controller: fix the controller

Replacing disks will not help. Check HBA firmware, cables, and backplane. Expect the errors to follow the controller, not the drives.

Scattered CKSUM errors across unrelated devices: test RAM

ZFS checksums on write too; bad memory produces bad data on good disks. Run memtest86 at the next maintenance window and check ECC logs. Do not clear the counters and hope; every write through bad RAM is new corruption.

Failing SLOG or L2ARC: replace the device

Neither failure causes data loss, and neither changes pool state to DEGRADED, which is exactly why -x misses them. A dead SLOG silently moves the ZIL onto the main pool and sync-heavy workloads (databases, NFS) fall off a latency cliff. Replace the SLOG; mirror it if you were not already. A dead L2ARC just costs you cache. Remove it with zpool remove <pool> <device>.

Uncorrectable errors: you are in recovery, not repair

If the permanent error list is non-empty, stop optimizing hardware and restore the listed files from backup. zpool status -v names them.

Prevention

Alert on counters, not state. zpool status -x is survival-level monitoring, and it is insufficient by itself. Alert on any non-zero READ/WRITE/CKSUM per device, and alert harder on growth.
Scrub on a schedule and alert on the results. Every 7 to 14 days on production pools. Alerting on “scrub found errors” without alerting on “scrub has not run in 30 days” leaves you with unknown integrity that reports as zero errors.
Never clear without a diagnosis. Make zpool clear a deliberate step in the runbook after root cause, not a reflex. Counters also reset on export/import and module reload, so track them externally if you need history across reboots.
Export counters to time series. zpool status is a point-in-time snapshot with clearable counters and no trend. Continuous collection turns “is this device failing” into a rate-of-change question you can answer in seconds.
Watch SMART alongside ZFS. ZFS errors plus SMART reallocation trends on the same device is your highest-confidence early replacement signal.

How Netdata helps

Per-device error counters as first-class metrics. Netdata tracks READ, WRITE, and CKSUM per vdev continuously, so the ONLINE-with-errors state that -x hides shows up as a visible, alertable line instead of a surprise in a manual zpool status.
Counter growth over time. Because counters reset on zpool clear and export/import, time-series history is the only way to distinguish a one-time transient from a steadily dying disk. Netdata gives you that rate of change without scripting.
Pool state and counters side by side. Correlating “pool ONLINE” with “CKSUM climbing on one device” on the same dashboard is the exact pattern this article is about, and it is the correlation -x structurally cannot make.
Scrub and resilver visibility. Scan progress, repair counts, and completion status are collected alongside I/O latency, so you can see both that a scrub found corruption and what it cost production I/O while it ran.
System context in one place. ZFS error counters next to per-disk latency and hardware-level metrics lets you confirm “ZFS blames sdX, sdX is also slow and resetting” without pivoting across three tools.