ZFS scrub repaired errors: correctable rot versus permanent data loss

A scrub finishes and the scan: line in zpool status reads something like scrub repaired 8.09M in 04:12:33 with 0 errors on Sun Jul 20 04:12:34 2026. Two numbers in that line decide your next 24 hours: how much data ZFS had to fix, and whether anything was lost for good. Operators misread this line in both directions: panicking over a large repaired count that cost zero data, or shrugging at a small error count that means files are permanently damaged.

This guide is about reading that line correctly, finding the hardware that caused it, and knowing when “repaired” is a warning and when “errors” is an incident. For the broader failure catalogue and mental model, see How ZFS actually works in production.

What this means

A scrub walks every allocated block in the pool, recomputes each block’s checksum, and compares it against the checksum stored in the block pointer. When the data on a device does not match its checksum, ZFS has found silent corruption: the disk returned bytes that are not the bytes ZFS wrote.

What happens next depends on redundancy:

Repaired > 0, errors = 0. ZFS found checksum mismatches and rewrote the bad blocks from a good copy (mirror partner or RAIDZ parity). No data was lost. This is correctable rot, and it means at least one device is returning bad data. The pool is safe today; the hardware question is not answered.
Errors > 0. ZFS found blocks it could not repair, because the redundant copy was also bad, missing, or never existed (no redundancy, or degraded redundancy at the time of the scrub). Those blocks are permanently lost. zpool status -v lists the affected files and objects under errors: Permanent errors have been detected in the following files:.
Repaired = 0B, errors = 0. Nothing found. This says nothing about blocks the scrub did not reach (a cancelled scrub guarantees nothing) and nothing about pools that have not been scrubbed recently.

Two properties of the counters matter before you interpret anything. First, the per-device READ/WRITE/CKSUM counters in zpool status are cumulative since the last zpool clear, held in memory, and can reset on pool export/import. Second, there is a known OpenZFS bug (#11545) where scrub-repaired checksum errors do not always increment the per-vdev CKSUM counter. Do not treat CKSUM = 0 as proof a device is clean when the scan line says otherwise.

flowchart TD
  A["scan: scrub repaired N with M errors"] --> B{M greater than 0?}
  B -->|"No, N > 0"| C[Correctable rot, data intact]
  B -->|Yes| D[Permanent data loss]
  C --> E[Find source vdev: CKSUM, SMART, dmesg]
  D --> F["zpool status -v: list affected files"]
  E --> G[Plan proactive replacement]
  F --> H[Restore from backup, then zpool clear]

Common causes

Cause	What it looks like	First thing to check
Dying disk, slow rot	Repaired count grows scrub over scrub, CKSUM climbs on one device, that device slower than peers	SMART attributes on that device (`Reallocated_Sector_Ct`, `Current_Pending_Sector`)
Bad cable, backplane, or connector	CKSUM and/or READ errors on one device, SATA reset messages in `dmesg`	`dmesg` for link resets and task aborts; reseat or swap the cable
Bad RAM	CKSUM errors on multiple unrelated devices at once, no pattern by device	Run memtest86; check for ECC error reports
HBA or controller corruption	Errors across several devices on the same controller	Group errors by controller; check firmware and controller logs
Transient event recorded as permanent	Old “permanent errors” list after a cable pull or iSCSI reconfiguration, new scrub is clean	Re-run the scrub; the permanent error list is historical until cleared
Errors repaired during normal reads	Scrub reports `repaired 0B` but a device shows non-zero CKSUM	Counter is cumulative; corruption was caught on reads before the scrub ran

If CKSUM errors appear on multiple devices that share nothing but the host, suspect RAM or the controller before you suspect two disks dying simultaneously. Disks fail independently; memory and controllers corrupt everything they touch.

Quick checks

# Read the scan line and error summary for one pool
zpool status tank

# Full detail: per-vdev errors and the permanent error list
zpool status -v tank

# Print exact numeric values (no human-readable units) for scripting
zpool status -p tank

# Check what the scan section is doing right now: scrub or resilver
zpool status tank | grep -A5 "scan:"

# Per-vdev throughput and latency: is one device slower than its peers?
zpool iostat -v 1 5

# Kernel view of the same devices: link resets, aborts, timeouts
dmesg | grep -i -E "ata|sas|reset|timeout" | tail -30

# SMART health on the suspect device
smartctl -a /dev/sdX

All of these are read-only and safe to run during production hours. Note that zpool iostat includes scrub I/O in its totals, so elevated read throughput during a scrub window is expected, not a symptom.

How to diagnose it

Capture the scan line verbatim. Record repaired bytes, error count, duration, and the completion timestamp. This is your baseline for trend comparison. There is no zpool get last_scrub_time in OpenZFS; the scan: line in zpool status is the record.
Split the verdict. Errors = 0 means correctable rot: go to step 3. Errors > 0 means permanent loss: skip to step 5.
Attribute the corruption to a device. Look at per-vdev CKSUM in zpool status -v. One device with a growing count points at that disk. Several devices at once points at RAM, cabling, the shared backplane, or the controller. Remember bug #11545: a zero CKSUM column does not fully exonerate a device, so weight the SMART data too.
Confirm with hardware signals. Run smartctl -a on the suspect device and check reallocated and pending sector counts. Correlate with dmesg for resets on that device’s link. If errors span multiple devices, schedule a memtest86 run and review ECC logs before blaming any disk.
For permanent errors, enumerate the damage. The errors: section of zpool status -v lists affected files and objects. That list is your recovery checklist: every entry needs a restore decision. See ZFS permanent errors have been detected in the following files.
Decide if the permanent error list is stale. The list persists until zpool clear, and it is historical: a transient event (cable pull, target reconfiguration) can leave entries even though the data is fine now. Re-run a full scrub. If it completes with 0 errors and no new entries, the old entries refer to an event, not ongoing rot.
Check the trend, not just the incident. Compare this scrub’s duration against previous runs on similar data volume. A scrub that took 4 hours last month and 11 hours now is telling you a device or the pool layout is degrading, even when the error count is zero.
Check scrub cadence. If the last completed scrub is more than 30 days old, the correctable-versus-permanent question has been unanswerable for that whole window. Zero detected errors on an unscrubbed pool means unknown integrity, not integrity.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Scrub repaired bytes per run	Measures how much silent corruption redundancy is absorbing	Any non-zero value; growth across consecutive scrubs
Scrub error count per run	Binary data-loss signal	Anything above 0, page immediately
Scrub duration	Proxy for pool and device I/O health	Climbing duration on similar data volume
Time since last completed scrub	Bounds the undetected-corruption window	More than 30 days on a production pool
Per-vdev CKSUM counter	Attributes corruption to hardware	Any non-zero value; sustained increment
Per-vdev latency (`zpool iostat -l`)	A device with rot often slows before it dies	One vdev consistently slower than peers
SMART reallocated and pending sectors	Independent confirmation of media failure	Rising counts on the same device ZFS flags

The severity split: uncorrectable errors or a non-empty permanent error list is a page, because ZFS attempted repair and could not. Correctable repairs are a ticket: the data is safe, but the hardware question must be answered within business hours, before a second failure removes the redundancy that just saved you.

Fixes

Single device with growing repairs or CKSUM

Replace the disk proactively. Check SMART first to confirm, then schedule the replacement. For a RAIDZ1 pool treat this as urgent: the scrub repairs prove redundancy is already being consumed, and one more failure in that vdev converts correctable rot into permanent loss. Use zpool replace rather than waiting for the device to fault on its own; a controlled replacement keeps the old device available as a second copy during the resilver if it is still partly functional.

Errors on multiple devices

Do not replace disks. Stop and test the shared components: run memtest86 against the RAM, check ECC reporting, inspect or swap cables and the HBA. ZFS checksums catch bad data in flight, so a bad stick of RAM can stamp plausible-looking corruption onto every disk in the pool. Replacing disks in this scenario replaces healthy hardware and leaves the cause in place.

Permanent errors (unrepaired)

Work the list from zpool status -v. Restore each affected file from backup or snapshot replicas, then deal with the hardware cause, then run another scrub to confirm no new damage. Only after the pool scrubs clean should you consider zpool clear to reset the counters and the error list. zpool clear destroys the historical record of what was damaged; run it only when you have documented everything you need from it.

Scrub shows repaired 0B but CKSUM is non-zero

The corruption was detected and repaired during normal reads before the scrub ever reached those blocks. The counter is cumulative and will sit there until zpool clear. Investigate the device exactly as if the scrub had done the repairs: the signal about the hardware is identical.

Prevention

Scrub on a schedule. Production pools should complete a scrub every 7 to 14 days. Alert when no scrub has completed in 30 days, not just on errors. Alerting on results without verifying execution is the classic gap.
Alert on any repaired count above zero. A repair means redundancy was consumed. Trending the repaired bytes per scrub turns “the disk died suddenly” into a three-month warning.
Trend scrub duration. Rising duration at constant data volume is an early degradation signal that fires before error counters move.
Do not trust zpool status -x. A pool with thousands of repairs is still ONLINE. See ZFS pool ONLINE with non-zero errors.
Replace on evidence, not on failure. A device that scrubs keep repairing is already failing. Proactive replacement while redundancy is intact is a routine ticket; reactive replacement after a second fault is an incident.
Use ECC memory where you can. It does not eliminate in-flight corruption from cables and controllers, but it removes the most common multi-device corruption source.

How Netdata helps

Per-vdev READ, WRITE, and CKSUM counters over time, so a slowly climbing checksum count is visible as a trend instead of a surprise in a manual zpool status.
Scrub state and duration tracking per pool, which makes “this scrub took twice as long as the last one” an alertable condition rather than something you notice by accident.
Correlation between ZFS error counters and disk-level SMART and latency metrics on the same dashboard, so you can confirm “ZFS says CKSUM, SMART says pending sectors” in one view instead of two terminals.
Pool state alerting that separates ONLINE-with-errors from genuinely healthy, closing the zpool status -x blind spot.
Alerts when the time since the last completed scrub exceeds your policy window, catching the “scrub cron died months ago” failure mode.