ZFS silent data corruption: how bit rot happens and why scrubs catch it

A disk can return the wrong bytes without returning an error. The drive reports success, the controller reports success, and the application gets corrupt data. Conventional filesystems have no way to notice: they trust the storage stack, and at scale the storage stack lies often enough that corruption is a when, not an if.

ZFS is designed around the assumption that every layer below it will eventually return wrong data. Every block carries a checksum, and every read verifies it. That is why ZFS operators see corruption events that ext4 or XFS operators never see: not because ZFS systems corrupt more, but because ZFS is the only layer looking.

The catch is that detection only happens when a block is read. Data that sits untouched for months can rot silently, and if the redundant copy rots too, the data is gone before anyone knows there was a problem. The time between corruption and detection is the danger window, and scrubs exist to close it.

What silent corruption is and why ZFS sees it

Silent data corruption is any case where the bytes returned by the storage stack differ from the bytes written, with no error surfaced to the operating system. “Bit rot” is the popular name for one source (gradual magnetic or charge decay on media), but corruption can also be introduced by the disk, the cable, the controller, the firmware, or the RAM the data passed through on the way to disk.

ZFS detects this because of two design decisions working together:

Copy-on-write. ZFS never overwrites a block in place. Every write goes to a new location and pointers are updated atomically, so the checksum stored for a block always corresponds to exactly one immutable version of that block’s contents.
End-to-end checksums. The checksum for every block is stored in the parent block pointer, not alongside the data. This forms a checksum tree rooted in the uberblock. On every read, ZFS recomputes the checksum and compares it against the value stored in the parent. A mismatch means the block, or the path that delivered it, is wrong.

Storing the checksum in the parent rather than next to the data matters. It catches whole classes of misdirection: a block written to the wrong location, a stale block returned for the right address, a phantom write that never landed. The parent pointer independently says what the child must look like.

flowchart TD
    W[Write path] --> C1[Checksum computed at write time]
    C1 --> P[Checksum stored in parent block pointer]
    P --> D[Data + parent tree written to disk]
    D --> R[Read path: block fetched]
    R --> V[Checksum recomputed and compared to parent]
    V -->|match| OK[Data returned to application]
    V -->|mismatch| Q{Redundancy available?}
    Q -->|mirror or RAIDZ| FIX[Repair from good copy, rewrite bad block, CKSUM counter increments]
    Q -->|no redundancy| LOST[Read error returned, permanent error logged]

How bit rot happens: the corruption sources

Each corruption source has a distinct operational signature, and the signature is what tells you what to replace.

Source	What it looks like	First thing to check
Media bit rot (failing sectors)	CKSUM errors on one device, slowly rising; SMART reallocated or pending sectors climbing	`smartctl -A` on the device; plan replacement
Bad cable or backplane	Errors on one or two devices sharing a path, sometimes with READ/WRITE errors mixed in	`dmesg` for SATA link resets and task aborts; reseat or replace the cable
Controller or HBA memory corruption	CKSUM errors on multiple devices attached to the same controller	Errors correlated by controller, not by disk model or age
System RAM errors	CKSUM errors on multiple unrelated devices simultaneously, across controllers	ECC error counts; run a memory test. Bad RAM writes corrupt data to every disk it touches
Firmware bugs	Errors clustered by drive model or firmware revision, sometimes after a specific event (resilver, power loss)	Check vendor advisories for the affected model and firmware

The multi-device patterns are the ones operators misdiagnose most often. If checksum errors appear on several unrelated disks at once, the disks are almost never the problem: the common elements are RAM, the controller, and the backplane. Replacing disks in that situation replaces innocent hardware while the real source keeps corrupting new data.

The danger window

Detection happens at read time. That has two consequences that define the risk model.

Hot data protects itself. Blocks that are read frequently get verified frequently. If one mirror leg rots, the next read catches it and repairs it from the good copy.

Cold data has no protection between scrubs. Archived datasets, old snapshots, backup targets, anything written once and read rarely: corruption on these blocks is invisible until something reads them. If the primary copy rots in March and the redundant copy fails in June, a read in July discovers the corruption at the exact moment there is nothing left to repair from. A scrub in April would have caught it while redundancy was still intact.

This is why “the pool is ONLINE and nothing complains” tells you nothing about integrity. The pool stays ONLINE throughout the entire corruption lifecycle. Corruption without scrubs is a Schrodinger state: the data is fine or destroyed, and you find out when someone finally reads it.

There is a second subtlety. A known OpenZFS bug (#11545) means scrub-repaired checksum errors do not always increment the per-vdev CKSUM counter. A counter of zero is evidence, not proof.

Why scrubs catch it

A scrub walks the entire allocated block tree, reads every data and metadata block (including blocks held only by snapshots), and verifies each checksum against the parent pointer. When redundancy exists, a mismatch is repaired on the spot: ZFS reconstructs the block from the mirror or parity copy and writes the corrected version back to the failing device. When no redundancy exists, the mismatch becomes a permanent error naming the affected file or object.

# Start a scrub (safe, but I/O intensive; schedule for low-load windows)
zpool scrub tank

# Check progress and last result
zpool status tank

The scan: line tells you what you need: scrub repaired 512K in 04:12:33 with 0 errors on Tue Jul 14 18:22:41 2026 means corruption was found and fixed. The repaired byte count matters even when the final error count is zero; it means a device returned bad data and redundancy absorbed it.

Properties worth knowing:

Scrubs and resilvers are mutually exclusive per pool. A resilver preempts a running scrub.
A scrub verifies block-level integrity, not application consistency. It proves the bytes match the checksums, not that the bytes are meaningful.
A cancelled scrub guarantees nothing for the portion of the tree it never reached.
Scrub does not defragment and does not verify L2ARC. It reads and verifies only.

Cadence is the whole game. A workable default is a completed scrub every 7 to 14 days on production pools, with an alert if no scrub has completed in 30 days. Vendor guidance (Klara Systems, among others) sets monthly as the floor. The right interval depends on how much cold data you hold and how much redundancy you have: a RAIDZ1 pool full of archives deserves weekly scrubs, because its entire margin is one device.

What scrubs do not catch

Scrubs verify that stored blocks match their checksums. They cannot catch corruption that is checksummed correctly on the way in, meaning bugs in the ZFS write path itself. Recent history has several:

Block cloning corruption (OpenZFS 2.2.0, fixed in 2.2.2 and 2.1.14). Files copied by tools using copy_file_range could have chunks silently replaced with zeros. Scrubs reported zero errors because the zeroed blocks were validly checksummed. See OpenZFS issue #15526.
Dedup write-path regression (2.4.0 release candidates). A race in the DDT write path produced zeroed blocks with dedup=on, again invisible to scrubs. See OpenZFS issue #18366.
Metaslab space-map corruption. A years-old issue that surfaces as a kernel panic (“adding existent segment to range tree”) on snapshot deletion. Scrubs complete cleanly because the corruption is in allocation metadata, not in checksummed payload blocks. There is no supported repair path; the practical fix is pool recreation from backup.

The lesson is not that scrubs are useless. It is that scrubs are one layer. Version currency, tested backups, and application-level verification (your own hashes of critical archives) cover the failure modes that block-level checksums structurally cannot.

Signals to watch in production

Signal	Why it matters	Warning sign
Per-vdev CKSUM counter (`zpool status -v`)	The definitive corruption signal, per device	Any non-zero value; growth over time is active corruption
Scrub result line (`zpool status` scan)	Confirms verification is actually happening and whether repairs occurred	Bytes repaired > 0, any uncorrectable errors, or no scrub in 30+ days
Permanent error list (`zpool status -v`)	Names the files/objects with irrecoverable damage	Any entries at all; this is data loss, page
Pool health state	DEGRADED plus rising errors means the repair margin is shrinking	DEGRADED combined with CKSUM growth on surviving devices
SMART reallocated/pending sectors	Correlates media rot with ZFS-level CKSUM evidence	Rising counts on the same device showing CKSUM errors
Memory ECC errors (`edac-util`, `mcelog`, or BMC)	Distinguishes RAM-sourced corruption from disk-sourced	ECC corrections climbing alongside multi-device CKSUM errors
READ/WRITE error columns	Transport failures often precede or accompany checksum failures	Non-zero counts on a device or shared controller path

The correlation patterns carry the diagnosis. CKSUM plus SMART reallocated sectors on one device is a dying disk. CKSUM on multiple unrelated devices plus ECC events is RAM. CKSUM plus READ/WRITE errors on devices sharing a controller or backplane is the interconnect, not the media.

How Netdata helps

Tracks per-vdev READ, WRITE, and CKSUM error counters over time, so a slow accumulation that point-in-time zpool status would hide becomes a visible trend.
Monitors scrub execution and results, closing a common gap: teams alert on scrub errors but never check whether scrubs are actually running.
Correlates ZFS checksum errors with disk SMART attributes and system memory metrics on the same dashboard, which is the join you need to decide between “replace this disk” and “the RAM is corrupting everything.”
Surfaces pool health state transitions alongside error counters, so a DEGRADED pool with climbing CKSUM on the surviving leg escalates instead of sitting as a silent ticket.
Retains history across zpool clear, preserving the error timeline after ZFS’s counters are reset.