ZFS checksum errors (CKSUM): the definitive signal of silent corruption

A non-zero number in the CKSUM column of zpool status means a block read from that device did not match the checksum ZFS stored for it. The data came back wrong, and ZFS can prove it. This is the only signal in your storage stack that definitively says “silent corruption happened here”, and zero is the only acceptable value in production.

What happens next depends on redundancy. On a mirror or RAIDZ vdev, ZFS reads the good copy or reconstructs from parity, repairs the bad block in place, and serves correct data to the application. The pool stays ONLINE, nothing page-fires, and the only evidence is the incrementing counter. On a single-disk vdev with no redundancy, the same counter means unrecoverable corruption: ZFS cannot repair what it has no second copy of.

This is why CKSUM is both easy to ignore and dangerous to ignore. The system keeps working. The counter keeps climbing. The device keeps getting worse.

What this means

Every block ZFS writes is checksummed, and the checksum is stored separately from the data, in the parent block pointer. On every read, ZFS recomputes the checksum and compares. A mismatch means one of three things: the device returned different bytes than were written (media degradation, firmware bug), the bytes were corrupted in flight (cable, backplane, controller, HBA), or the bytes were corrupted before they reached the disk (bad RAM). The counter alone cannot tell you which one. That is your job.

Two properties of the counter matter before you interpret anything:

• Counters are cumulative and volatile. They accumulate since the last zpool clear and live in memory; they may also reset on pool export/import or module reload. A device showing CKSUM=0 today may have a cleared history of errors. Conversely, old errors stay attributed to a replaced device until you run zpool clear.
• CKSUM=0 is not proof of integrity. Checksums are only verified on blocks that are actually read. Known OpenZFS bug #11545 means scrub repairs may not always increment the CKSUM counter. And if no scrub has run in months, most of your data has never been re-verified. Zero errors means zero errors detected.

Metadata versus data errors matter for severity. A corrupt data block costs you one file or one range of one file. Corrupt metadata can make entire datasets inaccessible, because metadata is the block tree that describes everything else. ZFS stores extra copies of metadata by default, which makes metadata corruption more survivable, but treat metadata checksum failures as a more serious incident than the same count on user data.

Common causes

The most useful heuristic: errors on one device point at that device or its cable; errors on several devices at once point at something shared. Disks do not coordinate their failures. Controllers, backplanes, power supplies, and RAM are shared.

Quick checks

All read-only and safe to run during production, except where noted.

# Full pool state with per-device error counters and permanent error list
zpool status -v

# Machine-parseable counters for trending
zpool status -p

# Only pools with problems (note: this does NOT flag non-zero error counters
# on an ONLINE pool; always read the full status)
zpool status -x

# Scrub history and current scan state
zpool status | grep -A 3 "scan:"

# Kernel-level storage errors that correlate with ZFS counters
dmesg | grep -i -E "ata|sas|reset|timeout|error" | tail -50

# SMART attributes for the suspect device
smartctl -A /dev/sdX

# Kick off a short SMART self-test on the suspect device (non-destructive,
# but adds some I/O load for a couple of minutes)
smartctl -t short /dev/sdX

A note on zpool status -x: it reports “all pools are healthy” based on pool state only. A pool with a thousand checksum errors is still ONLINE and still passes -x. If this is your only integrity check, you are blind to exactly the failure this article is about.

How to diagnose it

Work through these in order. The goal is to answer three questions: is corruption still happening, which component is at fault, and was any data actually lost.

1. Snapshot the current state. Run zpool status -v and save the output. Record per-device CKSUM values and the time. You need a baseline to measure growth.

2. Check for actual data loss first. Read the errors: section at the bottom of zpool status -v. If it says “No known data errors”, everything detected so far was repaired from redundancy. If it lists files or objects with permanent errors, that is unrecoverable data loss: document the affected files and start planning restore from backup before you continue triaging hardware. This list persists until zpool clear and is your recovery checklist.

3. Map the error distribution. One device with errors, or several? Several devices on the same controller points at the controller, cabling, or backplane. Several devices with no common hardware points at RAM. One device points at the disk itself or its individual cable.

4. Correlate with SMART. On the suspect device, check Reallocated_Sector_Ct, Current_Pending_Sector, and Offline_Uncorrectable with smartctl -A. Rising reallocated or pending sectors alongside rising CKSUM means the drive is dying: plan replacement. Clean SMART with rising CKSUM points at the connection, not the media. Treat a rising UDMA_CRC_Error_Count as corroborating cabling/link evidence, not sole proof: the port, backplane, or controller can also produce CRC errors.

5. Check the kernel log. dmesg showing SATA link resets, SAS task aborts, or device timeouts on the same device or controller confirms a transport problem. ZFS CKSUM errors plus kernel link resets on one port is usually a cable or connector, not a disk.

6. Measure the growth rate. Re-run zpool status after hours or a day of normal load. A counter that incremented once and stayed flat is a transient. A counter that climbs every scrub or every day is active corruption. Rate of accumulation determines urgency.

7. Run a scrub to force verification. zpool scrub <pool> reads every allocated block and verifies checksums, surfacing latent corruption instead of waiting for applications to trip over it. Schedule it for a low-load window; scrubs are I/O intensive. Watch both the repair count and which devices accumulate errors.

8. If multiple devices are implicated, test memory. Take the host through a maintenance window and run memtest86. RAM corruption writes bad data to disk with valid checksums computed over the corrupted bytes; ZFS then faithfully stores and verifies the wrong data. CKSUM on multiple unrelated devices means suspect RAM, not disks.

flowchart TD
    A[CKSUM > 0 in zpool status] --> B{Permanent errors in zpool status -v?}
    B -->|yes| C[Data loss: document files, restore from backup]
    B -->|no| D{Errors on one device or many?}
    D -->|one| E[Check SMART reallocated/pending sectors]
    E -->|rising| F[Replace the disk]
    E -->|clean| G[Suspect cable, connector, backplane slot]
    D -->|many| H{Common controller or backplane?}
    H -->|yes| I[Suspect HBA, cabling, expander, power]
    H -->|no| J[Suspect RAM: check ECC, run memtest86]

Metrics and signals to monitor

Severity: a non-zero or growing CKSUM counter is a ticket, not a page, because redundancy means the data is currently safe. It becomes a page through composite signals: uncorrectable scrub errors, a non-empty permanent error list, or the pool going DEGRADED or FAULTED.

Fixes

Failing single disk

Replace it before it faults. With the replacement installed, zpool replace <pool> <old> <new> starts the resilver. On RAIDZ1 this is urgent: you are one device failure from data loss while the disk degrades. Tradeoff: the resilver competes with production I/O for hours to days on large spinning disks, but every day you wait is a day the second failure can happen.

Cable, connector, or backplane

Reseat or swap the cable, or move the drive to a known-good slot. Then zpool clear <pool> to reset the counters and watch. If CKSUM stays at zero through the next scrub and normal load, the connection was the problem and the disk is fine. Do not clear counters before you have recorded the old values.

Controller or HBA

Check firmware, reseat the card, verify power and cooling to the enclosure. If multiple devices on one controller accumulate errors, replacing individual disks accomplishes nothing; the new disks will start accumulating errors too.

Bad RAM

Confirm with ECC logs and memtest86, replace the DIMM, then scrub the entire pool. RAM corruption is the worst case for data already on disk: blocks written while the RAM was bad carry checksums computed over corrupt data, so they verify cleanly and ZFS has no way to know. Only known-good copies (redundancy written before the corruption, or backups) can restore them.

Permanent errors already present

For files listed in the permanent error list, delete the file and restore from backup, or restore over it in place. ZFS cannot repair a block it has no good copy of. If the bad block is still referenced by a snapshot and has no good copy, the error can reappear on every scrub until the last referencing snapshot is destroyed.

About zpool clear

zpool clear <pool> resets the error counters. It repairs nothing. Clearing without fixing the root cause just restarts the clock and destroys the history you need for trending. Clear after the fix, never instead of it.

Prevention

• Scrub on a schedule and alert on results. Production pools should complete a scrub every 7 to 14 days. Alerting only on “did a scrub run” without checking the repair count is the classic failure: teams discover months of accumulated repairs only when a second disk dies and the resilver cannot reconstruct the data.
• Trend counters, do not snapshot them. zpool status is point-in-time with clearable, volatile counters. Export per-device READ/WRITE/CKSUM to a time-series store continuously so growth rate, not absolute value, drives alerts.
• Alert on any non-zero CKSUM on data-bearing devices. Zero is the only acceptable value. Tolerating small counts normalizes deviance until the count that matters gets ignored.
• Monitor SMART alongside ZFS counters. Reallocated and pending sector trends give you the confirmation layer that turns “CKSUM is climbing” into “replace this specific disk this week.”
• Use ECC memory on ZFS hosts. Without ECC you have no detection layer between bad RAM and your pool, and multi-device CKSUM becomes your first symptom instead of your confirmation.
• Never treat CKSUM=0 as an integrity attestation. With bug #11545 in the wild and scrubs that may not have run recently, zero means “nothing detected”, full stop.

How Netdata helps

• Netdata collects per-device READ, WRITE, and CKSUM counters from zpool status continuously, so you see the growth trend rather than a single cleared snapshot.
• Scrub state, completion, and repair counts are tracked per pool, which closes the “scrubs run but nobody reads the results” gap.
• Pool state transitions (ONLINE to DEGRADED to FAULTED) are alerted on directly, covering the escalation path from correctable errors to lost redundancy.
• Because Netdata also collects SMART attributes and system memory errors from the same host, you can correlate a climbing CKSUM counter with reallocated sectors or ECC events on one dashboard instead of three SSH sessions.
• ZED-driven events and deadman alerts complement counter polling for the failure modes that happen between samples.

Related guides

• How ZFS actually works in production: a mental model for operators
• ZFS monitoring checklist: the signals every production pool needs
• ZFS monitoring maturity model: from survival to expert
• ZFS pool DEGRADED: redundancy lost and one failure from data loss
• ZFS pool FAULTED: when the pool can no longer serve I/O
• ZFS pool I/O is currently suspended: a hung pool and blocked I/O
• ZFS READ and WRITE errors: transport-level device failures in zpool status
• ZFS device FAULTED - too many errors: a disk ejected from the pool