$ guides / zfs / zfs-cksum-errors-multiple-devices ▌

Operations Guides

ZFS checksum errors on multiple devices: suspect RAM or the controller, not the disks

zpool status shows non-zero CKSUM counters, and not on one disk. Two, four, or every device in the vdev has them. The instinct is to start RMA-ing drives. Stop. Independent disks do not fail in the same way at the same time. When checksum errors appear on multiple unrelated devices simultaneously, the failure is almost always upstream of the disks: something is corrupting data between the application and the platters, and every disk downstream of it is faithfully recording the damage.

The usual suspects, in rough order of probability: bad RAM corrupting data before it reaches the disks, a failing HBA or RAID controller, or a shared backplane or cabling path. ZFS cannot tell you which, because ZFS only sees what it was handed. The error pattern plus a few hardware-level checks separates them quickly.

This article covers the triage: confirming the multi-device pattern, discriminating RAM from controller from cabling, and handling the data written while corruption was active.

What this means

ZFS checksums every block and verifies that checksum on read. A CKSUM error means the bytes the device returned do not match the checksum stored in metadata. On a single device, that points at the device: dying media, bad sectors, firmware returning stale data. The per-device failure modes are covered in ZFS checksum errors (CKSUM).

When several devices report CKSUM errors in the same window, the probability of multiple independent disks developing checksum-visible faults simultaneously is negligible. What is not negligible:

One bad DIMM corrupting every write. ZFS computes checksums over data in memory. If the data is already corrupt in RAM, ZFS computes a perfectly valid checksum over corrupt data and writes both to disk; later reads may also be corrupted in memory before verification. Every device in the pool receives its share of poisoned blocks. This is why ECC RAM is so strongly recommended for ZFS: ZFS trusts what memory hands it.
One HBA or controller corrupting I/O in flight. Every device attached to that controller sits behind the same failing component. A dying HBA can produce CKSUM errors on all twelve drives behind it while every drive is healthy.
A shared backplane or cabling fault. Less common, but a degraded backplane or a batch of marginal cables affects every device routed through it.

The critical consequence: if the root cause is RAM, the corruption is not limited to blocks that failed checksums. Data corrupted in memory and written with a matching checksum is invisible to scrubs. The CKSUM counter only catches the subset where corruption happened after checksumming, or where verification itself read back different bytes. Treat the counter as a symptom of a much larger problem.

flowchart TD
  A[CKSUM errors on multiple devices] --> B{Errors only on devices behind one HBA or backplane?}
  B -->|Yes| C{SMART UDMA_CRC errors or SATA/SAS resets in dmesg?}
  C -->|Yes| D[Suspect cabling or backplane]
  C -->|No| E[Suspect HBA or controller]
  B -->|No, spread across controllers| F[Suspect RAM]
  F --> G[Run memtest86, check ECC/EDAC logs]
  B -->|Identical counts on mirror members| F

Common causes

Cause	What it looks like	First thing to check
Bad RAM (non-ECC)	CKSUM errors across devices on different controllers, often symmetric counts on mirror members, no SMART anomalies, no bus resets	Boot memtest86, run multiple passes
Failing HBA / controller	CKSUM (and often READ/WRITE) errors on every device attached to one controller	`dmesg` for SAS/ATA errors; controller firmware level
Cabling or backplane	CKSUM plus non-zero UDMA_CRC in SMART, link resets in kernel logs	`smartctl -A` UDMA_CRC_Error_Count; reseat cables
Marginal PSU or power delivery	Intermittent multi-device errors under load, devices dropping and reappearing	Kernel logs for link downs correlated with load
Software edge case	Errors only during scrub or after rebuild, hardware replacement changes nothing	OpenZFS version and release notes for checksum fixes

Two notes on the table. First, a single bad HBA producing CKSUM errors on every drive behind it is a well-documented failure mode: operators have replaced drives, cables, and firmware without improvement, and only replacing the HBA stopped the errors. Second, there are rare OpenZFS-side causes. If CKSUM errors appear on all members of a raidz vdev during scrubs and persist across complete hardware replacement, you may be hitting a parity reconstruction edge case rather than a hardware fault. Check that you are on a current OpenZFS release; OpenZFS 2.3.7 and 2.4.2 included fixes for rare checksum errors after rebuilds. On those branches, go to 2.3.7+ or 2.4.2+ before attributing the pattern to hardware.

Quick checks

All read-only. Run these before touching hardware.

# 1. Full error picture: which devices, which counters, how many
zpool status -v

# 2. Machine-parseable counters for trending
zpool status -p

# 3. Kernel view of the storage bus: resets, aborts, timeouts
dmesg | grep -i -E "ata|sas|reset|timeout" | tail -50

# 4. SMART per device: reallocated sectors, pending sectors, UDMA_CRC
smartctl -A /dev/sdX

# 5. Which devices share a controller (PCI topology)
ls -l /dev/disk/by-path/

# 6. ECC/EDAC error logs, if the platform has ECC
dmesg | grep -i edac
grep -i edac /var/log/messages 2>/dev/null | tail -20

What to look for in the output:

CKSUM counters roughly equal across mirror members (for example, identical counts on both legs of a two-way mirror) is a widely reported indicator of corruption outside the disks, typically RAM. The diagnostic value is the shared-source pattern rather than the raw counter arithmetic; mirror read placement is load- and implementation-dependent. SMART on both devices will usually be spotless.
Errors clustered on devices behind one PCI device in by-path output points at the HBA or the cabling/backplane downstream of it, not the disks.
Non-zero UDMA_CRC_Error_Count in SMART means the drive itself detected corruption on the SATA link. That is a cable, connector, or backplane signal, not a RAM signal. Zero UDMA_CRC across all devices with rising CKSUM pushes suspicion toward RAM or the controller.
SATA link resets or SAS task aborts in dmesg correlate transport instability with the ZFS error window. READ/WRITE counters rising alongside CKSUM on the same controller’s devices reinforces a controller or cabling cause.
EDAC reports of corrected or uncorrected memory errors settle the RAM question immediately on ECC systems. Uncorrected ECC errors plus multi-device CKSUM means replace the DIMM, not the disks.

Note on counters: READ/WRITE/CKSUM counters in zpool status are cumulative since the last zpool clear, and there is a reported OpenZFS issue (#11545) where scrub-repaired checksum errors may not increment the counter. PR #11609 improved counting and was in 2.1.0, but #11545 remains open, so do not assume the behavior is identical in every release. A zero counter is not proof of no corruption; trend the counters rather than reading them once.

How to diagnose it

Confirm the pattern is multi-device. Run zpool status -v. If only one device has non-zero CKSUM and it is growing, this is a dying-disk problem, not this article. Replace the disk. If two or more unrelated devices have errors in the same window, continue.
Map the topology. Use ls -l /dev/disk/by-path/ to see which PCI device each affected disk sits behind. Errors confined to one HBA or one expander point at the controller path. Errors spanning controllers point at RAM.
Check SMART on every affected device. smartctl -A per disk. Rising Reallocated_Sector_Ct or Current_Pending_Sector on one device plus CKSUM means that device is genuinely dying. Clean SMART with zero UDMA_CRC everywhere means the disks are bystanders.
Check the kernel log for transport errors. dmesg resets, aborts, and timeouts on a shared port implicate cabling, backplane, or HBA. A completely quiet kernel log with climbing CKSUM on multiple devices implicates RAM.
Check ECC logs if available. Any EDAC activity, corrected or not, makes the DIMM a replacement candidate. Absence of EDAC data usually means non-ECC RAM, which is itself the finding: you have no way to detect memory corruption except the pattern you are already looking at.
Test memory offline. Schedule a maintenance window and boot memtest86 (or memtest86+). Run several full passes; four is a practical starting point, and marginal DIMMs can pass a single pass. Also remove any memory or CPU overclocking, including XMP profiles, before concluding the DIMMs are bad. Unstable memory clocks produce exactly this symptom on otherwise good hardware.
Rule out software before replacing everything. If errors appear only during scrubs or only after a resilver, and hardware tests clean, check your OpenZFS version against release notes for checksum-related fixes. Do not skip this step: there are documented cases of operators replacing HBA, PSU, motherboard, CPU, and RAM while the actual cause was software.
Fix the root cause, then repair the pool. After the faulty component is replaced, run a full scrub to let ZFS repair corrupted blocks from redundancy. Only after the scrub completes clean should you consider zpool clear to reset the counters. Clearing counters before the root cause is fixed destroys your trend data.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Per-device CKSUM counters, trended	The single-versus-multiple-device split is only visible over time	CKSUM rising on 2+ devices in the same window
Per-device READ/WRITE error counters	Transport failures alongside CKSUM point at controller or cabling	READ/WRITE growing on all devices behind one HBA
SMART UDMA_CRC_Error_Count	Discriminates link-layer corruption from in-memory corruption	Non-zero and climbing on multiple devices
SMART Reallocated/Pending sectors	Proves or clears individual disks	Rising on a device you suspected was a bystander
EDAC corrected/uncorrected error counts	Direct evidence of DIMM failure on ECC systems	Any uncorrected error; rising corrected counts
Kernel storage messages (resets, aborts, timeouts)	Hardware-side confirmation of bus instability	Resets correlating with CKSUM increments
Scrub repair counts and permanent error list	Measures how much damage was written while the fault was active	`zpool status -v` listing files with permanent errors

Fixes

Bad RAM

Replace the failing DIMM. If the system runs non-ECC memory and memtest86 passes but suspicion remains high (symmetric mirror errors, clean hardware everywhere else), swap the DIMMs anyway or test on known-good ECC-capable hardware. Marginal non-ECC memory can pass testers and still corrupt under production load patterns.

After replacement, run a full scrub. Blocks corrupted between the device and memory will be repaired from redundancy. Blocks corrupted in memory before checksumming cannot be detected by the scrub, because their checksums match. If the corruption window was long and the data matters, validate critical datasets against backups or replicas. Check zpool status -v for permanent errors; that list is your recovery inventory.

Going forward, use ECC RAM on any host running ZFS. Every write passes through memory before checksumming, so memory integrity is the root of the trust chain.

Failing HBA or controller

Update or reflash controller firmware to a known-good level, reseat the card, and watch counters. If errors continue, replace the HBA. Do not replace the drives behind it first: they will all test clean individually, and replacing them wastes money and triggers pointless resilvers, which are themselves risk windows. After the controller is replaced, scrub, then clear counters.

Cabling or backplane

Reseat or replace cables for affected devices, starting with any device showing UDMA_CRC. On backplane-based chassis, check for expander firmware updates and reseat the affected slots. These errors are often load- or temperature-dependent, so a quiet hour after reseating is not proof of a fix. Trend UDMA_CRC for days.

After any fix

Do not clear counters until the root cause is confirmed fixed and a full scrub has completed. zpool clear resets the evidence. If permanent errors were recorded, work through the file list from zpool status -v and restore affected files from backup before clearing. See ZFS permanent errors have been detected for that recovery process.

Prevention

Run ECC memory on ZFS hosts. The single highest-leverage prevention for this failure class. Non-ECC systems have no way to detect the corruption source that produces this exact symptom.
Alert on error counter growth, not just pool state. A pool stays ONLINE the entire time this is happening. zpool status -x will say “all pools are healthy” while RAM poisons every vdev. Any non-zero CKSUM on a production pool warrants investigation; growth on multiple devices warrants urgency.
Scrub on a schedule, weekly to biweekly for production. Multi-device CKSUM patterns are usually discovered by a scrub. Longer gaps mean longer corruption windows.
Trend, do not snapshot. Export per-device counters to a time-series system. The diagnostic value is in the pattern across devices over time, which a single zpool status cannot show.
Keep a hardware map. Knowing which drives sit behind which HBA, expander, and backplane turns a multi-hour triage into minutes. Record it before the incident.
Do not overclock memory on storage hosts. XMP and memory overclocks trade exactly the stability ZFS depends on for bandwidth a fileserver rarely needs.

How Netdata helps

Per-device error counter trends: Netdata tracks READ, WRITE, and CKSUM counters per vdev over time, which is the view needed to distinguish “one disk dying” from “everything behind one component being corrupted”.
Correlation with system-level signals: multi-device CKSUM growth can be viewed alongside SMART attributes and kernel error activity on the same node, so the RAM-versus-controller discrimination happens on one dashboard instead of across several terminals.
Scrub visibility: tracking scrub completion and repair counts closes the loop after the faulty component is replaced, confirming the pool is repaired rather than merely quiet.
Alerting on growth, not state: alerts fire when error counters increment on any device, so the pattern surfaces even while the pool reports ONLINE.
Memory and hardware error context: on ECC systems, hardware-level memory error signals on the host can be correlated with the first CKSUM increments to pinpoint when the DIMM started failing.