ZFS permanent errors have been detected in the following files: recovering from data loss

You ran zpool status -v and at the bottom, under the errors section, you see:

errors: Permanent errors have been detected in the following files:

        tank/data@daily-2026-07-18:backups/db.dump
        tank/data/logs/app.log

This is the most severe per-file integrity message ZFS produces. It means one or more data blocks failed checksum validation and ZFS could not repair them from redundancy. The data in those blocks is gone. This is not a warning, not a transient condition, and not something a reboot or zpool clear will fix. Irrecoverable data loss has already occurred.

The message is also your recovery checklist. ZFS is telling you exactly which files or objects are damaged, which is more than most storage stacks will ever give you. Your job now is to quantify the loss, stop it from growing, restore what you can, and find the hardware that caused it before it takes more data with it.

What this means

Every block ZFS writes carries a checksum. On every read, and during every scrub, ZFS re-verifies the block against that checksum. When a block fails verification, ZFS tries to repair it from a redundant copy: the other side of a mirror, or RAIDZ parity. If that repair succeeds, you get a correctable error (the CKSUM counter increments, the bad copy is rewritten, and no data is lost). If there is no good copy to repair from, the error is uncorrectable and the affected file or object is added to the permanent error list you are looking at.

Key properties of this state:

• It is binary and cannot false-fire. No workload, idle period, backup job, or batch process can produce a permanent error report. ZFS attempted repair and failed. Treat it as a page-level event.
• The pool usually stays ONLINE. ZFS isolates the damage to the affected blocks. The rest of the pool keeps serving I/O, which is why this often goes unnoticed until someone reads zpool status -v or a scrub completes.
• The listed files are unrecoverable from the pool. Redundancy already failed to save them. Recovery means restoring from backups, replicas, or snapshots taken before the corruption.
• The list persists. The permanent error list in zpool status -v persists until zpool clear. Until you clear it, the entries remain as a record of what was lost.

The entry format matters for scoping the damage:

• A plain path like tank/data/logs/app.log is a live file in a mounted dataset.
• A path containing @, like tank/data@daily-2026-07-18:backups/db.dump, is a file as it exists inside a snapshot. You cannot simply rm it; you have to deal with the snapshot.
• A hex object reference such as <0x2f1d1c> means ZFS could not map the damaged block back to a path, typically because the file was deleted but the object is still referenced (for example by a snapshot or an open file handle).
• Entries of the form <metadata>:<0x...> indicate damage in pool metadata rather than file contents. These are the most serious, because metadata corruption can make entire datasets or snapshots inaccessible, and there is no file to delete and restore.

flowchart TD
  A[Block fails checksum] --> B{Redundant copy available?}
  B -->|yes| C[Repair from mirror or parity
CKSUM counter increments
no data loss]
  B -->|no| D[Uncorrectable error]
  D --> E[File or object added to
permanent error list]
  E --> F{Recovery path}
  F --> G[Restore file from backup or replica]
  F --> H[Roll back or destroy affected snapshot]
  F --> I[Accept loss and delete damaged file]

Common causes

Permanent errors are an outcome, not a root cause. The root cause is whatever destroyed the data on every redundant copy at once.

Quick checks

All read-only. Run these before changing anything.

# Full damage report: vdev states, error counters, and the permanent error list
zpool status -v <pool>

# Last scrub result and whether one is running now
zpool status <pool> | grep -A 5 "scan:"

# Same status, but with full device paths instead of shortened names
zpool status -p <pool>

# Pool-level health summary
zpool list -H -o name,health,cap,frag <pool>

# Which snapshots reference the affected datasets
zfs list -t snapshot -o name,used,refer -s used -r <pool>

# Kernel-level hardware errors that correlate with ZFS error counters
dmesg | grep -i -E "ata|sas|reset|timeout|uncorrect"

# SMART health of the devices showing non-zero error counters
smartctl -A /dev/sdX

Two notes on interpretation:

• Error counters (READ, WRITE, CKSUM) are cumulative since the last zpool clear. A device showing old errors may have already been replaced; check timestamps and history before blaming current hardware.
• There is a known OpenZFS bug (#11545) where scrub-repaired checksum errors may not increment the per-vdev CKSUM counter. Do not treat CKSUM=0 as proof that nothing was ever wrong.

How to diagnose it

Work through this in order. The goal is to answer three questions: what exactly was lost, is the loss still growing, and what hardware caused it.

1. Capture the full error list for the incident record. Save zpool status -v <pool> output to a file before you touch anything. Once you clear or repair, this evidence is gone, and you will want it for restore validation and post-incident review.
2. Classify each entry. Split the list into live files, snapshot-referenced files (paths with @), unresolvable object IDs (<0x...>), and metadata entries. The recovery action differs per class.
3. Check pool and vdev state. zpool status -v: is the pool ONLINE, DEGRADED, or FAULTED? Are error counters concentrated on one device or spread across several? One device points to that disk. Several unrelated devices with simultaneous CKSUM errors point to RAM or the controller, not the disks.
4. Check whether the loss is still growing. Re-run zpool status after some I/O or after the next scrub and compare counters. Rising counts mean an active failure; static counts mean the damage may be historical (for example from a device that already failed and was replaced).
5. Correlate with hardware telemetry. SMART data on the devices with errors, plus dmesg for link resets and timeouts. If CKSUM errors appear on multiple devices at once with no disk-level evidence, schedule a memory test: bad RAM produces bad data with valid checksums written to every disk, and ZFS faithfully stores the corruption.
6. Verify your backups before restoring. Confirm that your backup or replica actually contains a good copy of each listed file, from before the corruption window. Restoring a corrupted backup over the top just re-imports the damage.
7. Determine the detection gap. Compare the last scrub date against the likely corruption window. If scrubs were not running, the corruption sat undetected and a single later failure converted correctable damage into permanent loss. This gap is a process failure to fix, not just a hardware one.

Metrics and signals to monitor

Fixes

There is no repair for the damaged blocks themselves. Every action below is about restoring data, stopping the bleed, and clearing the record.

Restore the affected files from backup

This is the primary recovery path. For each live file in the list, restore a known-good copy from backup, replica, or a snapshot that predates the corruption. After restoring, verify the file (application-level check, hash comparison against the backup source, or a test restore for databases). Do not assume a restore succeeded because the command exited 0.

Handle snapshot-referenced corruption

You cannot delete a file out of a snapshot. If the damaged block only exists inside snapshots, your options are:

• Roll back the dataset to a good snapshot (zfs rollback), accepting the loss of everything written since. Disruptive; confirm the target snapshot is itself clean first.
• Destroy the affected snapshots if they are expendable. Note that a block shared across many snapshots produces many list entries, and every snapshot referencing it must go before the entry can clear.
• Leave read-only snapshots in place if they are retention-critical and the damaged content is tolerable, and document the decision.

Handle unresolvable object IDs and metadata entries

For <0x...> entries with no path, the file is already deleted from the live filesystem. The reference is usually held by a snapshot or an open handle; destroying the referencing snapshot or closing the handle releases it. For <metadata> entries, there is no user-facing file to fix. If the pool still imports and serves I/O, plan a migration: zfs send | zfs recv the healthy datasets to a new pool, then retire the damaged one. Metadata corruption does not heal.

Replace the failing hardware

Do this before or alongside data restoration, not after. If one device shows concentrated errors, replace it (zpool replace) and let the resilver complete. If errors point to RAM or the controller, fix that first; replacing disks against a corrupting memory path accomplishes nothing. Until the root cause is removed, every restored file is at risk of being corrupted again.

Clear the error record

Once the data is restored or the loss is accepted, and the hardware is fixed:

# Reset device error counters after the root cause is fixed
zpool clear <pool>

# Verify integrity end to end and confirm no new damage
zpool scrub <pool>

Do not run zpool clear while counters are still climbing. It resets the odometer while the failure is in progress and destroys your ability to quantify the problem. The scrub after the clear is not optional: it is the only way to confirm the pool is clean going forward. On OpenZFS 2.2.0 and later, zpool scrub -e scrubs only the blocks in the error log, which is much faster than a full scrub for validating the previously damaged regions.

Prevention

• Run scrubs on a schedule and alert on the result. Production pools should complete a scrub every 7 to 14 days. Alerting on “scrub completed with errors” is not enough; also alert on “no scrub completed in 30 days.” Zero errors without a recent scrub means zero errors detected, not zero errors present.
• Treat correctable errors as hardware tickets. Every repaired CKSUM error is a device telling you it is failing. Replacing that device in business hours is what prevents the next scrub from reporting permanent errors.
• Respond to DEGRADED same-day. A DEGRADED pool is one failure away from exactly this article. Permanent errors are what DEGRADED turns into when you wait.
• Never run production data without redundancy. On a single-disk stripe, ZFS can detect corruption but can never repair it. Every checksum error is data loss by definition.
• Use ECC memory. Non-ECC RAM can hand ZFS corrupted data with a valid checksum, which ZFS then writes to every redundant copy faithfully.
• Keep tested, restorable backups. ZFS redundancy protects against device failure, not against corruption that reaches all copies. The permanent error list is the moment you find out whether your backups actually work.

How Netdata helps

• Scrub and error-state visibility: Netdata surfaces pool health state, per-vdev READ/WRITE/CKSUM counters, and scrub status continuously, so a permanent error condition shows up on a dashboard and in alerts minutes after a scrub or read detects it, not weeks later when someone happens to run zpool status -v.
• Correctable versus uncorrectable trend lines: Historical CKSUM counter graphs show whether you are in the “redundancy is absorbing damage” phase or the “damage is now permanent” phase, which is the difference between a hardware ticket and a data-loss incident.
• Hardware correlation: ZFS error counters charted alongside disk SMART attributes and system-level I/O errors let you tie a specific failing device to the corruption window, instead of guessing which of twelve disks to replace.
• Scrub recency as an alertable signal: Alerting when a pool has not completed a scrub within your policy window closes the detection gap that turns correctable errors into permanent ones.
• DEGRADED state paging: Pool state transitions out of ONLINE page immediately, giving you the chance to restore redundancy before the next error becomes unrecoverable.

Related guides

• How ZFS actually works in production: a mental model for operators
• ZFS monitoring checklist: the signals every production pool needs
• ZFS monitoring maturity model: from survival to expert
• ZFS pool DEGRADED: redundancy lost and one failure from data loss
• ZFS pool FAULTED: when the pool can no longer serve I/O
• ZFS pool I/O is currently suspended: a hung pool and blocked I/O
• ZFS READ and WRITE errors: transport-level device failures in zpool status
• ZFS device FAULTED - too many errors: a disk ejected from the pool