ZFS scrub versus resilver: why one preempts the other

You kicked off the weekly scrub, checked back an hour later, and zpool status no longer says “scrub in progress”. It says “resilver in progress”, the pool is DEGRADED, and zpool scrub -s refuses to cancel anything. Or the reverse: you replaced a disk, the resilver is running, and your monitoring keeps alerting that the pool has not scrubbed in 30 days.

This is not a bug and not two independent problems. Scrub and resilver share the same block-tree-walking engine inside ZFS, and that engine runs exactly one scan per pool at a time. When both want to run, the resilver wins, every time. What state your pool is in during and after the handoff is the difference between a controlled maintenance window and a silent gap in integrity verification.

This article covers the preemption mechanism, how to tell a legitimate preempted scrub from a pool in trouble, and how to schedule both operations so neither one quietly cancels your data integrity guarantees.

What this means

A scrub and a resilver do conceptually different jobs:

• Scrub walks every allocated block in the pool, verifies checksums, and repairs correctable errors from redundancy. It is an integrity audit of data you already have.
• Resilver walks the block tree to reconstruct data onto a replacement or faulted device. It restores redundancy the pool has already lost.

Because both are I/O-intensive and share the same scan engine, ZFS makes them mutually exclusive per pool. The priority ordering reflects risk: a pool that has lost redundancy is one device failure away from data loss, so rebuilding that redundancy outranks auditing data that is still fully redundant. A resilver preempts a running scrub, and ZFS will not let you start a new scrub while a resilver is in progress.

Two consequences operators trip over:

1. The interrupted scrub is gone, not paused. When a device faults mid-scrub, the scrub is aborted and the resilver starts. After the resilver completes, the scrub does not resume on its own. You must restart it, and it starts over from the beginning.
2. A partial scrub proves nothing about the part it did not reach. If your scrub was preempted at 60%, you have zero verified coverage for the remaining 40% since your last completed scrub.

flowchart TD
  A[Pool ONLINE - no scan] -->|zpool scrub| B[Scrub running]
  B -->|completes| A
  B -->|device faults or is replaced| C[Scrub aborted]
  C --> D[Resilver running]
  A -->|device fault or replacement| D
  D -->|completes| E[Redundancy restored]
  E -->|operator restarts scrub manually| B
  D -->|scrub start attempt| F[Refused until resilver completes]
  F --> D

Common causes

A scrub that vanished or a resilver you did not expect usually traces to one of these:

The third row deserves emphasis: a scrub can turn into a resilver without any operator action. If a disk briefly drops off the bus and returns, the event daemon can react to the vdev state change and the scan flips from scrub to resilver. Operators see the transition in zpool status and assume someone replaced hardware. Check zpool events and the kernel log before assuming human involvement.

Quick checks

All read-only and safe to run during production load:

# Current scan state: scrub, resilver, progress, ETA
zpool status | grep -A 6 "scan:"

# Full vdev tree with per-device error counters
zpool status -v

# Recent ZFS events: vdev state changes, fault events
zpool events | tail -30

# Kernel log for device drop/reset events around the transition time
dmesg -T | grep -i -E "ata|sas|reset|timeout|offline" | tail -20

# Per-vdev I/O: is the resilver making progress, and how loaded is the pool
zpool iostat -v 5

# Last completed scrub result (parse the scan line; there is no last_scrub_time property)
zpool status | grep -E "scrub repaired|scrub in progress|resilver"

Two things to note while reading output:

• The scan percentage can exceed 100% and lose its ETA on a live pool, because data changes while the scan runs. That is expected, not a hung scan.
• zpool get last_scrub_time does not exist in OpenZFS. Last scrub time comes from parsing zpool status.

How to diagnose it

Work through these in order when a scrub was preempted or a resilver appeared unexpectedly.

1. Identify which scan is running and why. Read the full scan: line in zpool status. “resilver in progress” with a DEGRADED pool means a device was replaced or faulted and redundancy is being rebuilt. Note the start timestamp and progress rate.

2. Find the triggering event. Run zpool events -v and correlate with dmesg -T. Look for a device going UNAVAIL, I/O errors, or a deliberate zpool replace in zpool history. If nothing was replaced and a device flapped, you are looking at a hardware or cabling problem, not a scheduling issue.

3. Assess the risk window. While the resilver runs, the pool has reduced redundancy. On RAIDZ1 or a 2-way mirror, one more failure in that vdev is data loss. Check error counters on the remaining devices in the affected vdev. If any are non-zero and climbing, treat this as urgent.

4. Confirm the resilver is actually progressing. Sample zpool status twice a few minutes apart. Progress should advance. A resilver that stalls or keeps restarting from the beginning points at a second failing device or heavy I/O contention, not a scheduling problem.

5. Check what happened to the scrub. A preempted scrub leaves no partial result. Look at the last completed scrub line (“scrub repaired … with N errors on ”). That date is your real integrity baseline. If it is older than your scrub policy allows, plan a scrub immediately after the resilver finishes.

6. Do not fight the mutex. Attempting zpool scrub -s during a resilver returns an error along the lines of “cannot cancel scrubbing pool: currently resilvering” because the scrub is already suspended by the resilver. Attempting zpool scrub during a resilver is refused. Wait for the resilver, then start the scrub.

Metrics and signals to monitor

Fixes

Resilver preempted your scrub: let it finish, then re-scrub

The resilver is restoring redundancy. Do not stop it to protect your scrub schedule. When it completes:

# After resilver completes, restart the integrity scan
zpool scrub <pool>

There is no built-in way to queue a scrub to start automatically when the resilver finishes. If your scrub is cron-driven, either accept that this week’s run is skipped and run one manually, or add a wrapper that checks the scan: line and starts the scrub once no resilver is active. The restarted scrub runs from the beginning, so on a large pool budget the full duration.

Scrub refused because a resilver is running

This is by design. Check how far along the resilver is and its rate. If the resilver is crawling because production I/O is saturating the pool, the fix is not to force the scrub; it is to reduce contention or adjust scan throttling. Throttle tunables are version-dependent (older releases used zfs_scrub_delay and zfs_resilver_delay; OpenZFS 2.x replaced these with different I/O scheduling parameters), so check zpool-module-parameters or the module parameter documentation for your version before changing anything. Lowering resilver priority extends the reduced-redundancy window, which is usually the wrong trade.

Resilver keeps restarting or never completes

A resilver that restarts from the beginning or loops is almost always a hardware problem: a replacement device that is also faulty, a backplane or cable issue, or a surviving device in the same vdev throwing I/O errors under scan load. Check zpool status -v error counters, dmesg, and device SMART data. Replacing the same disk twice without checking cabling and the controller is a common time sink.

Scrub schedule keeps colliding with resilvers

Move scrub windows off known resilver windows and off peak I/O. Scrubs inflate latency for everything else on the pool. If your environment replaces disks frequently (large fleets), gate the scrub cron job on zpool status showing no active resilver, and alert when a pool goes beyond your policy interval without a completed scrub rather than alerting on scrub errors alone. Zero errors with no completed scrub in months is unknown integrity, not good health.

Neither operation defragments anything

Worth stating plainly because the misconception persists: neither scrub nor resilver defragments the pool. Scrub reads and verifies; resilver reconstructs onto a replacement device. If fragmentation is your problem, neither scan will help.

Prevention

• Schedule scrubs deliberately. Pick low-load windows, and keep scrub cadence inside your policy interval (a common cadence is every 7-14 days on production pools, 30 days at the outside).
• Alert on completed scrubs, not just scrub errors. Track the date of the last completed scrub per pool. A preempted or skipped scrub should alert before the integrity gap grows.
• Treat any DEGRADED state as same-day work. Every hour in DEGRADED is an hour where the next failure is data loss, and it is also an hour where your scrub is not running.
• Fix flapping devices. A device that drops and returns causes surprise resilvers that abort scrubs and compete with production I/O. Replace cables, reseat, or replace the device rather than tolerating the flap cycle.
• Baseline resilver duration. Know how long a resilver takes on each pool so you can size the risk window and detect abnormally slow rebuilds.
• After any interrupted scrub, record it. Your last-completed-scrub date is the integrity baseline. If audits or compliance care about scrub cadence, an aborted attempt does not count.

How Netdata helps

• Pool scan state as a time series: tracking whether each pool is scrubbing, resilvering, or idle, plus progress rate, turns “the scrub disappeared” from a surprise into a visible state transition with a timestamp.
• Per-vdev error counters alongside scan state: correlating READ/WRITE/CKSUM growth with the exact moment the scan flipped from scrub to resilver identifies the device that caused it without digging through logs after the fact.
• Last completed scrub age: alerting on time-since-last-completed-scrub catches the real risk of preemption, which is an integrity verification gap, not the preemption itself.
• Latency during scan windows: per-pool I/O latency with scrub/rebuild queue wait shows the production cost of scans and justifies moving scrub windows off peak.
• ZFS event monitoring: deadman and vdev state events surfaced as alerts catch device flapping before it cascades into repeated resilvers.

Related guides

• ZFS ARC hit ratio low: cache misses, cold caches, and working sets that outgrew RAM
• ZFS zfs_arc_max: capping the ARC without starving read performance
• ZFS ARC and the OOM killer: applications killed while the cache will not shrink fast enough
• ZFS ARC shrinking below c_max: reading memory pressure before latency hits
• ZFS ARC using all memory: the Linux default that eats your RAM
• ZFS capacity planning: runway estimation before the pool fills
• ZFS checksum errors (CKSUM): the definitive signal of silent corruption
• ZFS checksum errors on multiple devices: suspect RAM or the controller, not the disks
• ZFS deadman events: hung I/O and a stalled pool sync
• ZFS device UNAVAIL or REMOVED: a disk that fell off the bus
• ZFS dirty data throttling: the write delay that masquerades as slow disks
• How ZFS actually works in production: a mental model for operators