ZFS Operations Guides on Netdata

How ZFS actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

Every ZFS incident traces back to a small set of internal mechanisms. A write freeze at 3 a.m. is the transaction group pipeline stalling. A server that “runs out of memory” with gigabytes free is the ARC doing exactly what it was designed to do. A pool that was fine at 78% full and unusable at 88% is the metaslab allocator crossing a threshold, not a disk dying.

This article is the mental model layer: what ZFS is doing internally at all times, which resources it competes for, and why its characteristic failures look the way they do. It is the prerequisite for the operational runbooks. You cannot interpret a TXG sync time, an ARC hit ratio, or a fragmentation percentage without knowing the machinery that produces those numbers.

ZFS ARC and the OOM killer: applications killed while the cache will not shrink fast enough

Mon, 01 Jan 0001 00:00:00 +0000

Your database or application process just got OOM-killed on a ZFS host that “had plenty of memory.” dmesg shows the OOM killer invoked and a victim chosen, while the ARC was holding tens of gigabytes at the time. The machine was not out of memory. It was out of memory the kernel could get back fast enough.

The ARC is technically reclaimable, but it does not release memory instantly under a sudden allocation spike. The arc_prune shrinker callback has inherent latency. When a large allocation lands faster than the ARC can drain, direct reclaim gives up and the OOM killer fires, and it frequently picks the wrong victim, because the process that triggered the allocation is not necessarily the one the badness score ranks highest.

ZFS ARC hit ratio low: cache misses, cold caches, and working sets that outgrew RAM

Mon, 01 Jan 0001 00:00:00 +0000

Your monitoring says the ZFS ARC hit ratio dropped, or reads got slow and someone traced it to cache misses. The number itself tells you almost nothing until you know three things: which hit ratio you are looking at (demand vs prefetch), what the workload’s baseline is, and whether the ARC is actually at its intended size.

A low ARC hit ratio is one of the most misread signals in ZFS operations. Near-zero hit ratio after a reboot is normal. A 30% demand hit ratio on a database doing random reads over a dataset larger than RAM can be perfectly healthy. A file server that has run at 96% for months dropping to 78% is an incident. The absolute value is meaningless; the deviation from baseline is the signal.

ZFS ARC shrinking below c_max: reading memory pressure before latency hits

Mon, 01 Jan 0001 00:00:00 +0000

You have a ZFS box that “got slow” and nobody can say why. The disks are fine, the pool is ONLINE, no scrub is running, and zpool iostat shows nothing obviously wrong. Then you look at /proc/spl/kstat/zfs/arcstats and the ARC is sitting at a fraction of its configured maximum, the target size c keeps ratcheting downward, and arc_no_grow is 1.

This is not an ARC problem. The kernel is reclaiming ARC memory because something else wants it. It is the leading edge of the memory-pressure cascade: a smaller ARC means more reads go to disk, more in-flight I/O means more kernel buffers, more buffers mean more memory pressure, and the ARC shrinks further. By the time application read latency spikes or the OOM killer fires, you are already deep in the feedback loop.

ZFS ARC using all memory: the Linux default that eats your RAM

Mon, 01 Jan 0001 00:00:00 +0000

You run free -h on a ZFS host and see 58 of 64 GiB “used”, with almost nothing in buffers/cache. Your application processes account for maybe 8 GiB. Something is eating the machine, and you start hunting for a leak.

There is no leak. The missing memory is the ZFS ARC (Adaptive Replacement Cache), ZFS’s primary read cache in kernel memory. On Linux the ARC lives outside the kernel page cache, so free and top report it as used slab, not as reclaimable cache. Operators who do not know this tune swappiness, add swap, or kill innocent processes to “free” memory that was never in danger.

ZFS cannot destroy dataset is busy: clones, holds, and mounted filesystems

Mon, 01 Jan 0001 00:00:00 +0000

You run zfs destroy on a dataset or snapshot and get one of two errors back:

cannot destroy 'tank/data': dataset is busy

or:

cannot destroy 'tank/data@daily-2026-07-20': snapshot has dependent clones

Both mean the same thing at a mechanical level: something still references the object you are trying to remove, and ZFS refuses to guess which reference you are willing to lose. The frustrating part is that the error tells you almost nothing about what holds the reference. The job here is to find the dependent, understand why it exists, and remove it in the right order.

ZFS cannot import pool: missing devices, a damaged label, and root-pool boot failure

Mon, 01 Jan 0001 00:00:00 +0000

zpool import fails and the pool refuses to come online. The error is one of a small set: cannot import 'tank': no such pool available, one or more devices is currently unavailable, cannot import 'tank': insufficient replicas, or a complaint about an unsupported version or feature. Each message points at a different stage of the import process, and each has a different recovery path.

For a data pool this is an availability incident. For a root-on-ZFS system it is worse: the import happens inside initramfs, and a failure there means the machine does not boot at all. The initramfs message cannot import rpool: no such pool available is the same class of problem with a much higher cost.

ZFS capacity cliff: why the pool falls off a performance edge near 80-90% full

Mon, 01 Jan 0001 00:00:00 +0000

Your ZFS pool was fine last month. This week write latency is spiking, TXG syncs are stretching past their timeout, and applications are stalling on writes while reads still feel snappy. You check zpool list and see CAP at 87%. Nothing failed. No disk died. The pool just crossed a threshold it was never going to tell you about.

This is the ZFS capacity cliff, and it is the single most common ZFS incident in production. It is not a linear slowdown where you lose a few percent of throughput per percent of capacity. It is an algorithmic cliff: performance looks acceptable until it suddenly is not, and the distance between “fine” and “writes stalling for seconds” can be a few percentage points of pool capacity.

ZFS capacity planning: runway estimation before the pool fills

Mon, 01 Jan 0001 00:00:00 +0000

ZFS does not fail gracefully at full. Long before ENOSPC, the metaslab allocator starts working harder to find free space, write latency climbs, TXG syncs stretch, and the pool slides into the capacity-fragmentation cliff described in how ZFS actually works in production. By the time applications see errors, you are already in emergency territory, and the recovery options (destroying snapshots under I/O pressure, expanding a pool mid-incident) are the worst versions of themselves.

ZFS checksum errors (CKSUM): the definitive signal of silent corruption

Mon, 01 Jan 0001 00:00:00 +0000

A non-zero number in the CKSUM column of zpool status means a block read from that device did not match the checksum ZFS stored for it. The data came back wrong, and ZFS can prove it. This is the only signal in your storage stack that definitively says “silent corruption happened here”, and zero is the only acceptable value in production.

What happens next depends on redundancy. On a mirror or RAIDZ vdev, ZFS reads the good copy or reconstructs from parity, repairs the bad block in place, and serves correct data to the application. The pool stays ONLINE, nothing page-fires, and the only evidence is the incrementing counter. On a single-disk vdev with no redundancy, the same counter means unrecoverable corruption: ZFS cannot repair what it has no second copy of.

ZFS checksum errors on multiple devices: suspect RAM or the controller, not the disks

Mon, 01 Jan 0001 00:00:00 +0000

zpool status shows non-zero CKSUM counters, and not on one disk. Two, four, or every device in the vdev has them. The instinct is to start RMA-ing drives. Stop. Independent disks do not fail in the same way at the same time. When checksum errors appear on multiple unrelated devices simultaneously, the failure is almost always upstream of the disks: something is corrupting data between the application and the platters, and every disk downstream of it is faithfully recording the damage.

ZFS deadman events: hung I/O and a stalled pool sync

Mon, 01 Jan 0001 00:00:00 +0000

A ZFS deadman event means the kernel has watched an I/O operation sit incomplete for at least five minutes, or a pool sync sit incomplete for at least ten minutes, and has given up waiting quietly. The event shows up as FM_EREPORT_ZFS_DEADMAN in zpool events, and it is one of the few ZFS signals that justifies waking someone up immediately.

The deadman is not a performance heuristic. It does not fire because latency is high or throughput is low. It fires because a specific operation has made zero progress for a span of time that no healthy storage stack ever produces. A SATA disk completing a read in five minutes is not slow. It is stuck.

ZFS dedup memory exhaustion: when the DDT outgrows ARC and the pool crawls

Mon, 01 Jan 0001 00:00:00 +0000

The pool is ONLINE. zpool status -x says everything is healthy. Disk latencies look mediocre but not dead. Yet every write takes tens to hundreds of milliseconds, reads that used to come from cache now hit disk, and application latency is uniformly awful in both directions. Nothing is DEGRADED, no scrub is running, and capacity looks unremarkable.

If dedup=on is set anywhere on that pool, this is the signature of one of the most expensive mistakes in ZFS operations: the deduplication table (DDT) has outgrown the ARC, and the pool is doing a random disk read for nearly every write just to answer the question “have I seen this block before?”

ZFS deleted files but no space freed: snapshots holding the blocks

Mon, 01 Jan 0001 00:00:00 +0000

You deleted 100 GB of files. df shows the same free space as before. zpool list has not moved. Nothing is broken: ZFS is copy-on-write, and a snapshot taken before the delete still references every block you just removed. Until that snapshot is destroyed, the blocks stay allocated and the pool gains nothing.

This becomes an incident when the pool is already past 90%: the operator deletes files, gets nothing back, panics, and starts destroying snapshots at random. Mass snapshot destruction on a nearly-full pool triggers heavy asynchronous block freeing that competes for the same I/O bandwidth the pool is already short on. That is how a capacity annoyance turns into a write stall.

ZFS device FAULTED - too many errors: a disk ejected from the pool

Mon, 01 Jan 0001 00:00:00 +0000

You ran zpool status and found a device FAULTED with “too many errors”, and the pool has dropped to DEGRADED. ZFS did this on purpose: the device’s READ, WRITE, or CKSUM error counters crossed a threshold, and ZFS took the disk out of service to stop it from corrupting or stalling the pool further.

FAULTED is a verdict, not a glitch. ZFS only faults a device after repeated I/O or checksum failures, so the counters you see are the tail end of a problem that has been building. Your job is to find out whether the disk is dying or something between ZFS and the disk (cable, backplane, controller, power) is at fault, then replace or repair before the redundancy you have left disappears.

ZFS device UNAVAIL or REMOVED: a disk that fell off the bus

Mon, 01 Jan 0001 00:00:00 +0000

You ran zpool status and one of your devices is no longer ONLINE. It shows UNAVAIL or REMOVED, and the pool has flipped to DEGRADED or worse. This is not a ZFS software problem. Something between the kernel and the disk broke: the drive was pulled, the cable or backplane dropped it, the controller lost it, or a hot-plug event went badly.

The good news is that this failure mode is loud and binary. ZFS did not corrupt anything by itself. If the pool is DEGRADED, redundancy is still serving every read and write. Your job is to find out why the device disappeared, decide whether it is coming back, and either re-online it or replace it before a second device in the same vdev fails.

ZFS dirty data throttling: the write delay that masquerades as slow disks

Mon, 01 Jan 0001 00:00:00 +0000

Your applications report write latency spiking from microseconds to tens or hundreds of milliseconds. Throughput falls off a cliff under sustained write load. iostat shows the disks are not busy. SMART is clean. The NVMe drives benchmark fine. Everything points at the storage, and nothing is wrong with the storage.

This is the most misdiagnosed latency source in ZFS: the dirty data write throttle. When dirty (uncommitted) data in RAM crosses a threshold, ZFS deliberately injects artificial delay into every write syscall to slow writers down. If dirty data reaches the hard limit, writes stall completely until the syncing transaction group finishes. The system is working exactly as designed: it protects the pool from memory exhaustion by making applications wait.

ZFS encryption key not loaded: cannot mount an encrypted dataset

Mon, 01 Jan 0001 00:00:00 +0000

You rebooted a host, imported a pool, or tried to mount a dataset, and ZFS refused with a variation of cannot mount 'tank/secure': encryption key not loaded. The pool itself is ONLINE, zpool status is clean, and the data is intact. The dataset is simply locked: until the encryption key is loaded into the kernel, ZFS cannot decrypt the dataset’s metadata well enough to mount it.

This is one of the more benign ZFS errors, but it causes real outages because it shows up at the worst times: after an unplanned reboot, during a failover to a standby host, or after a zfs receive on a backup server nobody has logged into since. The fix is one command. Making sure you never have to type it at 3 a.m. takes a bit more work.

ZFS I/O queue depth: telling backend saturation apart from a hang

Mon, 01 Jan 0001 00:00:00 +0000

The pool is slow. Applications are timing out on writes, or reads that used to take milliseconds now take seconds. The first question that decides everything else is this: is the storage backend working hard and falling behind, or has something actually stopped moving? Both look identical from the application side. Both show up as “high latency.” The fix for one (more IOPS, better devices, workload shaping) is completely different from the fix for the other (a dead disk, a stuck controller, a hung I/O that needs intervention).

ZFS L2ARC ineffective: a cache that burns SSD endurance for nothing

Mon, 01 Jan 0001 00:00:00 +0000

You added an SSD as an L2ARC device expecting faster reads. Months later, read latency has not moved, the SSD’s wear indicator is climbing, and the ARC is smaller than it should be. The L2ARC is being written to constantly and read from almost never. You are paying for the cache in RAM and drive endurance and getting nothing back.

This is one of the most common ZFS mistakes: L2ARC is assumed to be beneficial by default, and almost nobody validates it after adding it. Many workloads get zero benefit from it, and for those workloads the device is a net negative: it consumes ARC memory for its index headers and burns SSD write endurance caching blocks that are never read again.

ZFS monitoring checklist: the signals every production pool needs

Mon, 01 Jan 0001 00:00:00 +0000

Most ZFS incidents are gaps in the basics: a pool DEGRADED for three weeks because nobody paged on it, a pool at 94% capacity discovered when writes started stalling, a disk accumulating checksum errors that a scrub would have caught months earlier. ZFS tells you almost everything you need to know, but only if you collect the right signals continuously instead of running zpool status by hand after something breaks.

ZFS monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Most ZFS incidents are not monitoring failures. They are monitoring-coverage failures. The pool was watched, but at the wrong level: the team had capacity graphs and no scrub result alerting, or pool state alerts and no TXG sync visibility, and the failure mode that actually fired lived exactly in the gap.

This article is a reference model with four levels: Survival, Operational, Mature, and Expert. Each level adds signals that catch failure classes the previous level structurally cannot see. Use it two ways: as an audit of what you monitor today, and as a roadmap for what to add next. The levels are cumulative. Skipping a level buys you alert noise, not insight.

ZFS No space left on device: ENOSPC, the slop reserve, and the pool you cannot delete from

Mon, 01 Jan 0001 00:00:00 +0000

Your application just failed with No space left on device. df shows free space. zpool list shows FREE above zero. You try to delete files to make room, and rm fails with the same error: rm: cannot remove 'file': No space left on device. The pool is in the worst state a ZFS pool can be in: full enough that even freeing space requires space you do not have.

This is not a bug and not a lying tool. It is the collision of two ZFS design decisions: the copy-on-write transactional model, and the slop reserve that exists specifically to keep a full pool administrable. Knowing which of the two you are hitting is the difference between a controlled recovery and a pool that stays wedged until you migrate data off it.

ZFS one slow disk in a vdev: the dying drive that drags the whole pool

Mon, 01 Jan 0001 00:00:00 +0000

The pool is ONLINE. zpool status -x says all pools are healthy. Error counters are zero or near zero. And yet write latency has doubled, TXG syncs are stretching, and applications are complaining about storage. When one disk in a vdev starts dying, it usually does not fail cleanly. It develops bad sectors, and the drive firmware starts retrying reads and remapping blocks internally. Each retry adds milliseconds to individual I/Os. The disk never errors out, so ZFS never faults it.

ZFS periodic write latency spikes: the TXG sync storm pattern

Mon, 01 Jan 0001 00:00:00 +0000

Your applications write fast for a few seconds, then every write stalls for 10 to 60 seconds, then everything is fast again. The stalls recur on a rough cycle, and between them the system looks completely healthy. Reads are fine. The pool is ONLINE. zpool status -x says all pools are healthy. Disk-level tools show the devices mostly idle, except for periodic bursts of intense write activity.

This is the TXG sync storm pattern, and the periodicity is the fingerprint. Write latency is bimodal: fast or blocked, with the blocked periods recurring roughly every zfs_txg_timeout seconds (default 5). It is distinct from steady-state slowness, which would indicate a simple throughput bottleneck. If your latency graph looks like a picket fence rather than a plateau, the write pipeline is backing up, not uniformly slow disks.

ZFS permanent errors have been detected in the following files: recovering from data loss

Mon, 01 Jan 0001 00:00:00 +0000

You ran zpool status -v and at the bottom, under the errors section, you see:

errors: Permanent errors have been detected in the following files:

 tank/data@daily-2026-07-18:backups/db.dump
 tank/data/logs/app.log

This is the most severe per-file integrity message ZFS produces. It means one or more data blocks failed checksum validation and ZFS could not repair them from redundancy. The data in those blocks is gone. This is not a warning, not a transient condition, and not something a reboot or zpool clear will fix. Irrecoverable data loss has already occurred.

ZFS pool DEGRADED: redundancy lost and one failure from data loss

Mon, 01 Jan 0001 00:00:00 +0000

Your monitoring fired, or a routine zpool status run shows the pool in DEGRADED state. One or more devices have failed or gone unavailable, but the pool is still serving I/O because a mirror partner or RAIDZ parity is covering the gap. Applications see no errors.

That is what makes DEGRADED dangerous. The pool is stable in this state and will run there indefinitely, so teams sit on it. But redundancy in the affected vdev group is gone or reduced: on a two-way mirror or RAIDZ1, the next failure in that group is unrecoverable data loss. On RAIDZ2 you have one fault of margin left, not two.

ZFS pool FAULTED: when the pool can no longer serve I/O

Mon, 01 Jan 0001 00:00:00 +0000

zpool status shows the pool state as FAULTED, the vdev tree shows devices FAULTED or UNAVAIL, and the status message says “insufficient replicas for the pool to continue functioning.” Applications cannot read or write anything on the pool. Unlike DEGRADED, where redundancy is still covering for a failed device, FAULTED means ZFS has lost more devices than the vdev topology can tolerate, or cannot open enough devices to guarantee data integrity.

ZFS pool fragmentation high: write amplification you cannot defragment away

Mon, 01 Jan 0001 00:00:00 +0000

You ran zpool list and the FRAG column on your production pool reads 55%, 65%, maybe higher. Writes feel slower than they used to, scrub takes longer every month, and nobody can say when it started. Now you are searching for the ZFS equivalent of defrag and discovering there is not one.

The FRAG percentage measures how scattered your pool’s free space is across metaslab spacemaps. It is not file fragmentation, and it says nothing about how fragmented already-written data is. It tells you how hard the allocator will have to work for every future write. On a pool with high fragmentation, logically sequential writes land in scattered physical locations, turning sequential workloads into random I/O at the device layer.

ZFS pool history: the forensic log almost nobody watches

Mon, 01 Jan 0001 00:00:00 +0000

Every ZFS pool keeps a built-in audit log. Every property change, every snapshot creation and destruction, every import, export, key operation, and delegation change is recorded by zpool history, with timestamps, and with the user and hostname if you ask for them. Almost nobody monitors it.

This is a problem in two directions. Forensically, when something goes wrong (a dataset destroyed at 3 a.m., sync suddenly disabled on a database dataset, a pool exported that nobody admits to exporting) the answers were in the history all along, often already rotated out by the time anyone looks. On the security side, mass snapshot destruction is exactly what ransomware or an attacker covering tracks looks like, and an unauthorised sync=disabled or delegation change is a privilege escalation signal. Both are visible in one place, and that place is not wired into anything by default.

ZFS pool I/O is currently suspended: a hung pool and blocked I/O

Mon, 01 Jan 0001 00:00:00 +0000

You ran a zpool command, or an application tried to touch a dataset, and you got this back:

cannot open 'tank': pool I/O is currently suspended

Or zpool status shows the pool in state SUSPENDED with an action line telling you to reconnect devices and run zpool clear. Every process touching the pool is stuck in uninterruptible sleep. New SSH sessions that touch the mountpoint hang. The system itself may still be responsive, but everything that depends on the pool is frozen.

ZFS pool I/O latency high: reading zpool iostat -l before blaming the disks

Mon, 01 Jan 0001 00:00:00 +0000

Applications are stalling on storage calls, zpool iostat -l shows ugly latency numbers, and someone has already said “the disks are dying.” Before you open a hardware ticket, look at which latency column is actually elevated. ZFS reports total wait, disk wait, and queue wait separately, and the split between them tells you whether the problem is the devices or something inside ZFS itself.

The most common misdiagnosis is treating high total_wait as a disk problem. total_wait includes queueing time inside ZFS: ZIL commits, TXG sync pressure, dirty data throttling, fragmentation-driven allocation cost, and competition from scrub or resilver. If disk_wait is low while total_wait is high, your disks are fine and the bottleneck is above them.

ZFS pool ONLINE with non-zero errors: why zpool status -x lies

Mon, 01 Jan 0001 00:00:00 +0000

Your monitoring script runs zpool status -x, gets back “all pools are healthy”, and moves on. Meanwhile, one disk in a mirror has 4,000 checksum errors, the pool is silently correcting every bad block from the good side, and the failing disk is weeks from dropping off the bus. You find out when the second disk in the vdev fails and the resilver uncovers corruption it can no longer reconstruct.

ZFS pool was previously in use from another system: MMP and multihost protection

Mon, 01 Jan 0001 00:00:00 +0000

You run zpool import tank and get:

cannot import 'tank': pool was previously in use from another system.
Last accessed by host2 (hostid=0x1a2b3c4d) at Wed Jul 22 03:12:44 2026
The pool can be imported, use 'zpool import -f' to import the pool.

Or, on a pool with multihost protection enabled, the harder version:

cannot import 'tank': pool is imported on host2 (hostid: 0x1a2b3c4d)

Both messages exist to stop one specific disaster: two hosts importing the same pool at the same time. ZFS has no distributed locking. If two systems write to the same vdevs concurrently, they allocate the same free blocks, overwrite each other’s metadata, and destroy the pool. Your job is to determine whether the fence is protecting you from a live peer, or blocking you on stale state.

ZFS READ and WRITE errors: transport-level device failures in zpool status

Mon, 01 Jan 0001 00:00:00 +0000

You ran zpool status and the READ or WRITE column on one or more devices is not zero. The pool may still show ONLINE. No application has complained yet. This is where ZFS lulls operators into inaction: redundancy is absorbing the failures, so nothing is visibly broken, but the counters are telling you a device, cable, controller, or power path is misbehaving.

The READ and WRITE columns count transport-level I/O failures: operations where the underlying device did not respond or returned an error status. The CKSUM column is different: it counts blocks where the device returned data but the data did not match the stored checksum. READ and WRITE errors mean the I/O itself failed. CKSUM errors mean the I/O succeeded and the payload was wrong. The distinction matters because it points at different hardware.

ZFS replacing a failed disk: zpool replace, autoreplace, and hot spares

Mon, 01 Jan 0001 00:00:00 +0000

A disk in your pool has failed or is failing. The pool shows DEGRADED in zpool status, which means redundancy is gone in that vdev group and the next failure in the same group is data loss. The pool keeps serving I/O, but you are on the clock.

This runbook covers getting a replacement disk in and resilvered: identifying the physical device correctly, choosing between manual zpool replace, autoreplace, and hot spares, what sequential resilver changes, and the extra steps a root pool needs before the new disk can boot the machine.

ZFS resilver in progress: the reduced-redundancy window after a disk replace

Mon, 01 Jan 0001 00:00:00 +0000

You replaced a failed disk, zpool status now shows scan: resilver in progress, and the pool is DEGRADED. This is normal recovery, but it is not a safe state. Until the resilver completes, the affected vdev is running with reduced redundancy, and every hour in this window is an hour where one more failure can take the pool down.

The playbook rule: a resilver in progress is a TICKET, not a page, but it is a ticket you actively manage. You do not close it because the replacement is physically seated. You close it when the scan finishes, the vdev is back at full redundancy, and the error counters on the surviving disks are still zero.

ZFS resilver slow or stalled: multi-day rebuilds and the second-failure race

Mon, 01 Jan 0001 00:00:00 +0000

A disk was replaced, zpool status shows resilver in progress, and the ETA says three days. Or worse: the scanned byte count has not moved in twenty minutes. Either way, the pool is running with reduced redundancy, and every hour the resilver takes is an hour where the next disk failure becomes a data-loss event.

Two facts frame everything below. First, ZFS deliberately throttles resilver I/O so production traffic wins. A slow resilver is often the system working as designed; the throttle-vs-risk trade-off is a decision you make, not an accident. Second, on large RAIDZ pools of spinning disks, a multi-day resilver is normal arithmetic, and a resilver that is decelerating frequently means a second device in the same vdev is also failing. That second case is the one that kills pools.

ZFS scrub not running: 'CKSUM 0' means nothing without regular scrubs

Mon, 01 Jan 0001 00:00:00 +0000

The pool looks healthy. zpool status shows every device ONLINE, and the READ, WRITE, and CKSUM columns are all zero. No scrub errors have ever been reported. So the data is safe, right?

Not necessarily. Zero checksum errors means zero errors detected, not zero corruption. ZFS only discovers silent corruption when it reads a block and verifies its checksum, and most blocks on a typical pool are read rarely or never. The scrub is the only mechanism that systematically reads every allocated block and checks it. A pool that has not scrubbed in six months has unknown integrity.

ZFS scrub repaired errors: correctable rot versus permanent data loss

Mon, 01 Jan 0001 00:00:00 +0000

A scrub finishes and the scan: line in zpool status reads something like scrub repaired 8.09M in 04:12:33 with 0 errors on Sun Jul 20 04:12:34 2026. Two numbers in that line decide your next 24 hours: how much data ZFS had to fix, and whether anything was lost for good. Operators misread this line in both directions: panicking over a large repaired count that cost zero data, or shrugging at a small error count that means files are permanently damaged.

ZFS scrub versus resilver: why one preempts the other

Mon, 01 Jan 0001 00:00:00 +0000

You kicked off the weekly scrub, checked back an hour later, and zpool status no longer says “scrub in progress”. It says “resilver in progress”, the pool is DEGRADED, and zpool scrub -s refuses to cancel anything. Or the reverse: you replaced a disk, the resilver is running, and your monitoring keeps alerting that the pool has not scrubbed in 30 days.

This is not a bug and not two independent problems. Scrub and resilver share the same block-tree-walking engine inside ZFS, and that engine runs exactly one scan per pool at a time. When both want to run, the resilver wins, every time. What state your pool is in during and after the handoff is the difference between a controlled maintenance window and a silent gap in integrity verification.

ZFS silent data corruption: how bit rot happens and why scrubs catch it

Mon, 01 Jan 0001 00:00:00 +0000

A disk can return the wrong bytes without returning an error. The drive reports success, the controller reports success, and the application gets corrupt data. Conventional filesystems have no way to notice: they trust the storage stack, and at scale the storage stack lies often enough that corruption is a when, not an if.

ZFS is designed around the assumption that every layer below it will eventually return wrong data. Every block carries a checksum, and every read verifies it. That is why ZFS operators see corruption events that ext4 or XFS operators never see: not because ZFS systems corrupt more, but because ZFS is the only layer looking.

ZFS SLOG device failed: sync latency 100x worse while the pool stays ONLINE

Mon, 01 Jan 0001 00:00:00 +0000

NFS clients are timing out. Database commit latency has jumped from microseconds to tens or hundreds of milliseconds. zpool status -x says “all pools are healthy.” Reads are fine, capacity is fine, scrubs are clean.

This is the SLOG failure trap. A SLOG (separate intent log) failure does not degrade pool redundancy, so the pool state stays ONLINE. But the ZIL has silently fallen back to writing on the main pool vdevs, and every synchronous write now pays the full latency of your data disks instead of the fast log device. For sync-heavy workloads, that is a 10x to 100x latency regression with zero change in pool health state.

ZFS SLOG endurance: the SSD that wears out from concentrated sync writes

Mon, 01 Jan 0001 00:00:00 +0000

A SLOG (Separate Intent Log) device is usually the smallest, fastest SSD in the box, and it is almost always the first one to die. Every synchronous write in the pool, from every dataset and every application, lands on this one device before the application gets its acknowledgment. That concentration is the point of a SLOG, and it is also why the device burns through write endurance far faster than the data vdevs around it.

ZFS slow pool import: space-map loading, ZIL replay, and import hangs

Mon, 01 Jan 0001 00:00:00 +0000

You ran zpool import tank (or the system is booting root-on-ZFS) and nothing has happened for ten minutes. No output, no prompt, no error. The instinct is that the import has hung and to reach for a reboot or a forced import. Both are usually wrong.

A slow import is usually ZFS replaying state it must replay before the pool is safe to use: loading space maps for metaslabs, replaying the ZIL after an unclean shutdown, and working through pending async destroy work left by deleted datasets or snapshots. On a large, fragmented, heavily snapshotted pool this can legitimately take minutes to, in extreme cases, an hour or more.

ZFS snapshot destroy slow: async destroy, the freeing property, and I/O contention

Mon, 01 Jan 0001 00:00:00 +0000

You ran zfs destroy on a large snapshot or a dataset full of snapshots. The command returned in seconds, but zpool list shows the space did not come back, write latency is climbing, and applications are starting to complain. Or the destroy itself hung, and now every zfs and zpool command against that pool is stuck in D state.

Both symptoms have the same root: snapshot destruction in ZFS is not a single operation. The command deletes the snapshot’s metadata quickly, but the actual block freeing runs asynchronously in the background, competing with production I/O for disk bandwidth, CPU, and dirty-data budget. On a busy or near-full pool, that background work can take hours and slows everything else while it runs.

ZFS snapshot space consumption: the 40%-of-the-pool blindspot

Mon, 01 Jan 0001 00:00:00 +0000

A ZFS pool reports 90% capacity. The operator deletes 500 GB of old files, and the pool does not get any emptier. This is one of the most common ZFS incidents, and it is not a bug. It is copy-on-write working exactly as designed.

ZFS never overwrites blocks in place, so a snapshot retains references to every block that existed when it was taken. When you delete a file in the live dataset, the block is freed only if no snapshot still references it. If snapshots exist, the space stays allocated until the last referencing snapshot is destroyed. Teams routinely discover, mid-incident, that snapshots are holding 40% or more of the pool and that the default tooling never showed them.

ZFS special vdev full: the hidden bottleneck while the pool looks empty

Mon, 01 Jan 0001 00:00:00 +0000

The pool is at 55% capacity, fragmentation is moderate, and zpool status -x says all pools are healthy. Yet anything touching lots of small files or metadata has gone from fast to miserable, and no pool-level dashboard explains why.

On pools with a special allocation class, the cause is often the special vdev itself: the (hopefully mirrored) fast SSDs that hold all pool metadata and, where special_small_blocks is set, small data blocks. When the special vdev fills, ZFS raises no alert. New metadata and small blocks silently land on the main pool vdevs, so metadata I/O now runs at bulk-storage latency while every pool-level capacity signal still shows headroom. The pool is ONLINE, capacity looks fine, and the bottleneck sits one level down in the vdev tree where most monitoring never looks.

ZFS synchronous write latency high: fsync, NFS, and database commits stalling

Mon, 01 Jan 0001 00:00:00 +0000

Your database commit latency just jumped from 2ms to 40ms. Or NFS clients are reporting sluggish writes while the pool itself looks healthy: zpool status -x says all pools are healthy, read latency is fine, and throughput has not collapsed. Applications that call fsync(), open files with O_SYNC, or run over NFS are stalling, while everything else seems normal.

This is the ZFS synchronous write latency problem. The bottleneck is almost never the data path. It is the ZIL (ZFS Intent Log) path: the mechanism ZFS uses to guarantee that synchronous writes survive a crash. When the ZIL backs up, every fsync, every NFS COMMIT, and every database transaction commit stalls behind it.

ZFS TXG sync time high: the most diagnostic write-path signal operators ignore

Mon, 01 Jan 0001 00:00:00 +0000

Applications are stalling on writes. zpool status -x says all pools are healthy. Disk latency looks mostly fine, or at least inconsistent with the severity of the application impact. The signal that explains what is actually happening sits in one file most operators never open: /proc/spl/kstat/zfs/<pool>/txgs, specifically the stime field, which records how long each transaction group took to commit to stable storage.

TXG sync time sits between “the application sees slow writes” and “a disk is slow” and explains the relationship between them. When sync time climbs, dirty data accumulates in the open TXG, ZFS starts throttling writers, and applications experience the periodic write freezes that are so often misattributed to hardware failure or application bugs.

ZFS zfs_arc_max: capping the ARC without starving read performance

Mon, 01 Jan 0001 00:00:00 +0000

On Linux, ZFS does not ship with a sane upper bound on the ARC for you. Left alone, the ARC aggressively consumes available memory, and because ARC memory is managed outside the kernel page cache, it shows up as “used” in free even though it is reclaimable. On a shared host, the outcome is familiar: a database or application process gets OOM-killed, and nothing in the storage layer looks wrong.

ZFS ZIL commit stalls and errors: reading the intent-log kstats

Mon, 01 Jan 0001 00:00:00 +0000

A database that normally commits in single-digit milliseconds suddenly takes seconds per commit. NFS clients time out. zpool status shows the pool ONLINE, read latency looks fine, and zpool status -x says all pools are healthy. The breakage is confined to synchronous writes, which points at one subsystem: the ZFS Intent Log.

The ZIL guarantees synchronous write semantics. Every fsync, O_SYNC write, and NFS commit goes to the ZIL before the application gets its acknowledgment. When the ZIL backs up, because the SLOG is slow or failed, the pool cannot absorb log writes, or the write pipeline is throttled, every synchronous writer stalls at once. Reads from the ARC keep working, which is why this failure mode is confusing on first contact.