$ guides / zfs ▌

ZFS · OPERATIONS PLAYBOOK

ZFS's copy-on-write bargain: end-to-end integrity, a batch-oriented write path, and a capacity cliff you fall off without warning

A combined filesystem and volume manager where every write lands somewhere new, all writes batch into transaction groups flushed every few seconds, a RAM cache will happily consume the whole machine, and performance collapses long before the pool is actually full. We trace how that design behaves under load, where it turns from graceful into a hard stop, and what to do when it does.

> Start with the monitoring checklist → # Jump to the full guide list

ZFS's defaults give you a self-healing, checksummed pool in minutes, then hand you a set of cliff-edges that most teams only discover during an incident.

The defaults work. Until the pool drifts past 80% full, the metaslab allocator quietly switches from first-fit to best-fit, and write latency falls off a cliff that was invisible a percent earlier. Until zfs_arc_max is left unset on Linux and the ARC consumes the RAM your database needed, so the OOM killer picks a victim. Until a disk fails, the pool goes DEGRADED, and it runs there for weeks one failure away from data loss. Until a scrub you never scheduled would have caught the bit rot that zpool status -v now reports as permanent errors in the following files. Until a SLOG dies, the pool stays ONLINE, and every synchronous write is suddenly a hundred times slower.

These guides are written for engineers who already run ZFS, not for people deciding whether to. The goal is the mental model of how the pool actually behaves under load — copy-on-write, transaction groups, the ARC, the ZIL, the metaslab allocator — the failure patterns that keep recurring, the monitoring story that catches them before they page anyone, and the runbooks you wish someone had handed you before your last incident.

How ZFS actually runs in production

ZFS is not just a filesystem. It is a copy-on-write storage stack where every write is checksummed, compressed, and batched into a transaction group before it ever reaches a disk, a RAM cache serves most reads, and a separate log underwrites every synchronous write. Most production failures live between these layers — in the write batching, the cache, the allocator, or the redundancy — not inside any single disk.

applications + syscalls

Reads, writes, and <code>fsync</code>/<code>O_SYNC</code> calls enter here. On Linux, ZFS keeps its own cache (the ARC) outside the kernel page cache, so <code>free</code> and <code>top</code> misrepresent ZFS memory. Synchronous callers (databases, NFS, mail) are the ones that feel ZIL latency.

CLIENT

DMU + ZIO pipeline

The Data Management Unit turns objects into blocks and runs every write through checksum, compression, and encryption stages. Copy-on-write means a block is never overwritten — it is written new and pointers are updated atomically. This is the source of ZFS's integrity guarantees and of its write amplification.

DMU

ARC (read cache)

The Adaptive Replacement Cache lives in RAM, balancing recently- and frequently-used blocks (MRU/MFU). It grows to fill memory and shrinks under pressure — but not instantly, which is how the OOM killer beats it on Linux. Hit ratio and size versus <code>c_max</code> are the read-performance story.

ARC

ZIL / SLOG

The intent log records synchronous writes for crash recovery and is the only path that makes <code>fsync</code> durable before the next TXG. A dedicated SLOG device moves it off the pool. It is write-only in normal operation and read only during import after an unclean shutdown.

ZIL

transaction groups (TXG)

All writes accumulate into transaction groups flushed every <code>zfs_txg_timeout</code> (5s). Three are always in flight: open, quiescing, syncing. When a sync runs long, the open TXG bloats with dirty data and writes get throttled — the periodic latency spike everyone blames on the disks.

TXG

SPA + metaslab allocator

The Storage Pool Allocator finds free space in metaslabs. Below ~80% full this is cheap; above it, the allocator switches to best-fit and searches harder, and fragmentation turns sequential writes into random I/O. The capacity cliff lives here, and it is non-linear.

SPA

vdevs + redundancy

Mirror, RAIDZ, and dRAID vdevs provide the redundancy that lets ZFS repair bad blocks from parity or copies. A device fault drops the pool to <code>DEGRADED</code>; too many faults, and it goes <code>FAULTED</code> or <code>SUSPENDED</code>. Resilver rebuilds a replacement — and is the reduced-redundancy danger window.

VDEV

physical devices

The actual disks and SSDs, reached through the kernel block layer. ZFS's checksums catch what they silently return wrong; SMART attributes and <code>dmesg</code> resets are where a dying device shows itself before ZFS faults it out.

DISK

Why this matters: 'ZFS is slow' or 'writes are hanging' can come from a pool near its capacity cliff, an ARC starved by another process, a TXG sync storm, a saturated or dying disk, a failed SLOG throwing sync writes back onto the pool, or fragmentation amplifying every write. The symptom rhymes but each layer has a different signal — and a different fix.

The failures you'll actually see

Most ZFS incidents fall into a small set of recurring patterns. Recognise the shape, and triage gets dramatically faster.

CRITICAL

The capacity cliff

The pool crosses its slop boundary and writes start failing with ENOSPC even though df shows free space. Because every operation is copy-on-write, even deleting files needs free space — so the pool can enter a death spiral where freeing space requires the space you don't have. Performance had already collapsed non-linearly on the approach past 80-90% full.

No space left on device with free space still showing in zpool list
Write latency spiking as capacity climbs past 85%
zfs destroy of snapshots slow or itself blocked
freeing property large as async reclaim struggles to keep up

Investigate →

CRITICAL

The suspended pool

Too many devices are lost or unreachable and the pool enters SUSPENDED — every I/O blocks indefinitely while ZFS waits for connectivity to return. Applications hang rather than error. The failmode property decides whether the pool waits, errors, or panics. This is a hard outage, not a slowdown, and it does not clear on its own.

cannot open POOL: pool I/O is currently suspended
Processes stuck in uninterruptible sleep on pool I/O
Multiple devices UNAVAIL or REMOVED at once (HBA, enclosure, cable)
SUSPENDED persisting until devices return or a forced re-import

Investigate →

CRITICAL

Silent data corruption surfaces

A scrub or a read finds blocks that fail their checksums and cannot be repaired from redundancy, and zpool status -v lists the affected files. The data was rotting invisibly — the pool stayed ONLINE the whole time. With redundancy ZFS would have repaired it; without enough of it, the loss is permanent and the file list is your recovery checklist.

permanent errors have been detected in the following files
Scrub completes with a non-zero unrepaired error count
CKSUM climbing on one device (or several — suspect RAM)
No scrub completed in months before the errors appeared

Investigate →

ACTIVE

The degraded pool

A vdev has lost a device and the pool is running DEGRADED — still serving I/O through redundancy, but one more failure in the same group means data loss. DEGRADED is stable and the pool will run there indefinitely, which is exactly why teams leave it and lose the race to the next failure. A resilver or hot spare may already be recovering it.

Pool state DEGRADED in zpool status
One device FAULTED, OFFLINE, or being resilvered
Per-vdev error counts non-zero and possibly climbing
No resilver in progress on a pool that has been DEGRADED for days

Investigate →

ACTIVE

ARC memory starvation

With no zfs_arc_max set, the ARC grows to consume the RAM other processes need. On Linux the OOM killer can fire before the ARC shrinks — killing a database instead of evicting reclaimable cache — because arc_prune has latency. Read performance also collapses as the working set no longer fits, and the whole thing feeds back on itself.

ARC size at or near total RAM with MemAvailable near zero
OOM killer activity in dmesg while the pool looks healthy
memory_throttle_count incrementing in arcstats
Read latency rising as the ARC hit ratio falls

Investigate →

IMMINENT

A disk faults out

A device's READ/WRITE/CKSUM counters cross the threshold and ZFS auto-faults it with too many errors, dropping the pool to DEGRADED. Often the drive was already the slow one in the vdev, dragging every stripe write while it retried and remapped internally. Catch it before the fault, from error counts and SMART, and replace it before a second device follows.

Device FAULTED with too many errors in zpool status
One vdev consistently 3x+ slower in zpool iostat -v
SATA/SAS reset, task abort, or timeout in dmesg
SMART reallocated or pending sectors rising on that device

Investigate →

Choosing a tool

Best ZFS Monitoring Tools: 7 Ranked (August 2026)

A ranked review of the tools teams actually shortlist here, what each one is genuinely good at, and how the pricing behaves as you scale.

Read the buyer's guide →

ZFS monitoring maturity levels

ZFS observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your data matters. Most production pools should land at the second level.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor. With these signals you can answer one question: is the pool up, is it full, and is the data still intact? You will not learn why something broke, but you will learn that it broke before users do. Survival is enough for scratch pools and non-critical storage.

Pool health state zpool status -x — ONLINE, DEGRADED, FAULTED, SUSPENDED.
Pool capacity utilization The non-linear cliff lives here; watch the approach past 80%.
Last scrub result and time Parsed from the scan: line — no scrub means unknown integrity.
READ / WRITE / CKSUM error counts Any non-zero value on a data-bearing device needs investigation.

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production pools should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident on its own: cache starvation, write stalls, fragmentation, a failing disk, a dead SLOG.

ARC size and hit ratio arcstats — the read-performance story; size vs c_max shows pressure.
TXG sync time txgs stime — the single most diagnostic write-path signal.
Pool fragmentation zpool list frag — write amplification you cannot defrag away.
Per-dataset and snapshot space zfs list -o space — where the capacity is actually going.
Per-vdev I/O latency zpool iostat -wl — finds the one slow disk in the vdev.
SMART on all backing devices Reallocated/pending sectors and wear precede ZFS faults.
Scrub running on schedule That scrubs execute, not just the last result.
SLOG / L2ARC device health A dead SLOG never shows DEGRADED but kills sync latency.

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone up. Dirty data creeping toward the throttle, ARC eviction churning, a resilver decelerating, capacity runway shrinking, a special vdev filling. None of these page you on day one; they become incidents on day thirty.

Dirty data vs zfs_dirty_data_max How close the write pipeline is to a hard stall.
Per-vdev queue depth zpool iostat -q — saturation versus a hang.
Latency histograms (p95/p99) zpool iostat -w — tail latency the averages hide.
ZIL commit latency and stalls The sync-write story for databases and NFS.
Capacity growth and runway Days to the threshold, not just today's percentage.
Fragmentation trend Rate of increase matters more than the absolute value.
ZFS deadman events zpool events — hung I/O and stalled sync.
Resilver speed and progress The reduced-redundancy window and whether it's stalling.

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. ARC ghost lists, DDT memory, space-map load times, async destroy backlog, per-device write endurance. Most teams never need every signal here. Add the ones your incident history says you do.

ARC ghost list sizes mru_ghost / mfu_ghost — working set beyond ARC capacity.
arc_meta_used vs arc_meta_limit Metadata filling the ARC forces data eviction.
DDT size and in-core memory Only with dedup on — the cliff nobody plans for.
Space map size per metaslab zdb -mmm — allocation cost and import-time driver.
Async destroy backlog freeing — pending reclaim not yet returned.
Pool import time trend Baseline it — critical for root-on-ZFS boot.
Device write amplification / endurance ZFS bytes vs SMART host writes; SLOG wears fastest.
Special vdev capacity / fragmentation Fills silently and falls back to the main pool.

Operating mistakes worth avoiding

The traps ZFS teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

Running a ZFS pool like it's ext4 near 90% full

The most common ZFS incident. Copy-on-write means fragmentation accelerates as the pool fills, and past ~80-85% the metaslab allocator switches from first-fit to best-fit — write latency degrades non-linearly, not gracefully. You cannot defragment ZFS in place; the only fix is <code>zfs send | zfs recv</code> to a fresh pool. Plan expansion at 75%, not at 95%.

Leaving zfs_arc_max unset on Linux

By default ZFS on Linux has no ARC cap, so the ARC grows to consume most of RAM and the OOM killer eventually kills an application — often the wrong one, because <code>arc_prune</code> cannot shrink the cache fast enough under a sudden allocation. Always set <code>zfs_arc_max</code> in <code>/etc/modprobe.d/zfs.conf</code> on any host that also runs memory-hungry processes.

Monitoring scrub errors but never checking scrubs ran

Teams alert on scrub errors and forget to check that scrubs actually execute. A pool reporting <code>CKSUM: 0</code> that has not scrubbed in six months has unknown integrity — zero errors means zero errors detected. <code>zpool get last_scrub_time</code> does not exist in OpenZFS, so parse the scan: line and alert when nothing has completed in 30+ days.

Treating DEGRADED as something to deal with later

A DEGRADED pool stays online and serves I/O, so it is tempting to leave it. But redundancy is already gone — the next failure in that vdev is data loss. DEGRADED is a same-day ticket, not a backlog item, and on RAIDZ1 it is urgent. Check whether a resilver or hot spare is recovering it, and replace the failed device immediately if not.

Blaming the disks for latency that's really the write path

Periodic write-latency spikes get attributed to failing hardware when the real cause is a TXG sync storm or dirty-data throttling — ZFS deliberately delaying writes so it doesn't exhaust memory. Check <code>txgs</code> sync time (<code>stime</code>) and dirty data against <code>zfs_dirty_data_max</code> before you touch a disk. The buffer is often just too small for fast NVMe.

Assuming a healthy pool means healthy performance

A SLOG failure does not trip <code>DEGRADED</code> — the pool stays <code>ONLINE</code> while NFS or database latency collapses a hundredfold because the ZIL fell back onto the main pool. Teams that only check <code>zpool status -x</code> see 'all pools are healthy' and miss it entirely. Monitor SLOG device health and sync-write latency explicitly, and mirror the SLOG.

Never tracking snapshot space

Automated snapshots (sanoid, zfs-auto-snapshot) accumulate silently because COW snapshots hold references to deleted blocks. Months later the pool is 90% full — not from new data, but from retention. Deleting files frees nothing while a snapshot references them. Track <code>usedbysnapshots</code> and <code>zfs list -o space</code>, and prune before the pool, not after.

Enabling dedup without budgeting DDT RAM

The dedup table needs about 320 bytes of RAM per unique block. Enable dedup on a large pool without doing that arithmetic and, once the DDT outgrows the ARC, every write does a disk read first and both reads and writes crawl. <code>dedup=off</code> only affects new writes, so the only real fix is migrating data out — agonisingly slow on a pool that no longer fits its DDT. Size it with <code>zdb -S</code> first.

ZFS runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

Pool health, vdevs, and device states

▸

Data integrity, checksums, and scrubs

▸

Capacity, fragmentation, and the allocation cliff

▸

ARC, memory pressure, and OOM

▸

The write path: TXG sync and throttling

▸

ZIL, SLOG, and synchronous writes

▸

Pool I/O latency and saturation

▸

Resilver, redundancy, and disk replacement

▸

Snapshots, clones, and space reclamation

▸

Dedup, L2ARC, encryption, and special vdevs

▸

Pool import, multihost, and history

WHERE TO GO NEXT

Setting up ZFS monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.

> Start with the checklist > Back to Operations Guides