Most ZFS incidents are not monitoring failures. They are monitoring-coverage failures. The pool was watched, but at the wrong level: the team had capacity graphs and no scrub result alerting, or pool state alerts and no TXG sync visibility, and the failure mode that actually fired lived exactly in the gap.

This article is a reference model with four levels: Survival, Operational, Mature, and Expert. Each level adds signals that catch failure classes the previous level structurally cannot see. Use it two ways: as an audit of what you monitor today, and as a roadmap for what to add next. The levels are cumulative. Skipping a level buys you alert noise, not insight.

flowchart TD
  S["Level 1: Survival - is the pool alive, full, verified, error-free?"]
  O["Level 2: Operational - is it performing, and are the devices healthy?"]
  M["Level 3: Mature - will it stall, fill, or hang in the near future?"]
  E["Level 4: Expert - why is it behaving this way internally?"]
  S --> O --> M --> E

Level 1: Survival

The minimum. These four signals tell you whether the pool is up, full, verified, and throwing device errors. Every one is binary or near-binary and cheap to alert on.

Pool health state. A FAULTED, SUSPENDED, or UNAVAIL pool is a page; DEGRADED is an urgent ticket because redundancy is gone and the next failure in the same vdev group is data loss.

# Show only pools with problems; "all pools are healthy" means ONLINE
zpool status -x

Pool capacity utilization. ZFS degrades non-linearly as it fills because the metaslab allocator switches from fast first-fit to expensive best-fit as free space fragments. Treat 75-80% as the planning threshold, 85% as the action threshold on write-heavy pools, and 96%+ with active writes and no reclaim in progress as an emergency.

zpool list -H -o name,size,alloc,free,cap,freeing

Per-vdev error counts. The READ, WRITE, and CKSUM columns in zpool status. Zero is the only acceptable production value. Any non-zero count is a hardware investigation; sustained growth is a dying device. Two caveats: counters are cumulative since the last zpool clear and may reset on pool export/import or module reload, so do not treat them as long-term history. And a known OpenZFS issue (#11545) means scrub-repaired checksum errors do not always increment the CKSUM counter, so CKSUM=0 is not proof of clean data.

Scrub completion status. Not just scrub results: whether scrubs are running at all. A pool showing 0 errors that has not scrubbed in six months has unknown integrity, not verified integrity. Alert on uncorrectable errors or a non-empty permanent error list (zpool status -v) as a page; correctable errors and no completed scrub in 30+ days as tickets. zpool get last_scrub_time does not exist in OpenZFS; parse the scan: line from zpool status.

What Level 1 cannot see: anything about performance. A pool can be ONLINE, at 50% capacity, freshly scrubbed, with zero errors, and still be stalling every write for 30 seconds.

Level 2: Operational

Everything in Level 1, plus performance visibility and per-device granularity.

ARC size and hit ratio. Read /proc/spl/kstat/zfs/arcstats (or use arcstat). Track size against c_max, and hit rate against a workload-appropriate target: above 80% for read-heavy, above 60% for mixed, lower is normal for write-heavy or sequential streaming. Two traps: ARC memory shows as used in free but is reclaimable, and a low hit rate in the first hours after boot is warmup, not a problem. Set zfs_arc_max explicitly on hosts running other services: the Linux default cap (half of RAM) is often too large for shared hosts, and ARC growth under pressure can trigger the OOM killer before the ARC shrinks.

TXG sync time. The most diagnostic write-path signal ZFS exposes. Read the stime field (sync duration, nanoseconds) from /proc/spl/kstat/zfs/<pool>/txgs. Compare it against zfs_txg_timeout (default 5 seconds): under 1x is healthy, 1-2x is elevated, sustained above 2x means the storage cannot keep up with the write rate.

Fragmentation. zpool list -H -o name,frag. Under 30% is normal, above 50% is concerning for write-heavy workloads, and it only ever gets fixed by recreating the pool (send/recv to new storage). Scrubs do not defragment. Trend it monthly; the rate of increase matters more than the absolute value.

Snapshot space distribution. Pool-level capacity cannot tell you whether the space is live data, snapshots, clones, or reservations, and the response is different for each.

zfs list -o space -r <pool>
zfs get -r usedbysnapshots <pool>

Deleting files does not free space while snapshots reference the blocks. The freeing pool property shows bytes still being asynchronously reclaimed after a destroy.

Per-vdev latency. zpool iostat -l 1 gives average latency columns (total_wait, disk_wait, syncq_wait, asyncq_wait, scrub, trim, rebuild) and zpool iostat -v breaks throughput down per device. In a mirror or RAIDZ group, the slowest device sets the pace, and a single device running 3x slower than its peers is a failing device that has not faulted yet. High syncq_wait with normal asyncq_wait points at the ZIL/SLOG, not the data vdevs.

SMART health for backing devices. ZFS error counters and SMART data corroborate each other: CKSUM errors plus rising Reallocated_Sector_Ct or Current_Pending_Sector means imminent disk failure; CKSUM errors on multiple unrelated devices means suspect RAM or the controller, not the disks.

SLOG and L2ARC device status. Check the logs and cache sections of zpool status -v. A failed SLOG does not change pool state: the pool stays ONLINE while synchronous write latency collapses because the ZIL falls back to the data vdevs. On NFS or database hosts, this is a page-level application event wearing a ticket-level disguise. L2ARC failure degrades read caching only; data is safe.

What Level 2 cannot see: the future. It tells you the pool is slow now, not that the write pipeline is about to stall, that the pool will fill in 12 days, or that an I/O has been hung for six minutes.

Level 3: Mature

Everything above, plus leading indicators and the internal state needed to catch stalls before applications feel them.

Dirty data pressure. ZFS throttles writers when dirty data reaches zfs_delay_min_dirty_percent (default 60%) of zfs_dirty_data_max, and hard-stalls at the limit. Watch the ndirty field in the txgs kstat and the limit in /sys/module/zfs/parameters/zfs_dirty_data_max. Sustained dirty data above 80% of the limit is the pre-cliff signal for a TXG sync hang: writes are about to block.

Queue depth. zpool iostat -q -v 1 shows pending and active I/O per queue type per vdev. This splits two diagnoses that look identical from the application side: pending queue growing plus latency rising means backend saturation; queue flat plus latency rising means the problem is inside ZFS (TXG, ARC, fragmentation).

Latency histograms. Averages from zpool iostat -l smooth out the spikes that users actually feel. zpool iostat -w gives bucket distributions for p50/p95/p99 analysis. Alert on tail-latency divergence from baseline, not on averages.

ZIL latency and stalls. There is no direct ZIL latency metric; infer it from application-level fsync timing and from the counters in /proc/spl/kstat/zfs/zil. On sync-heavy workloads without a SLOG, ZIL writes go to the main pool and fsync latency is pool latency.

Capacity runway. Trend capacity growth and compute Runway (days) = Free Space / Daily Allocation Rate, accounting for snapshot holds and async reclaim backlog. Ticket when the projection shows under 7 days to 96%. This turns capacity from an emergency into a calendar entry.

Deadman events. The deadman subsystem fires when an I/O hangs past zfs_deadman_ziotime_ms or a pool sync hangs past zfs_deadman_synctime_ms (defaults are minutes). Any deadman event is a page; nothing in normal operation hangs an I/O that long. Collect via zpool events -v, but the event buffer is in-memory only and lost on reboot. Configure ZED to react to and persist events.

Resilver progress. Track the scan: line during reconstruction. A resilver in progress is a ticket because redundancy is reduced for its duration. Compare observed speed against the device’s sequential capability, and remember resilvers and scrubs are mutually exclusive per pool. OpenZFS 2.0+ sequential resilver (zpool replace -s) is dramatically faster for mirrors and dRAID.

Memory throttling. Incrementing memory_throttle_count in arcstats means ZFS is actively throttling I/O due to memory pressure. Combined with ARC size pinned at c_max and low system MemAvailable, it is the early stage of the OOM cascade pattern.

Level 4: Expert

Deep signals most pools never need. Some pools are undebuggable without them.

ARC ghost lists. The ghost list sizes (mru_ghost_size, mfu_ghost_size in arcstats) reveal the working set that no longer fits in the ARC. Ghost hits mean the ARC evicted something it is now being asked for again: cache sizing evidence, visible nowhere else.

DDT memory (dedup pools only). The dedup table costs roughly 320 bytes of RAM per block and must stay resident in the ARC. If the DDT outgrows the ARC, every write does a disk read for the DDT lookup and the pool falls off a cliff. Check zpool status -D for DDT statistics and zdb -S for size analysis. Ticket when the DDT exceeds about 25% of ARC, or when the dedup ratio approaches 1.0x and you are paying the cost without the benefit. Dedup cannot be efficiently disabled retroactively; dedup=off only affects new writes.

Space maps. On heavily fragmented pools, space maps grow large and slow to load, which shows up as long pool import times and metaslab activation latency. zdb -mmm exposes per-metaslab detail.

Import time. Baseline it and trend it, especially on root-on-ZFS systems where import time is boot time. A growing import time is often the first visible symptom of space map growth, large pending ZIL replays, or DDT pressure.

Async destroy backlog. Snapshot and dataset destruction reclaims space asynchronously. A large, persistent freeing value means destructions are queued faster than the pool can reclaim, which matters both for capacity math (the space is not back yet) and because pending async destroy work competes with production I/O.

Device endurance. SLOG and L2ARC devices wear by design, and there is no ZFS-level kstat for SLOG write volume. Track SMART or NVMe wear indicators (Percentage_Used and equivalents) per device and trend the consumption rate. Common practice: replace production devices at 70-80% of rated endurance, more aggressively for SLOG devices, because endurance exhaustion is a sudden failure with no graceful degradation curve.

How to climb without creating noise

  • Automate the current level first. If Level 1 signals are checked by hand, do not add Level 2 signals. Manual checks decay to zero within a quarter.
  • Every new signal needs a consumer. An alert condition, a trend line, or a dashboard someone opens during incidents. Signals collected “because we might need them” become storage costs, not monitoring.
  • Keep point-in-time tools out of the alert path. zpool status shows current state with clearable, resettable counters. Export continuously to time-series storage and use ZED for event-driven alerts, or reboots and zpool clear will silently erase your evidence.
  • Move up when the level below is quiet. You are ready for the next level when the current one fires rarely and teaches you something when it fires. If Level 1 alerts still surprise you weekly, fix that first.

How Netdata helps

  • Correlates pool state, per-vdev error counts, and scrub results in one view, so a DEGRADED pool plus rising CKSUM on a remaining device is one picture, not three alerts.
  • Tracks ARC size, hit ratio, and memory_throttle_count alongside system memory, which makes the ARC-starvation-to-OOM cascade visible before the OOM killer fires.
  • Exposes TXG sync times and dirty data pressure next to zpool iostat latency, so the write-stall pattern (slow sync, rising dirty data, latency spike) reads as a single story.
  • Trends capacity and fragmentation together, which is how you see the capacity-fragmentation cliff approaching rather than arriving.
  • Captures ZFS events including deadman and scrub/resilver lifecycle events, closing the gap left by in-memory-only zpool events.