$ guides / zfs / zfs-monitoring-checklist ▌

Operations Guides

ZFS monitoring checklist: the signals every production pool needs

Most ZFS incidents are gaps in the basics: a pool DEGRADED for three weeks because nobody paged on it, a pool at 94% capacity discovered when writes started stalling, a disk accumulating checksum errors that a scrub would have caught months earlier. ZFS tells you almost everything you need to know, but only if you collect the right signals continuously instead of running zpool status by hand after something breaks.

This checklist is a maturity ladder. Level 1 is the floor: if you monitor nothing else, monitor these four signals. Levels 2 through 4 catch degradation before it becomes an incident. Alert severities use two tiers: PAGE means wake someone up, TICKET means handle during business hours.

flowchart TD
  L1["Level 1: Survival - pool state, capacity, scrub, error counters"]
  L2["Level 2: Operational - ARC, latency, throughput, fragmentation, snapshots"]
  L3["Level 3: Mature - TXG sync, dirty data, queue depth, deadman events"]
  L4["Level 4: Expert - ARC internals, ZIL counters, event stream analysis"]
  L1 --> L2 --> L3 --> L4

Level 1: survival

These four signals answer the only questions that matter at 3 a.m.: is the pool up, is it full, is the data verified, and are the devices healthy. All four come from zpool status and zpool list.

Pool health state. Any state other than ONLINE means hardware has failed or the pool cannot serve I/O. This is a binary signal that cannot false-fire on workload or load.

# Quick health check: prints "all pools are healthy" and nothing else when clean
zpool status -x

# Machine-readable state per pool
zpool list -H -o name,health

# Direct kstat read (Linux)
cat /proc/spl/kstat/zfs/<pool>/state

Alert: PAGE on FAULTED, SUSPENDED, or UNAVAIL. TICKET on DEGRADED, with same-day response: redundancy is reduced and another device failure in the same vdev may be data loss. DEGRADED with a resilver in progress is expected recovery; DEGRADED with no resilver means someone needs to replace a disk.

Pool capacity utilization. ZFS degrades non-linearly as pools fill. As metaslabs run low on free space, the allocator switches from fast first-fit to slow best-fit, and write latency climbs sharply. Treat 80% as the planning threshold, not the emergency threshold.

# Capacity, allocation, and async reclaim backlog
zpool list -H -o name,size,alloc,free,cap,freeing

Alert: TICKET above 85% on write-heavy workloads, or when growth rate projects under 7 days to 96%. PAGE when CAP exceeds 96% with active write I/O and the freeing property is not actively reclaiming. An idle pool at 96% is a ticket; a pool at 96% with writes in flight is an emergency, because ZFS reserves roughly 3.125% of pool capacity as slop space and regular writes start failing with ENOSPC below that boundary. Pool-level capacity also hides two things: per-dataset available (from zfs get available) accounts for quotas and reservations, and snapshots hold references to blocks that deleted files do not free.

Scrub completion and result. A pool with zero reported errors that has not scrubbed in six months has unknown integrity, not verified integrity. The scan: line in zpool status is the authoritative record: in-progress scrubs show percentage and ETA, completed scrubs show scrub repaired <N>B in <elapsed> with <N> errors on <timestamp>, and none requested means no scrub has ever run.

# Last scrub result, timing, and error count
zpool status <pool> | grep -A3 "scan:"

Alert: PAGE on uncorrectable errors or a non-empty permanent error list in zpool status -v (that list names the actual damaged files and objects; it is your recovery checklist). TICKET on correctable errors, since redundancy saved the data but hardware is failing. TICKET if no scrub has completed in 30+ days. Production pools should complete a scrub every 7 to 14 days. One critical collector note: zpool get last_scrub_time does not exist in OpenZFS (it is Oracle Solaris only). You must parse the scan: line.

Per-device READ, WRITE, and CKSUM error counters. Zero is the only acceptable value in production. These counters are how ZFS tells you a disk, cable, or controller is degrading before it takes the pool down.

# Vdev tree with per-device state and cumulative error counters
zpool status -v

Alert: TICKET on any non-zero value, and escalate when counts increase over time. A one-time increment from a SATA reset during heavy load can happen; sustained growth is dying hardware. Checksum errors on multiple unrelated devices simultaneously point at RAM or the controller, not the disks. PAGE escalation comes through the composite signals: counter growth combined with DEGRADED state or uncorrectable scrub errors.

Level 2: operational

Level 2 signals explain performance, not just survival, and catch failure modes that Level 1 structurally cannot see.

ARC size and hit rate. The ARC is ZFS’s primary read cache in RAM. Read /proc/spl/kstat/zfs/arcstats for size, c (target), c_max, hits, misses, and memory_throttle_count. On Linux, ARC memory shows as “used” in free but is reclaimable, which routinely causes misdiagnosis. TICKET when hit ratio stays below 80% on read-heavy workloads after warmup, and when memory_throttle_count is incrementing, because ZFS is actively throttling I/O due to memory pressure. If anything else memory-hungry runs on the host, set zfs_arc_max explicitly; an uncapped ARC competing with applications is how ZFS hosts end up in the OOM killer.

Pool I/O throughput and latency. zpool iostat -v 1 for per-vdev throughput and IOPS, zpool iostat -l 1 for average latency split into total_wait, disk_wait, syncq_wait, and asyncq_wait. The split matters: high syncq_wait with normal asyncq_wait points at the ZIL or SLOG, not the disks. TICKET on sustained latency above 2x your rolling baseline. Latency spikes during scrub and resilver windows are expected, so suppress or annotate those windows rather than paging on them.

Fragmentation. zpool list -H -o name,frag. Trend it monthly. Below 20% is healthy, 50%+ is concerning on write-heavy pools, and there is no in-place fix: scrub does not defragment, so the only remediation is zfs send | zfs recv into a new pool. Fragmentation rising together with capacity is the early warning of the capacity-fragmentation cliff.

Dataset and snapshot space distribution. Pool capacity alone cannot tell you why the pool is filling. zfs list -o space -r <pool> breaks usage into live data, snapshots, reservations, and children; zfs get -r usedbysnapshots <pool> pinpoints snapshot-heavy datasets. TICKET when snapshot space dominates a pool approaching capacity thresholds. The classic failure is a snapshot cron job with no pruning: the operator deletes 500 GB of files, gets zero space back, and discovers snapshots hold all the references.

SLOG and L2ARC device status. If you have them, monitor them. SLOG failure does not change pool state: the pool stays ONLINE while synchronous write latency jumps by orders of magnitude because the ZIL falls back to the main pool. Check the log and cache sections of zpool status -v, and track L2ARC effectiveness via l2_hits and l2_misses in arcstats. An L2ARC with a sustained hit ratio under 10% is consuming SSD endurance for nothing.

Level 3: mature

Level 3 adds the internal write-path signals that explain latency spikes instead of just reporting them.

TXG sync duration. ZFS batches writes into transaction groups that sync every 5 seconds by default (zfs_txg_timeout). When a sync takes longer than the timeout, the open TXG accumulates dirty data unbounded, and that is the primary source of ZFS write latency spikes. Read the stime field (sync duration, in nanoseconds) from /proc/spl/kstat/zfs/<pool>/txgs. TICKET when stime consistently exceeds 2x the timeout; sustained stime above 3x the timeout with no scrub or resilver running, combined with rising write latency, is the TXG sync hang pattern and should page.

Dirty data pressure. Track dirty bytes against zfs_dirty_data_max (in /sys/module/zfs/parameters/). OpenZFS starts syncing a TXG at 20% of the limit (zfs_dirty_data_sync_percent, default 20%) and delays transactions at 60% (zfs_delay_min_dirty_percent, default 60%); when zfs_dirty_data_max is exceeded, new writes halt until space frees. Sustained dirty data above 80% of the limit is the pre-cliff signal: the write pipeline is about to freeze, and reads from ARC will keep working while writes hang.

Latency histograms and queue depth. Averages hide tail latency. zpool iostat -w gives per-queue latency histograms for p95/p99 analysis, and zpool iostat -q -v 1 shows pending versus active I/O per vdev per queue. The combination discriminates the two most common latency causes: pending queues growing with latency rising means backend saturation; flat queues with latency rising means the problem is inside ZFS (TXG pressure, ARC starvation, fragmentation).

Deadman events. ZFS’s deadman subsystem fires when an I/O is stuck for 5+ minutes (zfs_deadman_ziotime_ms, default 5 minutes) or a pool sync exceeds zfs_deadman_synctime_ms (default 10 minutes). Any deadman event is a PAGE: the multi-minute thresholds are so conservative that no legitimate workload can trigger them. Collect via zpool events -v, but note that zpool events is in-memory only and lost on reboot, so ZED (the ZFS Event Daemon) needs to be running and configured to react.

Resilver tracking. A resilver in progress is a TICKET by itself: the pool runs at reduced redundancy for the duration. Track progress and speed against the device’s expected throughput, and treat extended DEGRADED time with no resilver as an urgent ticket.

Level 4: expert

Level 4 is where you go after your third or fourth incident teaches you what Level 3 missed.

ARC eviction behavior and MRU/MFU balance. Sustained eviction of frequently-used blocks indicates a working set that has outgrown the cache, and shows up before hit rate visibly degrades.
ZIL commit counters. zil_commit_stall_count and zil_commit_error_count in /proc/spl/kstat/zfs/zil catch sync-write path problems. Verify the fields exposed by your release with grep -E "^(zil_commit_stall_count|zil_commit_error_count)" /proc/spl/kstat/zfs/zil. There is no ZIL latency metric; infer it from application-level fsync timing.
DDT memory usage. Only if dedup is enabled. The dedup table costs roughly 320 bytes of RAM per block, and if it stops fitting in ARC, every write performs a disk read first. Check with zpool status -D <pool>.
Special vdev capacity. If you use a special allocation class, a full special vdev silently falls back to the main pool while pool-level capacity looks fine. On current OpenZFS, monitor class_special_capacity and the related class_special_* pool properties.
Event-stream analysis beyond deadman. I/O failure, probe failure, and delay events in the ZFS event stream are leading indicators that counters smooth over.
Capacity runway estimation. Free bytes divided by daily allocation rate, adjusted for snapshot holds and async reclaim backlog. Turn capacity from a threshold alert into a planning number.

Instrumentation gotchas that break collection

These are the traps that silently disable monitoring you think you have.

/proc/spl/kstat/zfs/<pool>/io was removed in OpenZFS 2.1.0. Tools referencing it return nothing without erroring. The replacement iostats file does not contain the traditional read/write bandwidth counters. Use zpool iostat output instead.
Error counters are volatile. They reset on zpool clear and can reset on pool export/import or module reload. A device showing zero errors today may have a cleared history. Export counters to a time-series store continuously; never treat a point-in-time zpool status as history.
CKSUM=0 is not proof of no corruption. A known OpenZFS bug (#11545) means scrub-repaired checksum errors do not always increment the per-vdev counter. Scrubs, not counters, are the integrity guarantee.
zpool status -x is not sufficient health monitoring. It reports pool state, not cumulative error counters. A pool with a thousand corrected checksum errors still shows ONLINE. You need both the state check and the counter check.
Scrub progress ETAs are unreliable. Early estimates are optimistically short, later estimates can be absurdly long. Alert on completion and results, not on ETA drift.

How Netdata helps

The recurring theme in this checklist is that ZFS signals are only useful as trends and combinations, not as point-in-time command output:

Netdata’s ZFS collector reads the kstat interfaces (arcstats, per-pool txgs, pool state) every second, so ARC pressure, hit rate drift, and TXG sync duration are captured as history rather than lost between manual checks.
Per-device READ/WRITE/CKSUM counters and pool state are collected continuously, which sidesteps the counter-reset problem: a zpool clear no longer erases your evidence.
Correlating TXG stime with dirty data, write latency, and queue depth on one dashboard is what separates “TXG sync hang” from “slow disk” in minutes instead of hours.
Capacity, fragmentation, and snapshot space trended together make the capacity-fragmentation cliff visible weeks ahead, while it is still a planning exercise.
Pool state transitions and deadman events are alertable as binary conditions, which is exactly what the PAGE-tier signals above require.