Every ZFS incident traces back to a small set of internal mechanisms. A write freeze at 3 a.m. is the transaction group pipeline stalling. A server that “runs out of memory” with gigabytes free is the ARC doing exactly what it was designed to do. A pool that was fine at 78% full and unusable at 88% is the metaslab allocator crossing a threshold, not a disk dying.

This article is the mental model layer: what ZFS is doing internally at all times, which resources it competes for, and why its characteristic failures look the way they do. It is the prerequisite for the operational runbooks. You cannot interpret a TXG sync time, an ARC hit ratio, or a fragmentation percentage without knowing the machinery that produces those numbers.

The examples use OpenZFS on Linux paths (/proc/spl/kstat/zfs/...). On FreeBSD and illumos the subsystems are the same but kstat paths, tunable names, and some defaults differ.

What ZFS is and why it matters

ZFS is a combined filesystem and volume manager. It owns the disks directly, manages redundancy (mirrors, RAIDZ, dRAID) without a hardware RAID controller, and layers datasets, snapshots, compression, checksumming, and optional encryption on top. Because it controls the whole stack from filesystem semantics down to block allocation, its failure modes are also end-to-end: a capacity problem becomes an allocator problem becomes a latency problem becomes an application outage, all inside one subsystem.

The operator consequence: you cannot monitor ZFS the way you monitor ext4 on LVM. Pool state, cache behavior, write batching, and physical layout are all first-class signals, and the interesting failures are almost always interactions between them.

Copy-on-write: the decision that drives everything

ZFS never overwrites existing data. Every write lands in a new location on disk, and the block pointers above it are updated atomically up the tree to the uberblock. This single decision cascades into nearly every operational behavior:

  • Snapshots are cheap to take, expensive to hold. A snapshot just pins the old block pointers. The cost arrives later, when the live dataset diverges and the snapshot holds references to blocks that have been logically deleted. Deleting files does not free pool space while a snapshot references those blocks.
  • There is no in-place defragmentation. Because data is always written to new locations, free space scatters over time. zpool scrub verifies checksums; it does not defragment anything. The only “defrag” is zfs send | zfs recv into a fresh pool.
  • Deletes need free space too. Deleting files or snapshots is itself a COW operation that must write new metadata. A pool that is completely full can enter a state where freeing space requires space. ZFS reserves slop space (about 3.125% of pool size, with a per-pool cap raised to 128 GiB in OpenZFS 2.1 ) for exactly this reason, and user writes fail before that reserve is consumed.
  • Crash consistency is structural. Since nothing is ever overwritten in place, a crash cannot leave a half-written block tree. There is no fsck. The ZIL (below) covers the synchronous-write window.

Transaction groups: the write path is a batch system

ZFS does not write when your application writes. It accumulates writes in memory and commits them to disk in transaction groups (TXGs), flushed periodically: default every 5 seconds, controlled by zfs_txg_timeout. Three TXGs are always in flight:

  • Open: accepting new writes.
  • Quiescing: finalizing, no longer accepting writes.
  • Syncing: being written to stable storage.

This is why ZFS write latency is bimodal. Most writes complete at memory speed. The pain arrives when the syncing TXG cannot finish within the timeout: the open TXG keeps accumulating dirty data, the next sync is even bigger, and the cycle compounds. ZFS defends itself with a dirty data limit (zfs_dirty_data_max): a throttle engages at zfs_delay_min_dirty_percent of that limit (default 60%) and a hard stall at the limit itself. Applications experience this as sudden write freezes lasting seconds to tens of seconds while reads from cache continue normally.

You can watch this directly. Each pool keeps a TXG history at /proc/spl/kstat/zfs/<pool>/txgs; the stime field is the sync phase duration in nanoseconds and ndirty is the dirty bytes per TXG. If stime consistently exceeds zfs_txg_timeout, the storage cannot keep up with the write rate.

The ARC: a read cache that lies to free

The Adaptive Replacement Cache is ZFS’s primary read cache, in kernel memory. It maintains MRU and MFU lists plus their ghost counterparts, and shifts the balance between recently-used and frequently-used blocks based on workload. It caches both data and metadata, and it grows and shrinks in response to memory pressure.

Two operational facts matter more than the algorithm:

  1. On Linux, ARC memory lives outside the kernel page cache. It appears as “used” (in slab) in free and top, but it is reclaimable. Operators routinely misdiagnose a healthy system as memory-starved because of this. The ground truth is /proc/spl/kstat/zfs/arcstats: size (current), c (dynamic target), c_min, c_max, plus hits and misses.
  2. Uncapped ARC plus other memory-hungry processes is an OOM cascade. The Linux OOM killer does not reliably account for ARC as reclaimable, and ARC does not shrink instantly under sudden pressure. On shared machines, set zfs_arc_max explicitly. The memory_throttle_count field in arcstats increments when I/O is being throttled due to memory pressure: that is your early warning.

Two optional extensions attach to the ARC. The L2ARC is an SSD-backed second tier fed by ARC evictions; it needs warmup (a cold L2ARC provides no benefit), it persists across reboots by default since OpenZFS 2.0 (l2arc_rebuild_enabled=1), and its index consumes ARC RAM, roughly 70 bytes per cached block, so an oversized L2ARC on a RAM-constrained host can be a net negative. If deduplication is enabled, the dedup table (DDT) lives in the ARC at roughly 320 bytes per block; a DDT that outgrows the ARC forces every write into random disk lookups for table entries and is one of the classic unrecoverable performance cliffs.

The ZIL and SLOG: underwriting synchronous writes

The TXG pipeline is asynchronous. Anything that demands synchronous semantics (O_SYNC, fsync, NFS, database commit logs) cannot wait up to 5 seconds for the next TXG, so ZFS writes an intent record to the ZFS Intent Log before acknowledging the write. The ZIL guarantees that if the system crashes between the acknowledgment and the TXG commit, the intent can be replayed on pool import.

Operationally:

  • In normal operation the ZIL is write-only. It is only read during import after an unclean shutdown. Once the TXG commits, the ZIL space is reclaimed.
  • Without a SLOG, the ZIL lives on the pool’s data vdevs. Sync write latency becomes pool latency: tens of milliseconds on spinning disk.
  • A SLOG (separate log device) moves the ZIL to a dedicated fast device. Sync write latency becomes SLOG latency. This is the entire reason SLOGs exist, and they only help synchronous workloads; an all-async workload gets nothing from one.
  • SLOG failure is a performance event, not a data event. ZFS falls back to the pool-based ZIL. The pool stays ONLINE, zpool status -x says everything is healthy, and your database’s commit latency just jumped by one or two orders of magnitude. Mirror your SLOG; a lone SLOG that dies concurrent with a crash can lose acknowledged writes.
  • There is no ZIL latency metric. The kstat at /proc/spl/kstat/zfs/zil exposes commit, writer, stall, and error counters, but latency must be inferred from application-level fsync timing.

The SPA and metaslab allocator: the capacity cliff is real

The Storage Pool Allocator owns physical allocation, vdev topology, redundancy, and I/O scheduling. Each top-level vdev is divided into metaslabs, fixed-size chunks, and every allocation picks a metaslab and carves space out of it.

This is the mechanism behind the most infamous ZFS failure: the capacity-fragmentation cliff. When an individual metaslab has only a few percent of free space left, its allocator switches from the fast first-fit path to an expensive best-fit scan , and fragmented free space forces sequential writes into scattered physical locations. The effect is non-linear: the pool looks fine at 75%, starts degrading through the 80s, and falls apart above 90%. The commonly cited “keep it under 80%” rule is a conservative proxy for per-metaslab behavior, which is why the actual pain threshold varies with workload and vdev size. Free space is tracked in space maps, log-structured allocation records, which on heavily fragmented pools also grow large and slow down pool import.

Two practical implications. First, monitor capacity and fragmentation together (zpool list -o name,cap,frag): a pool at 70% with 50% fragmentation is already degraded for write-heavy workloads. Second, fragmentation cannot be fixed in place; the remediation is send/recv to a new pool, so the only real strategy is not getting there.

The DMU and ZIO pipeline: where every block is processed

Between the filesystem semantics and the disks sit two layers. The Data Management Unit (DMU) translates logical objects (files, directories, zvols) into blocks and handles the operations applied to them: compression, checksumming, encryption, copy-on-write, and dnode management. The ZIO pipeline then moves every block through a staged pipeline: issue, wait, checksum, compress, encrypt, vdev I/O, done. Each stage queues independently with its own thread pool.

flowchart TD
  A[Application write] --> B[DMU: block mapping]
  B --> C{Sync write?}
  C -- yes --> D[ZIL / SLOG: record intent]
  C -- no --> E[Open TXG: accumulate dirty data]
  D --> E
  E --> F[TXG quiescing]
  F --> G[TXG syncing]
  G --> H[ZIO pipeline: checksum, compress, encrypt]
  H --> I[Vdevs: mirror / RAIDZ / dRAID]
  J[Application read] --> K{In ARC?}
  K -- yes --> L[Serve from RAM]
  K -- no --> I

This pipeline explains where CPU goes on a ZFS host: checksumming, compression, and encryption run on every block, on every read and every write. On a CPU-bound system, adding faster disks changes nothing. It also explains reported-versus-actual throughput: with compression on (the default), the bytes ZFS reports and the bytes the disks see are not the same number, and write amplification from metadata, parity, and COW overhead of 2-3x on small random writes to RAIDZ is structural, not a symptom.

Vdev redundancy: what failure actually looks like

The SPA builds redundancy from vdevs: mirrors, RAIDZ1/2/3, or dRAID (OpenZFS 2.0+). The operational differences that matter:

  • Failure is masked. A dead disk in a mirror or RAIDZ leaves the pool ONLINE and serving I/O. DEGRADED means redundancy is gone, not that something is down. The next failure in the same vdev group is data loss.
  • Resilver profiles differ fundamentally. Mirror resilver copies the device with sequential I/O. RAIDZ resilver walks the block tree and reconstructs allocated blocks, which is slow, random I/O proportional to used space. On large HDD RAIDZ pools, resilvers run for hours to days, and the entire window is spent at reduced redundancy while competing with production I/O.
  • Checksums turn silent corruption into a visible signal. Every block is verified against its checksum on read and on scrub. With redundancy, ZFS repairs bad blocks transparently and increments the per-device CKSUM counter in zpool status. Without redundancy, a checksum error is data loss, and the pool stays ONLINE the whole time. This is why scrubs (the scan: line in zpool status) are the only integrity assurance you have: CKSUM of zero with no recent scrub means zero errors detected, not zero errors.

Signals to watch in production

SignalSourceWhat it tells you
TXG sync duration (stime)/proc/spl/kstat/zfs/<pool>/txgsWhether storage keeps up with writes. Sustained stime over zfs_txg_timeout (default 5s) means write saturation.
Dirty data vs limitndirty in txgs; /sys/module/zfs/parameters/zfs_dirty_data_maxHow close the write pipeline is to throttling (60% of max) or a hard stall.
ARC size, target, hit rate/proc/spl/kstat/zfs/arcstats (size, c, c_max, hits, misses)Cache effectiveness and whether memory pressure is shrinking the ARC.
memory_throttle_count/proc/spl/kstat/zfs/arcstatsI/O actively throttled by memory pressure: the pre-OOM signal.
Pool capacity and fragmentationzpool list -o name,cap,fragDistance from the allocator cliff. Trend both together.
Pool and vdev state, error counterszpool status -vDEGRADED is lost redundancy. Any non-zero READ/WRITE/CKSUM needs hardware investigation.
Scrub status and permanent errorszpool status scan line and errors sectionThe only data-integrity verification. Uncorrectable errors mean data loss has already occurred.
Per-vdev latency and queue depthzpool iostat -l -w -q -vFinds the single slow device and separates backend saturation from internal ZFS stalls.
ZIL stall/error counters/proc/spl/kstat/zfs/zilSLOG trouble. Pair with application fsync latency; no direct ZIL latency metric exists.
Deadman eventszpool eventsAn I/O or TXG sync that exceeded the deadman timers (zfs_deadman_ziotime_ms, zfs_deadman_synctime_ms). Always real, always critical.

How Netdata helps

The mechanisms above produce signals that are only meaningful in combination, and that is where continuous per-second collection pays off:

  • TXG health: Netdata tracks TXG sync duration and dirty data per pool, so a write-pipeline stall shows up as a rising trend seconds before applications start freezing, not after.
  • ARC behavior: ARC size, target, hit rate, and memory_throttle_count are charted alongside system memory, which makes the “ARC squeezed the apps” cascade visible as a single correlated view instead of two unrelated dashboards.
  • The capacity cliff: capacity and fragmentation trended together, with growth-rate context, turns the non-linear allocator cliff into a linear planning problem weeks in advance.
  • Hardware degradation: per-vdev READ/WRITE/CKSUM counters and latency histograms are retained as time series, so the slow-dying-disk pattern (counters creeping, latency diverging from peers, pool still ONLINE) is caught while redundancy is still intact.
  • Scrub and SLOG blind spots: scrub recency and results, ZIL stall counters, and SLOG device state are monitored explicitly, covering the two failure modes that leave zpool status -x saying “all pools are healthy” while data or performance is actually at risk.