How ZFS actually works in production: a mental model for operators

Every ZFS incident traces back to a small set of internal mechanisms. A write freeze at 3 a.m. is the transaction group pipeline stalling. A server that “runs out of memory” with gigabytes free is the ARC doing exactly what it was designed to do. A pool that was fine at 78% full and unusable at 88% is the metaslab allocator crossing a threshold, not a disk dying.

This article is the mental model layer: what ZFS is doing internally at all times, which resources it competes for, and why its characteristic failures look the way they do. It is the prerequisite for the operational runbooks. You cannot interpret a TXG sync time, an ARC hit ratio, or a fragmentation percentage without knowing the machinery that produces those numbers.

The examples use OpenZFS on Linux paths (/proc/spl/kstat/zfs/...). On FreeBSD and illumos the subsystems are the same but kstat paths, tunable names, and some defaults differ.

What ZFS is and why it matters

ZFS is a combined filesystem and volume manager. It owns the disks directly, manages redundancy (mirrors, RAIDZ, dRAID) without a hardware RAID controller, and layers datasets, snapshots, compression, checksumming, and optional encryption on top. Because it controls the whole stack from filesystem semantics down to block allocation, its failure modes are also end-to-end: a capacity problem becomes an allocator problem becomes a latency problem becomes an application outage, all inside one subsystem.

The operator consequence: you cannot monitor ZFS the way you monitor ext4 on LVM. Pool state, cache behavior, write batching, and physical layout are all first-class signals, and the interesting failures are almost always interactions between them.

Copy-on-write: the decision that drives everything

ZFS never overwrites existing data. Every write lands in a new location on disk, and the block pointers above it are updated atomically up the tree to the uberblock. This single decision cascades into nearly every operational behavior:

• Snapshots are cheap to take, expensive to hold. A snapshot just pins the old block pointers. The cost arrives later, when the live dataset diverges and the snapshot holds references to blocks that have been logically deleted. Deleting files does not free pool space while a snapshot references those blocks.
• There is no in-place defragmentation. Because data is always written to new locations, free space scatters over time. zpool scrub verifies checksums; it does not defragment anything. The only “defrag” is zfs send | zfs recv into a fresh pool.
• Deletes need free space too. Deleting files or snapshots is itself a COW operation that must write new metadata. A pool that is completely full can enter a state where freeing space requires space. ZFS reserves slop space (about 3.125% of pool size, with a per-pool cap of 128 GiB added in OpenZFS 2.1) for exactly this reason, and user writes fail before that reserve is consumed.
• Crash consistency is structural. Since nothing is ever overwritten in place, a crash cannot leave a half-written block tree. There is no fsck. The ZIL (below) covers the synchronous-write window.

Transaction groups: the write path is a batch system

ZFS does not write when your application writes. It accumulates writes in memory and commits them to disk in transaction groups (TXGs), flushed periodically: default every 5 seconds, controlled by zfs_txg_timeout. Three TXGs are always in flight:

• Open: accepting new writes.
• Quiescing: finalizing, no longer accepting writes.
• Syncing: being written to stable storage.

This is why ZFS write latency is bimodal. Most writes complete at memory speed. The pain arrives when the syncing TXG cannot finish within the timeout: the open TXG keeps accumulating dirty data, the next sync is even bigger, and the cycle compounds. ZFS defends itself with a dirty data limit (zfs_dirty_data_max): a throttle engages at zfs_delay_min_dirty_percent of that limit (default 60%) and a hard stall at the limit itself. Applications experience this as sudden write freezes lasting seconds to tens of seconds while reads from cache continue normally.

You can watch this directly. Each pool keeps a TXG history at /proc/spl/kstat/zfs/<pool>/txgs; the stime field is the sync phase duration in nanoseconds and ndirty is the dirty bytes per TXG. If stime consistently exceeds zfs_txg_timeout, the storage cannot keep up with the write rate.

The ARC: a read cache that lies to free

The Adaptive Replacement Cache is ZFS’s primary read cache, in kernel memory. It maintains MRU and MFU lists plus their ghost counterparts, and shifts the balance between recently-used and frequently-used blocks based on workload. It caches both data and metadata, and it grows and shrinks in response to memory pressure.

Two operational facts matter more than the algorithm:

1. On Linux, ARC memory lives outside the kernel page cache. It appears as “used” (in slab) in free and top, but it is reclaimable. Operators routinely misdiagnose a healthy system as memory-starved because of this. The ground truth is /proc/spl/kstat/zfs/arcstats: size (current), c (dynamic target), c_min, c_max, plus hits and misses.
2. Uncapped ARC plus other memory-hungry processes is an OOM cascade. The Linux OOM killer does not reliably account for ARC as reclaimable, and ARC does not shrink instantly under sudden pressure. On shared machines, set zfs_arc_max explicitly. The memory_throttle_count field in arcstats increments when I/O is being throttled due to memory pressure: that is your early warning.

Two optional extensions attach to the ARC. The L2ARC is an SSD-backed second tier fed by ARC evictions; it needs warmup (a cold L2ARC provides no benefit), it persists across reboots by default since OpenZFS 2.0 (l2arc_rebuild_enabled=1), and its index consumes ARC RAM, roughly 70 bytes per cached block, so an oversized L2ARC on a RAM-constrained host can be a net negative. If deduplication is enabled, the dedup table (DDT) lives in the ARC at roughly 320 bytes per block; a DDT that outgrows the ARC forces every write into random disk lookups for table entries and is one of the classic unrecoverable performance cliffs.

The ZIL and SLOG: underwriting synchronous writes

The TXG pipeline is asynchronous. Anything that demands synchronous semantics (O_SYNC, fsync, NFS, database commit logs) cannot wait up to 5 seconds for the next TXG, so ZFS writes an intent record to the ZFS Intent Log before acknowledging the write. The ZIL guarantees that if the system crashes between the acknowledgment and the TXG commit, the intent can be replayed on pool import.

Operationally:

• In normal operation the ZIL is write-only. It is only read during import after an unclean shutdown. Once the TXG commits, the ZIL space is reclaimed.
• Without a SLOG, the ZIL lives on the pool’s data vdevs. Sync write latency becomes pool latency: tens of milliseconds on spinning disk.
• A SLOG (separate log device) moves the ZIL to a dedicated fast device. Sync write latency becomes SLOG latency. This is the entire reason SLOGs exist, and they only help synchronous workloads; an all-async workload gets nothing from one.
• SLOG failure is a performance event, not a data event. ZFS falls back to the pool-based ZIL. The pool stays ONLINE, zpool status -x says everything is healthy, and your database’s commit latency just jumped by one or two orders of magnitude. Mirror your SLOG; a lone SLOG that dies concurrent with a crash can lose acknowledged writes.
• There is no ZIL latency metric. The kstat at /proc/spl/kstat/zfs/zil exposes commit, writer, stall, and error counters, but latency must be inferred from application-level fsync timing.

The SPA and metaslab allocator: the capacity cliff is real

The Storage Pool Allocator owns physical allocation, vdev topology, redundancy, and I/O scheduling. Each top-level vdev is divided into metaslabs, fixed-size chunks, and every allocation picks a metaslab and carves space out of it.

This is the mechanism behind the most infamous ZFS failure: the capacity-fragmentation cliff. When an individual metaslab has only a few percent of free space left, its allocator switches from the fast first-fit path to an expensive best-fit scan below metaslab_df_free_pct=4% of metaslab free space, and fragmented free space forces sequential writes into scattered physical locations. The effect is non-linear: the pool looks fine at 75%, starts degrading through the 80s, and falls apart above 90%. The commonly cited “keep it under 80%” rule is a conservative proxy for per-metaslab behavior, which is why the actual pain threshold varies with workload and vdev size. Free space is tracked in space maps, log-structured allocation records, which on heavily fragmented pools also grow large and slow down pool import.

Two practical implications. First, monitor capacity and fragmentation together (zpool list -o name,cap,frag): a pool at 70% with 50% fragmentation is already degraded for write-heavy workloads. Second, fragmentation cannot be fixed in place; the remediation is send/recv to a new pool, so the only real strategy is not getting there.

The DMU and ZIO pipeline: where every block is processed

Between the filesystem semantics and the disks sit two layers. The Data Management Unit (DMU) translates logical objects (files, directories, zvols) into blocks and handles the operations applied to them: compression, checksumming, encryption, copy-on-write, and dnode management. The ZIO pipeline then moves every block through a staged pipeline: issue, wait, checksum, compress, encrypt, vdev I/O, done. Each stage queues independently with its own thread pool.

flowchart TD
  A[Application write] --> B[DMU: block mapping]
  B --> C{Sync write?}
  C -- yes --> D[ZIL / SLOG: record intent]
  C -- no --> E[Open TXG: accumulate dirty data]
  D --> E
  E --> F[TXG quiescing]
  F --> G[TXG syncing]
  G --> H[ZIO pipeline: checksum, compress, encrypt]
  H --> I[Vdevs: mirror / RAIDZ / dRAID]
  J[Application read] --> K{In ARC?}
  K -- yes --> L[Serve from RAM]
  K -- no --> I

This pipeline explains where CPU goes on a ZFS host: checksumming, compression, and encryption run on every block, on every read and every write. On a CPU-bound system, adding faster disks changes nothing. It also explains reported-versus-actual throughput: with compression on (the default), the bytes ZFS reports and the bytes the disks see are not the same number, and write amplification from metadata, parity, and COW overhead of 2-3x on small random writes to RAIDZ is structural, not a symptom.

Vdev redundancy: what failure actually looks like

The SPA builds redundancy from vdevs: mirrors, RAIDZ1/2/3, or dRAID (OpenZFS 2.0+). The operational differences that matter:

• Failure is masked. A dead disk in a mirror or RAIDZ leaves the pool DEGRADED and serving I/O. DEGRADED means redundancy is reduced, not that I/O is down. The next failure in the same vdev group is data loss.
• Resilver profiles differ fundamentally. Mirror resilver copies the device with sequential I/O. RAIDZ resilver walks the block tree and reconstructs allocated blocks, which is slow, random I/O proportional to used space. On large HDD RAIDZ pools, resilvers run for hours to days, and the entire window is spent at reduced redundancy while competing with production I/O.
• Checksums turn silent corruption into a visible signal. Every block is verified against its checksum on read and on scrub. With redundancy, ZFS repairs bad blocks transparently and increments the per-device CKSUM counter in zpool status. Without redundancy, a checksum error is data loss, and the pool stays ONLINE the whole time. This is why scrubs (the scan: line in zpool status) are the only integrity assurance you have: CKSUM of zero with no recent scrub means zero errors detected, not zero errors.

Signals to watch in production

How Netdata helps

The mechanisms above produce signals that are only meaningful in combination, and that is where continuous per-second collection pays off:

• TXG health: Netdata tracks TXG sync duration and dirty data per pool, so a write-pipeline stall shows up as a rising trend seconds before applications start freezing, not after.
• ARC behavior: ARC size, target, hit rate, and memory_throttle_count are charted alongside system memory, which makes the “ARC squeezed the apps” cascade visible as a single correlated view instead of two unrelated dashboards.
• The capacity cliff: capacity and fragmentation trended together, with growth-rate context, turns the non-linear allocator cliff into a linear planning problem weeks in advance.
• Hardware degradation: per-vdev READ/WRITE/CKSUM counters and latency histograms are retained as time series, so the slow-dying-disk pattern (counters creeping, latency diverging from peers, pool still ONLINE) is caught while redundancy is still intact.
• Scrub and SLOG blind spots: scrub recency and results, ZIL stall counters, and SLOG device state are monitored explicitly, covering the two failure modes that leave zpool status -x saying “all pools are healthy” while data or performance is actually at risk.

Related guides

• ZFS monitoring checklist: the signals every production pool needs
• ZFS monitoring maturity model: from survival to expert
• ZFS pool DEGRADED: redundancy lost and one failure from data loss
• ZFS pool I/O is currently suspended: a hung pool and blocked I/O
• ZFS pool FAULTED: when the pool can no longer serve I/O
• ZFS device FAULTED - too many errors: a disk ejected from the pool
• ZFS READ and WRITE errors: transport-level device failures in zpool status
• ZFS device UNAVAIL or REMOVED: a disk that fell off the bus
• ZFS pool ONLINE with non-zero errors: why zpool status -x lies
• ZFS permanent errors have been detected in the following files: recovering from data loss
• ZFS checksum errors (CKSUM): the definitive signal of silent corruption
• ZFS scrub not running: ‘CKSUM 0’ means nothing without regular scrubs