ZFS capacity cliff: why the pool falls off a performance edge near 80-90% full

Your ZFS pool was fine last month. This week write latency is spiking, TXG syncs are stretching past their timeout, and applications are stalling on writes while reads still feel snappy. You check zpool list and see CAP at 87%. Nothing failed. No disk died. The pool just crossed a threshold it was never going to tell you about.

This is the ZFS capacity cliff, and it is the single most common ZFS incident in production. It is not a linear slowdown where you lose a few percent of throughput per percent of capacity. It is an algorithmic cliff: performance looks acceptable until it suddenly is not, and the distance between “fine” and “writes stalling for seconds” can be a few percentage points of pool capacity.

The mechanism is well understood and the signals are visible well in advance. The catch: if you have been running hot for a long time, some of the damage (free-space fragmentation) does not heal when you free space.

What this means

ZFS is copy-on-write. Every write goes to a new location on disk; nothing is overwritten in place. That design is what gives you checksums, snapshots, and crash consistency, but it means every write must first find free space. Finding free space is the job of the metaslab allocator.

Each top-level vdev is divided into metaslabs, fixed-size chunks that are loaded and unloaded as the allocator works through them. While a metaslab has comfortable free space, the allocator uses a fast first-fit strategy: find a gap, take it, move on. When an individual metaslab’s free space drops below roughly 4% (the metaslab_df_free_pct threshold), that metaslab flips to a best-fit strategy: the allocator searches much harder to place allocations in a way that limits further fragmentation. Best-fit is dramatically more CPU-intensive and produces worse I/O patterns.

Two consequences follow:

The trigger is per-metaslab, not per-pool. A pool at 82% CAP can already have many metaslabs below the 4% free threshold while others are half empty. The widely cited “keep it under 80%” rule is a conservative approximation of this per-metaslab behavior, not the actual mechanism.
The degradation is non-linear. Each metaslab that flips to best-fit makes allocations more expensive. As more metaslabs flip, TXG syncs stretch, dirty data accumulates, and ZFS starts throttling writers. The feedback loop accelerates as you get closer to full.

flowchart TD
  A[Write enters TXG] --> B[Metaslab allocator picks target]
  B --> C{Metaslab free space}
  C -->|above ~4% free| D[First-fit allocation - fast]
  C -->|below ~4% free| E[Best-fit allocation - CPU intensive]
  D --> F[Normal write latency]
  E --> G[Slower allocation, more scatter]
  G --> H[TXG sync time stretches]
  H --> I[Dirty data accumulates, writer throttling]
  I --> J[Application write stalls]

Near the hard wall there is a second, nastier effect: the death spiral. Deletion in ZFS is also a copy-on-write operation that needs free space for metadata updates, and space reclamation after snapshot destruction is asynchronous. Past ~90%, writes and deletes both slow down, so you cannot efficiently free space because freeing space requires space and I/O bandwidth that are both exhausted. ZFS holds back a slop space reserve (1/32 of pool size, about 3.125%, floor 128 MiB, capped at 128 GiB since OpenZFS 2.0.6) to keep administrative operations possible, but if you routinely fill past it, you are betting on that reserve.

Common causes

Cause	What it looks like	First thing to check
Organic data growth with no capacity plan	CAP creeping up month over month, cliff arrives “suddenly”	`zpool list -H -o name,size,alloc,free,cap,freeing`
Snapshot retention out of control	Pool full but live datasets are not; deleting files frees nothing	`zfs get -r usedbysnapshots <pool>`
Free-space fragmentation from long-running churn	FRAG high even at moderate CAP; write-heavy pool that has been hot for years	`zpool list -H -o name,frag`
Special vdev full, fallback to main pool	Metadata/small-block writes slow while pool CAP looks comfortable	`zpool get class_special_capacity,class_special_free,class_special_fragmentation <pool>` where available; `zpool list -v` is the portable fallback
Write surge pushing an already-warm pool over the edge	Bulk load, rsync, or replication catch-up triggers the cliff	TXG `stime` in `/proc/spl/kstat/zfs/<pool>/txgs`

The class_special_* properties require OpenZFS support after the 2.4.4 release line; older releases rely on the per-class rows in zpool list -v or on zdb.

Quick checks

# Pool capacity, freeing backlog, and fragmentation in one shot
zpool list -H -o name,size,alloc,free,cap,freeing

# Fragmentation, pool-level and per-vdev
zpool list -H -o name,frag
zpool list -v

# Where the space actually is: live data vs snapshots vs reservations
zfs list -o space -r <pool>

# 20 largest snapshots (ascending order, largest last)
zfs list -t snapshot -o name,used,refer -s used -r <pool> | tail -20

# Snapshot-held space per dataset
zfs get -r usedbysnapshots <pool>

# Recent TXG sync times (stime is in nanoseconds)
cat /proc/spl/kstat/zfs/<pool>/txgs | tail -20

# Write latency and queue state
zpool iostat -l <pool> 5
zpool iostat -q -v <pool> 5

All of these are read-only and safe to run during an incident. Two notes: the freeing column tells you whether async reclaim from recent deletions is still in progress (a large value means space is coming back, slowly), and the available property per dataset (zfs get available <dataset>) is what actually gates new writes because it accounts for quotas, reservations, and slop space. Pool-level FREE does not.

How to diagnose it

Confirm you are on the cliff and not somewhere else. Check CAP and FRAG together. CAP above 85% with rising write latency is the classic signature. A pool at 70% CAP with 60% FRAG on a write-heavy workload can show the same symptoms, because the allocator is already struggling to find contiguous space.
Verify the write path, not the disks. Pull recent TXG entries from /proc/spl/kstat/zfs/<pool>/txgs. If stime (sync duration, nanoseconds) is consistently exceeding zfs_txg_timeout (default 5 seconds) and there is no scrub, resilver, or large snapshot destruction running, the allocator is the prime suspect. Per-vdev latency from zpool iostat -l -v rules out a single slow device.
Check whether deletes are also slow. This is the distinguishing feature of the death spiral. If writes are slow but deletes and snapshot destruction complete normally, suspect fragmentation or a device problem. If both writes and deletes crawl and CAP is above 90%, you are in the spiral.
Find the space. Use zfs list -o space -r <pool> to split usage into live data (USEDDS), snapshots (USEDSNAP), reservations (USEDREFRESERV), and children. The response is completely different depending on which one dominates.
Check the special vdev if you have one. A full special allocation class silently falls back to the main pool for metadata and small blocks. Pool-level capacity looks fine while the hidden bottleneck throttles everything. Check class_special_free and class_special_fragmentation where available, or the special class rows in zpool list -v.
Estimate runway. Runway (days) = free bytes / daily allocation rate. Use the actual growth trend, and account for snapshot holds: deleting files does not free space that snapshots still reference.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Pool CAP (`zpool list`)	Distance from the cliff edge	Above 85% on write-heavy pools; growth projecting under 7 days to 96%
Pool FRAG (`zpool list -o frag`)	How hard the allocator must work	Above 50% on write-heavy pools, or rising more than 5% per month
TXG `stime`	Direct measure of write-path cost	Consistently above 2x `zfs_txg_timeout` (default 5s), or trending upward
Write latency (`zpool iostat -l`, `-w`)	What applications actually feel	p95/p99 diverging from baseline; total_wait climbing
`freeing` property	Async reclaim backlog	Large value persisting; reclaim not keeping up with deletes
`usedbysnapshots` per dataset	Snapshot-driven capacity pressure	Snapshot space above 50% of pool allocation near the capacity threshold
Special vdev free/fragmentation	Hidden bottleneck on metadata	`class_special_free` near zero, where the property is available

One critical caveat: FRAG measures free-space fragmentation, not data fragmentation, and it is a pool-level average derived from metaslab space-map histograms. Individual metaslabs can be far worse than the average. For per-metaslab detail, zdb -mmm <pool> shows the histograms, but treat zdb output carefully on a loaded production pool.

Fixes

Free space the right way

Destroy the largest expendable snapshots first. zfs list -t snapshot -o name,used -s used -r <pool> ranks them. Snapshot destruction triggers asynchronous reclaim, so watch the freeing property rather than expecting instant recovery. Note that a snapshot’s used is only the space unique to it; intermediate snapshots share references.
Throttle application writes during recovery. If the pool is near the spiral, every writer competes with the async destroy thread for the same exhausted I/O bandwidth. Reducing foreground write load lets reclaim finish faster.
Do not expect zpool scrub to help. Scrubs verify checksums. They do not defragment, and they add I/O load to an already struggling pool. Postpone scheduled scrubs while you are recovering.

Expand the pool

Adding vdevs is the only way to genuinely raise the ceiling. Be aware that existing data is not rebalanced onto new vdevs automatically; new writes favor the emptier vdevs, so balance improves over time as data churns. Plan expansion at 75% CAP, not at 90%.

The honest fix for fragmentation damage

If the pool has run hot for a long time and FRAG is above ~70%, freeing space will stop the bleeding but will not restore the old write performance. Free-space fragmentation is not self-healing, and ZFS has no defragmentation tool. The only real fix is zfs send | zfs recv into a fresh pool, which rebuilds the data layout from scratch. This is disruptive and requires a second pool’s worth of capacity, which is exactly why the prevention section matters more than this one.

Prevention

Plan at 75%, act at 85%, treat 96% with active writes as an emergency. These breakpoints reflect the allocator’s behavior, not arbitrary caution. Pools with heavy random write workloads degrade earlier; mostly-sequential workloads tolerate slightly more.
Trend CAP and FRAG together. Capacity alone hides the fragmentation story; fragmentation alone hides the growth story. The composite is what predicts the cliff.
Alert on growth rate, not just level. A pool growing toward 96% in under 7 days is a ticket even if it is currently at 70%.
Keep snapshot retention deliberate. Automated snapshot jobs without pruning are the most common way pools fill with invisible data. Track usedbysnapshots per dataset.
Do not treat ZFS like ext4. Running a general-purpose filesystem at 95% is routine. Running ZFS at 95% is an incident in progress. Budget storage accordingly.
Monitor the special vdev separately if you use one; its failure mode is invisible in pool-level metrics.

How Netdata helps

Netdata collects pool CAP, ALLOC, FREE, and fragmentation continuously, so the slow drift toward the cliff is visible months before the latency spike, not during it.
Pool state and per-disk I/O latency are tracked side by side, which lets you distinguish “allocator struggling” (latency rising across all vdevs as CAP climbs) from “single slow device” (one vdev diverging from the rest).
Capacity growth-rate trending turns the 75% planning threshold into an automatic runway estimate, so expansion is scheduled work instead of an emergency.
Anomaly detection on write latency catches the non-linear onset of the cliff, which static thresholds on averages tend to miss until it is well underway.