$ guides / zfs / zfs-pool-fragmentation-high ▌

Operations Guides

ZFS pool fragmentation high: write amplification you cannot defragment away

You ran zpool list and the FRAG column on your production pool reads 55%, 65%, maybe higher. Writes feel slower than they used to, scrub takes longer every month, and nobody can say when it started. Now you are searching for the ZFS equivalent of defrag and discovering there is not one.

The FRAG percentage measures how scattered your pool’s free space is across metaslab spacemaps. It is not file fragmentation, and it says nothing about how fragmented already-written data is. It tells you how hard the allocator will have to work for every future write. On a pool with high fragmentation, logically sequential writes land in scattered physical locations, turning sequential workloads into random I/O at the device layer.

The uncomfortable part: ZFS has no in-place defragmenter. Scrub does not defragment. Resilver does not defragment. The only way to lower FRAG% is to rewrite the data, which in practice means zfs send | zfs recv to a fresh pool. That makes fragmentation a planning problem, not a fix-it-now problem, and it changes how you should respond.

What this means

Every vdev is divided into metaslabs. ZFS tracks free space per metaslab in spacemaps, and the FRAG value in zpool list is computed from the spacemap histograms. A low FRAG means free space exists in large contiguous ranges, so the allocator can satisfy a write with a single big allocation. A high FRAG means free space is a confetti of small gaps, so each write gets split across many small allocations in different locations.

The consequences compound:

Write amplification. One logical sequential write becomes many small physical writes to scattered offsets. On HDDs this converts sequential throughput into random I/O, the worst case for spinning media. SSDs absorb the IOPS penalty much better, but the metadata overhead and extra allocator work remain.
Slower allocation. As metaslabs fill and fragment, the allocator switches from fast first-fit selection to expensive best-fit searching. The commonly cited 80% capacity rule is a conservative approximation of this per-metaslab behavior; the actual switch happens when an individual metaslab drops below about 4% free.
Longer TXG syncs. More allocator work per write means longer transaction group sync times, which pushes the whole write pipeline toward throttling.
Slower scrubs. Fragmented pools get less sequential read during scrub, so scrub duration climbs even when data volume is flat.

Fragmentation also interacts with capacity. A pool at 40% capacity with 50% fragmentation has plenty of free space, just scattered. A pool at 85% capacity with rising fragmentation is approaching the capacity-fragmentation cliff, where allocator overhead, write amplification, and write throttling reinforce each other. That combination is the dangerous one. See the capacity and fragmentation sections of the mental model guide for the full mechanism.

flowchart TD
  A[Mixed writes, deletes, snapshot churn] --> B[Free space scatters across metaslabs]
  B --> C[FRAG percent rises in zpool list]
  C --> D[Allocator finds only small gaps]
  D --> E[Sequential writes split into random I/O]
  E --> F[Write latency and TXG stime climb]
  C --> G[Combined with capacity over 80 percent]
  G --> H[Best-fit allocation - allocator overhead cliff]
  H --> F
  F --> I[Only fix: rewrite data via send and recv to fresh pool]

Common causes

Cause	What it looks like	First thing to check
Normal COW churn over time	FRAG rising a few percent per month on an active pool, capacity stable	Trend of FRAG over 3-6 months, not a single reading
High capacity accelerating fragmentation	FRAG climbing faster as CAP passes 80%, write latency creeping up	`zpool list` CAP and FRAG side by side
Snapshot create/destroy churn	FRAG rising alongside heavy snapshot turnover; async freeing backlog	`zpool get freeing <pool>`, `zfs list -o space -r <pool>`
Small random write workload (databases, VMs)	High FRAG on a pool hosting zvols or database datasets; low FRAG on archival pools on the same hardware	Which datasets dominate writes; `zpool iostat -v` pattern
Sync-write-heavy workload without SLOG	FRAG elevated plus sync write latency problems	`zpool status` log section, ZIL stall counters in `/proc/spl/kstat/zfs/zil`

One configuration deserves special mention: logbias=throughput on a dataset without a SLOG device forces ZIL blocks into the main pool data path and is known to cause severe fragmentation with small block writes. If you inherited a pool with this set, check it early: zfs get logbias -r <pool>.

Quick checks

All read-only and safe to run during production.

# Pool-level and per-vdev fragmentation and capacity
zpool list -o name,size,alloc,free,cap,frag
zpool list -v

# Machine-parseable snapshot of the key numbers
zpool list -Hp -o name,cap,frag,freeing

# Where the space actually lives: live data vs snapshots vs reservations
zfs list -o space -r <pool> | sort -k3 -h | tail -20

# Recent TXG sync times; stime climbing alongside FRAG is the smoking gun
cat /proc/spl/kstat/zfs/<pool>/txgs | tail -20

# Write latency: is the pool actually slow, or just fragmented?
zpool iostat -l <pool> 5

# Per-vdev queue depths: fragmentation pain shows as queues without device faults
zpool iostat -q -v <pool> 5

# Check for logbias=throughput, a known fragmentation accelerator
zfs get logbias -r <pool>

Two things to note while you run these. First, a single zpool list reading tells you almost nothing; fragmentation is a trend metric. If you have no history, start recording zpool list -Hp -o name,cap,frag daily now. Second, if FRAG shows as - instead of a number, the spacemap_histogram feature is disabled on that pool; check with zpool get feature@spacemap_histogram <pool>. The feature@<feature_name> property format is used by OpenZFS 0.8.0 and later.

How to diagnose it

The goal is to answer three questions: is fragmentation actually hurting you, how fast is it growing, and are you near the cliff.

Establish the current state. Record CAP, FRAG, and FREE for the pool and each top-level vdev (zpool list -v). Per-vdev values matter because one badly fragmented vdev in a multi-vdev pool drags the whole pool.
Confirm real impact, not just a scary number. Correlate FRAG with write path signals:
- TXG stime from /proc/spl/kstat/zfs/<pool>/txgs trending upward over weeks
- Write total_wait in zpool iostat -l above baseline with syncq_wait and asyncq_wait both elevated
- Scrub duration increasing month over month on flat data volume
If FRAG is 60% but write latency, TXG stime, and scrub duration are all at baseline, you have a metric problem, not an incident. This is common on read-mostly and archival pools, where fragmentation barely matters.
Measure the rate of change. The rate matters more than the absolute value. A pool that went from 20% to 45% FRAG in six months will hit the degradation zone within a year even if nothing hurts today. A pool stable at 55% for two years has reached workload equilibrium. If you have no history, take two readings four weeks apart before concluding anything.
Check the capacity interaction. FRAG above 50% with CAP below 60% is a watch item. FRAG above 50% with CAP above 80% is the composite cliff pattern: allocator overhead plus write amplification plus approaching slop space. If you are in this zone, capacity relief (snapshot pruning, expansion) buys more than anything else.
Identify the workload driver. Small random writes (zvols, databases, VM images) fragment far faster than large sequential writes. Heavy snapshot create/destroy cycles also scatter free space. Use zfs list -o space -r <pool> and your knowledge of what runs on the pool to decide whether the fragmentation rate is inherent to the workload or caused by something fixable like logbias=throughput or a missing SLOG on a sync-heavy pool.
Decide: tolerate, slow, or rebuild. On SSDs, fragmentation is largely tolerable; the IOPS penalty is small and only the metadata overhead remains. On HDDs with a write-heavy workload and a rising trend, start planning the rebuild now, because the rebuild itself takes time and the pool will keep getting worse while you plan.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
FRAG% (`zpool list -o frag`)	The fragmentation metric itself; trend, not snapshot	Rising more than about 5% per month without mitigation; over 50% on write-heavy pools
CAP% alongside FRAG	Fragmentation accelerates as the pool fills; the two together define the cliff	CAP over 80% with FRAG rising
TXG stime (`/proc/spl/kstat/zfs/<pool>/txgs`)	Allocator overhead shows up here first as longer syncs	stime consistently above the 5s `zfs_txg_timeout`, trending up over weeks
Write latency (`zpool iostat -l`)	Confirms whether fragmentation has real user impact	Write `total_wait` over 2x rolling baseline sustained
Queue depth (`zpool iostat -q`)	Deep pending queues with healthy devices points at allocation cost, not hardware	Sustained pending much greater than active on data vdevs
Scrub duration	Degrades as the pool loses sequential read layout	Duration up 50%+ on flat data volume
`freeing` property	Async reclaim backlog from snapshot churn feeds fragmentation	Large `freeing` value persisting between checks

Fixes

There is no quick fix. In order of increasing effort:

Do nothing, deliberately

Valid when: the pool is on SSDs, write latency and TXG stime are at baseline, the workload is read-heavy or archival, or the FRAG trend is flat. Fragmentation on a read-mostly pool costs almost nothing because FRAG measures free space, and free space only matters to writes. Document the decision and set a trend alert so “do nothing” does not become “never look again.”

Tradeoff: you are betting the trend stays flat. If the workload changes (new database, new VM storage), the existing fragmentation amplifies the new write load immediately.

Relieve capacity pressure

If CAP is above 80%, freeing space is the highest-leverage action even though it does not lower FRAG. More total free space gives the allocator more options and delays the best-fit cliff. Prune old snapshots (zfs list -t snapshot -o name,used -s used to find the big ones), archive cold datasets, or expand the pool by adding vdevs.

Tradeoff: deleting data and snapshots does not defragment the freed space. The holes are still holes. This buys time, it does not fix the layout. Snapshot deletion also frees space asynchronously; watch the freeing property rather than expecting instant relief.

Fix fragmentation accelerators

Remove the things making it worse: unset logbias=throughput on datasets without a SLOG, add a SLOG for sync-heavy workloads so ZIL writes stop scattering across the pool, and keep OpenZFS current. OpenZFS 2.1 and later pin ZIL allocations to a dedicated metaslab, which significantly reduces fragmentation from synchronous write workloads. If you are on an older release with a sync-heavy pool, upgrading is a genuine mitigation.

Tradeoff: none of these reduce existing FRAG. They flatten the growth curve going forward.

Rebuild the pool (the only real fix)

The canonical defragmentation: zfs send | zfs recv every dataset to a fresh pool, then swap the new pool into service. The receive side allocates blocks sequentially as the stream arrives, so the new pool starts with FRAG near zero. In practice this means new hardware or new vdevs large enough to hold the data, a maintenance window for the final incremental send and cutover, and careful replication of dataset properties, snapshots, and mounts.

Tradeoffs you must plan for:

You need full additional capacity for the duration of the migration.
Send/recv of a multi-terabyte pool takes hours to days and competes with production I/O while it runs.
Snapshots, holds, clones, and dataset properties must all be carried over deliberately; a naive send loses them.
The received pool can show slightly different (sometimes higher) FRAG than expected immediately after receive, because allocation order during receive is not identical to the original.
Resilvering onto replacement disks does not achieve this. Resilver reconstructs blocks as they were, preserving the layout, gang blocks and all.

OpenZFS 2.3.4 and later ship a zfs rewrite subcommand that rewrites specified files into fresh block allocations without changing content. It may improve per-file contiguity, but it is not a pool-wide defragmenter; measure pool FRAG rather than assuming it drops.

Prevention

Fragmentation is cumulative and mostly irreversible, so the entire game is slowing the growth rate and seeing it early.

Trend FRAG and CAP together, forever. Record zpool list -Hp -o name,cap,frag into your time-series store at least daily. Alert on rate of change (more than ~5% FRAG per month) rather than absolute thresholds.
Respect the capacity ceiling. Keep write-heavy pools below 80% capacity. Fragmentation growth accelerates with fill level; capacity discipline is fragmentation discipline.
Match pool design to workload. Put small random write workloads (zvols, databases) on their own pools or on mirrors rather than sharing a RAIDZ pool with archival data. Mirrors fragment more gracefully and resilver sequentially.
Use a SLOG for sync-heavy workloads and never set logbias=throughput without one.
Stay current on OpenZFS. Allocator improvements (like the ZIL metaslab pinning in 2.1) are cumulative fragmentation mitigations you get for free.
Plan the rebuild before you need it. If your trend lines say the pool crosses into the degradation zone in 18 months, the hardware refresh conversation starts now, not during the incident.

How Netdata helps

Fragmentation is a slow-motion problem, which makes it exactly the kind of signal point-in-time zpool status checks miss. Useful correlations:

FRAG% and CAP% trended together over months, so the rate of fragmentation growth is visible and alertable before the pool enters the degradation zone.
TXG sync duration (stime) alongside FRAG, separating “fragmented but fine” from “fragmented and paying for it” at a glance.
Write latency histograms correlated with capacity milestones, catching the non-linear allocator cliff as it develops rather than after users complain.
Scrub duration history, where a steady month-over-month increase on flat data volume is an early fragmentation symptom.
Per-vdev I/O and queue depth to confirm that write slowness is allocation cost spread across healthy devices, not a single dying disk masquerading as a pool problem.