ZFS snapshot destroy slow: async destroy, the freeing property, and I/O contention

You ran zfs destroy on a large snapshot or a dataset full of snapshots. The command returned in seconds, but zpool list shows the space did not come back, write latency is climbing, and applications are starting to complain. Or the destroy itself hung, and now every zfs and zpool command against that pool is stuck in D state.

Both symptoms have the same root: snapshot destruction in ZFS is not a single operation. The command deletes the snapshot’s metadata quickly, but the actual block freeing runs asynchronously in the background, competing with production I/O for disk bandwidth, CPU, and dirty-data budget. On a busy or near-full pool, that background work can take hours and slows everything else while it runs.

The backlog is visible, measurable, and tunable. The pool property freeing tells you exactly how much space is still waiting to be reclaimed, and a small set of module parameters controls how aggressively ZFS spends I/O reclaiming it.

What this means

ZFS is copy-on-write. Destroying a snapshot means walking its block tree and marking every block that is no longer referenced as free. On a snapshot that references hundreds of gigabytes or terabytes, that is a lot of block pointer work, and each free is itself a metadata write that must go through the normal transaction group (TXG) pipeline.

With the feature@async_destroy pool feature (enabled by default on pools created with OpenZFS), zfs destroy does not wait for this work. It records the blocks to be freed in an on-disk list (the free bpobj) and returns. A background thread then works through that list, spending a bounded amount of time freeing blocks in each TXG commit.

Consequences an operator needs to internalize:

• Space returns gradually, not instantly. zpool get freeing <pool> shows the outstanding backlog. Over time freeing decreases while free increases. If you destroyed 2 TB and freeing says 1.4 TB, reclaim is roughly a third done.
• The backlog survives export/import. The free list is on-disk state. Rebooting or re-importing the pool does not cancel it; freeing resumes where it left off.
• Freeing is not free. Every freed block dirties metadata. On a near-full pool, those metadata writes consume the same dirty-data budget and disk bandwidth your foreground writes need. This is how a “cleanup” operation makes the capacity death spiral worse before it makes it better.

flowchart TD
  A[zfs destroy on large snapshot] --> B[metadata deleted, command returns]
  B --> C[blocks queued in on-disk free list]
  C --> D[async destroy thread frees blocks per TXG]
  D --> E[metadata writes consume dirty-data budget]
  D --> F[extra I/O competes with foreground]
  E --> G[write latency rises, TXG sync extends]
  F --> G
  G --> H[apps slow while freeing drains]
  D --> I[freeing property counts down to 0]

Common causes

Quick checks

All read-only and safe to run during an incident.

# How much space is still waiting to be reclaimed?
zpool get freeing <pool>

# One line: capacity and freeing together
zpool list -H -o name,size,alloc,free,cap,freeing

# Is a scrub or resilver also competing for I/O right now?
zpool status <pool> | grep -A3 "scan:"

# Recent TXG sync times (stime column is sync duration in ns;
# the ndirty column is current dirty bytes)
cat /proc/spl/kstat/zfs/<pool>/txgs | tail -20

# Current async destroy tunables
cat /sys/module/zfs/parameters/zfs_free_min_time_ms
cat /sys/module/zfs/parameters/zfs_per_txg_dirty_frees_percent

# Per-vdev latency and queue depth while freeing runs
zpool iostat -l -q -v <pool> 5

# Dirty data limit; compare against ndirty from the txgs kstat above
cat /sys/module/zfs/parameters/zfs_dirty_data_max

What to look for:

• freeing decreasing steadily: reclaim is healthy. The destroy is not stuck; it is just not done.
• freeing static for many minutes on a pool with active writes: freeing is being starved or throttled, or you are hitting a stall condition.
• TXG stime consistently above the 5-second zfs_txg_timeout default while freeing runs: the write pipeline is saturated and foreground writes are queuing behind reclaim I/O.

How to diagnose it

1. Confirm the backlog exists. zpool get freeing <pool>. If freeing is 0, your slowness is not async destroy; look elsewhere (TXG sync pressure, a slow vdev, scrub contention).
2. Measure the drain rate. Sample freeing twice, a few minutes apart. Divide the delta by the interval to get reclaim throughput in bytes/sec. Divide the remaining backlog by that rate for a rough ETA. OpenZFS 2.0+ also provides zpool wait -t free <pool>; with an interval, it prints remaining work in bytes. Run zpool wait -h on your release to check its supported options.
3. Check pool capacity. zpool list -o name,cap,freeing. Above 85% CAP, async destroy competes with a metaslab allocator that is already working hard. Above ~96%, you are in the emergency zone where reclaim and foreground writes fight over slop-adjacent free space.
4. Check what else is running. Scrubs, resilvers, and zfs send all consume the same I/O budget. zpool status shows scan activity. A destroy during a scrub window will look far worse than the same destroy on an idle pool.
5. Look at the write pipeline, not just the disks. TXG stime in /proc/spl/kstat/zfs/<pool>/txgs and dirty data (ndirty) versus zfs_dirty_data_max tell you whether foreground writes are being throttled. If zpool iostat -q shows growing pending queues on data vdevs while freeing drains, the disks are the contention point. If queues are flat but latency is up, the contention is inside ZFS (CPU, locks, dirty-data budget).
6. If everything is hung, not just slow: all zfs/zpool commands on the pool stuck in D state after launching many concurrent destroys matches a reported async destroy stall whose documented unblock is setting zfs_free_min_time_ms to 0 (OpenZFS issue #12697). See Fixes below.
7. Check for dedup. On a dedup=on pool, every freed block may require a DDT update. Destroys are slower and zfs_max_async_dedup_frees (default 250,000) caps DDT frees per TXG. There is no fast path here; plan for it.

Metrics and signals to monitor

Fixes

Let it drain, on a schedule you choose

If the pool has headroom (CAP under ~80%) and applications are not measurably impacted, the correct fix is usually patience. Async destroy is throttled by design so that it does not monopolize the pool. Watch freeing trend to zero and confirm latency returns to baseline. Do not reboot or export to “clear” it: the backlog is on-disk and resumes after import, and you will have added an import to your problem.

Slow the reclaim down to protect foreground I/O

zfs_free_min_time_ms is the minimum number of milliseconds per TXG commit the freeing thread spends freeing blocks (default 1000 on OpenZFS 0.6 through 2.4.0; 500 since 2.4.1). Lowering it gives each TXG back to foreground work sooner, at the cost of a slower drain:

# Gentler reclaim during business hours (runtime-only; resets on reboot)
echo 200 > /sys/module/zfs/parameters/zfs_free_min_time_ms

For the full hang case (all pool commands in D state after mass concurrent destroys), the reported unblock is setting it to 0, which stops the freeing thread from consuming TXG time; restoring the default resumes cleanup:

# Emergency unblock only - stops async destroy from consuming TXG time
echo 0 > /sys/module/zfs/parameters/zfs_free_min_time_ms
# Once the pool is responsive, restore the release default
# (500 on OpenZFS 2.4.1+; 1000 on 0.6 through 2.4.0)
echo 500 > /sys/module/zfs/parameters/zfs_free_min_time_ms

This is a runtime module parameter change, not a destructive operation, but treat 0 as a temporary measure: with it set, the freeing backlog does not drain.

Speed the reclaim up when the pool can afford it

zfs_per_txg_dirty_frees_percent controls what percentage of the dirty-data budget (zfs_dirty_data_max) frees may consume in one TXG. Older releases shipped a conservative default of 5, which caused abysmal delete performance under load; OpenZFS 2.2.0 raised the stable-series default to 30. If you are on an affected version and the pool has I/O headroom, raising it makes reclaim finish sooner:

# Check first, then raise if you are on the old default
cat /sys/module/zfs/parameters/zfs_per_txg_dirty_frees_percent
echo 30 > /sys/module/zfs/parameters/zfs_per_txg_dirty_frees_percent

Do not raise this on a near-full pool. Frees consuming more dirty-data budget means foreground writes get throttled harder.

Near-full pool: relieve capacity pressure first

If CAP is above ~90%, async destroy is fighting the metaslab allocator for scraps. The death-spiral failure mode is that reclaiming space requires space and I/O, both of which are exhausted. In order:

1. Reduce or pause foreground write load where possible. Every write you remove is I/O reclaim can use.
2. Destroy a few large, old snapshots rather than everything at once. Each completed reclaim adds real free space that makes the next reclaim cheaper.
3. Do not launch parallel zfs destroy -r storms. Serialize destroys; concurrency here has produced full pool hangs in the field.
4. If a scrub or resilver is running and the situation is urgent, consider whether it can wait; both compete for the same bandwidth.

Scripting around destroys

For automation, gate follow-up work on the backlog instead of guessing:

# Wait for reclaim to finish before the next destructive step
# (OpenZFS 2.0+; add an interval to print remaining bytes)
zpool wait -t free <pool>

For older releases, poll zpool get -Hp -o value freeing <pool> in a loop until it reads 0.

Prevention

• Stagger snapshot pruning. Retention jobs that destroy thousands of snapshots in one run create exactly the burst that stalls pools. Spread destroys across the maintenance window, and prune continuously rather than in monthly bulk runs.
• Alert on freeing, not just CAP. A pool at 92% with a large freeing backlog is a different situation from a pool at 92% with none. Track the property as a first-class metric so a destroy started by someone else does not surprise you.
• Keep capacity headroom. The 85% action threshold exists partly so that operations like large destroys remain cheap. A pool that lives above 90% turns every cleanup into an incident.
• Size destroys before running them. zfs list -t snapshot -o name,used,refer -s used -r <pool> tells you what you are about to set in motion. A 5 TB snapshot destroy on a Friday afternoon is a choice.
• Track snapshot-held space. zfs get -r usedbysnapshots <pool> and zfs list -o space -r <pool> show how much of your pool is locked behind snapshots before it becomes an emergency.

How Netdata helps

• Freeing backlog as a time series: the freeing pool property charted over time turns “the destroy seems slow” into a drain rate and an ETA, and catches destroys launched by other automation.
• Correlation with pool latency: overlaying freeing against zpool iostat read/write latency shows exactly when reclaim started impacting foreground I/O, separating destroy contention from device problems.
• TXG and dirty-data context: TXG sync duration and dirty-data pressure alongside the backlog tell you whether the write pipeline is absorbing the reclaim or throttling applications because of it.
• Capacity trend in the same view: CAP, fragmentation, and snapshot space next to the freeing backlog let you see the death-spiral preconditions (near-full pool plus large pending reclaim) before writes start failing.

Related guides

• ZFS capacity planning: runway estimation before the pool fills
• ZFS dirty data throttling: the write delay that masquerades as slow disks
• ZFS deadman events: hung I/O and a stalled pool sync
• How ZFS actually works in production: a mental model for operators