ZFS deleted files but no space freed: snapshots holding the blocks

You deleted 100 GB of files. df shows the same free space as before. zpool list has not moved. Nothing is broken: ZFS is copy-on-write, and a snapshot taken before the delete still references every block you just removed. Until that snapshot is destroyed, the blocks stay allocated and the pool gains nothing.

This becomes an incident when the pool is already past 90%: the operator deletes files, gets nothing back, panics, and starts destroying snapshots at random. Mass snapshot destruction on a nearly-full pool triggers heavy asynchronous block freeing that competes for the same I/O bandwidth the pool is already short on. That is how a capacity annoyance turns into a write stall.

This guide covers why the space does not come back, how to find exactly which snapshots are holding it, and how to reclaim it without stalling the pool.

What this means

ZFS never overwrites blocks in place. When you modify or delete a file, the old blocks stay put and new metadata points elsewhere. A snapshot is a frozen reference to a block tree: every block reachable from the snapshot must be kept until the snapshot is destroyed. Deleting a file from the live dataset removes it from the live view, but if any snapshot was taken while that file existed, the snapshot still references its blocks.

The visible symptoms:

• The dataset’s referenced space (REFER) drops after the delete, but the dataset’s total USED does not.
• The pool’s ALLOC and CAP in zpool list do not change.
• zfs get usedbysnapshots <dataset> shows exactly the space that will not come back until snapshots are destroyed.

One accounting trap: the USED column of an individual snapshot shows only blocks unique to that snapshot. Blocks shared between several snapshots are not counted in any single snapshot’s USED. The sum of all snapshot USED values can look far smaller than the space actually held. Do not trust per-snapshot USED for reclaim estimates; use a dry-run destroy instead (below).

flowchart TD
  A[Operator deletes 100 GB of files] --> B{Any snapshot referencing those blocks?}
  B -->|Yes| C[Blocks stay allocated
pool FREE unchanged]
  B -->|No| D[Blocks freed asynchronously
freeing property drains to zero]
  C --> E[usedbysnapshots shows held space]
  E --> F[Destroy the holding snapshots]
  F --> G{Clones or holds on the snapshot?}
  G -->|Yes| H[Destroy fails: dataset is busy
check zfs holds and clones]
  G -->|No| D
  D --> I[Monitor zpool get freeing
until it returns to 0]

Common causes

Quick checks

All of these are read-only and safe to run during an incident.

# Per-dataset space breakdown: live data vs snapshots vs children
zfs list -o space -r <pool>

# Snapshot-held space per dataset, the number that answers "where did my delete go"
zfs get -r usedbysnapshots <pool>

# Snapshots sorted by unique space, largest last
zfs list -t snapshot -o name,used,refer -s used -r <pool>

# Is space currently being reclaimed asynchronously?
zpool get freeing <pool>

# Any holds anywhere on the pool?
zfs holds -r <pool>

# Pool-level view: allocated, free, capacity, pending reclaim
zpool list -H -o name,size,alloc,free,cap,freeing

Two things while reading output. First, USEDSNAP (or usedbysnapshots) is the authoritative number for space pinned by snapshots of a dataset. Second, a non-zero freeing value means reclaim is already running; the space is coming back, just not yet.

How to diagnose it

1. Confirm the accounting. Run zfs list -o space -r <pool> and find the dataset you deleted from. If USEDSNAP is large (or jumped) while USEDDS dropped by about the size of your delete, snapshots are holding the blocks. There is no leak and nothing is corrupt.

2. Estimate true reclaimable space before destroying anything. Use a dry-run destroy with range syntax. -n makes it a no-op and -v prints what would be reclaimed:

   # Dry-run: how much space would destroying this snapshot range return?
   zfs destroy -nv <pool>/<dataset>@<oldest-snap>%<newest-snap>
   

   This is the reliable way to get a real number, because per-snapshot USED under-counts shared blocks.

3. Check for blockers on the snapshots you plan to destroy. List holds with zfs holds <dataset>@<snap> and check for clones with zfs get clones <dataset>@<snap>. A snapshot with a hold returns “dataset is busy” on destroy; a snapshot with a clone cannot be destroyed until the clone is promoted or destroyed.

4. Check pool headroom before a bulk destroy. If zpool list shows capacity above 90%, plan the destruction in batches rather than one large recursive destroy (see Fixes). Note the current freeing value so you can watch reclaim progress.

5. If you already destroyed snapshots and the space is not back yet, check zpool get freeing <pool>. A large value means asynchronous block freeing is still walking the deadlists. Wait for it to drain before concluding something is wrong.

Metrics and signals to monitor

Fixes

Destroy the right snapshots, in the right order

Identify the snapshots covering the period when the deleted data existed, then destroy them. Prefer range syntax over one-by-one deletion when the snapshots are contiguous:

# Dry-run first
zfs destroy -nv tank/data@daily-2026-06-01%daily-2026-06-30

# Then the real destroy (destructive: permanently deletes the snapshots)
zfs destroy tank/data@daily-2026-06-01%daily-2026-06-30

Tradeoff: destroying a snapshot deletes your rollback point. Confirm with your backup and replication owners that the snapshots are expendable before running the non-dry-run form.

Release holds first if destroy reports “dataset is busy”

# List holds, then release the tag (destructive: removes the protection)
zfs holds tank/data@daily-2026-06-01
zfs release <tag> tank/data@daily-2026-06-01

Holds are placed deliberately, often by replication or backup tooling. Find out what placed the hold before releasing it.

Handle clones

If zfs get clones shows a clone depending on the snapshot, either destroy the clone or promote it (zfs promote <clone>) so it no longer depends on the snapshot. Promotion restructures the dependency; make sure you understand which dataset should be the canonical one before promoting.

On a nearly-full pool, destroy in small batches

Bulk or recursive snapshot destruction on a pool above 90% capacity makes things worse. The destroy itself is a copy-on-write metadata operation that needs free space, and the async freeing it triggers consumes I/O and CPU on an already-saturated pool. Large destroys on pools with hundreds of snapshots have caused pool-wide I/O stalls lasting minutes.

Batch the work:

# Destroy snapshots a few at a time, watching freeing drain between batches
zfs destroy tank/data@snap-1%snap-5
zpool get freeing tank
# wait for freeing to approach 0, then next batch

If the pool is under active production load, reduce application write traffic during cleanup to give async freeing bandwidth. zpool wait -t free <pool> (available since OpenZFS 2.0.0) blocks until pending reclamation completes, which is useful in scripts between batches.

Prevention

• Track snapshot space as a first-class metric. Pool capacity alone hides snapshot growth. Alert when usedbysnapshots exceeds a set fraction of pool allocation; teams routinely discover only after the fact that snapshots consumed 40% of the pool.
• Verify pruning actually runs. Automated snapshot tools create reliably and prune with silent failures. Trend snapshot count and total snapshot space weekly; a monotonically rising count means pruning is broken.
• Plan capacity against the cliff, not against 100%. Keep pools under the 75-85% range so a cleanup operation has room to work. Deleting snapshots at 96% is slow and risky; at 70% it is routine.
• Set retention policy before enabling snapshots. Every snapshot schedule needs a matching destroy schedule, and replication snapshots need a pruning agreement on both sides.
• Watch freeing after any large destroy. A persistently large freeing backlog on a full pool means reclaim is not keeping up; your next capacity estimate should subtract only what has actually drained.

How Netdata helps

• Netdata collects ZFS pool capacity, allocation, and per-dataset space usage, so usedbysnapshots growth is visible as a trend weeks before the pool fills, not as a surprise during a delete.
• Correlating pool capacity with TXG sync duration separates “pool is full” from “pool is full and the write path is stalling,” which is the difference between a ticket and a page.
• During snapshot cleanup, per-second I/O latency and throughput charts show immediately whether async freeing is competing with production writes, so you can pace batch destroys against real impact instead of guessing.
• Alerting on capacity thresholds (75% plan, 85% act) with the snapshot-space breakdown alongside means the first alert names the cause, not just the symptom.
• Historical retention lets you confirm after the fact that a destroy actually reclaimed what the dry-run predicted.

Related guides

• ZFS capacity planning: runway estimation before the pool fills
• How ZFS actually works in production: a mental model for operators
• ZFS dirty data throttling: the write delay that masquerades as slow disks
• ZFS deadman events: hung I/O and a stalled pool sync