ZFS snapshot space consumption: the 40%-of-the-pool blindspot

A ZFS pool reports 90% capacity. The operator deletes 500 GB of old files, and the pool does not get any emptier. This is one of the most common ZFS incidents, and it is not a bug. It is copy-on-write working exactly as designed.

ZFS never overwrites blocks in place, so a snapshot retains references to every block that existed when it was taken. When you delete a file in the live dataset, the block is freed only if no snapshot still references it. If snapshots exist, the space stays allocated until the last referencing snapshot is destroyed. Teams routinely discover, mid-incident, that snapshots are holding 40% or more of the pool and that the default tooling never showed them.

The blindspot has a specific cause: zfs list in its default form does not show snapshot space at the dataset level, and it does not list snapshots at all unless you pass -t snapshot. You can run zfs list daily and never see the problem until the pool crosses the fragmentation cliff.

What this means

Snapshot-held space is not overhead. It is real allocated capacity, counted in the pool’s CAP percentage, and it contributes to every capacity-related failure mode: metaslab allocator slowdowns above 80-85%, write latency degradation, and eventually ENOSPC for applications.

Two accounting subtleties cause most of the confusion:

• Dataset-level snapshot space is hidden by default. You must ask for it with zfs list -o space (the USEDSNAP column) or zfs get usedbysnapshots.
• Per-snapshot used is not cumulative. An individual snapshot’s used shows only the space unique to that snapshot: the blocks freed if only that snapshot were destroyed. Blocks shared between adjacent snapshots are counted in no single snapshot’s used. The sum of per-snapshot used values therefore does not equal the dataset’s usedbysnapshots, and destroying one intermediate snapshot can increase the reported used of its neighbors, because formerly shared blocks become unique to the survivors.

The practical consequence: you cannot eyeball which snapshots to destroy from their individual used values, and deleting files in the live dataset tells you nothing about how much space you will actually get back.

flowchart TD
  A[Application deletes file] --> B{Any snapshot still references the block?}
  B -- yes --> C[Block stays allocated, held by snapshot]
  B -- no --> D[Block freed to pool]
  C --> E[Destroy last referencing snapshot]
  E --> F[Async reclaim runs in background]
  F --> D
  F -. monitor .-> G[zpool get freeing]

Common causes

Quick checks

All read-only and safe to run during an incident.

# Pool-level capacity and async reclaim in progress
zpool list -H -o name,size,alloc,free,cap,freeing

# The canonical space breakdown: USEDSNAP is the snapshot hold per dataset
zfs list -o space -r <pool>

# Same value as a property, per dataset
zfs get -r usedbysnapshots <pool>

# All snapshots, largest unique-space holders last
zfs list -t snapshot -o name,used,refer -s used -r <pool> | tail -20

# Snapshot count sanity check
zfs list -t snapshot -r <pool> | wc -l

# Holds that prevent destruction
zfs holds -r <pool>

# Dry-run: how much space would destroying THIS snapshot actually free?
zfs destroy -nv <pool>/<dataset>@<snapshot>

Two things about the last command. First, -n makes it a dry run; nothing is destroyed. Second, the reported reclaimable value matches that snapshot’s unique used, which can be far smaller than you expect if neighboring snapshots share the blocks. Do not be surprised when destroying one snapshot frees less than the “deleted data” you were hoping to reclaim.

How to diagnose it

1. Confirm the pool is filling from snapshots, not live data. Run zfs list -o space -r <pool> and compare USEDSNAP against USEDDS (live dataset data) per dataset. If USEDSNAP dominates on one or two datasets, you have a retention or churn problem there, not a capacity problem.
2. Check whether reclaim is already running. zpool get freeing <pool> shows bytes being asynchronously reclaimed. A large freeing value means a recent destruction is still being processed; the space is coming back, just not yet. Do not start more destruction on top of a large backlog without watching this drain first.
3. Find the retention offender. List snapshots by creation time and look for a naming pattern: hourly snapshots months old, replication snapshots with no pruning, or a cron job that creates but never destroys. Auto-snapshot and replication retention are the two classic sources.
4. Quantify the churn. For the suspect dataset, look at the written property between snapshots (zfs list -t snapshot -o name,written -r <dataset>). A dataset writing tens of GB between snapshots makes even modest retention expensive. This tells you whether to fix retention depth, snapshot frequency, or both.
5. Check for blockers. Before planning mass destruction, verify no clones depend on the snapshots (zfs list -t all, look at origin) and no holds exist (zfs holds -r <pool>). Destroying a snapshot that a clone is based on will fail; holds silently prevent destruction.
6. Estimate real reclaim before acting. Use zfs destroy -nv on candidate snapshots. Destroying intermediate snapshots redistributes shared blocks to neighbors, so total reclaim from a batch is only visible as it executes. Watch zpool get freeing and pool CAP as you go.

Metrics and signals to monitor

Fixes

Prune snapshots with a staggered retention policy

The durable fix for auto-snapshot sprawl is a staggered scheme: keep many recent snapshots and progressively fewer older ones (for example, a day of hourlies, a month of dailies, a few monthlies). This preserves rollback granularity where it matters while bounding total snapshot count. Keeping every hourly snapshot for months is how pools end up with thousands of snapshots holding ephemeral churn: lockfiles, temp files, and rewritten blocks no one will ever roll back to.

Destroy in scripted batches by age and naming pattern rather than ad hoc, and watch zpool get freeing between batches. Snapshot destruction is destructive and irreversible; verify your age filter and naming pattern with a listing pass before piping anything to zfs destroy. Large destructions trigger async block freeing that competes with production I/O, so on a pool already above 85% capacity, pace the work.

Fix replication retention at both ends

If replication snapshots are the problem, check the pruning logic on both source and target. A common failure is a send-side script that creates snapshots for incremental sends but never destroys them after the target confirms receipt. Align the hold/destroy logic with your replication tool’s requirements before deleting anything the tool expects to find; removing a snapshot the next incremental send depends on forces a full resend.

Address churn where retention cannot shrink

If the dataset genuinely churns heavily and you need the retention depth for compliance or recovery, the fix is not fewer snapshots but more capacity or a different layout. Move high-churn, low-value data (build artifacts, temp tables, logs with their own rotation) to a separate dataset with minimal or no snapshot schedule, so it stops inflating the snapshot hold of the valuable data.

Do not expect instant space back

Reclamation after destruction is asynchronous. zpool get freeing tells you the backlog. On a nearly full pool, freeing can be slow because reclaim competes for allocator and I/O resources. If the pool is already past the cliff, reduce application write load while the backlog drains rather than stacking more destruction jobs.

Prevention

• Monitor usedbysnapshots per dataset continuously, not just pool CAP. Capacity alone cannot tell you whether live data or snapshots are filling the pool, and the response is completely different.
• Set an explicit snapshot budget. Treat snapshot space above 50% of pool allocation on a pool approaching capacity as a ticket, not a curiosity.
• Alert on snapshot count and age drift. A steady upward trend in count without a policy change means pruning is broken, even if space is fine today.
• Track freeing after every destruction event so async reclaim backlogs are visible rather than surprising.
• Include snapshot space in runway estimation. Project dataset growth + snapshot retention growth - scheduled pruning, not just live-data growth. See the capacity planning guide linked below.

How Netdata helps

• Netdata collects ZFS pool capacity and allocation metrics continuously, so snapshot-driven growth trends are visible over time instead of being discovered during an incident.
• Pool CAP trending alongside dataset-level space breakdown makes it obvious when growth is snapshot-driven rather than live-data-driven, which is the first diagnostic fork.
• freeing backlog visibility lets you confirm that async reclaim after snapshot destruction is actually draining, rather than assuming space came back.
• Correlating snapshot-space growth with write latency shows when the snapshot hold is pushing the pool toward the metaslab allocation cliff, before applications feel it.
• Alerting on capacity growth rate catches broken pruning automation weeks before the pool fills.

Related guides

• ZFS capacity planning: runway estimation before the pool fills
• How ZFS actually works in production: a mental model for operators
• ZFS dirty data throttling: the write delay that masquerades as slow disks
• ZFS deadman events: hung I/O and a stalled pool sync
• ZFS ARC using all memory: the Linux default that eats your RAM