ZFS No space left on device: ENOSPC, the slop reserve, and the pool you cannot delete from

Your application just failed with No space left on device. df shows free space. zpool list shows FREE above zero. You try to delete files to make room, and rm fails with the same error: rm: cannot remove 'file': No space left on device. The pool is in the worst state a ZFS pool can be in: full enough that even freeing space requires space you do not have.

This is not a bug and not a lying tool. It is the collision of two ZFS design decisions: the copy-on-write transactional model, and the slop reserve that exists specifically to keep a full pool administrable. Knowing which of the two you are hitting is the difference between a controlled recovery and a pool that stays wedged until you migrate data off it.

This guide covers why ENOSPC appears before the pool is physically full, why deletes fail on a full pool, how to tell which situation you are in, and the recovery paths that work.

What this means

ZFS never overwrites data in place. Every write, including the metadata updates required to unlink a file, allocates new blocks and then updates pointers atomically. Deleting a file is itself a write operation: the dnode and indirect block tree must be updated, and those updates need free space. On a pool with genuine headroom this is invisible. On a pool at the wall, rm can return ENOSPC.

To keep the pool from becoming completely unmanageable, ZFS reserves a fraction of pool capacity as “slop” space, controlled by the spa_slop_shift module parameter. The default shift is 5, which reserves 1/2^5, or about 3.125% of pool size, bounded by a floor of 128 MiB and (since OpenZFS 2.0.6) a cap of 128 GiB. When free space drops below the full slop threshold, writes, creates, and similar allocations fail with ENOSPC. File removal and most administrative actions may use half the slop space, and space-freeing operations such as zfs destroy may use three quarters, so recovery can remain possible after ordinary writes fail; those narrower thresholds can still be exhausted.

In practice, on a badly fragmented pool or one pushed past the slop boundary by a burst of writes, even administrative operations can fail. That is the death spiral: freeing space requires space, and there is none.

flowchart TD
  A[Pool fills past slop boundary] --> B[User writes fail: ENOSPC]
  A --> C[Operator tries rm to free space]
  C --> D[Delete needs COW metadata writes]
  D --> E{Free space above slop reserve?}
  E -->|Yes| F[Delete succeeds, space freed asynchronously]
  E -->|No| G[rm fails with ENOSPC]
  G --> H[Death spiral: freeing space requires space]
  F --> I[freeing property drains as blocks reclaim]

One more source of confusion: the number you are looking at is probably the wrong one. Pool-level FREE from zpool list does not account for the slop reserve, quotas, or reservations. The per-dataset available property does. When zfs get available says 0 and zpool list says gigabytes free, believe available.

Common causes

Quick checks

All read-only and safe to run during an incident.

# Pool-level capacity and async reclaim backlog
zpool list -H -o name,size,alloc,free,cap,freeing

# The number that actually governs writes, per dataset
zfs get -r available <pool>

# Full space accounting: live data vs snapshots vs reservations vs children
zfs list -o space -r <pool>

# Snapshots sorted by space they uniquely hold
zfs list -t snapshot -o name,used,refer -s used -r <pool> | tail -20

# Snapshot-held space per dataset
zfs get -r usedbysnapshots <pool>

# Fragmentation
zpool list -H -o name,frag

# Current slop reserve shift (Linux)
cat /sys/module/zfs/parameters/spa_slop_shift

Interpretation notes:

• CAP at 96% or higher with write I/O active and freeing at zero is the emergency combination. The pool is at the slop boundary and nothing is being reclaimed.
• A large freeing value is good news during recovery: a destroy has been issued and blocks are being reclaimed asynchronously. Wait for it to drain before concluding the recovery failed.
• USEDSNAP dominating USED tells you the capacity problem is snapshot retention, not live data. The fix is destroying snapshots, not deleting more files.
• Fragmentation above 50% on a write-heavy pool above 85% capacity means the allocator is struggling even before the slop boundary. Space exists but is not usable efficiently.

How to diagnose it

1. Confirm which number is zero. Run zfs get available <dataset> for the dataset the application writes to. If available is 0 but zpool list shows FREE, you are at the slop boundary or constrained by a quota. If available is non-zero and writes still fail, check quota and refquota on that dataset and its parents.

2. Determine what is holding the space. Run zfs list -o space -r <pool>. If USEDSNAP is large relative to USEDDS, snapshots are the target. If USEDREFRESERV is large, reservations are committing space invisibly. Deleting files in the live filesystem will not help in either case.

3. Check whether reclaim is already in flight. Run zpool get freeing <pool>. A non-zero value means a previous destroy is still being processed. Space reclaim from snapshot destruction is asynchronous and competes with production I/O; on a nearly full, fragmented pool it can be slow. Do not stack more destroys on top blindly, but do not panic either.

4. Assess fragmentation. Run zpool list -H -o name,frag. High fragmentation at high capacity amplifies everything: allocation is expensive, TXG syncs slow down, and the async destroy that would save you is throttled by the same congested I/O path.

5. Determine if you are in the death spiral. Try a small, cheap delete. If rm of a small file fails with ENOSPC, and zfs destroy of a snapshot also fails with out-of-space errors, the pool is below the workable threshold and you need the recovery procedures below, not more deletes.

Metrics and signals to monitor

The composite emergency: CAP above 96%, write I/O active, freeing not reclaiming. That is the page-worthy condition. Capacity alone at 96% on an idle pool is a ticket, not a page.

Fixes

Destroy snapshots, not files

If USEDSNAP is significant, this is the primary recovery. Destroying a snapshot releases the blocks it uniquely holds via asynchronous freeing. Identify the largest candidates:

# Largest snapshots by uniquely held space
zfs list -t snapshot -o name,used -s used -r <pool> | tail -10

# Destructive: destroy one snapshot, then watch freeing drain
zfs destroy <pool>/<dataset>@<snapshot>
zpool get freeing <pool>

Destroy one large snapshot, watch freeing rise and then drain, and confirm available recovers before destroying the next. Note that used for a snapshot is the space unique to it; with long snapshot chains, destroying one intermediate snapshot frees less than you might expect because neighboring snapshots still reference the blocks. Check for holds (zfs holds -r <pool>) and clones, which pin snapshots and prevent destruction.

Reduce application I/O during reclaim

Async destroy competes with production I/O for the same congested devices. If reclaim is crawling, throttle or pause the heaviest writers to give the free process bandwidth. This is a real tradeoff: you are trading application throughput for recovery speed.

Release part of the slop reserve (recovery valve)

If the pool is wedged such that even destroys fail, you can temporarily shrink the slop reserve to hand some reserved space back to the allocator. spa_slop_shift is a runtime module parameter on Linux, system-wide, not persistent across reboot or module reload:

# Check current value (default 5)
cat /sys/module/zfs/parameters/spa_slop_shift

# Recovery move: raise the shift to shrink the reserve
echo 6 > /sys/module/zfs/parameters/spa_slop_shift

Raising the shift by 1 halves the reserve, releasing roughly 1.5% of pool capacity back as writable space. Then immediately destroy snapshots or delete data to build real headroom, and restore the default:

# Restore default after recovering headroom
echo 5 > /sys/module/zfs/parameters/spa_slop_shift

Warnings: this affects every pool on the host, it removes the safety margin that keeps the pool administrable, and it does not survive a reboot. Free substantially more than you released before restoring the default. Treat it as an emergency valve, not a tuning knob.

Expand the pool if the underlying storage grew

If the pool sits on devices or LUNs that were expanded, zpool online -e <pool> <device> grows the vdev into the new capacity, which immediately adds free space above the slop boundary. This is the cleanest escape when it is available, but it only applies if expandable backing storage actually exists.

Truncate rather than delete, as a last resort

On a fully wedged pool, truncating a file to zero can succeed where rm fails, because it requires fewer metadata allocations than a full unlink. Target large files. This is slow, surgical work, and it only buys enough room to make snapshot destroys possible. If you reach this point, plan the follow-up: the pool needs capacity relief, not just tonight’s rescue.

What does not work

• zpool scrub does not free space and does not defragment.
• There is no online defragmentation. Severe fragmentation is only fixed by zfs send | zfs recv to a new pool.
• Deleting files while snapshots reference their blocks frees nothing. Verify with zfs list -o space before spending incident time on rm.

Prevention

• Alert on capacity well before the wall. Planning at 75%, action at 85%, emergency at 96% with active writes. The degradation is non-linear; “we still have 10%” is already the degraded zone on a write-heavy pool.
• Monitor available, not just pool FREE. Per-dataset available is the number applications actually experience.
• Track snapshot space as its own line item. Trend usedbysnapshots per dataset and alert when it exceeds a set fraction of pool allocation. Pair every snapshot automation with a tested pruning policy.
• Trend fragmentation alongside capacity. Capacity rising plus fragmentation rising is the accelerating-degradation pattern; either one alone underestimates the risk.
• Watch freeing after bulk destroys. A large persistent freeing backlog means async reclaim is not keeping up, and your capacity numbers are optimistic.
• Model runway honestly. Free space divided by daily allocation rate, adjusted for snapshot retention growth. Snapshot-held space grows as the live dataset diverges, so growth is not linear.

How Netdata helps

• Netdata collects pool-level capacity, allocation, and fragmentation so you see the approach to the 85% and 96% boundaries as a trend, not a surprise during an incident.
• Correlating capacity growth against snapshot counts and snapshot-held space separates “data is growing” from “retention is leaking”, which have completely different fixes.
• Tracking the freeing property over time shows whether async reclaim after a destroy is progressing or stalled behind production I/O, which is the key question mid-recovery.
• TXG sync duration and write latency trends reveal the capacity-fragmentation cliff forming weeks before ENOSPC: sync times creep up as metaslabs starve, long before the pool reports full.
• Per-dataset space breakdowns put available next to pool FREE on one dashboard, so the “df says free, writes say ENOSPC” confusion resolves in seconds.
• Alerting on the composite condition, high capacity plus active writes plus no reclaim in progress, pages on the actual emergency instead of on capacity alone.

Related guides

• ZFS monitoring checklist: the signals every production pool needs
• ZFS monitoring maturity model: from survival to expert
• How ZFS actually works in production: a mental model for operators
• ZFS pool ONLINE with non-zero errors: why zpool status -x lies
• ZFS checksum errors (CKSUM): the definitive signal of silent corruption
• ZFS permanent errors have been detected in the following files: recovering from data loss
• ZFS pool DEGRADED: redundancy lost and one failure from data loss
• ZFS pool FAULTED: when the pool can no longer serve I/O
• ZFS pool I/O is currently suspended: a hung pool and blocked I/O
• ZFS device UNAVAIL or REMOVED: a disk that fell off the bus
• ZFS READ and WRITE errors: transport-level device failures in zpool status
• ZFS scrub repaired errors: correctable rot versus permanent data loss