ZFS special vdev full: the hidden bottleneck while the pool looks empty

The pool is at 55% capacity, fragmentation is moderate, and zpool status -x says all pools are healthy. Yet anything touching lots of small files or metadata has gone from fast to miserable, and no pool-level dashboard explains why.

On pools with a special allocation class, the cause is often the special vdev itself: the (hopefully mirrored) fast SSDs that hold all pool metadata and, where special_small_blocks is set, small data blocks. When the special vdev fills, ZFS raises no alert. New metadata and small blocks silently land on the main pool vdevs, so metadata I/O now runs at bulk-storage latency while every pool-level capacity signal still shows headroom. The pool is ONLINE, capacity looks fine, and the bottleneck sits one level down in the vdev tree where most monitoring never looks.

What this means

A special allocation class vdev intercepts two categories of blocks and keeps them on fast media:

• All pool metadata: dnodes, directory entries, indirect blocks, space maps, and (if dedup is enabled) the DDT.
• Small data blocks, on datasets where special_small_blocks is set to a non-zero block size cutoff.

Metadata touches almost every operation: file creation, directory listing, snapshot work, zfs send, even reads of uncached data. When the special vdev fills, ZFS stops directing new allocations there and falls back to the main pool vdevs. Blocks already on the special vdev stay put; ZFS does not evict or rebalance them. You end up in a split state where old metadata is fast and all new metadata is slow, which is harder to reason about than a clean failure.

There is also a permanent risk dimension: a non-mirrored special vdev is a single point of failure. If it dies, the pool’s metadata dies with it and the pool is lost. That is true whether the vdev is full or not, but operators who added a single SSD as a “cache-like” special vdev often do not realize this until an incident.

flowchart TD
  A[New metadata or small block write] --> B{Special vdev has headroom?}
  B -->|yes| C[Allocated on special vdev - fast SSD]
  B -->|no, special vdev full| D[Silent fallback to main pool vdevs]
  D --> E[Metadata I/O at HDD/slow-SSD latency]
  E --> F[Directory ops, snapshots, sends, small-file reads all slow]
  F --> H[Pool CAP still looks fine - root cause hidden]

Common causes

Quick checks

These are all read-only and safe to run during an incident.

# Per-vdev capacity and fragmentation: look at the 'special' rows, not just the pool total
zpool list -v <pool>

# Special allocation class fill level (see note on property names below)
zpool get special_capacity,special_free,special_fragmentation <pool>

# Confirm which datasets are allowed to place small blocks on the special vdev
zfs get special_small_blocks -r <pool>

# Per-vdev I/O: is the special vdev idle while main vdevs absorb metadata-ish I/O?
zpool iostat -v <pool> 5

# Latency histograms: rising total_wait on data vdevs with no scrub/resilver running
zpool iostat -l <pool> 5

# Pool-level view for contrast: CAP will look comfortable, that is the trap
zpool list -o name,size,alloc,free,cap,frag <pool>

# Recent administrative history: did a send/recv, property change, or migration coincide with the slowdown?
zpool history <pool> | tail -30

Two notes on interpretation:

• ZFS stops directing new allocations to the special vdev before it reaches a literal 100% full state, so “not completely full” does not mean “still being used.” Community testing puts the fallback watermark around 75% of the vdev, adjustable by a module parameter. Treat special-class capacity climbing into that range as the warning, not 100%.
• zpool list -v is the fastest way to see the split: a pool at 55% CAP with special rows at 95%+ CAP is the signature of this problem.

How to diagnose it

1. Confirm the special class state. Check the special-class capacity, free, and fragmentation properties. If capacity is high and free is near zero (or near the effective allocation cutoff), the special vdev has stopped accepting new allocations and fallback is active.

2. Look at the vdev tree, not the pool total. zpool list -v <pool> shows ALLOC, FREE, CAP, and FRAG per top-level vdev, including special rows. Pool CAP alone hides everything that matters here.

3. Establish when it filled. Use zpool history <pool> to find recent zfs receive, zfs set special_small_blocks, dataset creation, or large migrations. A large zfs send/receive of a dataset tree with millions of files can fill a special vdev surprisingly fast, because every file brings dnodes and directory metadata even when special_small_blocks=0.

4. Check what is allowed onto the vdev. zfs get special_small_blocks -r <pool>. Any dataset with a non-zero value pushes small data blocks to the special vdev on top of metadata. A value set at the pool root and inherited everywhere is a common way to exhaust the vdev years ahead of plan.

5. Check for dedup. If zfs get dedup <pool> shows on, the DDT is competing for the same special vdev. zpool status -D <pool> shows DDT size. Dedup plus a special vdev is a fast path to this failure mode.

6. Verify the performance side. zpool iostat -v <pool> 5 and zpool iostat -l <pool> 5. With fallback active, the special vdev sits relatively idle on writes while main vdev latency climbs under metadata-heavy operations. If queue depth and latency are fine and only capacity is high, you have a capacity problem waiting to become a performance problem.

7. Quantify the metadata footprint for planning. On OpenZFS 2.2.0 and later, zdb -bb <pool> prints a “Metadata Total” line giving the allocated size of all metadata blocks, which is the number you size a special vdev against.

Metrics and signals to monitor

Fixes

Free space on the special vdev

The special vdev only starts accepting allocations again once it has headroom. Options, in increasing order of disruption:

• Reduce special_small_blocks to 0 on datasets where it is not earning its keep (zfs set special_small_blocks=0 <dataset>). This stops new small data blocks from landing on the vdev. Existing blocks stay put; you are stopping the bleeding, not draining.
• Destroy expendable snapshots. Snapshot destruction frees metadata (and any small blocks) on the special vdev. Destruction is irreversible, so confirm nothing depends on the snapshots first. Space returns asynchronously; watch the pool freeing property while reclaim runs.
• Rewrite data to move it. ZFS never migrates existing blocks off or onto a special vdev. Moving metadata placement for existing data requires rewriting it, typically zfs send | zfs recv into a fresh dataset. Plan this as a maintenance task, not an incident fix.

Expand or rebuild the special vdev

• Grow by replacing devices. You cannot resize a special vdev in place. The realistic path is replacing devices with larger ones: replace one mirror leg, resilver, replace the other.
• Remove and re-add at the right size. zpool remove on a special vdev rewrites its metadata and small blocks onto the main pool. Warning: this only works if the special vdev is a mirror (redundant) and the pool layout supports removal, and it is I/O intensive with a measurable performance cost while it runs. It is a planned maintenance operation, not a mid-incident move.

If the special vdev is a single device

Fix the redundancy now, independent of the capacity issue: zpool attach a second device to turn it into a mirror, then wait for the resilver to complete. A full single-device special vdev is two incidents stacked: the performance problem you can see, and total pool loss waiting for one SSD failure.

Prevention

• Size from measured metadata, not vibes. Use zdb -bb (OpenZFS 2.2.0+) to measure the actual metadata footprint, then give the special vdev generous headroom on top. Metadata grows with file count, snapshot count, and small-file churn, all of which tend to surprise upward.
• Mirror the special vdev, always. Treat this as a hard requirement. The special vdev holds the pool’s metadata; it is not a cache.
• Be deliberate with special_small_blocks. Set it per-dataset where small-block latency actually matters, not at the pool root by default. Every non-zero value is a capacity claim on the vdev.
• Alert on the special-class properties. Trend special-class capacity and free space and alert well before the fallback watermark, exactly as you would alert on pool capacity before 85%.
• Watch fill events. Large replications, migrations, and enabling dedup all change the metadata growth rate. Re-check special vdev headroom after any of them.
• Include the special class in capacity runway reviews. The pool runway and the special vdev runway are different clocks. The special vdev almost always runs out first.

How Netdata helps

• Netdata collects ZFS pool and per-vdev capacity, fragmentation, and I/O metrics continuously, so the special vdev’s fill trend is visible as a time series rather than discovered after fallback has been degrading performance for weeks.
• Correlating per-vdev latency with pool-level capacity shows the signature directly: data vdev latency rising while pool CAP sits flat and comfortable.
• TXG sync duration trending upward alongside a plateaued special vdev I/O rate points at metadata fallback rather than a dying disk, saving a wrong hardware replacement.
• Alerts on special-class capacity and free space fire while there is still time to plan a resize, instead of during the performance incident.
• Historical retention lets you line up the fill event with the change that caused it, such as a replication job or a special_small_blocks change, using zpool history as the cross-reference.

Related guides

• ZFS capacity planning: runway estimation before the pool fills
• ZFS deleted files but no space freed: snapshots holding the blocks
• ZFS dedup memory exhaustion: when the DDT outgrows ARC and the pool crawls
• ZFS deadman events: hung I/O and a stalled pool sync
• ZFS ARC hit ratio low: cache misses, cold caches, and working sets that outgrew RAM
• ZFS checksum errors (CKSUM): the definitive signal of silent corruption