ZFS special vdev full: the hidden bottleneck while the pool looks empty

The pool is at 55% capacity, fragmentation is moderate, and zpool status -x says all pools are healthy. Yet anything touching lots of small files or metadata has gone from fast to miserable, and no pool-level dashboard explains why.

On pools with a special allocation class, the cause is often the special vdev itself: the (hopefully mirrored) fast SSDs that hold all pool metadata and, where special_small_blocks is set, small data blocks. When the special vdev fills, ZFS raises no alert. By default, new small data blocks stop entering it once it is about 75% allocated; once the class is actually full, new metadata spills back to the main pool vdevs too. Metadata I/O then runs at bulk-storage latency while every pool-level capacity signal still shows headroom. The pool is ONLINE, capacity looks fine, and the bottleneck sits one level down in the vdev tree where most monitoring never looks.

What this means

A special allocation class vdev intercepts two categories of blocks and keeps them on fast media:

• All pool metadata: dnodes, directory entries, indirect blocks, space maps, and (if dedup is enabled) the DDT.
• Small data blocks, on datasets where special_small_blocks is set to a non-zero block size cutoff.

Metadata touches almost every operation: file creation, directory listing, snapshot work, zfs send, even reads of uncached data. When the special vdev fills, ZFS stops directing new metadata there and falls back to the main pool vdevs; at 75% allocation it already reserves the last quarter for metadata by excluding new small data blocks. Blocks already on the special vdev stay put; ZFS does not evict or rebalance them. You end up in a split state where old metadata is fast and all new metadata is slow, which is harder to reason about than a clean failure.

There is also a permanent risk dimension: a non-mirrored special vdev is a single point of failure. If it dies, the pool’s metadata dies with it and the pool is lost. That is true whether the vdev is full or not, but operators who added a single SSD as a “cache-like” special vdev often do not realize this until an incident.

flowchart TD
  A[New metadata or small block write] --> B{Special vdev has headroom?}
  B -->|yes| C[Allocated on special vdev - fast SSD]
  B -->|no, special vdev full| D[Silent fallback to main pool vdevs]
  D --> E[Metadata I/O at HDD/slow-SSD latency]
  E --> F[Directory ops, snapshots, sends, small-file reads all slow]
  F --> H[Pool CAP still looks fine - root cause hidden]

Common causes

Quick checks

These are all read-only and safe to run during an incident.

# Per-vdev capacity and fragmentation: look at the 'special' rows, not just the pool total
zpool list -v <pool>

# Current OpenZFS source has allocation-class properties; use zpool list -v on older releases
zpool get class_special_capacity,class_special_free,class_special_fragmentation <pool>

# Confirm which datasets are allowed to place small blocks on the special vdev
zfs get special_small_blocks -r <pool>

# Per-vdev I/O: is the special vdev idle while main vdevs absorb metadata-ish I/O?
zpool iostat -v <pool> 5

# Latency histograms: rising total_wait on data vdevs with no scrub/resilver running
zpool iostat -l <pool> 5

# Pool-level view for contrast: CAP will look comfortable, that is the trap
zpool list -o name,size,alloc,free,cap,frag <pool>

# Recent administrative history: did a send/recv, property change, or migration coincide with the slowdown?
zpool history <pool> | tail -30

The current OpenZFS source documents class_special_capacity, class_special_free, and class_special_fragmentation; those properties are not present in OpenZFS 2.4.0, so use the per-vdev columns from zpool list -v when the property is unavailable.

Two notes on interpretation:

• ZFS stops directing new allocations to the special vdev before it reaches a literal 100% full state, so “not completely full” does not mean “still being used.” The exact control is zfs_special_class_metadata_reserve_pct, default 25%: once free space falls to that level, the last quarter of the class is reserved for metadata and new small data blocks use the normal class. Metadata itself continues until the class is full, then spills back. Treat special-class capacity climbing toward 75% as the small-block warning, not 100%.
• zpool list -v is the fastest way to see the split: a pool at 55% CAP with special rows at 95%+ CAP is the signature of this problem.

How to diagnose it

1. Confirm the special class state. Check the special-class capacity, free, and fragmentation properties. If capacity is high and free is near zero (or near the effective allocation cutoff), the special vdev has stopped accepting new allocations and fallback is active.

2. Look at the vdev tree, not the pool total. zpool list -v <pool> shows ALLOC, FREE, CAP, and FRAG per top-level vdev, including special rows. Pool CAP alone hides everything that matters here.

3. Establish when it filled. Use zpool history <pool> to find recent zfs receive, zfs set special_small_blocks, dataset creation, or large migrations. A large zfs send/receive of a dataset tree with millions of files can fill a special vdev surprisingly fast, because every file brings dnodes and directory metadata even when special_small_blocks=0.

4. Check what is allowed onto the vdev. zfs get special_small_blocks -r <pool>. Any dataset with a non-zero value pushes small data blocks to the special vdev on top of metadata. A value set at the pool root and inherited everywhere is a common way to exhaust the vdev years ahead of plan.

5. Check for dedup. If zfs get dedup <pool> shows on, the DDT is competing for the same special vdev. zpool status -D <pool> shows DDT size. Dedup plus a special vdev is a fast path to this failure mode.

6. Verify the performance side. zpool iostat -v <pool> 5 and zpool iostat -l <pool> 5. With fallback active, the special vdev sits relatively idle on writes while main vdev latency climbs under metadata-heavy operations. If queue depth and latency are fine and only capacity is high, you have a capacity problem waiting to become a performance problem.

7. Quantify the metadata footprint for planning. On OpenZFS 2.2.0 and later, zdb -bb <pool> prints a “Metadata Total” line giving the allocated size of all metadata blocks, which is the number you size a special vdev against.

Metrics and signals to monitor

Fixes

Free space on the special vdev

The special vdev only starts accepting allocations again once it has headroom. Options, in increasing order of disruption:

• Reduce special_small_blocks to 0 on datasets where it is not earning its keep (zfs set special_small_blocks=0 <dataset>). This stops new small data blocks from landing on the vdev. Existing blocks stay put; you are stopping the bleeding, not draining.
• Destroy expendable snapshots. Snapshot destruction frees metadata (and any small blocks) on the special vdev. Destruction is irreversible, so confirm nothing depends on the snapshots first. Space returns asynchronously; watch the pool freeing property while reclaim runs.
• Rewrite data to move it. ZFS never migrates existing blocks off or onto a special vdev. Moving metadata placement for existing data requires rewriting it, typically zfs send | zfs recv into a fresh dataset. Plan this as a maintenance task, not an incident fix.

Expand or rebuild the special vdev

• Grow by replacing devices. You cannot resize a special vdev in place. The realistic path is replacing devices with larger ones: replace one mirror leg, resilver, replace the other.
• Remove and re-add at the right size. zpool remove can evacuate a special top-level vdev back to the main pool. This does not require a mirrored special vdev, but the pool must satisfy top-level removal constraints: no raidz/dRAID in primary pool storage, matching top-level ashift, loaded keys for encrypted datasets, and the device_removal feature enabled. Removal is I/O intensive and is planned maintenance, not a mid-incident move.

If the special vdev is a single device

Fix the redundancy now, independent of the capacity issue: zpool attach a second device to turn it into a mirror, then wait for the resilver to complete. A full single-device special vdev is two incidents stacked: the performance problem you can see, and total pool loss waiting for one SSD failure.

Prevention

• Size from measured metadata, not vibes. Use zdb -bb (OpenZFS 2.2.0+) to measure the actual metadata footprint, then give the special vdev generous headroom on top. Metadata grows with file count, snapshot count, and small-file churn, all of which tend to surprise upward.
• Mirror the special vdev, always. Treat this as a hard requirement. The special vdev holds the pool’s metadata; it is not a cache.
• Be deliberate with special_small_blocks. Set it per-dataset where small-block latency actually matters, not at the pool root by default. Every non-zero value is a capacity claim on the vdev.
• Alert on the special-class properties. Trend special-class capacity and free space and alert well before the fallback watermark, exactly as you would alert on pool capacity before 85%.
• Watch fill events. Large replications, migrations, and enabling dedup all change the metadata growth rate. Re-check special vdev headroom after any of them.
• Include the special class in capacity runway reviews. The pool runway and the special vdev runway are different clocks. The special vdev almost always runs out first.

How Netdata helps

• Netdata collects ZFS pool and per-vdev capacity, fragmentation, and I/O metrics continuously, so the special vdev’s fill trend is visible as a time series rather than discovered after fallback has been degrading performance for weeks.
• Correlating per-vdev latency with pool-level capacity shows the signature directly: data vdev latency rising while pool CAP sits flat and comfortable.
• TXG sync duration trending upward alongside a plateaued special vdev I/O rate points at metadata fallback rather than a dying disk, saving a wrong hardware replacement.
• Alerts on special-class capacity and free space fire while there is still time to plan a resize, instead of during the performance incident.
• Historical retention lets you line up the fill event with the change that caused it, such as a replication job or a special_small_blocks change, using zpool history as the cross-reference.

Related guides

• ZFS capacity planning: runway estimation before the pool fills
• ZFS deleted files but no space freed: snapshots holding the blocks
• ZFS dedup memory exhaustion: when the DDT outgrows ARC and the pool crawls
• ZFS deadman events: hung I/O and a stalled pool sync
• ZFS ARC hit ratio low: cache misses, cold caches, and working sets that outgrew RAM
• ZFS checksum errors (CKSUM): the definitive signal of silent corruption