ZFS pool I/O is currently suspended: a hung pool and blocked I/O

You ran a zpool command, or an application tried to touch a dataset, and you got this back:

cannot open 'tank': pool I/O is currently suspended

Or zpool status shows the pool in state SUSPENDED with an action line telling you to reconnect devices and run zpool clear. Every process touching the pool is stuck in uninterruptible sleep. New SSH sessions that touch the mountpoint hang. The system itself may still be responsive, but everything that depends on the pool is frozen.

This is one of the few ZFS states that is a genuine page-the-human emergency. SUSPENDED is a hardware-driven binary state. It cannot be caused by load, workload, or configuration drift, and it will not resolve itself. The pool has lost too many devices (or lost connectivity to the storage fabric) and ZFS has deliberately frozen all I/O rather than risk writing to a pool it can no longer trust.

This article covers what the state actually means, how to tell it apart from DEGRADED and FAULTED, what to check before you touch anything, and the recovery paths that work in practice, including the ones the man pages undersell.

What this means

ZFS tracks pool health as a small set of states. The ones operators see most are ONLINE and DEGRADED. SUSPENDED is different in kind:

• ONLINE: all vdevs functioning. Normal operation.
• DEGRADED: one or more vdevs failed, but redundancy covers the gap. The pool still serves I/O. You are one failure from data loss, but the lights are on.
• FAULTED: the pool cannot serve I/O at all. Too many devices are gone for redundancy to reconstruct data. Recovery is uncertain.
• SUSPENDED: the pool has encountered an uncorrectable I/O failure condition, and ZFS has blocked all I/O while it waits for device connectivity to be restored. The pool is not necessarily destroyed. It is frozen, waiting for a human.

The mental model that matters: DEGRADED is “serving I/O with reduced safety”, FAULTED is “cannot serve I/O”, and SUSPENDED is “refusing to serve I/O until you tell it the devices are back”. SUSPENDED exists because ZFS would rather hang forever than silently corrupt a pool whose devices vanished and might come back.

The trigger is the deadman subsystem. ZFS watches how long individual I/O operations and pool syncs take. If an I/O sits uncompleted past zfs_deadman_ziotime_ms (default 300000, five minutes) or a sync stalls past zfs_deadman_synctime_ms (default 600000, ten minutes), the deadman fires and the pool is suspended. You can see these events with zpool events. The response to a deadman is controlled by zfs_deadman_failmode (default wait), and the pool-level failmode property governs what happens to I/O when devices fail: wait blocks all I/O until connectivity returns and errors are cleared, continue attempts to return errors to new I/O, and panic forces a kernel panic so a failover or watchdog can take over.

stateDiagram-v2
    ONLINE --> DEGRADED: device fails, redundancy covers it
    DEGRADED --> ONLINE: replace device, resilver completes
    DEGRADED --> FAULTED: redundancy exhausted
    ONLINE --> SUSPENDED: deadman fires, devices unreachable
    DEGRADED --> SUSPENDED: further loss, I/O cannot complete
    SUSPENDED --> ONLINE: devices restored + zpool clear
    SUSPENDED --> FAULTED: devices never return, forced import

One documentation gap worth knowing: the zpoolconcepts man page lists only ONLINE, DEGRADED, and FAULTED as pool health states. SUSPENDED is real, shows up in zpool status, and is documented in zpool-status and zpool-clear, but it is a transient blocked condition layered on top of the persistent states, not a first-class health state in the concepts documentation.

Common causes

The common thread: SUSPENDED is almost always a physical or fabric-layer event, not a ZFS-internal logic fault. Deadman events are deliberately conservative. An idle pool generates no hung I/O and cannot false-fire; even the heaviest backup completes individual I/Os in seconds. Five minutes of hung I/O is always real.

Quick checks

These are read-only and safe on a hung pool. One caveat up front: on some OpenZFS versions, zpool status without a pool name can itself hang in uninterruptible sleep if any pool has an inaccessible vdev (OpenZFS issue #18446, observed on 2.4.1). Specify the pool name explicitly, and be prepared for the command to block anyway.

# Pool state and per-vdev errors - name the pool explicitly
zpool status tank

# Pool health as a single string
zpool list -H -o name,health

# Raw state from kstat (bypasses some ioctl paths)
cat /proc/spl/kstat/zfs/tank/state

# Deadman and I/O failure events
zpool events -v | grep -i -E "deadman|io_failure|probe_failure"

# Kernel view of what happened to the devices
dmesg | grep -i -E "ata|sas|nvme|reset|timeout|offline"

# Deadman tunables and current failmode
cat /sys/module/zfs/parameters/zfs_deadman_enabled
cat /sys/module/zfs/parameters/zfs_deadman_ziotime_ms
cat /sys/module/zfs/parameters/zfs_deadman_synctime_ms
cat /sys/module/zfs/parameters/zfs_deadman_failmode

# Pool-level failmode property
zpool get failmode tank

Also check which processes are stuck before you plan recovery:

# Processes blocked on the pool (uninterruptible sleep)
ps -eo pid,stat,wchan:30,comm | awk '$2 ~ /D/'

Anything in D state against the pool’s mountpoints will not respond to signals. You cannot kill your way out of this cleanly.

How to diagnose it

1. Confirm the state. zpool status tank (or cat /proc/spl/kstat/zfs/tank/state). You want SUSPENDED confirmed, plus the list of which vdevs are faulted or unavailable. The action line in zpool status will read roughly: “One or more devices are faulted in response to IO failures. Make sure the affected devices are connected, then run ‘zpool clear’.”

2. Identify what was lost. From zpool status -v, note which vdevs are UNAVAIL or FAULTED and whether they share a controller, enclosure, or fabric path. If everything on one HBA dropped, the pool hardware is probably fine and the controller is the failure domain.

3. Correlate with the kernel log. dmesg will show link resets, task aborts, NVMe controller resets, or device removals that line up with the suspension time. This tells you whether the devices are dead, disconnected, or merely reset-looping.

4. Check deadman history. zpool events -v shows the deadman and I/O failure events. Note that zpool events is in-memory only and is lost on reboot, so capture it now. ZED should be persisting these; check its logs if configured.

5. Check whether devices are visible to the OS now. If the devices have physically returned (link restored, enclosure repaired, fabric reconnected), recovery via zpool clear is on the table. If they have not, nothing ZFS-side will help until they are.

6. Check for multihost/MMP. If the pool has multihost=on, zpool clear will refuse to resume the pool if there is evidence it was imported by another host. Verify no other host has the pool imported before attempting recovery. The /proc/spl/kstat/zfs/<pool>/multihost kstat provides MMP statistics.

7. Check for layered storage. If the pool sits on LUKS, mdraid, or a SAN layer, the restore path runs through that layer first. For LUKS, the reported working sequence is: close the LUKS device, rescan the bus, re-decrypt, then zpool clear.

Metrics and signals to monitor

The correlation that shortens diagnosis: pool state SUSPENDED plus deadman events plus a burst of controller-level errors in dmesg points at the fabric or HBA. SUSPENDED plus climbing per-device error counts on one disk in the hours before points at a dying device that finally dropped off the bus.

Fixes

Restore connectivity, then zpool clear

This is the documented and preferred path. Once the affected devices are genuinely back (cable reseated, controller replaced, fabric restored, enclosure repaired):

# Resume a suspended pool after devices are accessible again
zpool clear tank

If the devices can be accessed, the pool resumes in place with no export/import cycle. Two caveats from the field:

• zpool clear can itself hang if the device is not fully accessible. “Reconnected” is not the same as “healthy”; a device that is reset-looping will wedge the clear. Confirm the device is stable at the OS level first.
• With multihost=on, the clear is refused if there is evidence the pool was imported elsewhere. Resolve the fencing question first.

Note that zpool reopen does not work on a suspended pool; it returns the same “pool I/O is currently suspended” error.

Reboot

Uncomfortable but true: across community reports, a reboot is the most reliable way out of a fully suspended pool. Once suspended, processes hold unkillable D-state references, unmount paths block, and zpool destroy on a suspended pool is a known broken path (OpenZFS issue #2878, open for years). A clean shutdown may itself hang when it tries to unmount the suspended pool; plan for a hard power-off as a possibility.

After reboot, the pool imports fresh and, if the underlying devices are healthy, comes back ONLINE. If they are not, you are in DEGRADED or FAULTED territory and the recovery runbook changes. See ZFS pool DEGRADED: redundancy lost and one failure from data loss.

Read-only import for data evacuation

If the devices are marginal and you do not trust them for continued production, import read-only and copy data off:

# Import read-only to evacuate data from a questionable pool
zpool import -o readonly=on tank

This is a salvage tactic, not a fix. If individual disks are failing, clone them with ddrescue first and import against the clones.

failmode=panic and failmode=continue: know what you are choosing

If you are configuring a pool in advance, the failmode property changes what “suspended” means for you:

• failmode=wait (default): blocks all I/O indefinitely until devices return and errors are cleared. Safest for data, worst for availability.
• failmode=panic: panics the kernel on pool failure. Sounds extreme, but for HA pairs with automated failover or a watchdog, a panic is a clean, fast, unambiguous failure signal. Better than a host that is half-alive with all I/O hung.
• failmode=continue: intended to return EIO to new I/O instead of blocking. Be aware of the long-standing defect (OpenZFS issue #7990, still open): in-flight and queued writes still block, and processes still end up in D state, so in practice it behaves much like wait when the pool suspends. Do not rely on it to keep applications responsive.

Prevention

You cannot prevent hardware from failing, but you can control how bad a suspension is when it happens.

• Multipath and fabric redundancy. Most suspensions are single-path fabric or controller events. Redundant paths convert a suspension into a brief blip.
• Choose failmode deliberately. For clustered or HA storage, panic plus automated failover beats an indefinitely hung host. For standalone archive boxes, wait protects the pool.
• Alert on deadman events and pool state. SUSPENDED is a PAGE state. Deadman events are PAGE. Wire ZED to notify on both; zpool events alone is in-memory and lost on reboot.
• Monitor per-vdev errors as leading indicators. Devices rarely drop off the bus with zero warning. Climbing READ/WRITE/CKSUM counts on one device, or latency divergence on one vdev, usually precede the drop by days.
• Baseline device and controller health. SMART data, enclosure management, and HBA event logs catch the hardware that is about to take a pool down.
• Test the recovery path. Know before the incident whether your shutdown hangs on a suspended pool and how long a hard reset plus import takes for your pool size.

How Netdata helps

Netdata surfaces the signals that let you catch a pool sliding toward suspension and diagnose it faster once suspended:

• Pool state per pool, read from the ZFS kstats, so a transition to SUSPENDED (or DEGRADED) pages immediately rather than waiting for an application timeout report.
• Per-vdev error counters (READ, WRITE, CKSUM) trended over time, so the dying device that preceded the drop is visible in history, not just in the current zpool status snapshot.
• Pool I/O throughput and latency, so you can see I/O flatline at the moment of suspension and correlate it with what applications were doing.
• Kernel and hardware error context alongside pool metrics, shortening the “is it the disk, the controller, or the fabric” question.
• Historical retention, which matters because the first thing a suspended pool destroys is your ability to investigate it live; the hours of rising error counts before the event are the real diagnosis.

Related guides

• How ZFS actually works in production: a mental model for operators
• ZFS monitoring checklist: the signals every production pool needs
• ZFS monitoring maturity model: from survival to expert
• ZFS pool DEGRADED: redundancy lost and one failure from data loss