ZFS deadman events: hung I/O and a stalled pool sync

A ZFS deadman event means the kernel has watched an I/O operation sit incomplete for at least five minutes, or a pool sync sit incomplete for at least ten minutes, and has given up waiting quietly. The event shows up as FM_EREPORT_ZFS_DEADMAN in zpool events, and it is one of the few ZFS signals that justifies waking someone up immediately.

The deadman is not a performance heuristic. It does not fire because latency is high or throughput is low. It fires because a specific operation has made zero progress for a span of time that no healthy storage stack ever produces. A SATA disk completing a read in five minutes is not slow. It is stuck.

Treat every deadman event as evidence of serious hardware, driver, or controller trouble until you have proven otherwise. This guide covers what the event measures, how to find the guilty component, and how to control what ZFS does next.

What this means

ZFS tracks outstanding I/O through its internal ZIO pipeline and tracks pool sync progress through the transaction group (TXG) machinery. The deadman subsystem watches both:

• Hung I/O (zio path): if an individual I/O operation has not completed within zfs_deadman_ziotime_ms (default 300000 ms, 5 minutes), the deadman fires.
• Stalled pool sync: if a TXG sync has not completed within zfs_deadman_synctime_ms (default 600000 ms, 10 minutes), the deadman fires.

Two properties make this a clean page-level signal:

1. It cannot false-fire on an idle system. No I/O means no hung I/O. An idle pool generates no deadman events, ever.
2. The thresholds are multi-minute. Even the heaviest backup, scrub, or resilver completes individual I/Os in seconds. A five-minute hung I/O is not a busy system. It is a stuck one.

The sync timer also resets whenever a TXG completes, so a slow-but-progressing sync will not trigger the deadman. It only fires on genuine stalls.

When the deadman fires, the response is governed by zfs_deadman_failmode:

• wait (default): log the event and keep waiting for the operation to complete.
• continue: log the event and let the system keep running rather than blocking on the hung operation.
• panic: panic the system, on the theory that a clustered failover is safer than a hung node.

flowchart TD
    A[ZFS issues I/O or starts TXG sync] --> B{Progress within threshold?}
    B -->|I/O completes under 5 min| C[Normal operation]
    B -->|Sync completes under 10 min| C
    B -->|I/O hung 5+ min| D[Deadman fires: FM_EREPORT_ZFS_DEADMAN]
    B -->|Sync stalled 10+ min| D
    D --> E{zfs_deadman_failmode}
    E -->|wait - default| F[Log event, keep waiting]
    E -->|continue - crash risk, avoid| G[Log event, keep running]
    E -->|panic| H[Kernel panic for failover]

Common causes

The deadman is a symptom, not a root cause. The event tells you something is stuck; the surrounding signals tell you what.

Quick checks

All of these are read-only and safe to run during an incident.

# 1. Find the deadman events themselves
zpool events -v | grep -A 20 deadman

# 2. Check pool and per-vdev state and error counters
zpool status -v

# 3. Watch per-vdev I/O in real time: which vdev is not moving?
zpool iostat -v 1

# 4. Check per-vdev latency averages: one device far slower than peers?
zpool iostat -l 1

# 5. Check queue depths: pending piling up on one vdev?
zpool iostat -q -v 1

# 6. Kernel log around the event: link resets, timeouts, aborts
dmesg | grep -i -E "ata|sas|reset|timeout" | tail -50

# 7. Confirm the deadman tunables in effect
for p in zfs_deadman_enabled zfs_deadman_ziotime_ms \
         zfs_deadman_synctime_ms zfs_deadman_failmode; do
  echo "$p = $(cat /sys/module/zfs/parameters/$p)"
done

# 8. Check TXG sync times: is a sync in progress and how long has it run?
cat /proc/spl/kstat/zfs/<pool>/txgs | tail -5

# 9. Confirm whether a scrub or resilver is running (rare cause, but rule it out)
zpool status | grep -A 5 "scan:"

Two notes on these checks:

• zpool events is in-memory only. The event buffer is lost on reboot. If the node panicked (failmode panic) or was power-cycled before you collected events, they are gone. This is why ZED-based persistence matters, covered under Prevention.
• A pool in SUSPENDED state is a different escalation path: I/O is blocked waiting for device connectivity. Check zpool status and see ZFS device UNAVAIL or REMOVED if a disk has fallen off the bus entirely.

How to diagnose it

1. Pull the event first. Run zpool events -v and capture the full deadman ereport, including the pool name, vdev path, and timestamps. Copy it somewhere off the host. If the machine later hangs or panics, this buffer is lost.

2. Determine which path fired. A hung-I/O deadman points at a specific operation (and usually a specific vdev). A stalled-sync deadman means the TXG sync has not completed in ten minutes. Check /proc/spl/kstat/zfs/<pool>/txgs: if the most recent TXG shows an extremely long sync phase or a sync still in flight, you are on the sync path.

3. Isolate the device. zpool iostat -v 1 for a few seconds shows which vdevs are moving and which are frozen. The frozen vdev is your suspect. Correlate with zpool iostat -l 1: a vdev whose average latency dwarfs its peers has been degrading before it hung.

4. Correlate with the kernel log. Match the event timestamp against dmesg. SATA link resets, SAS task aborts, and I/O timeouts on one device point at the disk, cable, or backplane. The same errors across many devices on one controller point at the HBA. Silence in dmesg combined with a deadman event points at the driver or at something below the guest (hypervisor, SAN, network target).

5. Check error counters. zpool status -v READ, WRITE, and CKSUM columns on the suspect vdev tell you whether this hang is the latest symptom of an ongoing failure. Non-zero and climbing means dying hardware. Zero everywhere with a hung I/O leans toward link, controller, or driver.

6. Rule out benign explanations. Confirm no scrub or resilver is running (zpool status scan line). Confirm the pool is not at the capacity-fragmentation cliff, where sync times can stretch badly. Neither should produce a true deadman (the sync timer resets on TXG completion, and even degraded scrubs complete I/Os), but ruling them out sharpens the hardware conclusion.

7. Decide the immediate action based on redundancy. If the hung vdev is part of a mirror or RAIDZ group with surviving redundancy, offlining or replacing the device is viable. If the pool has no redundancy left, prioritize getting data off before touching anything. Do not reboot as a first move: on the wait failmode the pool may recover on its own if the device comes back, and a reboot destroys the event buffer and any diagnostic state.

Metrics and signals to monitor

The leading indicators are the latency and queue-depth signals. A disk that will hang at 3 a.m. usually spent the prior days getting slower. Catching that drift is the difference between a planned replacement and a deadman page.

Fixes

If a specific disk is the cause

Confirm with error counters and dmesg, then replace it. If the pool has redundancy, you can offline the device to stop the hangs while you source a replacement. Follow your normal replacement procedure (zpool replace with the new device), then watch the resilver. If checksum errors are part of the picture, read ZFS checksum errors (CKSUM) for the corruption angle, and if errors appear on multiple unrelated devices at once, suspect RAM or the controller per ZFS checksum errors on multiple devices.

If the link, cable, or backplane is the cause

Link reset storms in dmesg with a healthy-looking disk usually mean the path, not the platter. Reseat or replace cables, try a different port or slot. These faults are often intermittent, which is exactly the profile that produces occasional deadman events with no permanent device failure.

If the controller is the cause

Multiple vdevs hanging together, controller-level errors in dmesg, or hangs that move between disks on the same HBA point at the controller or its firmware. Update firmware, check driver versions against your kernel, and plan a controller replacement. A failing controller can also corrupt data in flight, so scrub the pool after the hardware is stable.

If the sync path stalled without a device culprit

A stalled-sync deadman with clean dmesg and clean per-vdev stats is rare and usually points at the driver, the module, or something below the filesystem (hypervisor storage, network block device). Collect zpool events -v output and TXG history before any reboot, because this is the scenario where you will want evidence for a bug report.

Changing the failmode

You can change the deadman response at runtime:

# View current failmode
cat /sys/module/zfs/parameters/zfs_deadman_failmode

# Change it (takes effect immediately; persist via /etc/modprobe.d/zfs.conf)
echo wait > /sys/module/zfs/parameters/zfs_deadman_failmode

Choose deliberately:

• wait (default) is right for almost everything. The pool hangs, the event fires, an operator investigates. Data is not at additional risk while waiting.
• continue: avoid this in production. There is a long-standing OpenZFS bug (#10838) in which continue leads to a crash when a hung I/O actually occurs, reported reproducible on releases through 2.3.x. The recovery mode intended to keep you up can take the machine down.
• panic is for clustered designs where a panicked node triggers fencing and failover to a healthy peer. On a standalone host it converts a hung pool into a downed server. Only use it where an automated failover layer exists to catch the panic.

Do not tune the time thresholds downward to catch problems earlier. The five- and ten-minute defaults are what make the signal pageable without false positives. If you want earlier warning, alert on per-vdev latency and queue depth drift instead.

Prevention

• Persist events with ZED. zpool events is a RAM-only ring buffer. Run the ZFS Event Daemon and configure it to forward events to syslog or your alerting pipeline so deadman ereports survive a hang, panic, or reboot. This is the single most important step: without it, the worst incidents leave no evidence.
• Alert on the deadman event itself. Any occurrence pages. There is no threshold tuning to do.
• Watch the drift signals. Trend per-vdev latency and error counters so the slowly dying disk gets replaced during business hours, before it hangs at night.
• Keep scrubs on schedule. Scrubs surface device corruption early, while redundancy can still repair it.
• Baseline the whole pool. The ZFS monitoring checklist covers the full signal set, of which deadman monitoring is the top tier.

How Netdata helps

• Pool and vdev state collection from zpool status gives you per-vdev READ/WRITE/CKSUM counters as time series, so you can see the error growth that preceded the hang instead of a single cleared snapshot.
• Latency and throughput tracking per pool and per vdev surfaces the slow-drift pattern: one device pulling away from its peers in the days before a deadman event.
• TXG sync duration from /proc/spl/kstat/zfs/<pool>/txgs lets you distinguish a stalled sync from a slow one, and see sync times stretching before the ten-minute deadman threshold is reached.
• Event correlation in one timeline: when a deadman page fires, you can line up vdev latency, queue depth, error counters, and TXG sync time around the event timestamp instead of reconstructing it from six terminals.
• Kernel log context alongside the metrics connects dmesg link resets and timeouts to the ZFS-level symptoms, which is usually where the root cause actually lives.

Related guides

• ZFS monitoring checklist: the signals every production pool needs
• ZFS device UNAVAIL or REMOVED: a disk that fell off the bus
• ZFS dirty data throttling: the write delay that masquerades as slow disks
• ZFS checksum errors (CKSUM): the definitive signal of silent corruption
• ZFS checksum errors on multiple devices: suspect RAM or the controller, not the disks
• How ZFS actually works in production: a mental model for operators