$ guides / lvm / lvm-io-hang-suspended-device-d-state ▌

Operations Guides

LVM I/O hang: a suspended dm device and processes stuck in D state

Processes are stuck in D state. kill -9 does nothing. lvs hangs. Load average is climbing. The root cause is almost always a device-mapper device stuck in suspended state.

A suspended dm device blocks all I/O to the underlying logical volume. Every process performing I/O to that volume enters uninterruptible sleep (D state) and cannot be killed, even with SIGKILL. The only way to release them is to resolve the underlying I/O blockage and let queued I/O drain.

The critical diagnostic insight: lvs, pvs, and vgs take LVM metadata locks and perform disk I/O. During this incident, those tools hang alongside everything else. dmsetup talks directly to the kernel device-mapper interface with no locks and no disk I/O. It is the tool that works when nothing else does.

Suspension is normal dm operation. It happens during every resize, snapshot creation, and table reload. The key distinction is duration: normal suspension is sub-second. Sustained suspension for more than a few seconds is abnormal.

What this means

When a dm device is suspended, the kernel device-mapper target queues all incoming I/O requests without processing them. No reads complete. No writes complete. The kernel marks every process waiting on that I/O as TASK_UNINTERRUPTIBLE (D state). SIGKILL cannot interrupt this state because the kernel cannot safely free resources that may be in use by in-flight I/O.

flowchart TD
    A[dm device suspended] --> B[All I/O to LV blocked]
    B --> C[Processes enter D state - unkillable]
    C --> D[D-state count grows, load climbs]
    D --> E[Possible full system hang]
    B --> F[lvs pvs vgs also hang - need disk I/O]
    F --> G[dmsetup still works - reads kernel memory]
    G --> H[dmsetup status pinpoints cause]
    H --> I[Resolve blockage: extend pool or resume device]
    I --> J[Queued I/O drains, D-state processes release]

The cascade builds quickly. A dm device enters suspended state and stays there. I/O requests queue. Processes waiting on those requests go D state. More processes touch the affected filesystem, they queue too. Load average climbs. If the affected LV backs the root filesystem, database storage, or /var/log, the system may become fully unresponsive.

Common causes

Cause	What it looks like	First thing to check
Thin pool exhaustion (queue_if_no_space)	All thin LVs in the pool affected simultaneously; dmsetup status shows pool data at 100%; `D` in lv_attr health position 9	`dmsetup status --target thin-pool` for used_data/total_data ratio
Failed snapshot removal	Specific snapshot device SUSPENDED after lvremove failed or was interrupted; not the whole pool	`dmsetup info -c` for suspended flag on snapshot device
Stuck resize or table reload	An lvextend, lvconvert, or pvmove was running when the hang started; specific LV affected	`ps aux \| grep -E 'lvextend\|lvconvert\|pvmove'`
Multipath queue_if_no_path	All paths to a multipath device down; pvs/vgs/lvs hang on device scanning; any LV behind the device affected	`multipath -ll` for path status

Thin pool exhaustion is the single most common cause. With the default queue_if_no_space policy, a full thin pool does not return errors to applications. It hangs silently. Operators investigate “server hung” as a kernel or hardware problem when LVM is the cause.

Quick checks

These commands are read-only and safe to run during an incident.

# Check which dm devices are suspended (works when lvs hangs)
dmsetup info -c -o name,suspended,open,attr

# Check thin pool data and metadata utilization from kernel memory
dmsetup status --target thin-pool

# Count D-state processes
ps aux | awk '$8 ~ /D/' | wc -l

# Identify D-state processes and their wait channels
ps -eo pid,stat,wchan:30,comm | grep ' D'

# Check kernel stack of a specific D-state process
cat /proc/<pid>/stack | grep -i 'dm\|thin\|mapper'

# Check for kernel I/O errors on underlying devices
dmesg | grep -i 'I/O error\|offline\|not ready' | tail -50

# Check if dmeventd is running (the auto-extend safety net)
pgrep -x dmeventd

# Check LVM lock file age (old lock file = stuck operation)
ls -la /run/lock/lvm/

# Time an LVM command to see if the management plane is responsive
time vgs --noheadings 2>&1

The single most important command is dmsetup status --target thin-pool. It reads pool utilization from kernel memory without touching disk. If used_data_blocks equals total_data_blocks, you have found the cause.

How to diagnose it

Step 1: Bypass LVM tools entirely. Start with dmsetup, not lvs.

dmsetup info -c -o name,suspended,open

Look for devices where the SUSPENDED column shows s. The OPEN column shows how many processes have the device open. High open count on a suspended device means many processes are stuck on it.

Step 2: If you use thin provisioning, check pool status.

dmsetup status --target thin-pool

The output includes used_data_blocks/total_data_blocks and used_metadata_blocks/total_metadata_blocks. If used_data equals total_data, the pool is full. The output also shows the active policy: queue_if_no_space (hangs on full) or error_if_no_space (returns errors on full).

Step 3: Confirm the D-state correlation.

# D-state processes and their wait channels
ps -eo pid,stat,wchan:30,comm | grep ' D'
# Stack trace for a specific stuck process
cat /proc/<pid>/stack

If D-state processes cluster around the same dm device minor number and that device is suspended, the diagnosis is confirmed. Check /proc/<pid>/stack for dm_ or thin_ function names to verify the process is blocked in the device-mapper layer.

Step 4: Check whether an LVM operation is stuck.

# Look for running LVM operations
ps aux | grep -E 'lvextend|lvconvert|pvmove|lvremove|lvcreate'
# Check lock files
ls -la /run/lock/lvm/
# If a lock file is old and the holding PID no longer exists, it may be stale
fuser /run/lock/lvm/* 2>/dev/null

Step 5: Check underlying storage for hardware errors.

dmesg | grep -i 'I/O error\|device offline\|link down' | tail -50

Kernel I/O errors on the underlying PV device can cause the dm layer to suspend I/O as a protective measure.

Step 6: Check multipath if applicable.

multipath -ll

If all paths to a multipath device are down and queue_if_no_path is enabled, LVM commands hang during device scanning. This is not a dm-suspend issue but presents identically.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
dm device suspended state	Suspended dm device blocks all I/O; every process touching it goes D state	Any device SUSPENDED for more than 5 seconds with no approved LVM operation in progress
D-state process count	Growing count means incident is escalating; processes are unkillable	Count increasing over time, processes clustered on same dm device
Thin pool data utilization	At 100% with queue_if_no_space, all writes to all thin LVs freeze	data_percent above 95% and climbing
Thin pool metadata utilization	Metadata exhaustion can cause pool corruption, not just I/O freeze	metadata_percent above 75%
LVM command execution time	Slow commands mean management plane is degraded; monitoring goes blind	pvs/vgs/lvs taking more than 5 seconds
LV health status (attr position 9)	`D` = thin pool out of data space; `F` = failed; `M` = metadata read-only	Any non-`-` value in position 9
Kernel I/O errors in dmesg	Hardware failures, SAN path loss, cable issues	New I/O error messages for PV backing devices

Fixes

Thin pool exhaustion (the most common cause)

If dmsetup status --target thin-pool shows the pool is full, extend the pool LV, not the individual thin volumes.

# Extend the thin pool data LV (the pool, not the thin volumes)
lvextend -L +<size>G <vg>/<thinpool>

# If VG has no free space, add a PV first
pvcreate /dev/<new_device>
vgextend <vg> /dev/<new_device>
lvextend -L +<size>G <vg>/<thinpool>

If you cannot extend the pool immediately, deleting unnecessary thin snapshots frees pool blocks:

# List thin snapshots sharing the pool
lvs -o lv_name,origin -S 'lv_layout=thin,sparse'
# Remove a snapshot to reclaim pool space
lvremove <vg>/<snapshot_name>

Running fstrim on thin LV filesystems can reclaim discarded blocks:

fstrim /mountpoint

Once the pool has free space, queued I/O begins completing automatically. D-state processes release as their writes finish. No dmsetup resume is needed for thin pool cases; the device un-suspends once the pool can accept allocations again.

The kernel dm-thin module queues writes for up to 60 seconds by default when the pool is full (configurable via the no_space_timeout module parameter); after that, queued writes fail.

Failed snapshot removal leaving a suspended device

If dmsetup info shows a snapshot device in SUSPENDED state after a failed or interrupted lvremove, manually resume the device:

# Resume a stuck snapshot device
dmsetup resume /dev/mapper/<vg>-<snapshot>

This is the documented workaround for the case where a failed snapshot removal leaves the dm device suspended, blocking all further I/O.

Stuck LVM operation

If an lvextend, lvconvert, or pvmove is holding a lock and not completing, identify the holder:

fuser /run/lock/lvm/* 2>/dev/null

If the operation is legitimately long-running (a large pvmove), it may eventually complete. If it is stuck on I/O, resolve the underlying storage issue first.

For pvmove, use pvmove --abort to cleanly unwind. Never kill a pvmove with kill -9.

Multipath queue_if_no_path

If all paths to a multipath device are down, restore storage connectivity first. LVM commands and dm devices recover once paths return. Check multipath -ll for path status and investigate the storage fabric, HBA, or SAN presentation.

Last resort: forced removal or reboot

dmsetup remove --force replaces the device table with one that fails all I/O. This may allow D-state processes to error out and be killed. This is destructive: all in-flight data is lost.

The --force option replaces the device table with one that fails all I/O, per dmsetup(8). Stuck D-state processes then receive I/O errors and can exit.

If the root filesystem is on the affected LV and the system is fully unresponsive, the only remaining option may be a forced reboot via out-of-band management (IPMI, console). Use SysRq if possible:

# Emergency sync (may not complete if I/O fully blocked)
echo s > /proc/sysrq-trigger
# Remount filesystems read-only
echo u > /proc/sysrq-trigger
# Reboot immediately
echo b > /proc/sysrq-trigger

SysRq requires kernel.sysrq enabled. If I/O is completely blocked, even SysRq sync may not complete. A hard reset via IPMI may be the only option.

Prevention

Enable thin pool auto-extend explicitly. The default thin_pool_autoextend_threshold is 100, which means disabled. Set it to 80 or lower in /etc/lvm/lvm.conf with a nonzero thin_pool_autoextend_percent.
Verify dmeventd is running. Auto-extend only works if dmeventd is alive and monitoring the pool. Check pgrep -x dmeventd and the lvm2-monitor.service systemd unit.
Monitor thin pool data AND metadata utilization. Data exhaustion causes I/O freeze. Metadata exhaustion can cause pool corruption. Both must be watched independently.
Use dmsetup for monitoring, not lvs. LVM tools take locks and perform disk I/O. During the incidents you most need visibility, they hang. Build monitoring around dmsetup status for production thin pool tracking.
Audit pool policies before the incident. Know which pools use queue_if_no_space (hang on full) versus error_if_no_space (return errors on full). Use dmsetup status --target thin-pool to see the active policy. For pools backing stateful workloads that can handle write errors cleanly, consider switching to error_if_no_space via lvchange --errorwhenfull y.
Maintain VG free space headroom. Auto-extend silently fails if the VG has no free extents. Keep enough free space for at least two thin pool extension cycles.

How Netdata helps

Per-second dm device metrics: Netdata collects disk I/O statistics for every dm device at one-second resolution, including inflight I/O and latency. A sudden spike in inflight I/O with zero completed operations is the signature of a suspended device.
D-state process tracking: Netdata monitors process states across the system. A rising D-state count correlated with a specific dm device identifies the affected volume within seconds.
Thin pool utilization trends: Netdata collects data_percent and metadata_percent for thin pools. Rate-of-change analysis projects when the pool will reach 100%, giving hours of lead time.
Kernel log correlation: Netdata collects kernel log events alongside metrics. Block I/O errors from dmesg correlate with dm device state changes, pinpointing whether the cause is LVM-level (thin pool full) or storage-level (hardware failure).
LVM command execution latency: Netdata can detect when the management plane is degrading. A vgs command taking 10 seconds instead of the usual sub-second is an early warning that device scanning is hitting unresponsive hardware.