LVM commands hang: when lvs, vgs, and pvs block on locks or dead devices

You type lvs on a host during an incident and the prompt never comes back. Ctrl-C does nothing. kill -9 from another shell does nothing. The process is in D state, uninterruptible sleep, waiting on storage I/O or a metadata lock. Now vgs and pvs hang too, and your monitoring agent, which polls lvs every 60 seconds, has gone silent on exactly the host that is on fire.

This is not a bug in LVM. lvs, vgs, pvs, and every other LVM command acquire a VG lock and then read metadata from the header of every PV in the group. If the lock is held by a stuck operation, or if any PV is slow, dead, or behind a suspended device-mapper target, the command blocks. On a system with many PVs, a slow SAN, or one unresponsive device, “blocks” can mean minutes, or forever.

The dangerous part is the observability gap. The conditions that hang LVM commands (lock contention, dead devices, full thin pools queuing I/O, suspended dm devices) are precisely the conditions you need to observe during an incident. If your diagnostics and your monitoring both depend on the LVM management plane, you are blind at the worst moment. This article covers why the commands hang, how to tell lock waits from device waits, and how to keep diagnosing with dmsetup, which reads kernel memory directly and takes no LVM lock.

What this means

LVM is a userspace management layer over the kernel’s device-mapper subsystem. There are two planes:

• Management plane: lvs, vgs, pvs, lvchange, and friends. They open lock files under /run/lock/lvm/, serialize on the VG lock, and read metadata from PV headers on disk. They do real I/O.
• Kernel plane: device-mapper tables already loaded in kernel memory. dmsetup status, dmsetup info, and dmsetup ls query this directly. No LVM lock, no metadata I/O to PVs.

When the management plane hangs, the kernel plane is usually still responsive. That asymmetry is your escape hatch.

Two distinct hang mechanisms get conflated:

1. Lock waits. Another LVM process (a long pvmove, an lvconvert, a monitoring script, an automated snapshot job) holds the VG lock. Your command queues behind it. Devices are fine; the lock is not.
2. Device waits. The command got its lock and is now doing I/O to read PV metadata, and one device is unresponsive: a dead SAN LUN, a multipath device with all paths failed, a suspended dm device, or a thin pool that has run out of space and is queuing I/O under a queue_if_no_space policy. The process sits in D state and cannot be killed.

flowchart TD
  A[lvs / vgs / pvs invoked] --> B{VG lock free?}
  B -- no --> C[Queue behind lock holder
pvmove, lvconvert, other lvm procs]
  B -- yes --> D[Read metadata from every PV header]
  D --> E{All PVs responsive?}
  E -- no --> F[D state on dead/slow device
SAN loss, suspended dm, full thin pool]
  E -- yes --> G[Command returns]
  C --> F
  H[dmsetup status / info] --> I[Reads kernel dm tables
no lock, no PV I/O]
  I --> G

Common causes

Quick checks

Run these from a second session. All are read-only.

# 1. Is anything holding an LVM lock right now?
ps aux | grep -E 'lvm|pvs|vgs|lvs|lvcreate|lvextend|pvmove|lvconvert' | grep -v grep

# 2. Who owns the lock files?
ls -la /run/lock/lvm/
fuser /run/lock/lvm/* 2>/dev/null

# 3. Are processes stuck in D state (device wait rather than lock wait)?
ps -eo pid,stat,wchan:30,comm | grep ' D'

# 4. Does dmsetup still work? (It should. No LVM lock, no PV I/O.)
dmsetup ls
dmsetup info -c -o name,suspended

# 5. Is any dm device suspended?
dmsetup info -c --noheadings -o name,suspended

# 6. Thin pool state straight from the kernel
dmsetup status --target thin-pool

# 7. Kernel evidence of a dying device
dmesg | grep -iE 'I/O error|offline|not ready' | tail -50

# 8. Multipath health, if applicable
multipath -ll

# 9. How slow is a minimal LVM query when it does complete?
time pvs --noheadings

The fork in the road is check 3 versus check 1: if ps shows your hung lvs in D state with an I/O wait channel, you have a device problem. If it is sleeping on a lock file, you have a lock problem.

How to diagnose it

1. Confirm dmsetup works. Run dmsetup ls and dmsetup info -c. If these return instantly, the kernel side of LVM is healthy and your problem is in the management plane: locks or PV metadata I/O. If dmsetup also hangs, suspect the block layer or a kernel-level dm issue, not LVM locking.

2. Classify the hang: lock or device. List all LVM-related processes and their states. A hung lvs sleeping on a file lock points at another LVM process as the blocker. A hung lvs in D state points at device I/O. Check fuser /run/lock/lvm/* and match the PID to a live process. If the lock-holder PID does not exist, you may have a stale lock file (see Fixes).

3. If lock contention: identify the holder. The most common legitimate long holder is pvmove, which can run for hours on large volumes while holding serialization. lvconvert operations (mirror sync, cache conversion) on slow storage behave the same way. Check lvs -o lv_name,copy_percent from a working shell if you can get one, or find the operation in the process list. Also check for automation: snapshot scripts, backup jobs, and your own monitoring agent polling lvs every few seconds all take the same locks and can queue into a stampede.

4. If device wait: find the offending device. Correlate three sources: dmesg for I/O errors naming a specific sd*/nvme* device, multipath -ll for path failures, and dmsetup info -c -o name,suspended for a suspended target. A multipath device that has lost all paths with queuing enabled will block every I/O, including LVM metadata reads, indefinitely.

5. Check thin pools explicitly. dmsetup status --target thin-pool shows used_data/total_data and used_meta/total_meta. If data blocks are exhausted and the pool queues I/O on full, all writes to thin LVs queue in the kernel. Processes go to D state, and LVM commands that need I/O on those devices hang too. This is the thin pool death spiral, and it is why you reach for dmsetup status over lvs during incidents: lvs may itself need I/O to the hung storage. The kernel dm-thin module queues writes for up to 60 seconds by default when the pool is full (configurable via the no_space_timeout module parameter); after that, queued writes fail.

6. Rule out scale. If commands complete but take 10 to 60 seconds, count your block devices. LVM scans devices to find PVs, and on hosts with hundreds of SAN LUNs or multipath paths, the scan itself is the delay. This is degraded, not broken, but the fix (device filtering) differs from the fixes for locks and dead devices. On RHEL 9 the devices file is enabled by default, and LVM only scans devices listed in it, so a dead device not in the file is not scanned at all.

Metrics and signals to monitor

Fixes

Lock contention from a legitimate long operation

If the holder is a real pvmove or lvconvert, your options are to wait or to unwind cleanly. For pvmove, use pvmove --abort; never kill -9 a pvmove, since an interrupted migration leaves the LV in a temporary mirror state (lv_attr position 1 shows p) that you then have to complete or abort anyway. While the operation runs, use dmsetup for any diagnostics you need.

Automation and monitoring stampede

Reduce LVM polling frequency, serialize your scripts so only one runs at a time, and move high-frequency collection to dmsetup status, which takes no lock. A monitoring agent that shells out to lvs every 10 seconds takes the VG lock six times a minute and is part of the contention it reports on.

Stale lock files

If fuser shows a lock file held by a PID that no longer exists, a killed LVM process left it behind. Removing the lock file can unblock everything, but verify with ps that no LVM process is alive and mid-operation before deleting anything. Deleting a lock file under a live operation risks metadata corruption.

Dead or unresponsive device

Work the layer below LVM. For SAN, verify LUN presentation and rescan: echo '- - -' > /sys/class/scsi_host/host*/scan. For multipath, multipath -ll tells you whether all paths failed; a map with queuing enabled and no live paths blocks I/O indefinitely, and clearing or reconfiguring the map is the unblock. For a truly dead local disk, the PV is gone; see the missing-PV guide for the recovery path. Do not run vgreduce --removemissing until you have assessed which LVs lose data.

Suspended dm device

dmsetup resume <vg>-<lv> can unstick a suspended device, but investigate first. Suspension is brief and normal during resize (sub-second); a device that stays suspended for minutes with no operation in progress usually means a stuck table reload or an underlying device problem. Resuming into a still-broken backing device can make things worse.

Thin pool full

Extend the pool: lvextend -L +<size>G <vg>/<pool> if the VG has free extents. If the VG is also full, add a PV first (vgextend) or delete thin snapshots and run fstrim on thin LV filesystems to reclaim blocks. Note the uncomfortable catch: if the pool hang is severe enough that LVM commands block on I/O to the pool, the lvextend may also hang. This is the worst case of the observability gap and the reason capacity alerts at 85% exist: fix this an hour before it hangs, not during.

Prevention

• Monitor the management plane itself. Track how long pvs/vgs/lvs take. A healthy system answers in under a second; creeping latency is the earliest warning of scan scope growth, device trouble, or lock contention.
• Collect incident data via dmsetup. dmsetup status baselines (thin pool used/total blocks, suspended state) keep working when LVM tools do not. Collect them before you need them, or you will have no normal to compare against during an incident.
• Serialize LVM operations. One LVM job at a time per host. Snapshot scripts, backup jobs, and monitoring polls should never overlap.
• Bound multipath queuing. An all-paths-down device that queues I/O forever converts a path failure into a full LVM management-plane hang. Choose retry/queue policies deliberately.
• Keep thin pools out of the red zone. Alert at 85% data and 75% metadata, and verify auto-extend actually works (dmeventd running, threshold below 100, VG free space available), because a hung pool hangs your tooling too.
• Constrain device scans. On hosts with many SAN LUNs or multipath paths, use the lvm.conf filter (or the devices file on newer releases) so LVM only scans devices that can actually be PVs. This shrinks both scan time and the blast radius of a single dead device. On RHEL 9 the devices file is enabled by default (use_devicesfile=1); on RHEL 8 it is disabled by default.
• Never leave pvmove unwatched. It holds serialization for hours on large volumes. Monitor progress and I/O impact until it completes.

How Netdata helps

• Per-second block-layer metrics for dm devices from /proc/diskstats (IOPS, throughput, inflight I/O, accumulated I/O time) keep flowing during an LVM management-plane hang, because they come from the kernel, not from lvs.
• D-state process visibility lets you correlate a hung-lvs report with a growing count of processes in uninterruptible sleep, which is the tell that this is a device wait, not a lock wait.
• Thin pool capacity trending (data and metadata percent) gives you the 85%/90% early warning that prevents the queue-on-full hang in the first place.
• I/O latency on dm devices versus underlying PVs pinpoints whether the LVM layer or the physical layer is introducing the delay that is slowing your commands.
• Alerting on missing PVs and LV health flags catches the dead-device condition upstream of the hang, while pvs still works.

Related guides

• How LVM actually works in production: a mental model for operators
• LVM I/O hang: a suspended dm device and processes stuck in D state
• LVM Couldn’t find device with uuid: a physical volume has gone missing
• LVM logical volume partial (p) flag: which LVs the missing disk took down
• LVM logical volume not active: lvchange -ay and why activation failed
• LVM boot activation failure: emergency shell and missing mount points
• LVM metadata corruption recovery: vgcfgrestore from /etc/lvm/archive
• LVM Found duplicate PV: multipath devices and the lvm.conf filter
• LVM Insufficient free extents: the volume group is out of space
• LVM cannot extend a logical volume: adding a PV when the VG is full
• LVM filesystem full while the volume group has space: the resize step everyone forgets
• LVM dm-N device numbers change after reboot: use /dev/mapper, not /dev/dm-N