$ guides / lvm / lvm-lock-contention ▌

Operations Guides

LVM lock contention: stale lock files and blocked commands

You run lvs and it sits there. You try vgs in another terminal and it hangs too. Storage is fine, VMs are running, application I/O is flowing, but every LVM command blocks. Your monitoring, which polls lvs every 10 seconds, goes stale at the exact moment you need it.

This is LVM metadata lock contention. Every LVM command (pvs, vgs, lvs, lvcreate, lvextend, pvmove) acquires locks before reading or modifying VG metadata, using lock files in /run/lock/lvm/. When one command holds a lock and stalls, everything else queues behind it. Because the monitoring commands are LVM commands too, the management plane goes blind under the same condition it is supposed to be observing.

The kernel’s device-mapper layer does not participate in this locking. The data plane keeps running. dmsetup talks directly to the kernel with no LVM locks and no metadata disk I/O, so it remains your working diagnostic tool while every lvs variant is stuck.

What this means

LVM serializes metadata access to prevent concurrent commands from corrupting the VG mapping tables. On a single host this is file-based locking under /run/lock/lvm/. On clustered setups (lvmlockd with sanlock or DLM), lock acquisition involves network round-trips to a distributed lock manager, which adds latency and another failure domain.

Contention follows a predictable cascade:

flowchart TD
  A[Long LVM operation starts
pvmove, lvconvert, resize] -->|acquires and holds VG lock| B[VG lock held]
  C[Killed LVM process
leaves stale lock file] -->|lock never released| B
  B --> D[Subsequent LVM commands
queue behind the lock]
  E[Monitoring polls lvs
every 10-60s] -->|adds more waiters| D
  D --> F[pvs / vgs / lvs hang
monitoring data goes stale]
  F --> G[Operator blind during
any concurrent incident]
  H[dmsetup status] -.->|no LVM locks,
reads kernel memory| I[Diagnostics still work]

Two distinct situations produce the same symptom, and telling them apart is the whole diagnosis:

A live holder. A legitimate long-running operation (pvmove on a large volume, lvconvert mirror sync on slow storage) holds the lock for minutes or hours. The fix is to wait, or to abort the operation cleanly.
A stale lock. An LVM process was killed (or the box was hard-bounced mid-operation) and the lock file was never released. The holder PID no longer exists. Every subsequent command blocks on a lock that no live process owns. The fix is careful manual removal.

Common causes

Cause	What it looks like	First thing to check
Long pvmove holding the VG lock	All LVM commands slow or hung; pvmove process running for hours	`pvs --noheadings -o pv_name,move_pv` and `lvs -o lv_name,copy_percent`
lvconvert (mirror sync, cache conversion) stalled on slow storage	Lock held with no visible progress	`dmsetup status` for the affected device; check sync progress
Stale lock file from a killed LVM process	Everything hangs; the PID in `fuser` output does not exist in `ps`	`fuser /run/lock/lvm/*` then `ls -la /run/lock/lvm/` for lock file age
High-frequency lvs/vgs polling by monitoring or scripts	Contention spikes align with poll interval; many short-lived lvm processes	`ps aux` for repeated `lvs` invocations; check cron/systemd timers
Concurrent automation fighting for locks	Backup scripts, snapshot managers, provisioning jobs queued up	Cron and systemd timer inventory; process list depth of LVM commands
Clustered lock manager latency (sanlock/DLM)	Lock acquisition slow on all nodes, not just one	`lvmlockctl -i` or `dlm_tool ls` for lock manager state

Quick checks

All of these are read-only and safe to run during an incident.

# 1. Who is holding LVM locks right now?
fuser /run/lock/lvm/* 2>/dev/null

# 2. Lock file inventory and age. A lock file older than a few
#    seconds is abnormal for routine operations.
ls -la /run/lock/lvm/

# 3. What LVM processes exist, and what state are they in?
ps aux | grep -E 'lvm|pvs|vgs|lvs|lvcreate|lvextend|pvmove|lvconvert' | grep -v grep

# 4. Do the PIDs from step 1 actually exist?
#    A PID printed by fuser that does not appear in ps = stale lock.
ps -o pid,stat,wchan:30,etime,cmd -p <pid_from_fuser>

# 5. dmsetup bypasses LVM locks entirely. Use it to keep working.
dmsetup info -c -o name,attr,open,segments,uuid

# 6. Is a dm device suspended (a related but different failure)?
dmsetup info -c -o name,suspended

# 7. How slow is the management plane? Time a trivial query.
time pvs --noheadings 2>&1

# 8. Clustered setups only: inspect the lock manager.
lvmlockctl -i     # lvmlockd lock state
dlm_tool ls       # DLM lock status

Interpretation notes:

A healthy pvs completes in under a second. 1-5 seconds is slow. More than 10 seconds means contention or an unresponsive device in the scan path. A command that never returns is the signature of a blocked lock.
fuser printing a PID that exists in ps means a live holder: look at what it is running (wchan, elapsed time) before deciding anything.
fuser printing a PID that does not exist in ps, with an old lock file in /run/lock/lvm/, means a stale lock.

How to diagnose it

Confirm the symptom is locking, not storage. Run dmsetup status. If it returns instantly with sane output, the kernel data plane is healthy and you are dealing with a management-plane lock problem. If dmsetup also hangs or shows suspended devices, you may have an I/O hang instead. See LVM I/O hang: a suspended dm device and processes stuck in D state.
Enumerate the locks. ls -la /run/lock/lvm/ shows which locks exist and their timestamps. On modern systems you will see a global lock and per-VG locks.
Identify holders. fuser /run/lock/lvm/* maps lock files to PIDs. For each PID, check ps -o pid,stat,wchan:30,etime,cmd -p <pid>.
Classify each holder.
- Live long operation. etime is large, the command is pvmove or lvconvert, and lvs -o lv_name,copy_percent (it may be slow, but it should eventually return) shows progress. Decision: wait it out, or abort cleanly.
- Stuck live process. The process exists but is in D state on an unresponsive device. The lock is a symptom; the real fault is underneath (multipath, SAN, failing disk). Check dmesg for I/O errors before touching the lock.
- Stale lock. The PID from fuser does not exist. The lock file is orphaned.
Check for polling pressure. Even after clearing the immediate blockage, count how often something invokes LVM commands: cron entries, systemd timers, monitoring agents, backup scripts. A poller running lvs every 10 seconds acquires the VG lock 6 times a minute and can re-create the contention you just cleared.
Clustered only: if this VG is managed by lvmlockd, lock acquisition crosses the network to sanlock or DLM. Check lvmlockctl -i and cluster quorum before assuming a local stale file.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
LVM command execution time (`time pvs`)	Direct measure of management-plane health	Over 5-10 seconds, or a hang
Lock file presence and age in `/run/lock/lvm/`	Stale locks block everything	Lock file older than a few seconds with no live holder
Processes holding locks (`fuser`)	Identifies who is serializing the queue	Holder in D state, or holder PID absent from `ps`
pvmove / mirror sync progress (`copy_percent`)	Tells you whether a long lock hold is making progress	Progress unchanged for over an hour
dm device suspended state	Distinguishes lock contention from an I/O hang	Any device suspended for more than minutes
Count of concurrent LVM processes	Reveals automation pile-up	Many queued lvm commands from timers or scripts
D-state process count on dm devices	Corroboration that blockage is reaching applications	Growing count, processes stuck over 60 seconds

Fixes

Live holder: legitimate long operation

If the holder is a pvmove or lvconvert making progress, the safest fix is patience. These operations are I/O-bound and can legitimately hold a lock for a long time on large volumes or slow storage.

If you cannot wait, abort cleanly rather than killing the process:

# Abort an in-progress pvmove. Do NOT kill -9 a pvmove:
# an interrupted pvmove leaves the LV in a temporary mirror
# configuration that you will then have to unwind.
pvmove --abort

Tradeoff: an aborted pvmove has already moved some extents; re-running it later re-does work. But it releases the lock and restores the management plane.

Stale lock file

Only after you have verified (step 4 in diagnosis) that the holder PID does not exist and no live LVM operation is in progress:

# Verify the holder is gone, then remove the orphaned lock file.
# This is disruptive if you are wrong: removing a lock that a live
# process holds breaks mutual exclusion and risks metadata corruption.
# Double-check ps output for ANY running LVM command first.
rm /run/lock/lvm/<lock_file_name>

After removal, run a trivial pvs and confirm it returns promptly. Then figure out why the process was killed (OOM killer, operator Ctrl-C during a hung wait, hard reboot) so it does not recur.

Polling pressure

Replace high-frequency lvs/vgs/pvs collection with dmsetup-based collection for the signals that support it. dmsetup status reads kernel memory: no LVM locks, no metadata disk I/O. Thin pool data and metadata usage, device state, and I/O counters are all available without touching the lock. Reserve LVM commands for low-frequency checks (VG free space, LV attribute audits) that genuinely need metadata.

Clustered lock contention

If sanlock or DLM round-trips are the bottleneck, the lock itself is behaving correctly and removing local files will not help. Verify cluster quorum and lock manager health first. The default is 3 retries (global/lvmlockd_lock_retries in lvm.conf); sanlock itself manages its own retry timing for the internal lock LV. Cluster locking behavior is version-dependent; check the lvmlockd documentation for your distribution before changing retry behavior.

Prevention

Collect via dmsetup where possible. Thin pool data percent, metadata percent, and dm device state should come from dmsetup status, not lvs. This removes your monitoring from the lock queue entirely.
Rate-limit LVM metadata queries. If you must run lvs/vgs/pvs, do it on a slow cadence (minutes, not seconds) and never from multiple agents on the same host.
Inventory your automation. Snapshot managers, backup jobs, provisioning scripts, and cron jobs that invoke LVM commands should be serialized so they cannot pile up behind each other and a long operation at the same time.
Never kill -9 LVM operations. A killed pvmove leaves temporary mirror state; a killed metadata-writing command can leave a stale lock. Use the operation’s own abort path (pvmove --abort) or let it finish.
Alert on the management plane itself. Track LVM command execution time and lock file age so you learn about contention from your monitoring, not from a hung terminal at 3 a.m.
Monitor pvmove while it runs. A pvmove on a large volume holds the lock for hours. Watch copy_percent, watch I/O impact, and do not schedule other LVM work in the same window.

How Netdata helps

Lock contention is the failure mode where your monitoring tool can become part of the problem, so how you collect matters as much as what you collect:

Device-mapper metrics without LVM locks. Netdata reads dm device I/O, latency, and queue depth from kernel interfaces (/proc/diskstats, sysfs) rather than shelling out to lvs, so per-second visibility continues while LVM commands are blocked.
Process state correlation. A rising count of processes in uninterruptible sleep (D state), correlated with dm device activity, distinguishes a lock pile-up from a true I/O hang.
Latency divergence as an early warning. When dm device latency stays normal but management-plane operations slow down, the problem is locking, not storage. Comparing per-device latency against its baseline makes that split visible.
Long-operation context. Sustained I/O on specific PVs matching an in-progress pvmove or mirror resync tells you the lock hold is legitimate and progressing, versus stalled.
Historical trend for post-incident review. LVM itself keeps no history. Per-second I/O and process data from before the hang lets you reconstruct which operation started the queue.