$ guides / lvm ▌

LVM · OPERATIONS PLAYBOOK

LVM's cliff-edges: a thin pool that freezes at 100%, a snapshot that vanishes, and a disk you can no longer see

A storage virtualization layer where the kernel's device-mapper routes I/O from virtual volumes to physical extents, thin pools overcommit real space, and snapshots copy-on-write. We trace how that design behaves under pressure, where it turns from flexible into a hard stop, and what to do when a pool fills, a disk drops, or a volume group runs out of room.

> Start with the monitoring checklist → # Jump to the full guide list

LVM gives you flexible storage — resize on the fly, snapshot for backups, overcommit with thin provisioning — and in exchange hands you a set of cliff-edges that most teams meet for the first time during an incident.

It works until a thin pool crosses 100% and every volume backed by it freezes at once — not errors, silence, with processes wedged in D state. Until a traditional snapshot's copy-on-write store fills and the snapshot is invalidated instantly, taking the backup that depended on it. Until a disk drops and Couldn't find device with uuid turns a volume group partial. Until a mirror loses a leg and keeps running with zero redundancy and no visible error, one failure from total loss. Until the volume group runs out of free extents and lvextend, auto-extend, and snapshot creation fail together. Until lvs itself hangs on a VG lock during the exact incident you needed it to report.

These guides are written for engineers who already run LVM, not for people learning what a logical volume is. The goal is the mental model of how device-mapper, thin pools, snapshots, and volume groups actually behave under pressure, the failure patterns that keep recurring, the monitoring story that catches them before they wake anyone, and the runbooks you wish you had the last time a pool filled at 3AM.

How LVM actually runs in production

LVM is not a filesystem and not a disk. It is a mapping layer where the kernel's device-mapper routes I/O from virtual volumes to physical extents, thin pools overcommit real space, and snapshots copy-on-write. Most production failures live between these layers — a full pool, a missing disk, an exhausted volume group — not inside any one of them.

filesystem + application

What sits above LVM. <code>df</code> measures the filesystem, not the volume — a filesystem can be 100% full on a half-empty LV, or have room while the volume group underneath is exhausted. Extending storage is a two-step chain: grow the LV, then grow the filesystem (<code>resize2fs</code> / <code>xfs_growfs</code>).

logical volumes (LVs)

Virtual block devices exposed as <code>/dev/VG/LV</code>. Each LV is a mapping table describing which physical extents back it. Types — linear, striped, mirror/RAID, thin, snapshot, cache — have completely different storage and failure modes. The 11-character <code>lv_attr</code> string encodes type, activation state (position 5), and health (position 9).

device-mapper (dm)

The kernel subsystem that implements every active LV as a <code>/dev/dm-N</code> device with a mapping table in kernel memory. Table reloads during resize/reshape briefly suspend I/O; a device suspended for more than seconds wedges every process touching it in uninterruptible sleep. <code>dmsetup</code> talks to it directly — no locks, no disk I/O.

thin pool (data + metadata)

The overcommit layer: a data LV and a separate, much smaller metadata LV. Either can hit 100% independently. Data exhaustion freezes every thin LV in the pool; metadata exhaustion can corrupt it. Auto-extend is disabled by default (<code>thin_pool_autoextend_threshold = 100</code>).

POOL

snapshots + copy-on-write

Traditional snapshots keep a fixed-size COW exception store — it fills from writes to the <em>origin</em>, and at 100% the snapshot is invalidated instantly and irreversibly. Thin snapshots share the pool instead, avoiding the fixed store but inheriting the pool-full problem.

SNAP

volume group (VG)

The pool of physical extents that LVs are carved from. When free extents reach zero, LV extension, thin pool auto-extend, and snapshot creation all fail — a cliff-edge, not a slowdown. VG metadata is replicated across every member PV, with a sequence number that increments on each change.

physical volumes (PVs)

Raw block devices stamped with an LVM metadata header and divided into physical extents (4 MiB default). Identified by UUID, not device path. A PV that disappears makes the VG partial; LVs with extents on it go inaccessible (linear/striped) or degraded (RAID/mirror).

block devices below LVM

Disks, partitions, md RAID arrays, multipath maps, iSCSI/SAN LUNs, NVMe namespaces. LVM has little visibility into this layer, so a SAN path loss or a failing drive surfaces as an LVM PV problem two layers up — always check underneath before blaming LVM.

BLOCK

Why this matters: 'writes are hung', 'a volume disappeared', or 'the disk is full but df disagrees' can each come from a full thin pool, an exhausted volume group, a missing physical volume, a degraded mirror, an invalidated snapshot, a suspended device-mapper device, or a stuck LVM lock. The symptom rhymes — each layer has a different signal, and a different fix.

The failures you'll actually see

Most LVM incidents fall into a small set of recurring patterns. Recognise the shape, and triage gets dramatically faster.

CRITICAL

The thin pool wall

A thin pool's data_percent reaches 100% and every thin LV backed by it fails writes at the same instant. With the default queue_if_no_space policy the kernel queues I/O for 60 seconds and then errors — processes pile up in D state and, if root is on the pool, the box can freeze. This is a cliff-edge: the allocator only struggles slightly near 95%, then stops dead at 100%.

data_percent at or near 100% on the thin pool
lv_health_status showing D (out of data space)
Processes stuck in D state, writes to every thin LV hanging
dmsetup status shows used_data equal to total_data

Investigate →

CRITICAL

The vanished disk

A physical volume disappears — a failed disk, a lost SAN path, an unplugged cable, a detached cloud volume — and LVM commands start printing Couldn't find device with uuid. The volume group goes partial. Linear and striped LVs with extents on the missing PV return I/O errors; RAID and mirror LVs degrade but keep serving. LVM identifies PVs by UUID, so a path change alone does not cause this — the device is genuinely gone.

"Couldn't find device with uuid" from LVM commands
PV listed as [unknown]; VG showing the partial flag
dmesg reporting I/O errors for the underlying sd*/nvme* device
Linear or striped LVs on the missing PV returning I/O errors

Investigate →

IMMINENT

The silent redundancy loss

A leg of a mirror or a member of a RAID LV fails. The array keeps running on its surviving copies with no application-visible error — which is exactly why it is dangerous. Redundancy is now zero, and the next device failure is total data loss. The distinguishing detail: a (alive, not in-sync) is normal during a rebuild, while D (dead) is the failure indicator.

dmsetup status showing a D (dead) health character on a leg
copy_percent below 100 on an array that was previously synced
No application errors — the array runs on its surviving copy
lv_health_status flagged, redundancy effectively at zero

Investigate →

CRITICAL

The metadata cliff

A thin pool's metadata LV fills while its data space may look fine. This is rarer than data exhaustion and worse: the pool can enter a structurally corrupted state, not merely run out of room. Recovery via lvconvert --repair is not guaranteed, and the man page warns data may be unrecoverable. Teams watching only data_percent never see it coming.

metadata_percent at or near 100% while data_percent may be low
lv_health_status showing M (metadata read-only) or F (failed)
Writes to thin LVs returning I/O errors; pool refusing to activate
thin_check reporting errors on the _tmeta device

Investigate →

ACTIVE

The vanishing snapshot

A traditional copy-on-write snapshot fills its fixed exception store — driven by writes to the origin, not the snapshot — and at 100% is invalidated instantly and irreversibly. The origin is unaffected, so nothing pages; but any backup reading the snapshot silently fails, and the only restore point is gone. Multiple snapshots on one origin compound the write amplification that fills them.

snap_percent at 100 and lv_attr position 5 showing I (invalid)
"Invalidating snapshot" in the kernel log
A backup job reading the snapshot silently failing
Origin volume unaffected — only the restore point is lost

Investigate →

IMMINENT

The volume group with no room

The volume group runs out of unallocated extents. lvextend fails with Insufficient free extents, thin pool auto-extend cannot fire, and snapshot creation is refused — all at once. This is capacity exhaustion in slow motion, but the failure is sharp: no graceful degradation between 99% and 100%. The fix is to add a PV (pvcreate + vgextend) or free extents.

"Insufficient free extents" from lvcreate/lvextend
vg_free at or near zero; thin pool auto-extend failing
Snapshot creation failing for lack of space
Filesystem extension blocked because the LV cannot grow

Investigate →

Choosing a tool

Best LVM & Linux Storage Monitoring Tools (9 Ranked)

A ranked review of the tools teams actually shortlist here, what each one is genuinely good at, and how the pricing behaves as you scale.

Read the buyer's guide →

LVM monitoring maturity levels

LVM observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your storage matters. Most production systems with thin provisioning or snapshots should land at the second level.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor. With these signals you can answer one question: can we still allocate storage, and is anything in a failed state? You will not learn why, but you will learn that something broke before applications do. Survival is enough for dev boxes and simple linear-LVM systems.

VG free space Room to allocate, extend, or snapshot at all?
Thin pool data usage How close is the pool to freezing every thin LV?
PV accessibility Are all physical disks present, or is the VG partial?
LV health status (lv_attr) Any p (partial), D (out of data space), or F (failed) flag?
LV activation state Are the LVs that should be serving workloads actually active?

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production systems should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident on its own: pool exhaustion, snapshot overflow, degraded redundancy, a missing disk, an offline safety net.

Thin pool metadata usage The silent second dimension; can exhaust while data looks fine.
Snapshot COW usage A traditional snapshot at 100% is gone, and its backup with it.
Mirror / RAID sync status Is redundancy intact, or is the array running on one leg?
VG metadata consistency (vgck) Do all PVs still agree about the volume group?
dmeventd running Is the auto-extend and failure-event safety net actually on?
D-state process count Are applications already hung behind an I/O blockage?
VG free space trend Direction and runway, not just today's number.
dm device I/O latency Is the LVM layer adding latency over the raw disk?

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone up. An overprovisioned pool one bulk-import from full, a forgotten snapshot creeping toward invalidation, free extents stranded on the wrong PV, auto-extend that was never really enabled. None of these page you on day one; they become incidents on day thirty.

Thin pool overprovisioning ratio How fast a runaway writer could fill the physical space.
Snapshot age and count per origin COW overhead exposure and forgotten backup snapshots.
Resync progress and rate Time to full redundancy after a rebuild.
Per-PV free-space distribution Free extents on the wrong PV block striped allocation.
Auto-extend configuration audit Threshold 100 means disabled; verify it actually fires.
Thin pool allocation rate dmsetup deltas turn data% into a time-to-exhaustion estimate.
LVM command execution time A slow pvs/vgs/lvs is the management plane degrading.
Lock contention A held VG lock stalls every command, monitoring included.

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. dmsetup baselines, no-space policy audits, D-state stack captures, cross-layer latency correlation. Most teams never need every signal here. Add the ones your incident history says you do.

dmsetup status baselines The diagnostic that still works when lvs hangs; know normal first.
queue vs error no-space policy audit Whether a full pool hangs or errors — per pool, before the incident.
D-state process stack capture /proc/<pid>/stack pins the hang to the dm layer, not the app.
VG seqno + backup freshness Inter-PV metadata consistency and a current recovery copy.
Per-PV I/O error rate Failing drives show up in kernel error counts before they vanish.
Cross-layer latency correlation app to FS to dm to PV to disk, to locate the slow layer.
dm-integrity violations Checksum mismatches: bit rot or tampering, if integrity LVs are used.
LVM audit trail Unexpected pvcreate/vgcreate/lvcreate from the archive timestamps.

Operating mistakes worth avoiding

The traps LVM teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

Monitoring df but not the LVM layer

Filesystem usage and LVM capacity are independent failure domains. A thin pool can be 100% full — every write frozen — while <code>df</code> shows 50%, because space is virtually allocated but not yet physically consumed. Conversely a volume group can be exhausted while a filesystem still shows room. Track filesystem usage, VG free space, and thin pool usage as three separate dimensions.

Ignoring thin pool metadata usage

Teams alert on <code>data_percent</code> and never look at <code>metadata_percent</code>. The metadata LV is tiny (64 MiB–1 GiB) and a high-churn random-write workload can fill it even when data usage is moderate — and metadata exhaustion can be unrecoverable, not just full. Alert aggressively: a ticket at 75%, urgent at 90%.

Assuming thin pool auto-extend is enabled

The default <code>thin_pool_autoextend_threshold</code> is 100, which means disabled. Many operators believe their pools auto-grow and have never verified it. Even when configured, auto-extend fails silently if <code>dmeventd</code> is not running or the VG has no free extents — and the first test is the production incident. Set the threshold to 70–80 and confirm an extend has actually fired.

Leaving zombie snapshots in place

A snapshot created for a backup and forgotten sits idle, slowly consuming COW space from origin writes. Then a burst of writes fills it in seconds, it invalidates, and the restore point is gone — often discovered only during a recovery attempt. Alert on any snapshot older than 24 hours on a high-churn origin, and on usage crossing 80%.

Not alerting on degraded mirrors or RAID

Systems limp along on a degraded array because 'it still works'. The first device failure is invisible to applications; only the second failure — total data loss with no redundancy — gets noticed. Alert on any mirror/RAID showing <code>copy_percent</code> below 100 that was previously synced, or any <code>D</code> in the device health characters.

Monitoring with lvs/pvs/vgs instead of dmsetup

The LVM tools take a VG lock and read metadata from every PV, so during the exact incidents you most need data — a full pool, a hung device, an unresponsive disk — they hang, and your monitoring goes blind. <code>dmsetup status</code> reads from kernel memory with no lock and no disk I/O. Use it as the primary high-frequency collection path.

Not knowing your no-space policy

With the default <code>queue_if_no_space</code>, a full thin pool freezes the system (processes in <code>D</code> state) instead of returning errors — and teams chase a phantom kernel or hardware fault while LVM is the cause. <code>error_if_no_space</code> returns write errors applications can handle. Audit which policy each pool uses before it matters.

No physical topology awareness

LVM abstracts away the physical layer, which is its job — but two PVs on the same controller, shelf, or SAN give a volume group no more redundancy than one. Extending a VG by 'adding another disk' without verifying physical independence builds false confidence. LVM metadata shows nothing about the topology underneath it.

LVM runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

Volume group capacity

▸

Thin pools: data and metadata

▸

Snapshots and copy-on-write

▸

Physical volume loss and partial VGs

▸

Mirror and RAID redundancy

▸

Activation, device-mapper, and I/O hangs

▸

Metadata, corruption, and recovery

▸

dmeventd, locks, and the management plane

▸

Security and integrity

WHERE TO GO NEXT

Setting up LVM monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.

> Start with the checklist > Back to Operations Guides