LVM Operations Guides on Netdata

How LVM actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

Most LVM incidents are caused by operators holding the wrong mental model: thinking of LVM as “partitioning with extra steps” rather than what it actually is, a userspace management layer that programs the kernel’s device-mapper subsystem. Once you internalize that, the confusing behaviors stop being confusing. A full thin pool freezing every process in D-state, a snapshot silently invalidating, an lvs command hanging during the exact incident you need it for: all of these follow directly from the architecture.

LVM boot activation failure: emergency shell and missing mount points

Mon, 01 Jan 0001 00:00:00 +0000

You boot a server and land in a dracut emergency shell or initramfs rescue prompt. Or the system boots but /mnt/data, /var/lib/postgresql, or other non-root filesystems are not mounted. The common thread: LVM volumes that should have activated during early boot did not.

The initramfs activates LVM volumes before mounting root and before systemd mounts the rest of fstab. If the initramfs lacks the right LVM tools or configuration, or if a required PV is slow to appear (iSCSI not connected, multipath not configured, SAN LUN not presented), activation fails before you have a running system to diagnose from.

LVM cannot extend a logical volume: adding a PV when the VG is full

Mon, 01 Jan 0001 00:00:00 +0000

You ran lvextend and it refused with some variation of “insufficient free space” or “insufficient free extents,” while the filesystem that prompted all this sits at 99% and application writes fail. The fix is mechanical once you know which of three independent constraints is actually blocking you.

The trap is that “no space” means three different things in an LVM stack. The filesystem can be full while the LV has room. The VG can be out of free extents while every filesystem looks fine. A thin pool can be 100% full while the VG reports free space. Each constraint has a different remediation, and running the wrong one either does nothing or makes the incident worse.

LVM commands hang: when lvs, vgs, and pvs block on locks or dead devices

Mon, 01 Jan 0001 00:00:00 +0000

You type lvs on a host during an incident and the prompt never comes back. Ctrl-C does nothing. kill -9 from another shell does nothing. The process is in D state, uninterruptible sleep, waiting on storage I/O or a metadata lock. Now vgs and pvs hang too, and your monitoring agent, which polls lvs every 60 seconds, has gone silent on exactly the host that is on fire.

This is not a bug in LVM. lvs, vgs, pvs, and every other LVM command acquire a VG lock and then read metadata from the header of every PV in the group. If the lock is held by a stuck operation, or if any PV is slow, dead, or behind a suspended device-mapper target, the command blocks. On a system with many PVs, a slow SAN, or one unresponsive device, “blocks” can mean minutes, or forever.

LVM Couldn't find device with uuid: a physical volume has gone missing

Mon, 01 Jan 0001 00:00:00 +0000

The error “Couldn’t find device with uuid” appears when LVM’s volume group metadata references a physical volume UUID that no block device on the system currently claims. Every LVM command that touches the affected VG prints the warning. The PV shows as [unknown] in pvs output, and the VG enters a partial state.

The recovery path depends entirely on why the device disappeared and what LV layout sits on top of it. The wrong recovery command, applied too quickly, causes permanent data loss. If system uptime is more than a few minutes and a PV is gone, something has failed at the hardware, fabric, cloud, or operator layer.

LVM device node permissions: raw block access that bypasses the filesystem

Mon, 01 Jan 0001 00:00:00 +0000

Every active logical volume on a Linux host exposes device nodes at /dev/mapper/<VG>-<LV> and /dev/<VG>/<LV>. The default permissions on these nodes are brw-rw---- root:disk (mode 0660). Anything more permissive than that, a mode like 0666, or a group assignment that includes unprivileged users, lets any local user read and write raw block data on the volume. Filesystem permissions, ACLs, and mount options do not apply at this layer. A user who can open the device node can read /etc/shadow straight off the disk or overwrite filesystem metadata directly.

LVM dm-N device numbers change after reboot: use /dev/mapper, not /dev/dm-N

Mon, 01 Jan 0001 00:00:00 +0000

After a reboot, you run lsblk or df -h and notice your logical volumes have different /dev/dm-N numbers. What was dm-0 before the reboot is now dm-2, and what was dm-1 is now dm-0. If anything on the system references /dev/dm-0 directly, it may now point at the wrong volume.

This is not a bug and not a sign of corruption. Device-mapper minor numbers are assigned dynamically at activation time based on the order devices are discovered and activated. The numbers are not stored persistently anywhere. The device-mapper tables that back logical volumes exist only in kernel memory and are torn down on every shutdown, then recreated on every boot.

LVM dmeventd not running: the auto-extend safety net is offline

Mon, 01 Jan 0001 00:00:00 +0000

You found this page because systemctl is-active lvm2-monitor.service returned something unexpected, or pgrep dmeventd came back empty, or lvs -o+seg_monitor showed “not monitored” on a thin pool you assumed was protected.

Here is the uncomfortable part: nothing is broken right now. No errors in lvs. No failed services your alerting noticed. No I/O hangs. The system looks healthy, and it is, right up until the moment a thin pool hits 100%, a mirror leg dies, or a snapshot overflows. Then you discover, during the incident, that the daemon which was supposed to fire the safety net has been dead for weeks.

LVM filesystem full while the volume group has space: the resize step everyone forgets

Mon, 01 Jan 0001 00:00:00 +0000

df -h says the filesystem is 100% full. Applications are getting ENOSPC. vgs shows plenty of free space. Maybe you already ran lvextend and df still shows the old size. Nothing here is broken. You are looking at three layers that measure three different things, and one of them was never told to grow.

lvextend grows the logical volume. The filesystem on top keeps its old size until you explicitly resize it. The LV is a bigger container; the filesystem inside has not expanded into the new space.

LVM Found duplicate PV: multipath devices and the lvm.conf filter

Mon, 01 Jan 0001 00:00:00 +0000

When LVM prints “Found duplicate PV” during boot or routine commands, it has found the same PV UUID on two or more block devices. On multipath-attached storage, this almost always means LVM is scanning both the raw SCSI paths (/dev/sdb, /dev/sdc) and the aggregated multipath device (/dev/mapper/mpatha). Each path carries the same PV metadata header.

The warning is advisory at first: LVM picks one device and proceeds. The danger is which one. If LVM activates a logical volume through a raw path instead of the multipath device, you lose path redundancy and failover. I/O runs through a single HBA link until that link fails, then the volume goes dark. The system can boot fine for weeks until a udev timing race changes which device LVM selects, and the next path failure takes down production.

LVM has free space but striped or mirrored allocation still fails

Mon, 01 Jan 0001 00:00:00 +0000

vgs shows the volume group 30% free. lvcreate for a striped or mirrored logical volume, or lvextend on an existing one, fails anyway:

Insufficient suitable allocatable extents for logical volume

The VG has headroom and the operation needs less than that headroom. Both facts are true at once, and that is the trap: vg_free is an aggregate across every physical volume in the group, but striped and mirrored allocations are not aggregate operations. They are per-PV placement problems. LVM must find extents on multiple distinct PVs at the same time, and if your free space is piled onto one PV, the allocation has nowhere legal to go.

LVM I/O hang: a suspended dm device and processes stuck in D state

Mon, 01 Jan 0001 00:00:00 +0000

Processes are stuck in D state. kill -9 does nothing. lvs hangs. Load average is climbing. The root cause is almost always a device-mapper device stuck in suspended state.

A suspended dm device blocks all I/O to the underlying logical volume. Every process performing I/O to that volume enters uninterruptible sleep (D state) and cannot be killed, even with SIGKILL. The only way to release them is to resolve the underlying I/O blockage and let queued I/O drain.

LVM Insufficient free extents: the volume group is out of space

Mon, 01 Jan 0001 00:00:00 +0000

You ran lvcreate, lvextend, or lvresize and got one of these:

Insufficient free extents
Insufficient free space: 12800 extents needed, but only 0 available

Both messages mean the same thing: the volume group has no unallocated physical extents left for the allocation you requested. Every physical extent in the VG is already assigned to some logical volume, and the operation fails outright.

This is a cliff-edge failure, not a gradual one. A VG at 99% allocated works exactly like a VG at 50% allocated. At 100%, every allocation operation fails immediately: LV creation, LV extension, snapshot creation, and thin pool auto-extend. Existing LVs keep serving I/O, so applications look fine until the moment something tries to grow.

LVM lock contention: stale lock files and blocked commands

Mon, 01 Jan 0001 00:00:00 +0000

You run lvs and it sits there. You try vgs in another terminal and it hangs too. Storage is fine, VMs are running, application I/O is flowing, but every LVM command blocks. Your monitoring, which polls lvs every 10 seconds, goes stale at the exact moment you need it.

This is LVM metadata lock contention. Every LVM command (pvs, vgs, lvs, lvcreate, lvextend, pvmove) acquires locks before reading or modifying VG metadata, using lock files in /run/lock/lvm/. When one command holds a lock and stalls, everything else queues behind it. Because the monitoring commands are LVM commands too, the management plane goes blind under the same condition it is supposed to be observing.

LVM logical volume not active: lvchange -ay and why activation failed

Mon, 01 Jan 0001 00:00:00 +0000

A logical volume that should be serving data shows as inactive. In lvs output, lv_attr position 5 is - where you expect a. No device-mapper table is loaded, no /dev/dm-N device exists, and nothing can mount, open, or write to that LV. Applications see a missing block device, and if this LV backs root or a critical service, the system may not have booted properly.

lvchange -ay is the standard activation command, but it can fail for several distinct reasons. With thin snapshots that carry the activation skip flag, lvchange -ay can even exit 0 without activating the LV. Always verify with dmsetup ls or lvs that the LV actually transitioned to active state.

LVM logical volume partial (p) flag: which LVs the missing disk took down

Mon, 01 Jan 0001 00:00:00 +0000

You run lvs and see a lowercase p in the ninth character of lv_attr. That means a physical volume (PV) backing extents in this logical volume has disappeared from the system. The VG is now in partial mode, and some LVs may be silently degraded or completely inaccessible.

The p flag is direct evidence of device loss: a disk failed, a SAN LUN was unpresented, a multipath device lost all paths, or a cloud volume was detached. The question is how bad the damage is and which volumes took the hit.

LVM metadata backup stale: /etc/lvm/backup out of sync with the VG

Mon, 01 Jan 0001 00:00:00 +0000

Every LVM command that changes metadata (lvcreate, lvextend, lvremove, vgextend, snapshot creation) writes two things to the local filesystem: an archived copy of the new metadata in /etc/lvm/archive/ and a current-state backup in /etc/lvm/backup/<vg>. That backup file is what vgcfgrestore reads when you recover a VG after metadata corruption.

The failure mode here is quiet: something blocks those writes, the backup file stops updating, and nothing in day-to-day operation tells you. LVM keeps working. Months later you hit metadata corruption, reach for vgcfgrestore, and restore a VG layout from six months ago. LVs created since then are missing. LVs removed since then come back pointing at extents that have been reallocated. You have traded a recovery exercise for a data integrity problem.

LVM metadata corruption recovery: vgcfgrestore from /etc/lvm/archive

Mon, 01 Jan 0001 00:00:00 +0000

LVM keeps the complete map of your logical volumes in a small metadata area at the start of each physical volume. If that metadata is corrupted, by a power loss during a metadata write or a bad sector in the first few megabytes of a PV, the entire volume group becomes unreadable. Every LV in the VG becomes inaccessible at once, regardless of how healthy the actual data extents are.

LVM mirror resync storm: multiple rebuilds saturating disk I/O

Mon, 01 Jan 0001 00:00:00 +0000

Production latency is climbing. Multiple applications are slow. Storage shows high I/O across all physical volumes, but there is no single failed disk and no obvious error in the logs. When you check LVM, several mirrored logical volumes show copy_percent below 100%. They are all resyncing at the same time.

This is a mirror resync storm. After a system crash, a disk replacement, or bulk mirror creation, multiple mirrored LVs may need to resynchronize simultaneously. Each resync generates substantial sequential and random I/O as the mirror copies dirty regions or entire leg contents to the recovering side. When several run in parallel, the combined I/O overwhelms storage bandwidth. Production workloads suffer elevated latency, and each resync takes longer than it would alone because they compete for the same physical devices.

LVM monitoring checklist: the signals every production volume manager needs

Mon, 01 Jan 0001 00:00:00 +0000

Most LVM incidents are not exotic. They are a thin pool that hit 100% because nobody watched metadata_percent, a mirror that ran degraded for three months because nobody alerted on copy_percent, or a PV that went missing on a Friday and paged nobody because the check only ran df. The failure modes are well known. What is usually missing is a deliberate, minimal set of signals with thresholds and severities attached.

LVM monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Most teams monitor LVM by accident. They alert on filesystem fullness, maybe on disk failure, and discover thin pool exhaustion or a silently degraded mirror only when applications start returning I/O errors. LVM has its own failure domains that filesystem metrics cannot see: a thin pool can be at 100% data usage while df shows 50% free, and a mirror can run on one leg for months with no visible symptom.

LVM pvmove stuck or interrupted: temporary mirror state and how to unwind it

Mon, 01 Jan 0001 00:00:00 +0000

You ran pvmove to drain a disk, it has been running for hours, and now something is wrong. Maybe the SSH session died, maybe the host rebooted, maybe someone killed the process. Now lvs shows a mirror LV you never created, and every LVM command against the volume group feels slow or blocked.

This is the pvmove temporary mirror state. It is recoverable in almost every case, but only if you unwind it the way LVM expects: resume it with a bare pvmove, or abort it with pvmove --abort. The one move that turns a tedious situation into a bad one is kill -9 on the pvmove process, or worse, deleting the mirror LV by hand.

LVM RAID mismatch count: data integrity after a scrub

Mon, 01 Jan 0001 00:00:00 +0000

A non-zero raid_mismatch_count means the scrub found blocks where data on one leg disagrees with data on the other leg(s). This surfaces as the m flag in position 9 of lv_attr and in the raid_mismatch_count field of lvs. It demands investigation, though some mismatches turn out to be benign.

The operational question is not “do we have mismatches” but “is this real corruption or expected noise.” RAID1 and RAID10 arrays can report non-zero mismatch counts from documented edge cases in the kernel write path that produce alignment differences in transient data areas. Mismatches appearing after an unclean shutdown are expected until the scrub completes and should not trigger paging on initial boot.

LVM RAID or mirror degraded: a leg is dead and you are one failure from data loss

Mon, 01 Jan 0001 00:00:00 +0000

Your LVM RAID1 or mirrored logical volume is still serving reads and writes. Applications see no errors. Filesystems are mounted. But one leg is dead and you are running on a single copy with zero redundancy. The next disk failure, cable disconnect, or SAN path loss is total data loss.

The default raid_fault_policy is warn. When a leg fails, dmeventd logs a warning but takes no repair action. No resync is triggered, no alert fires unless your monitoring explicitly checks for degraded state. The system runs indefinitely on the surviving leg until the second failure removes all copies.

LVM RAID resync stuck: copy_percent not progressing

Mon, 01 Jan 0001 00:00:00 +0000

You are watching lvs output on a RAID logical volume, and copy_percent has not moved for over an hour. The array is degraded, a resync or rebuild is supposed to be underway, and the percentage is frozen. Meanwhile, the volume is running on fewer healthy legs than intended, and every minute without full redundancy is a minute where a second disk failure means data loss.

This is one of the more frustrating LVM failure modes because the system appears functional. Applications are still running. The degraded array is still serving I/O. But the rebuild that should restore redundancy is not progressing, and without active diagnosis, it may never complete.

LVM reached low water mark for data device: the thin pool warning before the freeze

Mon, 01 Jan 0001 00:00:00 +0000

You found this line in dmesg or the journal:

device-mapper: thin: 253:4: reached low water mark for data device: sending event

This is not an error. It is the dm-thin kernel target reporting that a thin pool’s data device has crossed its low water mark and that a device-mapper event has been sent to userspace. dmeventd listens for that event and can auto-extend the pool. On a correctly configured system, you may see this message once, dmeventd extends the pool, and usage drops back below the threshold.

LVM recovering a volume group after permanent PV loss: vgreduce --removemissing

Mon, 01 Jan 0001 00:00:00 +0000

When a physical volume (PV) disappears from the system, the volume group (VG) enters a partial state. LVM commands start printing “Couldn’t find device with uuid” warnings, and any logical volume (LV) with extents on the missing PV becomes inaccessible (linear/striped) or degraded (mirror/RAID). Once you have confirmed the PV is permanently lost, vgreduce --removemissing cleans the missing PV out of the VG metadata.

This is a one-way door. The command permanently removes the missing PV from VG metadata and, with --force, destroys any partial LV that depended solely on that PV. There is no automatic undo. LVM archives metadata before destructive operations, and vgcfgrestore can reverse some changes, but thin pool LVs and cache pool LVs removed by --force are not recoverable through metadata restoration.

LVM snapshot COW usage climbing: extend or remove before it overflows

Mon, 01 Jan 0001 00:00:00 +0000

lvs shows a traditional snapshot with snap_percent at 78%. An hour ago it was 61%. You have a fixed-size COW exception store filling up from writes to the origin volume, and when it reaches 100% the snapshot is invalidated instantly and permanently. There is no warning state, no graceful degradation, and no recovery. If a backup is reading from that snapshot, the backup dies with it.

This is a countdown, not a trend. The right response is a quick calculation (how fast is it filling, how much time is left) followed by one of three actions: finish the backup faster, extend the snapshot, or remove it.

LVM snapshot invalid: the COW exception store filled and the snapshot is gone

Mon, 01 Jan 0001 00:00:00 +0000

You run lvs and see it: a snapshot with snap_percent at 100.00 and an I in the fifth position of lv_attr. In the kernel log there is a line like Invalidating snapshot: Unable to allocate exception. The snapshot is dead. It did not degrade, it did not warn you at the application layer, and it cannot be brought back.

This article covers traditional (non-thin) LVM snapshots only. Thin snapshots live in a thin pool and fail differently; if your snapshots are thin, the relevant failure is pool exhaustion, not this one. See LVM thin pool out of data space for that case.

LVM snapshot slowing the origin: copy-on-write write amplification

Mon, 01 Jan 0001 00:00:00 +0000

Write latency on a logical volume doubles or worse with no obvious cause. The disks are healthy, dmesg shows no I/O errors, no RAID resync is running, and the workload has not changed. The one thing that did change: someone created a snapshot, often hours or days ago, usually for a backup that has long since finished.

This is a traditional (thick) LVM snapshot working as designed. While the snapshot exists, the first write to every chunk of the origin triggers a copy-on-write (COW) cycle: read the original chunk, write it to the snapshot’s exception store, then write the new data. One application write becomes a minimum of three physical I/Os. Under write-heavy load, origin latency doubles or worse, and it degrades further as the exception store fills.

LVM thin pool auto-extend not working: threshold 100 means disabled

Mon, 01 Jan 0001 00:00:00 +0000

Your thin pool hit 100% data usage, writes to every thin LV in the pool started failing, and the auto-extend you thought was protecting you never fired. Or you are reading this before that happens, auditing a pool that has been “protected” for months without anyone ever confirming the mechanism works.

The most common root cause is a one-line surprise in /etc/lvm/lvm.conf: the default value of thin_pool_autoextend_threshold is 100, and a threshold of 100 does not mean “extend when the pool is full”. It means auto-extend is disabled. The commented-out line # thin_pool_autoextend_threshold = 100 in the shipped config looks like a sensible default. It is the off switch.

LVM thin pool check needed: thin_check and lvconvert --repair

Mon, 01 Jan 0001 00:00:00 +0000

A thin pool that reports “check needed” is telling you the kernel no longer trusts the pool’s metadata B-tree. The pool may still be serving I/O, or it may already be erroring writes. Either way, the wrong move (resizing metadata, forcing activation, running the wrong repair path on an oversized metadata LV) can turn a recoverable state into permanent data loss.

This state usually appears after metadata exhaustion, an unclean shutdown during a metadata update, or an underlying storage error. It shows up in lvs output as c (check needed) or C (check needed, suspended) in position 5 of lv_attr, often alongside a health flag in position 9. If you have not read the background on how thin pools are structured, see how LVM actually works in production first. This page covers only detection, validation, and repair.

LVM thin pool metadata full: the exhaustion that can corrupt the pool

Mon, 01 Jan 0001 00:00:00 +0000

Your thin pool’s metadata_percent has hit 100. Writes to every thin LV in the pool are failing or hanging, and yet lvs shows data_percent at 40%. That is not a contradiction. It is the defining feature of thin pool metadata exhaustion, and it is the LVM failure mode most likely to cost you data.

Metadata exhaustion is more dangerous than data exhaustion for one reason: a full data area stops new allocations, but a full metadata area can leave the pool structurally corrupted. The metadata LV holds the block mapping tables for every thin LV and snapshot in the pool. When the kernel can no longer allocate a metadata block mid-transaction, it aborts the transaction and switches the pool to read-only mode. From that point, recovery via lvconvert --repair is possible but not guaranteed, and the lvmthin(7) man page warns plainly that data from thin LVs may ultimately be unrecoverable.

LVM thin pool out of data space: every thin volume freezes at once

Mon, 01 Jan 0001 00:00:00 +0000

The symptom usually arrives as “the server hung” rather than “the storage filled up.” Processes writing to any thin volume in the pool go into uninterruptible sleep (D state). Applications stop mid-write without returning errors at first. Databases stall, VMs pause, containers wedge. If root or swap sits on a thin LV in the pool, the whole system becomes unresponsive and SSH sessions freeze.

The underlying event is simple: the thin pool’s data device reached 100% allocation, and the kernel’s device-mapper thin-pool target cannot satisfy a new block allocation for any thin volume in that pool. Because all thin LVs share the pool, they all stop at once. This is the cliff-edge failure behind most thin provisioning postmortems.

LVM thin pool overprovisioning: 60% full can still be dangerous

Mon, 01 Jan 0001 00:00:00 +0000

Your thin pool shows 60% data usage. Every dashboard is green, no alert has fired, and the pool has sat around that number for weeks. Then a single process on one thin volume starts writing aggressively, and eleven minutes later every thin LV in the pool is frozen: databases crash, VMs pause, and writes across the entire pool queue for 60 seconds and then fail with I/O errors.

This is one of the classic LVM states that looks normal but is silently dangerous. The number you are watching, data_percent, measures physical pool occupancy. It says nothing about how much data the thin volumes are allowed to write. That second number is the overprovisioning ratio, and it is the risk multiplier that data_percent hides.

LVM thin pool queue_if_no_space: why a full pool hangs instead of erroring

Mon, 01 Jan 0001 00:00:00 +0000

The server looks dead. Processes pile up in uninterruptible sleep, shells that touch certain filesystems never return, and even lvs hangs. There are no I/O errors in the application logs, no kernel oops, nothing in dmesg that screams hardware. Teams burn the first hour of this incident investigating a kernel or storage fault, when the actual cause is an LVM thin pool that ran out of data space.

The reason there are no errors is the pool’s when-full policy. The default for LVM thin pools is queue_if_no_space: when the pool fills, the kernel queues write I/O instead of failing it. The alternative, error_if_no_space, returns write errors immediately, which applications can see, log, and handle. Most operators have never checked which policy their pools use, because the default is silent until the pool fills.

LVM thin pool space not reclaimed: discard, TRIM, and fstrim

Mon, 01 Jan 0001 00:00:00 +0000

You deleted 200 GB of files from a filesystem on a thin LV. df shows the space as free, but lvs still reports the thin pool data_percent exactly where it was. Nothing is broken in the reporting: this is how thin provisioning works. Deleting a file only updates filesystem metadata. The pool blocks that held the data stay allocated until a discard (TRIM) operation travels from the filesystem, through the thin LV, into the pool.

LVM thin snapshots vs old-style COW snapshots: which failure mode you inherit

Mon, 01 Jan 0001 00:00:00 +0000

LVM offers two snapshot mechanisms that look similar from the command line but fail in fundamentally different ways. The choice between them is about which failure mode you inherit when something goes wrong.

Old-style COW (copy-on-write) snapshots allocate a fixed exception store at creation time. When that store fills to 100%, the snapshot is invalidated permanently and silently. The origin volume continues running. You lose the snapshot and whatever backup or rollback point it represented, but production keeps going.

LVM unexpected metadata changes: auditing pvcreate, vgcreate, and lvcreate

Mon, 01 Jan 0001 00:00:00 +0000

You found a logical volume nobody remembers creating. Or a volume group grew a new physical volume overnight. Or lvdisplay output does not match your configuration management records. None of your change tickets explain it, and now you need to answer two questions fast: what changed, and who changed it.

LVM metadata changes are a high-signal security event because every pvcreate, vgcreate, lvcreate, lvremove, and vgextend requires root. An unexpected LVM change outside a change window means root ran that command, which narrows the possibilities to an authorized root process you forgot about (automation, a package script, a provisioning tool) or an unauthorized one (a compromised service that escalated, a rogue script, an attacker with a shell).

LVM vgck metadata inconsistency: PVs disagree about the volume group

Mon, 01 Jan 0001 00:00:00 +0000

vgck returned non-zero on a production VG, or you are seeing warnings like “Inconsistent metadata found for VG” or “ignoring metadata seqno N on /dev/sdX for seqno M on /dev/sdY.” The physical volumes in the volume group hold different versions of the VG metadata. One or more PVs are stale: they missed a metadata update because they were offline, unreachable, or had bad blocks when the write happened.

LVM replicates complete VG metadata across every member PV. Each metadata change increments a sequence number (vg_seqno). When all PVs agree, the VG is consistent. When they disagree, you have split metadata. The impact ranges from noisy warnings on every LVM command to failed LV activation and, in the worst case, an unreadable VG.

LVM volume group is partial: operating a VG with a missing PV

Mon, 01 Jan 0001 00:00:00 +0000

You ran vgs or lvs and saw something wrong. The VG attributes show a p in the fourth position (wz--pn- instead of wz--n-). One or more PVs show as [unknown] in pvs output. Commands that touch the VG now print warnings about a missing device and may refuse to run.

The VG is in partial mode. LVM metadata still references a PV UUID that no block device on the system currently claims. The metadata itself is intact on surviving PVs, which is why you can still see the VG at all. But any LV that had physical extents on the missing PV now has a hole in its mapping table.

LVM volume group running low on free space: vg_free and runway estimation

Mon, 01 Jan 0001 00:00:00 +0000

Your volume group still works: LVs are active, filesystems are mounted, applications are writing. But vgs shows vg_free shrinking week over week, and the trend says you hit zero sometime next month. VG exhaustion is a cliff edge: there is no graceful degradation between 99% and 100% full. Everything works until an allocation request cannot be satisfied, and then it fails immediately. Watching vg_free is the warning the cliff never gives you.