$ guides / lvm / lvm-vgreduce-removemissing-recovery ▌

Operations Guides

LVM recovering a volume group after permanent PV loss: vgreduce --removemissing

When a physical volume (PV) disappears from the system, the volume group (VG) enters a partial state. LVM commands start printing “Couldn’t find device with uuid” warnings, and any logical volume (LV) with extents on the missing PV becomes inaccessible (linear/striped) or degraded (mirror/RAID). Once you have confirmed the PV is permanently lost, vgreduce --removemissing cleans the missing PV out of the VG metadata.

This is a one-way door. The command permanently removes the missing PV from VG metadata and, with --force, destroys any partial LV that depended solely on that PV. There is no automatic undo. LVM archives metadata before destructive operations, and vgcfgrestore can reverse some changes, but thin pool LVs and cache pool LVs removed by --force are not recoverable through metadata restoration.

What this command does

vgreduce --removemissing <VG> removes all missing PVs from VG metadata, but only if there are no LVs allocated on them. If LVs reference the missing PVs, the command refuses without --force.

With --force, any LV (or segment of an LV) that had extents on the missing PV is removed from VG metadata entirely. This is the destructive path.

The --mirrorsonly flag limits removal to missing PVs that are part of mirror or RAID LVs only, leaving non-redundant LVs intact. Use this when you want to clean up degraded RAID legs without destroying linear LVs that may still have recoverable data on surviving PVs.

Always run with --test first to preview exactly what will be destroyed.

Prerequisites: confirm the PV is permanently gone

The most common operator mistake is running this command when the PV is temporarily unreachable due to a loose cable, SAN path failure, multipath daemon outage, or a LUN that was temporarily unpresented.

# Check for missing PVs
pvs -o pv_name,vg_name,pv_attr,pv_size,pv_free
# Missing PVs show as "[unknown]" or with attr 'm'

# Check VG partial status
vgs -o vg_name,vg_attr,pv_count
# 'p' in vg_attr means partial (missing PV)

# Verify each PV device exists
for pv in $(pvs --noheadings -o pv_name); do
  [ -b "$pv" ] && echo "$pv: OK" || echo "$pv: MISSING"
done

# Check kernel log for device errors
dmesg | grep -i 'I/O error\|offline\|not ready\|device not found' | tail -50

Additional checks before concluding the PV is gone permanently:

SAN or iSCSI: Rescan the bus with echo '- - -' > /sys/class/scsi_host/host*/scan. Check multipath -ll for path status. Check iscsiadm for session state.
Multipath: Verify multipathd is running. A down multipath daemon can make PVs appear missing even when underlying paths exist.
Cable or connector: Physical inspection for loose SAS, SATA, or FC connections.
udev or systemd: Verify the device is not simply slow to appear after a reboot or rescan.

If the device reappears after any of these checks, do not run vgreduce --removemissing. The VG will recover automatically once the PV is visible again.

Assess the damage before running anything

Do not run vgreduce --removemissing until you have assessed the damage. Identify which LVs had extents on the missing PV and determine their redundancy status.

# Identify LVs with extents on the missing PV
pvs --segments -o pv_name,lv_name,seg_start_pe,seg_size_pe

# Check LV health status
lvs -o lv_name,vg_name,lv_attr,lv_health_status
# 'p' in position 9 of lv_attr means partial (missing PV)

# Check RAID health if applicable
lvs -o lv_name,lv_attr,copy_percent,raid_sync_action

flowchart TD
    A["PV reported missing"] --> B{"Permanently gone?"}
    B -- No --> C["Fix path, cable,\nor multipath"]
    C --> D["PV returns,\nVG recovers"]
    B -- Yes --> E["Assess damage:\nwhich LVs affected?"]
    E --> F{"Non-redundant LVs\non missing PV?"}
    F -- Yes --> G["Data is lost.\nRestore from backup."]
    F -- No --> H["RAID or mirror\ndegraded but functional"]
    G --> I["Run --test preview"]
    H --> I
    I --> J{"Acceptable\ndamage?"}
    J -- No --> K["Stop. Seek\nalternative recovery."]
    J -- Yes --> L["vgreduce --removemissing\n--force"]
    L --> M["Rebuild RAID or\nrestore from backup"]

The recovery path depends on the LV layout:

LV layout	Impact of missing PV	Recovery path
Linear or striped	Data on missing PV extents is lost. LV may be unrecoverable.	Restore from backup. `--force` removes the partial LV from metadata.
Mirror or RAID 1	Array degrades but continues on remaining leg(s).	Replace PV, rebuild with `lvconvert --repair`.
RAID 4/5/6	Degrades with reduced redundancy. RAID 6 tolerates two failures.	Replace PV, rebuild.
Thin pool (data or metadata LV on missing PV)	All thin LVs in the pool are affected.	Do not use `--force` without understanding that thin pool removal is irreversible.
Cache pool	Cache pool may be silently destroyed.	Recovery requires raw metadata extraction. See common pitfalls.

Procedure

Each step assumes you have already confirmed the PV is permanently gone and assessed the damage.

Step 1: Activate accessible LVs in partial mode.

# Activate LVs that do not depend on the missing PV
vgchange --activate y --partial <VG>

This brings up LVs whose extents are entirely on surviving PVs. LVs with extents on the missing PV remain inaccessible. If activation hangs, use dmsetup status to check device-mapper state directly, since dmsetup does not take LVM locks.

Step 2: Preview what will be destroyed.

# Always run --test first
vgreduce --removemissing --test <VG>

Review the output carefully. This shows exactly which PVs and LVs will be removed. If the output shows thin pool or cache pool LVs, stop and read the next section before proceeding.

Step 3: For RAID or mirror VGs, consider --mirrorsonly first.

# Remove only missing PVs from mirror/RAID LVs
# Leaves non-redundant LVs intact for separate recovery planning
vgreduce --removemissing --mirrorsonly --force <VG>

This preserves linear or striped LVs, giving you time to plan their recovery without committing to full metadata removal.

Step 4: Run the full removal.

# WARNING: permanently removes all partial LVs that depended on the missing PV
vgreduce --removemissing --force <VG>

Step 5: Verify the VG is consistent.

# Confirm no missing PVs remain
vgs -o vg_name,vg_attr,pv_count <VG>

# Confirm metadata is consistent
vgck <VG>

Step 6: Rebuild RAID or restore from backup.

# Add a replacement PV if hardware is available
pvcreate /dev/<new_device>
vgextend <VG> /dev/<new_device>

# Repair degraded RAID LV
lvconvert --repair <VG>/<raid_lv>
<!-- TODO: verify whether lvconvert --repair is the current preferred path for RAID recovery after vgreduce --removemissing, or if the error-segment behavior from RHEL 6.5 still applies -->

For non-redundant LVs whose data was on the missing PV, restore from backup. The metadata has been cleaned up, so you can recreate the LVs on surviving PVs.

Thin pools and cache pools are special

Two categories of LVs have irreversible consequences when removed by --force.

Thin pool LVs: If the thin pool’s data LV or metadata LV was on the missing PV, vgreduce --removemissing --force removes the thin pool from VG metadata. LVM cannot reverse this operation. vgcfgrestore will not recover a thin pool LV removed by this command. All thin volumes and thin snapshots in the pool are gone.

Cache pool LVs: vgreduce --removemissing on a VG with a cache pool LV can remove the entire cache LV without warning. Recovery in one documented case required extracting raw LVM metadata from disk images using dd, editing the metadata ring buffer, and using pvcreate --restorefile with a loop device. This is not a supported recovery path.

Before running --force, check whether the VG contains thin pools or cache pools:

# Check for thin pools
lvs -o lv_name,lv_attr -S 'lv_layout=thin,pool' <VG>

# Check for cache volumes
lvs -o lv_name,lv_attr -S 'lv_layout=cache' <VG>

If thin pools or cache pools exist and their backing PVs are among the missing, --force will destroy them. Consider whether the data or metadata LV can be recovered by other means before proceeding.

Undo with vgcfgrestore

LVM automatically archives metadata before every change. The archive file created before vgreduce --removemissing --force ran is the key recovery point if you need to reverse the operation.

# List available archive files for the VG
vgcfgrestore --list <VG>

# Restore from a specific archive file
vgcfgrestore -f /etc/lvm/archive/<VG>_<timestamp>.vg <VG>

Important limitations:

vgcfgrestore restores metadata, not data. It can bring back the LV’s mapping table, but the underlying physical extents on the missing PV are still gone.
vgcfgrestore will not recover thin pool LVs removed by --force. The thin pool removal is irreversible.
vgcfgrestore may refuse if the VG state changed while the device was missing, reporting an error about restoring metadata when a PV was missing during the last modification. The workaround is to create a loop device with the same UUID as the missing PV, then restore.
After vgcfgrestore, you must reactivate affected LVs manually.

vgcfgrestore is a safety net for metadata mistakes, not for data recovery. If a non-redundant LV was destroyed and its backing data was on a permanently lost PV, the data is gone regardless of metadata restoration.

Common pitfalls

Automated vgreduce in cluster failover scripts: Automated vgreduce --removemissing --force in cluster failover scripts can destroy VG metadata when LUNs are temporarily unseen on the secondary node. This has been documented in ClusterLabs resource-agents. If you run clustered LVM, never automate this command without a reliable quorum and device-visibility check.

Temporary path failure mistaken for permanent loss: A loose SATA connector, a multipath daemon restart, or a brief SAN fabric disruption can cause a PV to appear missing. Running vgreduce --removemissing in this state destroys the LV metadata even though the disk data is intact. Recovery is possible via vgcfgrestore from the archive taken before the command, but only if you notice quickly.

The 2.03.38 bug fix: LVM2 2.03.38 (December 2025) fixed an infinite loop in vgreduce --removemissing --force for RAID and mirror snapshots. If you are running an older version and the command hangs, this is a known bug.

LVM commands hanging during partial VG state: pvs, vgs, and lvs take a VG lock and read metadata from all PVs, including missing ones. During a partial VG state, these commands can hang for minutes waiting for I/O timeouts on the missing device. Use dmsetup status instead, which reads from kernel memory without taking LVM locks or performing disk I/O.

Signals to monitor

Signal	Why it matters	Warning sign
PV accessibility	Missing PV triggers the partial VG state that leads to this procedure	PV shows as `[unknown]` in `pvs` output
LV health status	Confirms which LVs have extents on the missing PV	Non-dash character in `lv_attr` position 9
VG metadata consistency	After recovery, verifies the VG is clean	`vgck` returns non-zero
RAID sync status	After rebuild, confirms redundancy is restored	`copy_percent` below 100 or devices showing `D` in health characters
LVM command execution time	Hung commands during partial VG state indicate the management plane is blocked	`pvs`, `vgs`, or `lvs` taking more than a few seconds
Kernel I/O errors	Confirms hardware failure versus presentation issue	`dmesg` showing I/O errors for specific device

How Netdata helps

Correlating PV accessibility with kernel I/O errors, multipath path status, and application I/O latency helps you distinguish permanent device loss from transient path failures before you reach for vgreduce --removemissing.

PV accessibility tracking (missing PV detection from pvs output) gives you the first signal that a device has disappeared, with severity escalation when non-redundant LVs are affected.
LV health status monitoring surfaces the p (partial) flag on affected LVs, telling you exactly which volumes are impacted before you assess the damage manually.
RAID sync progress and health character monitoring confirms whether redundant LVs are surviving on remaining legs, helping you decide between rebuild and restore.
LVM command execution time monitoring catches the management-plane hang that occurs when LVM tools try to read metadata from missing devices.
Kernel log collection provides the I/O error corroboration needed to confirm hardware failure versus SAN presentation or multipath issues.