LVM Couldn't find device with uuid: a physical volume has gone missing

The error “Couldn’t find device with uuid” appears when LVM’s volume group metadata references a physical volume UUID that no block device on the system currently claims. Every LVM command that touches the affected VG prints the warning. The PV shows as [unknown] in pvs output, and the VG enters a partial state.

The recovery path depends entirely on why the device disappeared and what LV layout sits on top of it. The wrong recovery command, applied too quickly, causes permanent data loss. If system uptime is more than a few minutes and a PV is gone, something has failed at the hardware, fabric, cloud, or operator layer.

What this means

LVM identifies physical volumes by UUID, not by device path. The PV metadata, stored in the first ~1 MiB of each device, contains a UUID that the volume group references in its own metadata. When you run pvs, vgs, or lvs, LVM scans block devices for PV labels and matches them against the VG’s expected PV list by UUID.

When the scan finds nothing claiming a UUID that the VG expects, LVM marks that PV as missing. The PV appears in pvs output as [unknown] with the missing attribute (m). The VG’s attribute field shows p in the partial position. Any LV with extents on the missing PV gets p in the health position of lv_attr (position 9 in current LVM2).

The impact depends on LV layout:

• Linear and striped LVs: any extent on the missing PV is inaccessible. If filesystem metadata (superblock, journal, inode tables) landed on those extents, the entire filesystem may be unreadable even if most extents were on surviving PVs.
• Mirrored and RAID LVs: the array degrades but continues operating on remaining legs. A single missing PV in RAID1 or RAID5 is survivable. A second failure during degraded state is total data loss.
• Thin pools: if the missing PV hosted the thin pool’s data LV or metadata LV, all thin volumes in the pool are affected simultaneously.

flowchart TD
    A["PV missing: [unknown] in pvs"] --> B{"Underlying device visible at OS level?"}
    B -->|"No"| C["Check dmesg, multipath -ll, SAN, cloud console"]
    B -->|"Yes"| D["Check lvm.conf filter, pvck"]
    C --> E{"Device returns after rescan?"}
    E -->|"Yes"| F["vgextend --restoremissing"]
    E -->|"No"| G["Assess LV damage: which extents were lost"]
    D --> H["Fix filter or restore PV label via pvcreate --uuid"]
    G --> I["RAID/mirror: rebuild on remaining legs"]
    G --> J["Linear/striped: restore from backup"]

Common causes

Quick checks

All read-only and safe to run on a production system.

# Confirm which PV is missing and its VG
pvs -o pv_name,vg_name,pv_attr,pv_size,pv_free
# Missing PVs show as [unknown] or with attr containing 'm'

# Check VG partial status and missing PV count
vgs -o vg_name,vg_attr,vg_missing_pv_count
# vg_missing_pv_count > 0 means a PV is gone

# Identify which LVs have extents on the missing PV
pvs --segments -o pv_name,lv_name,seg_start_pe,seg_size_pe

# Check LV health status (position 9 of lv_attr)
lvs -o lv_name,vg_name,lv_attr
# 'p' in position 9 means partial (extents on missing PV)

# Look for kernel-level device errors
dmesg | grep -i 'I/O error\|offline\|not ready\|device not found' | tail -50

# Check block device presence and multipath health
lsblk -o NAME,TYPE,SIZE,STATE,MOUNTPOINT
multipath -ll

# Verify VG metadata consistency on surviving PVs
vgck -v <vgname>

# Confirm dm devices are still active (works when LVM tools hang)
dmsetup ls | grep '<vgname>'

How to diagnose it

1. Identify the missing PV UUID. The error message includes the UUID. Confirm it in pvs output where the device shows as [unknown]. Note the VG name.

2. Determine if the underlying device is visible to the OS. Check lsblk, /sys/block, and /dev for the expected device. If the device is completely gone (no /dev entry, no /sys/block entry), the problem is below LVM: hardware, SAN, fabric, or cloud layer.

3. Check kernel logs for the root cause. Run dmesg | grep -i 'I/O error' to see if the kernel reported device failures. SCSI timeout, medium error, and link reset messages point to hardware. No kernel messages at all suggests the device was never presented (SAN zoning change, cloud detachment, or an lvm.conf filter blocking it).

4. Assess damage to LVs. Run pvs --segments -o pv_name,lv_name,seg_start_pe,seg_size_pe to see exactly which LVs had extents on the missing PV. Cross-reference with lvs -o lv_name,lv_attr to see which LVs show p (partial) in position 9. This tells you what data is at risk before you touch anything.

5. Check the storage layer below LVM. If multipath is in use, run multipath -ll to see path status. If LVM sits on md RAID, check mdadm --detail. If iSCSI, check iscsiadm. A SAN path failure shows as LVM PV issues, but the root cause is two layers down.

6. Try a device rescan if the device should be present. For SCSI/SAN, rescan the bus:

   # The shell expands the glob; this writes to all SCSI hosts
   echo '- - -' > /sys/class/scsi_host/host*/scan
   

   After any rescan, refresh LVM’s device cache with pvscan --cache.

7. Do not run vgreduce --removemissing yet. This is the most common destructive mistake. It permanently removes the missing PV from VG metadata. If any LVs have allocated extents on that PV, the command fails without --force. With --force, it removes those LVs entirely, including their extents on surviving PVs. Run this only after confirming the PV is permanently gone and data loss is acceptable.

Metrics and signals to monitor

Fixes

The recovery path depends on whether the device can come back and whether its PV metadata is intact.

The device has returned (transient failure)

If the underlying device reappears after a rescan, SAN re-presentation, or cable reseating, LVM does not automatically clear the MISSING flag when the PV has allocated data. You must explicitly reintegrate it:

# Verify the device has its PV label intact
pvck -v /dev/device

# Reintegrate the returned PV into the VG
vgextend --restoremissing <vgname> /dev/device

The --restoremissing flag adds the PV back without reinitializing it. This is the correct recovery path for transient SAN failures, multipath flaps, and brief device disappearances.

The device is present but the PV label is damaged

If the block device exists but someone ran pvremove, overwrote the partition table, or the PV metadata area has bad blocks, the PV label is gone but the underlying data extents are still on disk. Recovery requires recreating the PV header with the original UUID:

# Find the original UUID from the error message or VG metadata backup
grep -A5 'physical_volumes' /etc/lvm/backup/<vgname>

# Recreate the PV with the original UUID using a metadata archive.
# pvcreate writes a new PV header but does not touch data extents.
pvcreate --uuid <original-uuid> \
  --restorefile /etc/lvm/archive/<vgname>_<seqno>.vg /dev/device

# Restore VG metadata so LVM knows the full extent map again.
# WARNING: this writes metadata to ALL PVs in the VG, not just the repaired one.
# Verify that the backup file matches current VG state on surviving PVs first.
vgcfgrestore -f /etc/lvm/backup/<vgname> <vgname>

This only works if the underlying data extents were not overwritten.

The lvm.conf filter is blocking a present device

If the device is visible to the OS and has an intact PV label, but LVM still reports it missing, check the filter in /etc/lvm/lvm.conf. A restrictive filter that does not accept the PV device path causes LVM to ignore it.

# Check the current filter
grep -E '^\s*filter' /etc/lvm/lvm.conf

# After fixing the filter, refresh LVM's device cache
pvscan --cache

Common filter problems: accepting only one specific device path on multipath systems (where the PV is visible via multiple sd paths and a mapper device), or a filter written for device paths that shifted after adding new hardware.

The device is permanently gone

If the PV is permanently lost (disk failure with no recovery, cloud volume deleted, physical loss), the decision is whether to salvage surviving data or remove the missing PV from the VG.

To activate surviving LVs for read access in partial mode:

# WARNING: activates LVs even with missing PVs. Use for data salvage only.
# Reads to extents on the missing PV will return I/O errors.
vgchange -ay --partial <vgname>

To permanently remove the missing PV from the VG metadata:

# WARNING: destructive. Assess damage with pvs --segments first.
# Removes the missing PV from VG metadata.
# If LVs reference it, add --force to remove those LVs entirely.
# Do NOT add --force unless you accept losing those LVs.
vgreduce --removemissing <vgname>

For RAID or mirror LVs that had a leg on the missing PV, replace the failed device and rebuild:

# Add a new PV to the VG
pvcreate /dev/newdevice
vgextend <vgname> /dev/newdevice

# Repair degraded RAID legs onto the new PV
lvconvert --repair <vgname>/<raid-lv>

Prevention

• Monitor PV accessibility continuously. A PV going missing should trigger a page, not wait for an operator to notice LVM command warnings. Any PV in [unknown] state is abnormal after the boot scanning window.
• Use persistent device naming. Multipath mapper devices and /dev/disk/by-id paths are stable. Relying on /dev/sdX paths invites confusion after hardware changes.
• Verify lvm.conf filter on every multipath system. The filter must accept multipath mapper devices and reject individual paths to avoid duplicate PV warnings and metadata confusion.
• Keep VG metadata backups current. LVM writes backups to /etc/lvm/backup/ and archives to /etc/lvm/archive/ after every metadata change. Verify these are fresh. If /etc is itself on an LV in the same VG, store copies off-volume.
• Correlate across storage layers. When LVM reports PV issues, always check the layer below: multipath, md RAID, iSCSI, or physical disk health. The root cause is frequently not LVM.
• Document LV-to-service mappings. When a PV goes missing, you need to know immediately which services are affected to prioritize recovery.

How Netdata helps

• Cross-layer correlation: Netdata’s per-second metrics let you correlate the exact moment a PV disappeared with kernel I/O errors, disk latency spikes, multipath path changes, and D-state processes in a single timeline.
• dm device metrics: Netdata collects /proc/diskstats for dm devices, so I/O latency and throughput on LVs remain visible even when LVM tools are hung or slow.
• D-state process detection: processes stuck in uninterruptible sleep on dm devices are the user-visible symptom of LVM I/O hangs. Netdata surfaces process states alongside storage metrics.
• Historical baselines: knowing when the PV disappeared relative to other system events (deployment, config change, SAN maintenance window) shortens root cause analysis.

Related guides

• LVM cannot extend a logical volume: adding a PV when the VG is full
• LVM filesystem full while the volume group has space: the resize step everyone forgets
• How HVM actually works in production: a mental model for operators
• LVM insufficient free extents: the volume group is out of space
• LVM monitoring checklist: the signals every production volume manager needs
• LVM monitoring maturity model: from survival to expert
• LVM reached low water mark for data device: the thin pool warning before the freeze
• LVM thin pool space not reclaimed: discard, TRIM, and fstrim
• LVM snapshot invalid: the COW exception store filled and the snapshot is gone
• LVM thin pool auto-extend not working: threshold 100 means disabled
• LVM thin pool out of data space: every thin volume freezes at once
• LVM thin pool metadata full: the exhaustion that can corrupt the pool