You ran vgs or lvs and saw something wrong. The VG attributes show a p in the fourth position (wz--pn- instead of wz--n-). One or more PVs show as [unknown] in pvs output. Commands that touch the VG now print warnings about a missing device and may refuse to run.

The VG is in partial mode. LVM metadata still references a PV UUID that no block device on the system currently claims. The metadata itself is intact on surviving PVs, which is why you can still see the VG at all. But any LV that had physical extents on the missing PV now has a hole in its mapping table.

The critical first question is not “how do I fix this” but “is this disk gone for good or is this a path issue.” The answer determines whether you are doing data recovery or waiting for a fabric path to come back. The second question is “which LVs had extents on that PV and what is their layout type,” because linear and striped LVs are now broken, while mirror and RAID LVs are degraded but functional.

What this means

A volume group enters partial mode when at least one member PV disappears. LVM tracks PVs by UUID, not device path. If no block device presenting that UUID is found during the scan, the PV is marked missing. The VG metadata on surviving PVs still references the missing PV’s UUID and extent allocations.

What you can still do in partial mode:

  • Read VG and LV metadata from surviving PVs.
  • Activate LVs that have no extents on the missing PV, with --partial.
  • Operate RAID and mirror LVs in degraded mode.

What is blocked or broken:

  • LVs with extents on the missing PV: linear and striped LVs return I/O errors for any access to those extents.
  • Commands that modify VG metadata may refuse to run without --partial.
  • vgck may fail because it cannot read all PVs. This is not necessarily corruption.
  • Consistency checks are partially masked by --partial, so you lose some verification ability.

The --partial flag tells LVM to proceed despite the missing PV. It does not recover anything. It suppresses the refusal and allows operations on the remaining PVs. Because --partial masks some consistency problems, treat any operation done under it with caution.

Common causes

CauseWhat it looks likeFirst thing to check
Disk hardware failuredmesg shows I/O errors, medium errors, or link resets for a specific devicedmesg | grep -i error | tail -50, SMART data
SAN LUN unpresented or zoned awayPV disappears without kernel errors; device node may exist but returns no dataSAN management console, multipath -ll
Multipath all paths downmultipath -ll shows no active paths; brief or persistent I/O pausemultipath -ll, check fabric and HBA state
Cable or connector failureIntermittent I/O errors before complete disappearance; link resets in dmesgPhysical inspection, dmesg for transport errors
Accidental device removalHot-unplug event in dmesg; device was present, now gonedmesg | tail, check for recent administrative changes
NVMe namespace disappearanceNamespace removed or controller reset; device path changes or vanishesnvme list, check subsystem state
udev or device naming changePV UUID exists but device path shifted after rebootpvs showing [unknown], check /dev/disk/by-id/

Quick checks

These are read-only and safe to run on a production system.

# Check VG partial status: 'p' in attr position 4 means partial
vgs -o +vg_missing_pv_count

# Identify which PVs are missing: name shown as [unknown]
pvs -o pv_name,vg_name,pv_attr,pv_size,pv_free

# Verify each PV device node still exists
for pv in $(pvs --noheadings -o pv_name); do
  [ -b "$pv" ] && echo "$pv: OK" || echo "$pv: MISSING"
done

# Check LV health: 'p' in position 9 of lv_attr means partial
lvs -o lv_name,vg_name,lv_attr,lv_active

# Identify which LVs had segments on the missing PV
pvs --segments -o pv_name,lv_name,seg_start_pe,seg_size_pe

# Check kernel for I/O errors on block devices
dmesg | grep -i 'I/O error\|offline\|not ready\|device not found' | tail -50

# Check multipath status if SAN-attached
multipath -ll

# dmsetup works when LVM tools hang: no locks, no metadata I/O
dmsetup status

How to diagnose it

flowchart TD
    A["VG shows p in vg_attr"] --> B["PV shows unknown in pvs"]
    B --> C["Check dmesg for I/O errors"]
    C --> D{"Kernel errors for device?"}
    D -->|"Yes"| E["Likely hardware or cable failure"]
    D -->|"No"| F["Check multipath -ll"]
    F --> G{"Paths present?"}
    G -->|"No active paths"| H["Fabric, HBA, or zoning issue"]
    G -->|"Paths exist"| I["Check SAN presentation or udev"]
    I --> J{"Device reappears on rescan?"}
    J -->|"Yes"| K["Transient: rescan fixed it"]
    J -->|"No"| E
    E --> L["Permanent loss: assess LV layout"]
    K --> M["Verify with vgck"]

  1. Confirm the partial state. Run vgs -o +vg_missing_pv_count. If vg_missing_pv_count is non-zero and vg_attr shows p in position 4, the VG is partial.

  2. Identify the missing PV. Run pvs -o pv_name,vg_name,pv_attr,pv_size,pv_free. The missing PV shows as [unknown]. Record the PV UUID from VG metadata or archive files if you need to recreate the PV later.

  3. Determine impact on LVs. Run lvs -o lv_name,vg_name,lv_attr,lv_active. Any LV with p in position 9 of lv_attr has extents on the missing PV. Then run pvs --segments -o pv_name,lv_name,seg_start_pe,seg_size_pe to see exactly which segments are affected.

  4. Check for transient versus permanent loss. Examine dmesg for I/O errors, transport errors, or link resets on the specific device. Run multipath -ll for SAN-attached storage. Check /sys/class/block/*/device/state for SCSI device health.

  5. Attempt a rescan if SAN-attached. For SCSI and SAN devices, trigger a bus rescan and check whether the PV reappears:

# Rescan SCSI bus for disappeared devices
echo '- - -' > /sys/class/scsi_host/host*/scan
pvs -o pv_name,vg_name,pv_attr

  1. Assess LV layout type for affected LVs. If affected LVs are linear or striped, their data on the missing PV’s extents is inaccessible. If they are mirror or RAID, they should be degraded but operational on remaining legs. Check device health characters with dmsetup status for RAID volumes.

  2. Do not run vgreduce --removemissing yet. This command permanently removes the missing PV and any LVs with extents on it from VG metadata. It is destructive and irreversible. Run it only after confirming the PV is permanently lost and after assessing which LVs will be removed.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
PV accessibility (pvs name [unknown])Direct evidence of device lossAny PV in [unknown] state with uptime > 600 seconds
VG partial flag (vgs attr position 4 = p)Confirms VG is operating with a missing memberAny non-zero vg_missing_pv_count
LV health status (lvs attr position 9 = p)Shows which LVs have extents on the missing PVAny LV with p flag, especially non-redundant layouts
Kernel I/O errors in dmesgDistinguishes hardware failure from presentation issueNew I/O errors for a specific block device
Multipath path statusIdentifies fabric, HBA, or zoning problemsAll paths down for a device
D-state processes on dm devicesApplication-visible symptom of I/O to missing extentsGrowing count of processes stuck > 60 seconds
LVM command execution timeSlow commands indicate device scan hitting unresponsive devicespvs or vgs taking > 10 seconds

Fixes

Transient path issue (SAN, multipath, udev)

If the missing PV reappears after a rescan or path recovery, the VG should return to normal. Verify and clean up:

# Verify VG is no longer partial
vgs -o +vg_missing_pv_count

# Check metadata consistency now that all PVs are back
vgck -v <vgname>

# Refresh LV states if health flags persist after PV returns
lvchange --refresh <vg>/<lv>

For RAID LVs, a refresh also triggers a resync check on the returned device. Monitor copy_percent until it reaches 100%.

Degraded RAID or mirror LV

If the affected LVs are RAID or mirror type, they are running on remaining legs with reduced redundancy. One more failure means total data loss. Replace the failed PV and rebuild.

  1. Add a replacement PV: pvcreate <new_device>
  2. Add it to the VG: vgextend <vg> <new_device>
  3. Repair the RAID LV to use the new PV: lvconvert --repair <vg>/<raid_lv>
  4. After rebuild completes, remove the missing PV from metadata: vgreduce --removemissing <vg>

During rebuild, expect significant I/O overhead. The array is vulnerable to a second failure until resync completes. Monitor copy_percent and device health characters via dmsetup status.

Permanent loss of a non-redundant PV

If the PV is permanently lost and affected LVs are linear or striped, the extents on that PV are gone. The data is unrecoverable unless you have a backup.

Before doing anything destructive, preview the damage:

# Preview what vgreduce --removemissing will remove (does not change anything)
vgreduce --removemissing <vg> --test

Review the output carefully. Any LV with extents on the missing PV will be removed from VG metadata.

If you accept the data loss:

# WARNING: irreversible. Removes the missing PV and any LVs with extents on it
# from VG metadata. Those LVs are gone.
vgreduce --removemissing <vg> --force

Replacing a PV with metadata recovery

If you have a replacement disk and want to restore the VG metadata to its pre-loss state, you can recreate the PV with the original UUID and restore from metadata archives:

  1. Identify the missing PV’s UUID from archive files in /etc/lvm/archive/.
  2. Recreate the PV with the original UUID:
# Recreate PV with original UUID from archive metadata
pvcreate --uuid <uuid> --restorefile /etc/lvm/archive/<vgname>_<nnnnn>.vg <new_device>
  1. Restore VG metadata:
# Restore VG metadata from backup to all PVs in the VG
vgcfgrestore <vgname>
  1. Activate the VG:
# Activate all LVs in the VG
vgchange -ay <vgname>

This restores the metadata structure only. The data on the replacement disk is blank. This is useful when you need to reconstruct the VG layout to recover LVs that had no extents on the lost PV but could not activate while the VG was partial.

Operating in partial mode temporarily

If you need to keep the system running while waiting for a replacement disk or SAN fix, activate unaffected LVs:

# Activate VG in partial mode: unaffected LVs activate read-write
vgchange -ay --partial <vgname>

LVs with no extents on the missing PV activate normally. LVs with extents on the missing PV do not activate.

vgck consistency checks may fail in partial mode because not all PVs are accessible. This does not necessarily indicate metadata corruption. Once the missing PV returns or is properly removed, run vgck again to verify.

Prevention

  • Monitor PV accessibility. A PV going missing is never benign after boot. Alert on any PV showing [unknown] in pvs output.
  • Use RAID or mirror layouts for critical LVs. Linear and striped LVs have no redundancy. A single PV loss means immediate data loss for affected extents.
  • Verify physical topology independence. RAID legs on the same controller, shelf, or SAN fabric share a failure domain. Confirm PVs in redundant layouts are physically independent.
  • Check multipath filter configuration. Ensure the lvm.conf filter excludes raw paths so LVM sees only multipath devices. Misconfigured filters cause duplicate PV detection and metadata confusion.
  • Keep metadata backups current. Verify /etc/lvm/backup/ and /etc/lvm/archive/ are being updated on every metadata change. If /etc is on an LVM volume, store backups externally too.
  • Monitor LVM command latency. A pvs that normally takes under 1 second suddenly taking 10 seconds often signals a device becoming unresponsive before it disappears entirely.

How Netdata helps

  • Per-second PV accessibility checks surface the exact moment a PV transitions to missing, correlated with kernel I/O errors and multipath events in the same dashboard.
  • LV health status monitoring detects the p flag in lv_attr position 9 as soon as it appears, showing which LVs are affected without manual lvs queries.
  • D-state process tracking reveals the application-level impact of I/O to missing extents, distinguishing a storage event from an application issue.
  • dm device I/O latency from kernel counters means visibility continues even when lvs and vgs are blocked by the same storage problem they are trying to report on.
  • Cross-layer correlation across block device errors, multipath paths, dm device state, and filesystem usage pinpoints whether the cause is at the fabric, disk, or LVM layer.