LVM RAID or mirror degraded: a leg is dead and you are one failure from data loss

Your LVM RAID1 or mirrored logical volume is still serving reads and writes. Applications see no errors. Filesystems are mounted. But one leg is dead and you are running on a single copy with zero redundancy. The next disk failure, cable disconnect, or SAN path loss is total data loss.

The default raid_fault_policy is warn. When a leg fails, dmeventd logs a warning but takes no repair action. No resync is triggered, no alert fires unless your monitoring explicitly checks for degraded state. The system runs indefinitely on the surviving leg until the second failure removes all copies.

What this means

When a leg fails, device-mapper removes it from the active set and routes I/O to the surviving leg or legs. Data is still accessible, but the redundancy is gone.

The failure surfaces through two interfaces:

• dmsetup status <vg>-<lv> shows per-leg health characters. A means alive and in-sync. D means dead or failed. a means alive but not in-sync (normal during rebuild).
• lvs shows aggregate LV health. The lv_health_status field reports “partial” when a device is permanently gone, or “refresh needed” when a device was transiently missing and returned. The lv_attr field shows p in position 9 for partial, or r for refresh needed.

The critical distinction is between D and a in dmsetup status output. D means a device has failed and is no longer participating in the array. a means alive but not yet synchronized, which is expected during initial sync or rebuild. Alert on D, or on a copy_percent that was previously 100 and has dropped below 100.

Fixed in LVM 2.02.155 (WHATS_NEW: “Correcting value in copy_percent() for 100%”). On older LVM versions, copy_percent may report 100.00 due to rounding even when the device is not fully in-sync. For those versions, rely on dmsetup status health characters to determine true sync state.

flowchart TD
    A["copy_percent dropped below 100
or lv_attr shows p or r"] --> B{"Check dmsetup status
per-leg health chars"}
    B -->|D present| C["Leg is dead or failed"]
    B -->|a present, no D| D["Resync in progress
Check if progressing"]
    B -->|All A| E["Check pv_attr for
missing PV"]
    C --> F{"Is the failed device
visible to the OS?"}
    F -->|No, gone| G["lvconvert --repair
with replacement PV"]
    F -->|Yes, returned| H["lvchange --refresh
to trigger resync"]
    F -->|Yes, still present| I["lvconvert --replace
with new PV"]
    D -->|Progressing| J["Normal rebuild
Monitor to completion"]
    D -->|Stalled over 1 hour| K["Investigate device
errors blocking rebuild"]

Common causes

Quick checks

# Check copy_percent and health for all RAID/mirror LVs
lvs -o lv_name,vg_name,lv_attr,copy_percent,lv_health_status -S 'seg_type=~raid|seg_type=~mirror'

# Get per-leg health characters from device-mapper
dmsetup status <vg>-<lv>
# Look for D (dead), a (alive not in-sync), A (alive in-sync)

# Check which PVs are missing
pvs -o pv_name,vg_name,pv_attr,pv_size,pv_free
# Missing PVs show as [unknown]

# Check VG partial status
vgs -o vg_name,vg_attr,vg_missing_pv_count
# 'p' in attr position 4 means partial (missing PV)

# Verify each PV device exists
for pv in $(pvs --noheadings -o pv_name); do
  [ -b "$pv" ] && echo "$pv: OK" || echo "$pv: MISSING"
done

# Check for I/O errors on underlying devices
dmesg | grep -i 'I/O error\|offline\|not ready' | tail -30

# Check RAID-specific fields
lvs -o lv_name,raid_sync_action,raid_mismatch_count,copy_percent -S 'seg_type=~raid'

# Check multipath if SAN-attached
multipath -ll

# Check if resync is progressing (run twice, compare copy_percent)
lvs -o lv_name,copy_percent && sleep 30 && lvs -o lv_name,copy_percent

# Check dmeventd is running (relevant for raid_fault_policy)
systemctl is-active lvm2-monitor.service
pgrep -x dmeventd

How to diagnose it

1. Identify which LVs are degraded. Run lvs -o lv_name,vg_name,lv_attr,copy_percent,lv_health_status -S 'seg_type=~raid|seg_type=~mirror'. Any LV with copy_percent below 100, a non-empty lv_health_status, or lv_attr position 9 not - needs investigation.

2. Get per-leg health from device-mapper. Run dmsetup status <vg>-<lv> for each suspect LV. D confirms a dead leg. a without D means resync in progress (normal). All A with copy_percent below 100 may indicate the rounding bug on older LVM or the very start of a resync.

3. Determine which PV backed the failed leg. Check pvs for missing devices. A PV showing as [unknown] means the device disappeared from the system entirely. Cross-reference with pvs --segments -o pv_name,lv_name,seg_start_pe,seg_size_pe to map PVs to LV segments.

4. Check the kernel log for the cause. Run dmesg | grep -i 'I/O error\|offline\|not ready\|device not found' to find when and why the device disappeared. SCSI timeouts, link resets, and medium errors each produce distinct messages.

5. Check the physical layer. For SAN-attached storage, run multipath -ll to verify path status. For local disks, check /sys/block/<device>/device/state. For NVMe, check /sys/class/nvme/nvme*/state.

6. Distinguish permanent loss from transient failure. If lv_health_status shows “refresh needed”, the device was temporarily absent and has returned. This requires lvchange --refresh to clear, not a full repair. If the status shows “partial”, the device is gone and needs replacement.

7. Verify whether resync is progressing. If rebuilding (lowercase a in health chars, no D), check that copy_percent is advancing. Run lvs -o lv_name,copy_percent twice with a 30-second gap. If the value is unchanged after an hour on a reasonably sized LV, the rebuild is stalled and you need to investigate the underlying device.

Metrics and signals to monitor

Fixes

The failed PV is gone (not visible to the OS)

You need a replacement device with enough free extents to hold the failed leg’s data.

# DESTRUCTIVE: This destroys any existing data on the device
pvcreate /dev/new_device

# Add it to the VG if not already a member
vgextend <vg> /dev/new_device

# Repair the RAID LV, allocating the new leg from the replacement PV
lvconvert --repair <vg>/<lv> /dev/new_device

lvconvert --repair allocates a new leg from the specified PV (or from VG free space if no PV is named), copies data from the surviving leg, and brings the new leg into sync. The failed leg’s metadata is cleaned up.

Confirmed by lvmraid(7): integrity must be disabled before repair/replace commands can be used. If raidintegrity is enabled on the LV, you may need to disable it before repair: lvconvert --raidintegrity n <vg>/<lv>. Re-enable after repair completes.

The repair command requires free extents in the VG. If insufficient space exists, it fails with an error indicating how many extents are needed versus available.

The failed PV returned (transient failure)

If the device was temporarily disconnected (SAN path blip, cable reseat) and is now visible again, the LV shows “refresh needed” in lv_health_status. The array has not automatically re-synced the returned device.

# Non-destructive: trigger resync of the returned device
lvchange --refresh <vg>/<lv>

After refresh, monitor copy_percent to confirm the resync completes. The device rejoins as a (alive, not in-sync) and transitions to A (alive, in-sync) when the copy reaches 100 percent.

The failing PV is still visible but unreliable

When a device is present but showing hardware errors, bad sectors, or SMART warnings, use lvconvert --replace to swap it out without waiting for total failure.

# Prepare the replacement as a PV first
pvcreate /dev/new_device
vgextend <vg> /dev/new_device

# Replace the failing device
lvconvert --replace /dev/failing_device <vg>/<lv>

Unlike --repair, --replace works on devices that are still visible and participating in the array. Data is copied to the new device, then the old leg is removed.

Repairing an active system volume

For LVs that must stay active (root, swap, mounted application volumes), run lvconvert --repair with the LV active. The repair does not require deactivation. Monitor I/O latency during the resync, as rebuild traffic competes with production workload for disk bandwidth.

After any repair: verify full redundancy

# Confirm all legs show A (alive, in-sync)
dmsetup status <vg>-<lv>

# Confirm copy_percent is 100
lvs -o lv_name,copy_percent -S 'seg_type=~raid|seg_type=~mirror'

# Confirm lv_health_status is empty
lvs -o lv_name,lv_health_status

Do not consider the incident resolved until every leg shows A and copy_percent is 100 with no health flags.

Prevention

• Alert on any RAID/mirror LV with copy_percent below 100 that was previously at 100. This catches both new failures and stalled resyncs. Gate new LVs (which legitimately start below 100 during initial sync) by tracking which LVs have ever reached 100.
• Alert on any D health character in dmsetup status output. This is the definitive failure signal.
• Alert on lv_health_status showing any non-empty value. Both “partial” and “refresh needed” require operator action.
• Monitor resync progress rate. A stalled resync (copy_percent unchanged for over an hour on a reasonably sized LV) often indicates an underlying device problem.
• Verify dmeventd is running. With the default raid_fault_policy of warn, dmeventd logs the failure but does not auto-repair. Confirm the daemon is at least logging so you can correlate events.
• Document LV-to-PV-to-physical-device mappings. When a leg fails, you need to know immediately which physical device to replace. Maintain this mapping outside LVM metadata so it survives PV loss.
• Run periodic RAID scrubs. lvchange --syncaction check <vg>/<lv> verifies data consistency between legs and populates raid_mismatch_count. Investigate any nonzero mismatch count.
• Verify physical independence of legs. Two PVs on the same controller, disk shelf, or SAN fabric provide no real redundancy. Verify when the array is initially created.

How Netdata helps

• Tracks copy_percent trends across all RAID and mirror LVs, alerting when a previously synced LV drops below 100.
• Surfaces lv_health_status and lv_attr position 9 as discrete metrics, enabling alerts on any non-healthy state without parsing command output in scripts.
• Correlates per-device disk I/O latency and error rates with LVM degradation events, so you can see whether the surviving leg is also showing wear.
• Collects kernel log entries (dmesg) alongside LVM metrics, providing device-level context for LVM-level degradation without running multiple commands manually.
• Monitors dmeventd process presence and lvm2-monitor service state, so you know whether the safety net is actually running.

Related guides

• LVM cannot extend a logical volume: adding a PV when the VG is full
• LVM Couldn’t find device with uuid: a physical volume has gone missing
• LVM filesystem full while the volume group has space: the resize step everyone forgets
• How LVM actually works in production: a mental model for operators
• LVM Insufficient free extents: the volume group is out of space
• LVM monitoring checklist: the signals every production volume manager needs
• LVM monitoring maturity model: from survival to expert
• LVM reached low water mark for data device: the thin pool warning before the freeze
• LVM thin pool space not reclaimed: discard, TRIM, and fstrim
• LVM snapshot COW usage climbing: extend or remove before it overflows
• LVM snapshot invalid: the COW exception store filled and the snapshot is gone
• LVM snapshot slowing the origin: copy-on-write write amplification