$ guides / lvm / lvm-raid-resync-stuck ▌

Operations Guides

LVM RAID resync stuck: copy_percent not progressing

You are watching lvs output on a RAID logical volume, and copy_percent has not moved for over an hour. The array is degraded, a resync or rebuild is supposed to be underway, and the percentage is frozen. Meanwhile, the volume is running on fewer healthy legs than intended, and every minute without full redundancy is a minute where a second disk failure means data loss.

This is one of the more frustrating LVM failure modes because the system appears functional. Applications are still running. The degraded array is still serving I/O. But the rebuild that should restore redundancy is not progressing, and without active diagnosis, it may never complete.

The key question is not just “why is it stuck” but “is it actually stuck at all.” A resync competing with heavy production I/O can crawl at well under 1% per hour. A newly created mirror legitimately starts at 0%. Before applying any fix, you need to distinguish “slow but progressing” from “genuinely stalled.”

What this means

When copy_percent stops advancing, the kernel’s device-mapper RAID target (dm-raid, which uses the MD layer underneath) is unable to complete the synchronization of data across array legs. The raid_sync_action field tells you what the kernel thinks it is doing: idle, resync, check, or repair. If the action shows resync or repair but sync_percent is not moving, the rebuild is genuinely blocked.

During resync, the array is in a vulnerable state. For a RAID1 mirror, losing the single remaining healthy leg means total data loss. For RAID5, losing a second member means data loss. For RAID6, you can survive two failures, but a stalled resync extends the window where you are operating at reduced redundancy indefinitely. This is why a stalled rebuild is not merely a performance issue. It is a data protection risk.

Health characters from dmsetup status provide the per-device breakdown:

A = alive and in-sync (healthy)
a = alive but not in-sync (resync or rebuild in progress; normal during recovery)
D = dead or failed

An a leg is expected during rebuild. A D leg means the device has failed. The combination of a legs with a non-advancing copy_percent is the signature of a stalled resync.

flowchart TD
    A["copy_percent stuck
> 1 hour"] --> B{"raid_sync_action?"}
    B -->|"idle"| C["No resync running
Start one manually"]
    B -->|"resync or repair"| D{"dmsetup status
health chars"}
    D -->|"Any D (dead)"| E["Device failed
Check dmesg + pvs"]
    D -->|"All a (not in-sync)"| F{"I/O errors
in dmesg?"}
    F -->|"Yes"| G["Underlying device errors
blocking rebuild"]
    F -->|"No"| H["Throttled by prod I/O
or rate limit"]
    H --> I{"sync_speed
> 0?"}
    I -->|"Yes but slow"| J["Increase rate
or reduce prod load"]
    I -->|"No"| K["lvchange --syncaction repair
or check LV active state"]

Common causes

Cause	What it looks like	First thing to check
Underlying device errors	`dmsetup status` shows `D` or `a` legs; dmesg shows I/O errors, timeouts, or device resets for the PV device	`dmesg \| grep -i error \| tail -50`
Resync throttled by production I/O	`copy_percent` advances slowly but not zero; application latency elevated; `raid_sync_action` shows `resync`	Check `iostat` or disk latency on PVs; compare with production write load
Flapping hardware	`copy_percent` oscillates or restarts from low values; dmesg shows repeated device connect/disconnect cycles	`dmesg \| grep -iE 'link down\|link up\|reset\|removed' \| tail -30`
LV is inactive	`raid_sync_action` shows `idle`; `lv_attr` position 5 is not `a`	`lvs -o lv_name,lv_attr,copy_percent`
Rate limit set too low	`copy_percent` advances very slowly but consistently; no errors in dmesg	`lvs -o lv_name,raid_sync_action,sync_percent` plus check recovery rate settings

Quick checks

# Check resync action and progress for all RAID LVs
lvs -a -o name,vg_name,lv_attr,copy_percent,raid_sync_action,sync_percent,raid_mismatch_count

# Detailed per-device RAID health (bypasses LVM locks, reads from kernel)
dmsetup status <vg>-<lv>
# Look for health chars: A=healthy, a=resyncing, D=dead

# Check for mismatch count after scrub
lvs -o name,raid_mismatch_count

# Check kernel log for device errors, resets, or link flapping
dmesg | grep -iE 'error|reset|link down|link up|removed|timeout' | tail -50

# Verify all PVs are present and accessible
pvs -o pv_name,vg_name,pv_attr,pv_size,pv_free

# Check kernel MD sync speed limits
cat /proc/sys/dev/raid/speed_limit_min
cat /proc/sys/dev/raid/speed_limit_max

# Monitor progress over 60 seconds to see if it is actually moving
watch -n 5 'lvs -o lv_name,copy_percent,raid_sync_action,sync_percent'

How to diagnose it

Confirm the resync action. Run lvs -a -o name,raid_sync_action,sync_percent. If raid_sync_action is idle, no resync is running. The array may be degraded but the kernel is not attempting a rebuild. You need to trigger one.
Check device health characters. Run dmsetup status <vg>-<lv>. The output includes per-device health characters. Any D means a dead device. If all remaining devices show A but the action is idle, the kernel may need a nudge to start the rebuild.
Look for underlying device errors. Check dmesg for I/O errors, SCSI/NVMe resets, or link state changes on the physical devices backing the PVs. Device errors can block the rebuild entirely. Correlate the device names in dmesg with the PV paths from pvs.
Verify the LV is active. LVM does not synchronize inactive volumes. Check lv_attr position 5. If the LV is inactive (- instead of a), activate it with lvchange -ay <vg>/<lv> and the resync should begin.
Assess I/O contention. A resync operation is I/O intensive and competes with production workloads for disk bandwidth. If production write load is high, the resync may be throttled to near-zero throughput. Check disk utilization with iostat -x 5 on the underlying PV devices. If utilization is near 100% from application I/O, the resync is starving.
Check for flapping. If copy_percent resets to a low value or oscillates, the underlying device may be disconnecting and reconnecting. Each reconnection can restart the resync from scratch or from a checkpoint. Look for repeated link up/down messages in dmesg.
Check rate limits. LVM exposes minrecoveryrate and maxrecoveryrate settings on RAID LVs. If maxrecoveryrate is set very low, the resync throughput is artificially capped. Query the current settings and compare with the expected sync speed.
Check for multiple concurrent resyncs. If several RAID LVs are resyncing simultaneously, they compete for the same I/O bandwidth. Run lvs -o name,copy_percent,raid_sync_action across all LVs. Multiple active resyncs compound I/O impact and can make each one appear stalled.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`copy_percent` / `sync_percent`	Primary indicator of rebuild progress	Unchanged for more than 1 hour, or rate below 1% per hour for a reasonable LV size
`raid_sync_action`	Tells you what the kernel is doing (`idle`, `resync`, `check`, `repair`)	Shows `idle` on a degraded array, or shows `resync` with no progress
`dmsetup status` health chars	Per-device health (`A`/`a`/`D`)	Any `D` (dead device); all `a` with no progress
`raid_mismatch_count`	Data inconsistencies found during scrub	Nonzero value indicates data integrity concern
dmesg I/O errors and resets	Underlying device problems blocking rebuild	New error messages correlated with resync stall
Disk I/O utilization on PVs	Whether resync is competing with production load	Near 100% utilization from application I/O
`lv_attr` position 5 (activation)	LV must be active for resync to proceed	Not `a` (active)

Fixes

Resync not running at all (raid_sync_action is idle)

Force a resync or repair:

# Force a check and repair of the RAID LV
lvchange --syncaction repair <vg>/<lv>

The repair action reads all data and parity blocks and corrects discrepancies. Use check instead if you only want to count mismatches without modifying data:

# Count mismatches without repairing
lvchange --syncaction check <vg>/<lv>

After triggering, verify the action changed:

lvs -o name,raid_sync_action,sync_percent

If the LV has the R attribute (not initially synchronized) and --syncaction does not start a resync, lvchange --resync <vg>/<lv> forces a full mirror synchronization by resetting that attribute.

Underlying device errors

If dmesg shows I/O errors or device resets on the PV backing device, the device may be failing. The resync cannot complete until the device is healthy or replaced.

For a device that has failed (showing D in dmsetup status):

Identify which PV is affected: pvs -o pv_name,vg_name,pv_attr and look for missing or error attributes.
If the device is permanently failed, replace it and add the new PV to the VG.
Use targeted rebuild if your lvm2 supports it:

# Rebuild data on a specific PV (newer lvm2 only)
lvchange --rebuild <new_pv> <vg>/<lv>

If the device is flapping (connecting and disconnecting repeatedly), the resync may restart each time. Fix the hardware issue (cable, HBA, firmware, SAN path) before attempting another resync. Each restart wastes I/O and extends the vulnerability window.

Resync throttled by production I/O

If the resync is progressing but very slowly due to I/O contention, you have two options.

Increase the recovery rate ceiling. The kernel’s MD layer governs sync speed through /proc/sys/dev/raid/speed_limit_min and speed_limit_max:

# Check current limits
cat /proc/sys/dev/raid/speed_limit_min
cat /proc/sys/dev/raid/speed_limit_max

# Temporarily raise the minimum sync speed (KB/s)
echo 50000 > /proc/sys/dev/raid/speed_limit_min

These are system-wide settings affecting all MD-backed arrays. Changes do not persist across reboots.

LVM also exposes per-LV rate control:

# Set minimum recovery rate (e.g., 1 MiB/s per device)
lvchange --minrecoveryrate 1M <vg>/<lv>

# Set maximum recovery rate (e.g., 10 MiB/s per device)
lvchange --maxrecoveryrate 10M <vg>/<lv>

Reduce production I/O pressure. If the storage is saturated by application writes, temporarily reducing the write load (throttling batch jobs, deferring backups, moving workloads) allows the resync to progress faster. This is the safer approach when you cannot afford to impact production latency by raising the sync speed.

LV is inactive

If the LV is not active, the resync cannot proceed:

# Activate the LV
lvchange -ay <vg>/<lv>

After activation, verify the resync starts:

lvs -o name,raid_sync_action,sync_percent

Stuck at a high percentage (near 100%)

A known behavior exists where copy_percent or sync_percent stalls at 99.99% and never reaches exactly 100%. This has no well-documented root cause or reliable fix on the open web.

If the array is functionally healthy (all legs show A in dmsetup status, raid_mismatch_count is zero), the stuck percentage may be cosmetic. Verify actual health through dmsetup status rather than relying solely on copy_percent.

Multiple concurrent resyncs causing I/O thrashing

If multiple RAID LVs are resyncing on the same underlying devices, they compete for I/O bandwidth and can thrash. The MD layer may not detect the shared underlying devices through LVM’s device-mapper layer.

Prioritize critical LVs by serializing resyncs. There is no standard LVM command to pause an active resync, but you can influence which LVs are actively syncing by managing their activation state and syncaction. Let critical resyncs complete first, then trigger the rest.

Prevention

Monitor copy_percent and raid_sync_action as first-class signals. A stalled resync extends the vulnerability window. Alert when copy_percent has not advanced for more than 1 hour on a degraded array, or when the sync rate drops below 1% per hour for a reasonable LV size.
Alert on degraded arrays immediately. Any device showing D in dmsetup status health characters means the array has lost a leg. The clock starts ticking. Do not wait for a resync to stall before investigating.
Verify hardware health proactively. Monitor dmesg for device resets, link state changes, and I/O errors. Flapping hardware that causes repeated resync restarts is preventable if caught early through SMART monitoring, multipath path health checks, and controller log analysis.
Serialize resyncs after bulk recovery events. After a system crash, storage failure, or disk replacement, multiple arrays may need resyncing. Stagger them to avoid I/O thrashing.
Document LV-to-application mappings. Know which LVs back which services. During a stalled resync, this determines whether you can safely throttle production I/O to let the rebuild finish.

How Netdata helps

Netdata’s per-second metrics collection and anomaly detection shorten the diagnosis window for stalled RAID resyncs by surfacing correlated signals at the right granularity:

Block device I/O metrics collected per second let you see whether the underlying PV devices are saturated by application I/O, which would explain a throttled resync. Correlate disk utilization spikes with copy_percent stalls to confirm I/O contention.
Kernel log integration surfaces dmesg device resets, link flapping, and I/O errors in the same timeline as storage metrics. You see the device disconnect event and the resync restart side by side, without manually correlating across tools.
Anomaly detection on disk latency and I/O counters flags the onset of device degradation before a full failure. A slowly failing drive often shows increasing latency variance days before it produces hard errors.
D-state process tracking reveals when application processes are blocked on I/O to dm devices. A growing D-state count during a resync indicates the rebuild is starving production workloads.
Historical baselines for per-disk latency and throughput let you compare current performance against normal operation. This helps determine whether the resync is genuinely stalled or merely slow due to elevated baseline I/O.