$ guides / lvm / lvm-mirror-resync-storm ▌

Operations Guides

LVM mirror resync storm: multiple rebuilds saturating disk I/O

Production latency is climbing. Multiple applications are slow. Storage shows high I/O across all physical volumes, but there is no single failed disk and no obvious error in the logs. When you check LVM, several mirrored logical volumes show copy_percent below 100%. They are all resyncing at the same time.

This is a mirror resync storm. After a system crash, a disk replacement, or bulk mirror creation, multiple mirrored LVs may need to resynchronize simultaneously. Each resync generates substantial sequential and random I/O as the mirror copies dirty regions or entire leg contents to the recovering side. When several run in parallel, the combined I/O overwhelms storage bandwidth. Production workloads suffer elevated latency, and each resync takes longer than it would alone because they compete for the same physical devices.

The feedback loop compounds the problem. Resync I/O competes with production I/O, slowing production. The resyncs themselves slow from the contention, extending the degraded window. A resync that should take two hours can stretch to eight or more. The system limps along, neither failing nor recovering, just slow.

What this means

Multiple dm-mirror or dm-raid1 logical volumes are running synchronization operations in parallel. Each resync reads from a healthy leg and writes to the recovering leg, consuming read and write bandwidth on every PV involved. The kernel does not coordinate resync priority across independent LVs. Each LV’s resync runs independently, and the combined I/O load saturates the storage path.

The distinction between legacy --type mirror and --type raid1 matters. Legacy mirror uses dm-mirror with a separate log device and must block I/O during failure handling. RAID1 (dm-raid, md-based) does not block I/O during failure handling and supports per-LV recovery rate throttling via lvchange --maxrecoveryrate and lvchange --minrecoveryrate. Since LVM 2.02.100, raid1 is the default mirror segment type. Legacy mirror is deprecated. If your environment still uses legacy mirror, the throttling options below do not apply.

Key observable behaviors during a resync storm:

copy_percent is below 100% on multiple LVs simultaneously
dmsetup status on each RAID or mirror device shows health character a (alive but not in-sync) rather than A (alive and in-sync)
System-wide disk I/O is elevated across all PVs, not isolated to one device
I/O load correlates with copy_percent values advancing across multiple LVs

flowchart TD
    A[Crash or disk replacement] --> B[Multiple mirrors need resync]
    B --> C[Resyncs start simultaneously]
    C --> D[Combined I/O saturates storage]
    D --> E[Production latency spikes]
    D --> F[Resyncs slow from contention]
    F --> G[Resyncs take far longer]
    G --> D

Common causes

Cause	What it looks like	First thing to check
System crash or unclean shutdown	Multiple mirrors resync dirty regions at boot. `copy_percent` below 100% on previously healthy mirrors right after reboot.	`uptime` and boot logs. Resyncs appeared immediately after boot.
Disk replacement and rebuild	One or more mirrors recovering after a PV was replaced. Resyncs triggered by `lvconvert --repair` or automatic rebuild.	`lvs -o lv_health_status` for `r` (refresh needed) or recent hardware changes in dmesg.
Bulk mirror creation	Multiple new mirrored LVs created in quick succession, all running initial sync from zero.	LV creation timestamps. New mirrors start at `copy_percent = 0`.
pvmove on multiple LVs	Temporary mirror segments from interrupted or concurrent pvmove operations. `lv_attr` position 1 shows `p`.	`pvs -o pv_name,move_pv` for active pvmove state.
No recovery rate throttle configured	Resyncs run at maximum speed with no per-LV rate cap. Production I/O starved by resync bandwidth consumption.	Check whether `--maxrecoveryrate` is set on any RAID1 LV.

Quick checks

# Check which mirrors are resyncing (copy_percent below 100)
lvs -o lv_name,vg_name,lv_attr,copy_percent,lv_health_status

# Watch resync progress over time
watch -n 5 'lvs -o lv_name,copy_percent,lv_attr'

# Detailed RAID health per device: A = synced, a = resyncing, D = dead
dmsetup status <vg>-<lv>

# Check for active pvmove operations that create temporary mirrors
pvs -o pv_name,vg_name,move_pv

# Check per-PV I/O utilization and saturation
iostat -xm 5

# Check for D-state processes waiting on dm I/O
ps aux | awk '$8 ~ /D/'

# Verify underlying device health (errors may be blocking resync)
dmesg | grep -i 'I/O error\|device not found' | tail -20

# dmsetup works when lvs may hang (no LVM locks, reads from kernel memory)
dmsetup status --target raid
dmsetup status --target mirror

How to diagnose it

Identify all resyncing mirrors. Run lvs -o lv_name,vg_name,lv_attr,copy_percent,lv_health_status and note every LV with copy_percent below 100. Count how many are in progress. The a health character (alive but not in-sync) is normal during resync and should not be confused with D (dead or failed).
Verify resyncs are actually progressing. Run lvs -o lv_name,copy_percent twice, several minutes apart. If copy_percent has not changed, the resync may be stuck. A resync not progressing for more than one hour requires investigation. A stalled resync combined with a second device failure can result in data loss.
Check for stuck or failed resync. If copy_percent is not advancing, run dmsetup status <vg>-<lv> and inspect health characters. A D means a dead device, not a slow resync. Check dmesg for underlying device errors that could be blocking rebuild I/O. RAID1 resync can get stuck at 0% in some cases, requiring deactivation and reactivation of the LV.
Assess production impact. Check application latency, D-state process counts, and disk I/O latency via iostat or /proc/diskstats. If production workloads are timing out or D-state processes are accumulating on dm devices, treat this as high priority.
Identify the trigger. Check uptime to see if this started after a reboot, which points to crash recovery with dirty regions. Check dmesg for recent device errors suggesting disk replacement. Check LVM archive timestamps in /etc/lvm/archive/ for recent metadata changes indicating bulk mirror creation.
Distinguish pvmove from mirror resync. An interrupted pvmove leaves temporary mirror segments that look like mirror problems in lvs output. Check lv_attr position 1 for p (pvmove) and pvs -o pv_name,move_pv for active pvmove state. Cleanup uses pvmove --abort, not mirror repair commands.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`copy_percent` per mirror LV	Indicates whether a resync is in progress and how far along it is	Multiple LVs below 100% simultaneously
`dmsetup status` health characters	Distinguishes resync (`a`) from dead device (`D`) from healthy (`A`)	`a` persisting without `copy_percent` progress
Disk I/O latency per PV	Resync I/O directly increases read and write latency on backing devices	Latency 2-5x above baseline across multiple PVs
`iostat` `%util` per device	Shows storage bandwidth saturation from combined resync load	All backing devices near 100% utilization
D-state process count	Application processes stuck waiting on dm I/O	Growing count of D-state processes on dm devices
`lv_health_status`	Reveals structural issues driving the resync	`r` (refresh needed) or `p` (partial) flags on multiple LVs
LVM command execution time	Slow `lvs` or `pvs` indicates lock contention or I/O pressure	Commands taking more than 5 seconds to complete

Fixes

Throttle recovery rate per LV

For raid1 type LVs, lvchange --maxrecoveryrate sets the maximum resync speed. The rate is in KiB/sec per device. Setting it to 0 means unbounded, which is the default.

# Limit resync rate to 50 MiB/sec per device (51200 KiB/sec)
lvchange --maxrecoveryrate 51200 <vg>/<lv>

# Verify the setting is applied
lvs -o lv_name,raid_max_recovery_rate <vg>/<lv>

This does not pause the resync. It caps the bandwidth the resync consumes, leaving more I/O headroom for production workloads. Apply this to less critical LVs first, then to all resyncing LVs if needed. To remove the throttle after production stabilizes:

lvchange --maxrecoveryrate 0 <vg>/<lv>

This applies only to raid1 type LVs. Legacy --type mirror does not support recovery rate throttling.

There is no lvm.conf default for max_recovery_rate; the rate must be set per-LV with lvchange --maxrecoveryrate or at creation with lvcreate --maxrecoveryrate.

Prioritize critical resyncs by serializing

If throttling alone is not enough, reduce concurrency by letting production-critical mirrors resync first. Deactivate non-critical LVs to pause their resyncs, then reactivate them after the critical ones finish.

For RAID1 LVs that are not in active use, deactivating and reactivating later effectively pauses their resync:

# Deactivate a non-critical LV to pause its resync (DISRUPTIVE if in use)
lvchange -an <vg>/<non-critical-lv>

# Reactivate later to resume resync
lvchange -ay <vg>/<non-critical-lv>

Warning: Deactivating an LV that is mounted or has open file descriptors will fail or cause data corruption. Always verify the LV is not in use before deactivating. Check with lsof or fuser on the device path, and verify the mount table.

Use targeted rebuild instead of full resync

When replacing a specific disk in a RAID1 LV, lvchange --rebuild <pv> reconstructs data on that specific PV only, avoiding a full resync of the entire LV. This is more efficient than lvchange --resync, which forces a complete resynchronization of all legs.

# Rebuild data on a specific PV after disk replacement
lvchange --rebuild /dev/sde1 <vg>/<lv>

This reduces total resync I/O when the trigger is a single disk replacement rather than a crash affecting all mirrors.

Handle interrupted pvmove

If the resync storm includes temporary mirror segments from an interrupted pvmove, use pvmove --abort to cleanly unwind the operation. Never kill a pvmove with kill -9. It leaves the LV in an inconsistent temporary mirror state that persists across reboots.

# Abort an interrupted pvmove cleanly
pvmove --abort

# Resume a pvmove that was interrupted
pvmove

The allocation/pvmove_max_segment_size_mb setting in lvm.conf limits the maximum size of data mirrored in a single pvmove segment. The default is 0 (no limit), meaning a single pvmove segment can be very large. If interrupted, the entire segment must be re-mirrored from scratch. Setting this to a finite value limits each chunk and reduces rework after interruptions.

Tune raid_region_size

The activation/raid_region_size setting in /etc/lvm/lvm.conf controls the synchronization region size for RAID LVs. Larger region sizes reduce bitmap overhead during resync but increase the granularity of dirty region tracking, meaning more data must be resynced after an unclean shutdown. This is a tradeoff between resync efficiency and the amount of dirty data tracked.

Check the current default and its description on your system:

lvmconfig --type default --withcomments activation/raid_region_size

The default raid_region_size is 2048 KiB (possible values: 512, 1024, 2048, 4096).

Prevention

Stagger mirror creation. Create new mirrored LVs sequentially, not in bulk. Allow each initial sync to complete before creating the next. Bulk creation is the most preventable cause of resync storms.
Set recovery rate defaults proactively. Configure --maxrecoveryrate on RAID1 LVs before you need them. If a resync is triggered unexpectedly by a crash or disk failure, the throttle prevents it from consuming all available I/O bandwidth.
Monitor resync status routinely. Track copy_percent across all mirrored LVs. A single resync after disk replacement is expected. Multiple simultaneous resyncs are the pattern to catch early.
Verify resyncs are progressing. During any resync, check that copy_percent is advancing. A stalled resync combined with a second device failure can result in data loss.
Audit mirror segment type. Legacy --type mirror is deprecated and does not support recovery rate throttling. RAID1 (--type raid1) supports throttling and does not block I/O during failure handling. Plan migration from legacy mirror to raid1 where possible.
Document LV-to-service mappings. During a resync storm, you need to know which mirrors back production-critical services so you can prioritize their resyncs. Without this mapping, triage decisions become guesswork.

How Netdata helps

Per-second disk I/O metrics let you see storage bandwidth saturation as it begins, before application latency triggers pages.
Disk latency correlation between dm devices and underlying PVs pinpoints whether the storage path is saturated by resync I/O or by application load.
D-state process monitoring surfaces the user-visible impact of I/O contention, showing which services are stuck waiting on dm devices.
Historical baselines for disk I/O and latency let you compare current resync impact against normal operation, helping you decide whether to intervene or let the resync complete.