LVM pvmove stuck or interrupted: temporary mirror state and how to unwind it

You ran pvmove to drain a disk, it has been running for hours, and now something is wrong. Maybe the SSH session died, maybe the host rebooted, maybe someone killed the process. Now lvs shows a mirror LV you never created, and every LVM command against the volume group feels slow or blocked.

This is the pvmove temporary mirror state. It is recoverable in almost every case, but only if you unwind it the way LVM expects: resume it with a bare pvmove, or abort it with pvmove --abort. The one move that turns a tedious situation into a bad one is kill -9 on the pvmove process, or worse, deleting the mirror LV by hand.

What this means

pvmove does not copy extents directly from one PV to another. It creates a temporary internal mirror LV (conventionally named pvmove0), inserts mirror segments in front of the data being moved, syncs one segment at a time, then breaks the mirror and checkpoints the metadata. It repeats this until all extents have been relocated. On large volumes this runs for hours and holds the VG lock for much of that time, which is why unrelated lvs/vgs/pvs calls can stall while it runs.

Two observable markers tell you a migration is in progress or was interrupted:

• A pvmove0 LV appears in lvs -a output, looking exactly like a mirror you did not create.
• lv_attr position 1 shows p on the affected LVs, indicating pvmove-type volume (migration in progress).

If pvmove exits cleanly, both disappear. If pvmove is interrupted (crash, reboot, kill, OOM), the temporary mirror persists in the VG metadata and the kernel keeps using it. The system is usually still serving I/O correctly at this point. The mirror is functional; it is just unfinished business that blocks the source PV from being removed and confuses anyone reading lvs output later.

stateDiagram-v2
    [*] --> Normal: LVs on source PV
    Normal --> Migrating: pvmove started
    Migrating --> Normal: completed, mirror torn down
    Migrating --> Interrupted: crash, reboot, kill
    Interrupted --> Migrating: pvmove (no args) resumes from checkpoint
    Interrupted --> RolledBack: pvmove --abort
    RolledBack --> Normal: unmoved segments stay on source
    Normal --> [*]: source PV removable

Common causes

Quick checks

All read-only. Prefer dmsetup when LVM commands feel slow: it reads kernel state directly and does not take LVM metadata locks or scan devices.

# Find the temporary mirror and migration state
lvs -a -o lv_name,vg_name,lv_attr,copy_percent,seg_pe_ranges

# Confirm the pvmove attribute flag (position 1 = 'p')
lvs -o lv_name,vg_name,lv_attr

# Check which PV is the source of an in-flight move
lvs -a -o lv_name,vg_name,lv_attr,move_pv,copy_percent

# Is a pvmove process actually alive?
ps aux | grep -v grep | grep pvmove

# Kernel-side view without LVM locks
dmsetup status
dmsetup info -c -o name,suspended,open

# Who holds the VG lock?
ls -la /run/lock/lvm/
fuser /run/lock/lvm/* 2>/dev/null

# Underlying disk trouble on source or target?
dmesg | grep -iE 'I/O error|reset|offline' | tail -50

# dm-mirror target available?
lsmod | grep dm_mirror

How to diagnose it

1. Confirm you are looking at pvmove state, not a real mirror problem. The combination of a pvmove0 LV and p in lv_attr position 1 is the distinguishing signal. A mirror LV with a different name and no p flag is a different incident (see the mirror/RAID sync signal in the LVM mental model).

2. Determine whether the move is live or orphaned. If a pvmove process is running and copy_percent advances between samples taken a few minutes apart, the move is alive. Let it run or plan an abort; do not interrupt it. If no process exists and the state survived a reboot, it is orphaned and waiting for you to resume or abort.

3. Check progress and pace. lvs -o lv_name,copy_percent on the pvmove LV. A copy_percent that has not moved in an hour, with a live process, means the source or target disk is slow or erroring. Check dmesg before assuming LVM is the problem.

4. Check I/O impact on production. pvmove mirrors writes while syncing, so write latency on affected LVs rises during the move. Correlate dm device latency on the source and target PVs against your baseline before deciding to abort for performance reasons. Aborting a move that is 95% done because of latency complaints usually costs more than letting it finish.

5. Check for lock fallout. If the VG lock seems stuck and the holding PID no longer exists, you may have a stale lock file in /run/lock/lvm/. Confirm the process is truly gone before removing anything.

6. Decide: resume or abort. Resume when the goal (drain the source PV) still stands and the target disk is healthy. Abort when the target PV is failing, the move was a mistake, or you need the VG lock and I/O bandwidth back urgently.

Metrics and signals to monitor

Fixes

Resume an interrupted move

Run pvmove with no PV arguments. It restarts any in-progress operations from the last metadata checkpoint:

# Resume all interrupted pvmove operations in progress
pvmove

This is the default correct action after a reboot or session death when you still want the data off the source PV. Moved segments stay moved; the operation continues from where the metadata says it stopped.

Abort the move

# Abort all in-progress pvmove operations
pvmove --abort

Without --atomic, abort is not a rollback: segments already moved stay on the destination PV, unmoved segments stay on the source. Your LVs end up split across both PVs, which is functionally fine but means the source PV is not drained. If you started pvmove with --atomic, an abort leaves all affected LVs entirely on the source PV, which is the clean “pretend it never happened” outcome. --atomic must be specified when you start the move, not retroactively.

One operational caveat: there are longstanding reports of pvmove --abort hanging and only completing after a reboot. If abort appears stuck, confirm with dmsetup status whether mirror sync activity is actually progressing before assuming it is dead.

Never kill -9, and what to do if someone already did

kill -9 leaves the mirror running in the kernel with no userspace process managing it. The recovery is the same as any other interruption: run bare pvmove to resume, or pvmove --abort to unwind. Do not lvremove the pvmove0 LV and do not hand-edit the dm table. Both risk leaving LVs pointing at extents LVM no longer tracks.

Abort worked, but re-running pvmove fails with “Insufficient free space”

A known trap: after an abort, re-running the move can fail with an extents-needed-versus-available error even when the destination PV shows exactly enough free extents. The usual cause is allocation policy: an LV split into multiple segments (from a previous resize) cannot place two segments on the same PV under the normal allocation policy. The workaround is:

# Relax allocation policy for the move
pvmove --alloc anywhere /dev/sdX /dev/sdY

RAID LVs showing “refresh needed” after the move

After pvmove on RAID LVs, a leg can report lv_health_status of r (refresh needed), and lvconvert --repair may fail with insufficient-space errors because of stale references left by the move. The fix is:

# Refresh the affected LV
lvchange --refresh <vg>/<lv>

pvmove refuses to start: missing dm-mirror target

If pvmove fails with “Required device-mapper target(s) not detected in your kernel”, the dm-mirror module is not loaded:

# Load the mirror target
modprobe dm-mirror

Kernel-level crashes during pvmove

There are documented kernel faults triggered during pvmove, including a NULL pointer dereference in dm-mirror’s mirror_merge on RHEL 7.9 kernels later than 3.10.0-1160.10.1.el7, and a separate report involving the mq_deadline I/O scheduler on kernel 5.5.8. In the worst of these cases the pvmove0 LV is left in a state where neither resume nor --abort works. Manual recovery (deactivating, removing the pvmove LV, restoring VG metadata from /etc/lvm/archive/ via vgcfgrestore) has been described in field reports but is destructive-adjacent and version-specific. Treat this as a “restore from metadata backup, then verify data” scenario, and check your kernel against known pvmove issues before moving large volumes on older distributions.

Prevention

• Watch it while it runs. pvmove can run for hours holding the VG lock and adding mirror write overhead. Track copy_percent and source/target PV latency for the duration instead of starting it and walking away.
• Use --atomic when partial placement is unacceptable. If an abort must leave everything on the source, atomic is the only way to get that guarantee.
• Bound the rework with segment sizing. The allocation/pvmove_max_segment_size_mb setting in lvm.conf caps how much data is mirrored in a single operation (0 means no limit). A non-zero value limits how much work is lost or must be redone after an interruption. The default value is 0 (no limit), per pvmove(8).
• Verify the target before you start. Check dmesg for errors on the destination PV and confirm dm-mirror is loaded. Moving onto a failing disk converts a maintenance task into an incident.
• Plan for the lock. Schedule large moves in a window where stalled LVM commands and elevated write latency are tolerable, and use dmsetup status rather than lvs for high-frequency progress checks during the move.

How Netdata helps

• Per-disk and per-dm-device I/O latency and throughput, per second, so you can see the mirror write overhead of an active pvmove on source and target PVs and tell “slow because syncing” apart from “slow because the disk is dying.”
• Block device error and reset signals surfaced alongside the latency view, which is the corroboration you need before aborting a move against a suspect target PV.
• D-state process visibility, so a mirror stall shows up as accumulating uninterruptible processes before the system feels hung.
• Historical baselines for the affected devices, letting you quantify “2x normal latency” instead of guessing from the incident itself.
• Alerting that keeps working when LVM commands stall: Netdata reads kernel counters directly, so it does not go blind behind the VG lock the way an lvs-polling script does.

Related guides

• How LVM actually works in production: a mental model for operators
• LVM I/O hang: a suspended dm device and processes stuck in D state
• LVM Couldn’t find device with uuid: a physical volume has gone missing
• LVM logical volume partial (p) flag: which LVs the missing disk took down
• LVM Insufficient free extents: the volume group is out of space
• LVM dmeventd not running: the auto-extend safety net is offline