LVM keeps the complete map of your logical volumes in a small metadata area at the start of each physical volume. If that metadata is corrupted, by a power loss during a metadata write or a bad sector in the first few megabytes of a PV, the entire volume group becomes unreadable. Every LV in the VG becomes inaccessible at once, regardless of how healthy the actual data extents are.

The recovery path is built into LVM itself. After every metadata change, LVM writes the current configuration to /etc/lvm/backup/<vg> and appends a timestamped copy to /etc/lvm/archive/. The vgcfgrestore command reads these files and rewrites the on-disk metadata. If you have a current archive, recovery is usually a matter of minutes. If you do not, it becomes a data recovery exercise.

This guide covers the full workflow: confirming corruption versus a missing device, choosing the correct archive, handling damaged PV headers, the thin pool complication, and the chicken-and-egg problem when /etc lives on the VG you are trying to restore.

How the archive and backup files work

LVM metadata on disk lives in the volume group descriptor area at the start of each PV, stored as a text-based ring buffer that retains recent metadata versions until old entries are overwritten. The VG metadata carries a sequence number (seqno) that increments on every metadata-modifying operation: LV create, extend, remove, snapshot, pvmove, and so on.

On every change, LVM also writes two files on the root filesystem:

  • /etc/lvm/backup/<vg>: the current metadata, one file per VG. This is what vgcfgrestore uses when you do not specify a file.
  • /etc/lvm/archive/<vg>_<NNNNN>-<hash>.vg: timestamped history of metadata states. Each archive records the state before a specific change was applied.

Archive retention is controlled by the backup section of /etc/lvm/lvm.conf. The defaults keep a minimum of 10 archive files and purge files older than 30 days, so do not assume an archive from last quarter still exists.

You can inspect backup freshness at any time:

# Compare the backup file's seqno with the live VG seqno
grep seqno /etc/lvm/backup/<vg>
vgs --noheadings -o vg_name,vg_seqno

# See the archive history
ls -lt /etc/lvm/archive/ | head -20

If the backup seqno is behind the live seqno, something is preventing backups (read-only /etc, full filesystem, permissions), and your recovery gap is already growing.

Confirm it is corruption, not a missing device

Before restoring anything, rule out the two conditions that look like metadata corruption but are not.

First, verify all PVs are present. vgck requires all PV devices to be accessible, and a missing PV causes it to fail for reasons unrelated to corruption. Second, check for underlying device errors. If dmesg shows ongoing I/O errors on the PV device, you have a hardware problem and restoring metadata onto a dying device will not help.

# Check for missing PVs and partial VGs
pvs -o pv_name,vg_name,pv_attr,pv_size,pv_free
vgs -o vg_name,vg_attr

# Verify metadata consistency (non-zero exit means a problem)
vgck -v <vg>

# Rule out device failure
dmesg | grep -i 'I/O error' | tail -20

The corruption diagnosis is strongest when all PVs are present and readable at the block level, but vgck fails, vgs/lvs report metadata errors, and multiple LVs refuse to activate. If instead a PV shows as [unknown], you are dealing with a missing device, which is a different recovery path. See LVM Couldn’t find device with uuid.

One caution before any write operation: if the underlying device has hardware-level corruption (an SSD with FTL remap problems, for example), image the device first. Writes to a device with address-translation corruption can land in the wrong physical location and destroy the data you are trying to save.

The recovery workflow

The restore has one branching decision: whether the PV header itself is damaged. If the header is intact and only the metadata area is stale or inconsistent, vgcfgrestore alone is enough. If the header is gone (for example, someone ran pvcreate on the device, or the first sectors are destroyed), you must rebuild the header with the original UUID before restoring the metadata into it.

flowchart TD
  A[VG unreadable, vgck fails] --> B{All PVs present and readable?}
  B -- No --> C[Missing PV or device failure - different recovery path]
  B -- Yes --> D{PV header damaged?}
  D -- Yes --> E[pvcreate --uuid --restorefile to rebuild header]
  D -- No --> F[vgcfgrestore --list VG]
  E --> F
  F --> G[Select archive by seqno and description]
  G --> H[vgcfgrestore --test -f file VG]
  H --> I[vgcfgrestore -f file VG - add --force for thin pools]
  I --> J[vgchange -ay VG, then verify]

Step 1: List the available archives

# List all archived metadata states for the VG
vgcfgrestore --list <vg>

Each entry shows the archive file, a timestamp, and a description line of the form “Created before executing ’lvcreate …’” or similar. That description tells you exactly which operation each archive precedes.

Step 2: Choose the right seqno

You want the archive that represents the most recent good state: the one just before the corruption event, not the oldest one and not blindly the newest one.

The risk of choosing wrong is real. If you restore an archive from before a legitimate lvextend, the extent map in the restored metadata will point logical extents at physical extents that no longer match the on-disk layout. Data written after that extend will be read back from the wrong offsets. Read the description lines and reconstruct the timeline: what was the last operation you know succeeded, and what happened around the time things broke.

Step 3: Dry-run the restore

# Test restore without writing anything
vgcfgrestore --test -f /etc/lvm/archive/<vg>_00042-1234567890.vg <vg>

The --test flag validates the file and reports what would be restored without touching the PVs. If the dry run reports inconsistencies, stop and pick a different archive.

Step 4: Rebuild the PV header if needed (only when damaged)

Skip this step if the PV header is intact. If the header is destroyed, recreate it with the original UUID, which is recorded inside the archive file:

# Rebuild the PV header using the UUID recorded in the archive
pvcreate --uuid <original-pv-uuid> --restorefile /etc/lvm/archive/<vg>_00042-1234567890.vg /dev/<device>

This is a destructive write to the start of the device. Double-check the device path and the UUID. Running pvcreate without --restorefile and the original UUID creates a new, empty PV and discards your ability to rejoin the existing VG cleanly. Repeat per damaged PV, then run vgcfgrestore as below.

Step 5: Restore the metadata

# Restore the selected archive onto the VG
vgcfgrestore -f /etc/lvm/archive/<vg>_00042-1234567890.vg <vg>

If the VG contains thin pools, vgcfgrestore requires --force, and this is where you need to think hard. Changes to thin pool kernel metadata cannot be reverted. If thin metadata has changed since the archive was written (new blocks allocated, snapshots created or removed), the restored VG metadata may not match the pool’s internal metadata, and data loss can follow. Prefer the newest archive that is still consistent, and treat --force on a production thin pool as a decision you make after exhausting other options, not a reflex.

Step 6: Activate and verify

# Activate the restored VG
vgchange -ay <vg>

# Verify
vgs -o vg_name,vg_attr,vg_seqno
lvs -o lv_name,vg_name,lv_attr,lv_active
vgck -v <vg>

A warning about inconsistent metadata right after the restore is normal when PVs carried different seqno values. LVM resolves this by selecting the highest seqno. If the inconsistency persists, synchronize metadata across the PVs:

# Rewrite consistent metadata across all PVs
vgck --updatemetadata <vg>

After activation, check that LVs map to the devices you expect and that filesystems mount cleanly. Run a filesystem check (fsck -n for ext4, xfs_repair -n for XFS) read-only first rather than mounting straight into production. A successful metadata restore does not guarantee the data extents were untouched by whatever corrupted the metadata.

Common pitfalls

  • Restoring the wrong archive. The newest file is not automatically correct. If corruption happened during an operation, the archive written before that operation is the last known-good state. Read the description lines.
  • Restoring across a legitimate resize. An archive older than a successful lvextend produces an extent map that disagrees with reality. If you must use an older archive, plan on filesystem-level verification before trusting any data.
  • Forgetting --force on thin pools, or using it blindly. The flag is mandatory for VGs with thin pools, and it bypasses safety checks. The thin metadata in the kernel must match the restored VG metadata precisely.
  • Assuming vgcfgrestore undoes data destruction. It restores metadata, not data. If issue_discards = 1 is set in lvm.conf, an lvremove on an SSD may already have TRIMmed the blocks. No metadata restore brings that back.
  • Running pvcreate casually on a VG member. It overwrites the descriptor area. If the on-disk metadata ring is small, no prior copy survives to recover from.
  • Trusting a 30-day-old mental model of the archive. Default retention prunes old archives. Verify /etc/lvm/archive/ contents during routine operations, not during the incident.

The chicken-and-egg problem: when /etc is on the affected VG

The archive and backup files live on the root filesystem. If your root filesystem is itself an LV on the corrupted VG, the files you need to restore the VG are inside the VG you cannot read. The system may not even boot.

Solve this before the incident:

  • Keep an off-volume copy. Back up /etc/lvm/backup/ and /etc/lvm/archive/ to another host, configuration management, or object storage on a schedule and after every deliberate LVM change.
  • Export explicitly after major operations. vgcfgbackup -f <path-on-different-storage> <vg> writes a metadata copy wherever you point it. The playbook signal for this is backup freshness: compare the seqno in your latest external copy against vgs -o vg_seqno.
  • Boot from rescue media. With an off-volume copy, boot a rescue image, place the archive file where vgcfgrestore can read it, and run the workflow above.
  • Last resort without any archive. The metadata text also exists in the on-disk descriptor area ring buffer on each PV. pvck --dump metadata_search /dev/<device> can extract metadata text from the device when no file copy exists at all. This is slower and more fragile than restoring from a file, which is why the off-volume copy matters.

Signals to monitor

Metadata corruption is rare, but the conditions that make recovery easy or impossible are observable long before the event.

SignalWhy it mattersWarning sign
vgck exit code per VGCatches metadata inconsistency early, while archives are freshAny non-zero exit
Backup seqno vs live VG seqnoStale backups mean a growing recovery gap/etc/lvm/backup/<vg> seqno behind vgs -o vg_seqno
Archive directory stateDefines your recovery windowNewest archive file days old, or directory empty
LV activation stateCorruption surfaces as LVs that will not activateExpected-active LVs inactive after boot
Kernel I/O errors on PV devicesDistinguishes hardware failure from logical corruptionI/O errors in dmesg against a PV device

Run vgck periodically (from a cron job or your monitoring system) rather than waiting for activation failures. Corruption caught while the VG is still functioning gives you time to restore cleanly instead of under incident pressure.

How Netdata helps

  • Block device error visibility. Netdata collects disk-level I/O errors and latency, so you can correlate a vgck failure with hardware errors on the underlying PV and tell corruption apart from a dying disk.
  • LV and filesystem state in one place. Mount points disappearing, filesystems going read-only, and dm device behavior changing at the same timestamp narrows the blast radius to a specific VG quickly.
  • Boot-to-boot comparison. Per-second metrics retained across reboots let you see whether LVs failed to activate at boot or dropped later, which separates metadata problems from activation-ordering problems.
  • Alerting on filesystem and device anomalies gives you the early page when a VG silently degrades, before someone notices missing data.
  • Correlating D-state processes with device errors helps confirm whether an unresponsive LVM command is a symptom of the corruption event itself.