$ guides / lvm / lvm-vgck-metadata-inconsistent ▌

Operations Guides

LVM vgck metadata inconsistency: PVs disagree about the volume group

vgck returned non-zero on a production VG, or you are seeing warnings like “Inconsistent metadata found for VG” or “ignoring metadata seqno N on /dev/sdX for seqno M on /dev/sdY.” The physical volumes in the volume group hold different versions of the VG metadata. One or more PVs are stale: they missed a metadata update because they were offline, unreachable, or had bad blocks when the write happened.

LVM replicates complete VG metadata across every member PV. Each metadata change increments a sequence number (vg_seqno). When all PVs agree, the VG is consistent. When they disagree, you have split metadata. The impact ranges from noisy warnings on every LVM command to failed LV activation and, in the worst case, an unreadable VG.

Most metadata inconsistencies are repairable without data loss, provided all PVs are present and the damage is in the metadata text rather than the PV headers. The repair path depends on what caused the divergence and what tools your LVM version offers.

What this means

VG metadata lives in the first approximately 1 MiB of each PV, typically stored in duplicate. It contains the UUID, extent map, VG membership, the complete LV segment layout, and the vg_seqno counter. Every metadata-modifying operation (LV creation, resize, snapshot, pvmove) increments vg_seqno and writes the updated metadata to all PVs in the VG.

If a PV is unavailable when a metadata write occurs, it retains the old seqno. When that PV reappears, LVM detects the mismatch. Depending on LVM version and configuration, it may auto-repair on the next metadata write, or it may emit warnings and refuse certain operations until you intervene.

flowchart TD
    A["vgck non-zero exit"] --> B{"All PVs present?"}
    B -->|"No"| C["Different problem:
missing PV"]
    B -->|"Yes"| D{"seqno differs
between PVs?"}
    D -->|"Yes"| E["Stale PV missed
a metadata write"]
    D -->|"No"| F["Possible header
or text damage"]
    E --> G["vgck --updatemetadata"]
    G -->|"Fails or old LVM"| H["vgcfgrestore from
archive backup"]
    F --> I["pvck -v to inspect
metadata area"]

The canonical symptom is a vg_seqno mismatch between PVs. You may also see “WARNING: PV /dev/sdX in VG vgY is using an old PV header, modify the VG to update,” which is a related but distinct issue involving the pv_header_extension.version field rather than the VG metadata seqno. Both are repairable with vgck --updatemetadata on LVM 2.03.05 and later.

Common causes

Cause	What it looks like	First thing to check
PV offline during metadata write	One PV has a lower seqno than the rest. PV is now visible but stale.	`vgs -o vg_name,vg_seqno` and `pvs -o pv_name,vg_name`
Crash or power loss during metadata update	Multiple PVs may disagree. No single PV is clearly “old.”	`dmesg` for recent kernel panic or unclean shutdown markers
Bad blocks in PV metadata area	One PV reports read errors from its metadata region. I/O errors in `dmesg` for that device.	`dmesg
Clustered LVM split-brain	Different cluster nodes see different seqno values for the same VG.	Cluster quorum and lock manager (DLM or sanlock) state
Cloned or dd-copied PV without UUID regeneration	Two devices report the same PV UUID. Duplicate PV warnings.	`pvs -o pv_name,pv_uuid` for duplicate UUIDs
Old PV header format	“using an old PV header” warning on every LVM command. seqno agrees between PVs.	`vgck -v <vgname>` output

Quick checks

Run these read-only commands to characterize the inconsistency before attempting any repair.

# Check VG metadata consistency with verbose output
vgck -v <vgname>
# Exit code 0 = consistent, non-zero = problem

# Compare seqno across the VG
vgs -o vg_name,vg_seqno

# Verify all PVs are present and accessible
pvs -o pv_name,vg_name,pv_attr,pv_size,pv_free
# Missing PVs show as "[unknown]" or with attr 'm'

# Confirm each PV device actually exists as a block device
for pv in $(pvs --noheadings -o pv_name <vgname>); do
  [ -b "$pv" ] && echo "$pv: OK" || echo "$pv: MISSING"
done

# Check VG partial status (p in attr position 4 means a PV is missing)
vgs -o vg_name,vg_attr <vgname>

# Inspect raw metadata on individual PVs
pvck -v /dev/sdX1
pvck -v /dev/sdY1

# Check LV activation state (metadata problems can block activation)
lvs -o lv_name,vg_name,lv_attr,lv_active <vgname>

# Verify metadata backup freshness
grep seqno /etc/lvm/backup/<vgname>
ls -lt /etc/lvm/archive/<vgname>/ | head -10

# Check for kernel I/O errors on PV devices
dmesg | grep -i 'I/O error' | tail -30

How to diagnose it

Confirm all PVs are present. vgck requires all PV devices to be accessible. A missing PV causes vgck to fail for an unrelated reason. If any PV shows as [unknown] in pvs output, you have a missing-device problem, not a metadata corruption problem. See LVM Couldn’t find device with uuid for that path.
Run vgck with verbose output. vgck -v <vgname> prints which PVs it checked and any inconsistency details. A non-zero exit code confirms the problem. The output typically names the PVs that disagree and shows the conflicting seqno values.
Compare seqno across PVs. Use pvck -v /dev/sdX on each PV to read its metadata area. The seqno in each PV’s metadata should match. A PV with a lower seqno missed one or more metadata writes.
Check for header-level damage. If pvck -v reports errors reading the label header, disk label, or metadata area header, the problem is in the PV header structure, not just the metadata text. Header-level damage requires pvck --repair and is a more serious condition.
Rule out clustered split-brain. If you run clustered LVM (clvmd or lvmlockd), check cluster quorum and lock manager health before assuming local corruption. A node that lost quorum may have a stale view. Different nodes reporting different seqno values for the same VG points to a cluster communication issue, not disk corruption.
Check for duplicate PV UUIDs. If a PV was cloned (dd, VM disk snapshot, SAN LUN clone) without regenerating its UUID, LVM sees duplicate PVs and may read metadata from the wrong copy. pvs -o pv_name,pv_uuid reveals duplicates. See LVM Found duplicate PV for resolution.
Assess LV activation impact. Run lvs -o lv_name,lv_attr,lv_active <vgname>. If LVs that should be active are inactive, the metadata inconsistency may be blocking activation, which escalates urgency. If all LVs are active and serving I/O normally, the inconsistency is not yet causing an outage but still needs repair.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`vgck` exit code	Direct test of metadata consistency across all PVs	Any non-zero exit
`vg_seqno` per PV	Reveals which PVs hold stale metadata	seqno differs between PVs in same VG
PV accessibility	Missing PVs cause vgck failure unrelated to corruption	PV shows as `[unknown]` or attr `m`
LV activation state	Metadata corruption can block LV activation	Expected-active LVs showing inactive
LVM command latency	Slow `pvs`/`vgs`/`lvs` can indicate I/O problems in the metadata area	Commands taking more than 5 seconds
Metadata backup freshness	`/etc/lvm/backup/` seqno should match the live VG	Stale backup means the recovery fallback is compromised
dmesg I/O errors	Bad blocks in the metadata area cause read/write failures	I/O errors on PV device paths

Fixes

vgck –updatemetadata (LVM 2.03.05 and later)

Available since lvm2 version 2.03.05 (June 2019), this is the primary repair command. It rewrites VG metadata to correct the most common inconsistencies: stale seqno on a PV that was offline during a write, damaged metadata text with intact headers, and old PV header format warnings.

# Repair metadata inconsistency (requires all PVs present)
vgck --updatemetadata <vgname>

This command uses the same metadata write path as normal VG operations (vg_write). It copies the current authoritative metadata to all PVs, bringing stale ones up to date. It also handles the case where a PV was removed from the VG while missing and has reappeared: it clears the outdated metadata from that PV.

What --updatemetadata cannot repair: corruption in the label header, pv_header or its location fields, or corruption of certain mda_header fields. For those conditions, pvck --repair is required.

After the repair, verify:

# Confirm consistency is restored
vgck <vgname>
echo "Exit code: $?"

# Verify seqno now matches across PVs
vgs -o vg_name,vg_seqno

If LVs failed to activate while the VG was inconsistent, retry activation:

lvchange -ay <vgname>/<lvname>

vgcfgrestore from archive (fallback for older LVM or severe cases)

On systems running LVM older than 2.03.05 (legacy RHEL 7, Ubuntu 16.04), or when --updatemetadata fails, restore from the LVM metadata archive. LVM automatically backs up metadata to /etc/lvm/backup/ (latest copy) and /etc/lvm/archive/ (historical copies) after every metadata change.

# List available archive files for the VG
ls -lt /etc/lvm/archive/<vgname>/

# Preview what a restore would do (does not write)
vgcfgrestore --test -f /etc/lvm/archive/<vgname>/<archive_file>.vg <vgname>

# Perform the restore
vgcfgrestore -f /etc/lvm/archive/<vgname>/<archive_file>.vg <vgname>

# Rescan to pick up the restored metadata
pvscan --cache
vgscan

Choose an archive file whose seqno matches the majority of your PVs (the authoritative version). Verify the archive’s seqno before restoring:

grep seqno /etc/lvm/archive/<vgname>/<archive_file>.vg

Warning: vgcfgrestore overwrites metadata on all PVs. If the archive predates recent legitimate changes (LV creation, resize), those changes are lost from metadata. The data on disk is not destroyed, but LVM will not know about LVs created after the archive timestamp. Identify every operation between the archive date and the incident before restoring.

When a PV is genuinely missing

If vgck fails because a PV is absent (not stale, but truly gone), this is a different problem. Do not run vgck --updatemetadata or vgcfgrestore until you understand why the PV is missing. See LVM Couldn’t find device with uuid for the missing-PV diagnostic path.

If the PV is permanently lost and you have confirmed its data is either migrated or expendable, vgreduce --removemissing <vgname> removes the missing PV from VG metadata. This is destructive: any LV extents on that PV are permanently abandoned.

Header-level damage (pvck repair)

When pvck -v /dev/sdX shows damage in the label header, PV header fields, or MDA header (not just the metadata text), vgck --updatemetadata will not fix it. These are lower-level structures that vg_write does not rewrite.

Use pvck --repair on the affected PV. This is an advanced repair that works on the PV header structures. Consult the pvck(8) man page for your specific LVM version, as the repair options have evolved.

Prevention

Monitor vg_seqno across PVs. Any discrepancy is abnormal. Trending the seqno also reveals excessive automated LVM operations: many rapid increments suggest scripts doing too many LV operations. See LVM monitoring checklist for what to instrument.
Ensure clean shutdowns. The most common cause of stale PV metadata is a crash or power loss during a metadata write. After any crash, run vgck on all VGs before returning the system to production.
Verify PV return after temporary absence. When a PV goes missing and comes back (SAN path flap, cable reseating, multipath recovery), it may reappear with stale metadata. Run vgck after any PV disappearance and reappearance event.
Maintain metadata backup freshness. Verify periodically that the /etc/lvm/backup/ seqno matches the live VG seqno. If /etc is on an LVM volume, store backup copies off-volume or off-host to avoid a chicken-and-egg problem during recovery.
Clustered LVM discipline. In clustered setups, verify quorum before performing VG operations. Concurrent metadata writes from nodes that have lost quorum are a primary cause of split metadata. See LVM monitoring maturity model for cluster-specific monitoring guidance.
Never clone PVs without UUID regeneration. If you clone a disk or LUN that contains a PV, use pvchange -u to regenerate the UUID, or filter the clone out of LVM’s device scan in lvm.conf.

How Netdata helps

Correlate PV disappearance with metadata warnings. When a PV flaps, Netdata shows the device state change alongside any LVM command latency spike or error, helping you identify which event produced the stale metadata.
Track LVM command execution time. If pvs or vgs suddenly takes 10 seconds instead of sub-second, it can indicate I/O problems in a PV metadata area. Netdata timing data across the LVM management plane surfaces this degradation early.
Monitor dmesg for I/O errors. Bad blocks in a PV metadata area produce kernel I/O errors. Netdata correlates these errors with specific block devices, pinpointing which PV is developing hardware problems before metadata corruption occurs.
Detect LV activation failures. If metadata inconsistency blocks LV activation, Netdata catches the inactive LV state and correlates it with the timing of preceding storage events, shrinking the diagnostic window.