$ guides / lvm / lvm-metadata-backup-stale ▌

Operations Guides

LVM metadata backup stale: /etc/lvm/backup out of sync with the VG

Every LVM command that changes metadata (lvcreate, lvextend, lvremove, vgextend, snapshot creation) writes two things to the local filesystem: an archived copy of the new metadata in /etc/lvm/archive/ and a current-state backup in /etc/lvm/backup/<vg>. That backup file is what vgcfgrestore reads when you recover a VG after metadata corruption.

The failure mode here is quiet: something blocks those writes, the backup file stops updating, and nothing in day-to-day operation tells you. LVM keeps working. Months later you hit metadata corruption, reach for vgcfgrestore, and restore a VG layout from six months ago. LVs created since then are missing. LVs removed since then come back pointing at extents that have been reallocated. You have traded a recovery exercise for a data integrity problem.

Detection is a two-command check: compare the seqno field inside /etc/lvm/backup/<vg> against the live vg_seqno from vgs. If they differ, your recovery lifeline is stale and you need to find out why before anything else goes wrong.

What this means

VG metadata carries a sequence number that increments on every metadata change. The backup file records the metadata as of some past sequence number in its seqno field. In a healthy system the backup is rewritten after every change, so the two numbers match.

When the backup seqno lags, every intervening metadata operation is missing from your recovery point. The gap matters most in exactly the environments that change LVM layout often: automated snapshot rotation, container or VM hosts creating and destroying thin volumes, provisioning scripts touching LVM.

The common thread behind staleness is that LVM could not write to /etc/lvm/. The write failure is usually silent at the severity level operators watch: the metadata change itself succeeded on disk, the command returns success, and the workload is unaffected. Only the on-host backup copy failed.

flowchart LR
  op["LVM metadata change
(lvcreate, lvextend, ...)"] --> md["Metadata written
to all PVs"]
  md --> arch["Archive copy
/etc/lvm/archive/"]
  md --> bak["Backup copy
/etc/lvm/backup/<vg>"]
  bak --> blocked{"Write blocked?"}
  blocked -->|"read-only /etc, missing dir,
permissions, full fs, backup=0"| stale["Backup seqno lags live vg_seqno
recovery point erodes"]
  blocked -->|"no"| ok["seqno matches
vgcfgrestore is trustworthy"]

Common causes

Cause	What it looks like	First thing to check
/etc (or its filesystem) mounted read-only	LVM commands succeed but backup file mtime is old; occasional “Cannot archive volume group metadata for %s to read-only filesystem.” type messages in logs	`mount \| grep ' / '` and `findmnt -no OPTIONS /etc` for `ro`
`/etc/lvm/backup` directory missing or wrong permissions	Backup write fails on every metadata change	`ls -ld /etc/lvm /etc/lvm/backup /etc/lvm/archive`
Filesystem containing /etc full or out of inodes	Backup writes fail; other small-file writes on the same fs also fail	`df -h /etc` and `df -i /etc`
`backup = 0` set in lvm.conf	No backup file is written at all; archive may still update depending on `archive` setting	`lvmconfig backup/backup backup/archive` or grep the backup section of `/etc/lvm/lvm.conf`
Hardened or image-based provisioning wipes /etc/lvm state	Backup file absent or ancient relative to VG age	Compare backup mtime to the VG’s earliest LV creation time in `lvs -o lv_name,lv_time`

The most dangerous variant is backup = 0, because there is no stale file to notice. There is no file at all. If a provisioning template or hardening guide set this, you have no local recovery point for any VG on the host.

Quick checks

# 1. Live sequence number for each VG
vgs --noheadings -o vg_name,vg_seqno

# 2. Sequence number recorded in each backup file
grep -H 'seqno' /etc/lvm/backup/*

# 3. Backup file timestamps vs recent LVM activity
ls -la /etc/lvm/backup/
ls -lt /etc/lvm/archive/ | head -20

# 4. Is /etc writable right now?
findmnt -no TARGET,OPTIONS /etc
touch /etc/lvm/.writetest && rm /etc/lvm/.writetest && echo "writable" || echo "NOT writable"

# 5. Space and inode headroom on the filesystem holding /etc
df -h /etc
df -i /etc

# 6. Backup and archive configuration actually in effect
lvmconfig backup/backup backup/backup_dir backup/archive backup/archive_dir
# (on older LVM2: lvm dumpconfig backup)

# 7. Retention settings
lvmconfig backup/retain_min backup/retain_days

# 8. Recent archive churn: are archives being written while backup is not?
stat -c '%y %n' /etc/lvm/archive/* | sort -r | head -5

Healthy result: the seqno in /etc/lvm/backup/<vg> equals vg_seqno from vgs, the backup mtime tracks the newest archive file, and both directories are writable with free space and inodes.

How to diagnose it

Quantify the gap. For each VG, subtract the backup seqno from the live vg_seqno. A difference of 1 or 2 on a quiet system may be a single failed write. A difference in the hundreds means backups have been broken for a long time or the system churns metadata constantly (automated snapshots are the usual driver).
Check the archive directory. If /etc/lvm/archive/ has recent files but /etc/lvm/backup/<vg> is old, the write path works in general and something specific to the backup file is wrong (permissions on the file itself, or the backup was manually removed and LVM could not recreate it). If both are stale, the whole /etc/lvm write path is broken: read-only mount, full filesystem, or backup/archive disabled.
Check writability and capacity. Run checks 4 and 5 above. A read-only root after filesystem errors is a classic cause: the system remounted / read-only weeks ago, everything else degraded gracefully, and nobody noticed because applications keep running. Check dmesg and the journal for remount-ro events.
Audit the effective configuration. Use lvmconfig (or lvm dumpconfig) rather than reading lvm.conf by eye. Configuration can come from profile directories or command-line overrides, and a commented-out default in the file does not tell you what the tools will do. Confirm backup = 1 and archive = 1.
Test the write path explicitly. Run vgcfgbackup <vg> and watch for errors. This is safe: it writes the current metadata to /etc/lvm/backup/<vg> (or to a file you name with -f). If it fails, the error message points at the cause. If it succeeds, the path is healthy now and the staleness was a past condition; check whether it recurs after the next real metadata change.
Fix the root cause, then re-baseline. Remount the filesystem read-write if it flipped, recreate missing directories, fix ownership, free space or inodes, or restore backup = 1. Then run vgcfgbackup for every VG so the backup reflects reality.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Backup seqno vs live vg_seqno (per VG)	Direct measure of recovery-point staleness	Any non-zero gap persisting across a metadata change
`/etc/lvm/backup/` file mtimes	Cheap proxy when you cannot parse seqno	Backup mtime older than newest archive file
`/etc/lvm/archive/` file count and total size	Archive accumulates a file per metadata change; retention is a floor, not a ceiling	Thousands of files, or near-zero history on an active system
vg_seqno rate of change	Tells you how fast a stale backup decays and how aggressive archive churn is	Sustained high rate (automated tooling hammering LVM)
Filesystem space and inodes on the fs holding /etc	Backup writes fail silently when either runs out	>90% used or inodes nearly exhausted
/etc mount options	Read-only remount is a top cause of silent staleness	`ro` present where `rw` is expected

One nuance on archive retention: retain_min and retain_days are minimums. LVM keeps at least retain_min archive files and does not prune files younger than retain_days, but it does not automatically delete older archives beyond that guarantee. On hosts with automated snapshot rotation, archives can accumulate into the thousands of files over years. Prune them deliberately (by age, with a scheduled job), not by lowering retention settings and expecting LVM to garbage collect. Conversely, if an external cleanup job prunes too aggressively, you destroy the rollback history that makes vgcfgrestore useful for anything except the latest state.

Fixes

Re-sync the backup now

# Safe: rewrites the backup from live metadata
vgcfgbackup

# Or per VG, and to a scratch file first if you want to inspect before overwriting
vgcfgbackup -f /tmp/myvg.backup myvg

vgcfgbackup does not modify VG metadata. It reads the current on-disk metadata and writes the backup file. Do this after every root-cause fix so your recovery point is current.

Restore a writable /etc

If the filesystem remounted read-only, find out why before remounting: dmesg and the journal will show the I/O errors that triggered it. Remounting failing media read-write without addressing the underlying errors can make things worse. Once the fs is healthy, mount -o remount,rw / (or the relevant mount) and re-run vgcfgbackup.

Recreate missing directories and permissions

mkdir -p /etc/lvm/backup /etc/lvm/archive
chmod 700 /etc/lvm/backup /etc/lvm/archive   # root-only is the normal posture for these

Then vgcfgbackup to repopulate.

Re-enable backup in lvm.conf

Set backup = 1 and archive = 1 in the backup section of /etc/lvm/lvm.conf, verify with lvmconfig, and re-run vgcfgbackup. Then find out what set it to 0, because it will come back on the next image rebuild or config management run.

Copy backups off the volume

/etc/lvm/backup lives on the root filesystem. If the root filesystem is itself on LVM in the affected VG, you have a chicken-and-egg problem: the recovery file for the VG sits on storage provided by that VG. Ship the backup and archive directories to another host or object store on a schedule. A stale local backup plus a current off-host copy is a recoverable position; a stale local backup alone is not.

Prevention

Alert on seqno divergence. A small script comparing grep seqno /etc/lvm/backup/<vg> against vgs -o vg_seqno per VG, run from cron or your monitoring agent, catches this the day it starts instead of the day you need the backup. Any persistent non-zero gap should open a ticket.
Alert on the write path, not just the result. Watch mount options and free space/inodes on the filesystem holding /etc. Read-only remounts and inode exhaustion break backups long before they break anything operators notice.
Audit lvm.conf after provisioning changes. backup, archive, retain_min, and retain_days belong in a configuration baseline. Image builds and hardening scripts are the usual source of silent changes.
Manage archive growth deliberately. Pick a retention window (for example, 90 days of archives on hosts with automated snapshot churn), enforce it with a scheduled cleanup, and monitor archive directory size so you notice both runaway growth and accidental total pruning.
Include vgcfgbackup in change automation. Any playbook that creates or removes LVs should end with vgcfgbackup and a push of /etc/lvm/backup/ to off-host storage. The automatic backup is a safety net, not a distribution mechanism.

How Netdata helps

Filesystem and inode saturation on the /etc filesystem is collected continuously, so the capacity and inode exhaustion that silently break backup writes show up as trends and alerts before the write path fails.
Mount state and filesystem error visibility lets you correlate a read-only remount event with the moment backup mtimes stopped advancing, instead of discovering both during an incident.
Custom seqno and file-age collection: export the per-VG seqno gap and backup mtime from the divergence script above via a custom collector or textfile-style endpoint, and “backup staleness” becomes a monitored signal with history: you can see when the gap opened and which metadata changes fell into it.
Because Netdata keeps per-second history locally, the sequence “remount-ro event, backup writes stop, seqno gap grows” is reconstructable after the fact for the postmortem.