LVM unexpected metadata changes: auditing pvcreate, vgcreate, and lvcreate

You found a logical volume nobody remembers creating. Or a volume group grew a new physical volume overnight. Or lvdisplay output does not match your configuration management records. None of your change tickets explain it, and now you need to answer two questions fast: what changed, and who changed it.

LVM metadata changes are a high-signal security event because every pvcreate, vgcreate, lvcreate, lvremove, and vgextend requires root. An unexpected LVM change outside a change window means root ran that command, which narrows the possibilities to an authorized root process you forgot about (automation, a package script, a provisioning tool) or an unauthorized one (a compromised service that escalated, a rogue script, an attacker with a shell).

LVM keeps a detailed local audit trail, and the kernel keeps another one. Both live on the same box the attacker controls. This guide covers how to read the trail, how to reconstruct what happened, and how to make sure the trail survives long enough to be useful.

What this means

Every metadata-modifying LVM operation writes a timestamped copy of the full VG metadata to /etc/lvm/archive/ before executing the change, and a fresh copy of the latest metadata to /etc/lvm/backup/ afterward. The archive files are plain text, named <vg_name>_<seqno>-<random>.vg, and the description field inside each one records the command that triggered the change. The VG sequence number increments on every modification, so the archive gives you an ordered, per-change history of the VG.

That makes /etc/lvm/archive/ a comprehensive per-change audit trail, with two caveats. First, it is a local file: a root attacker can delete or edit it, and the files are not checksummed or signed. Second, it records what changed, not which process or user invoked the command. For attribution you need the systemd journal or auditd.

A further wrinkle: a hidden logical volume can be created and used with almost no application-visible trace. An LV created with activation disabled (lvcreate -an) has no entry in /dev/mapper/ until someone activates it. An attacker can create an LV, format it, activate and mount it, use it, then deactivate and remove it. Nothing in df output from your applications would ever show it. The only footprints are the archive files, the journal, audit records, and the VG sequence number. For background on the LVM layers involved, see how LVM actually works in production.

Common causes

Quick checks

All of these are read-only.

# Recent archive files: one per metadata change, newest first
ls -lt /etc/lvm/archive/ | head -20

# Current sequence number per VG
vgs --noheadings -o vg_name,vg_seqno

# LVM messages in the systemd journal
journalctl -t lvm | tail -50

# Search audit logs for LVM command executions (if auditd is running)
grep -iE 'pvcreate|vgcreate|lvcreate|lvremove|vgextend' /var/log/audit/audit.log

# All LVs including hidden/internal ones
lvs -a -o lv_name,vg_name,lv_attr,lv_size,origin,seg_monitor

# Compare backup seqno with live seqno (should match)
grep seqno /etc/lvm/backup/<vgname>

Two things to note while you check. First, lvs -a shows internal LVs (thin pool _tdata/_tmeta and similar reserved suffixes) that are legitimate; an LV with a normal-looking name that matches no provisioning record is the anomaly. Second, if pvs/vgs/lvs hang during your investigation, that is a separate problem: see LVM commands hang: when lvs, vgs, and pvs block on locks or dead devices.

How to diagnose it

Work from broadest to narrowest. The goal is a timeline: each change, its sequence number, its timestamp, and the process that caused it.

flowchart TD
  A[Unexpected LVM change detected] --> B[Read /etc/lvm/archive/ timeline]
  B --> C{Archive intact?}
  C -->|yes| D[Extract description fields per change]
  C -->|gaps or deleted files| E[Treat as potential compromise: preserve evidence]
  D --> F[Correlate with journalctl -t lvm and auditd]
  F --> G{Change matches authorized automation?}
  G -->|yes| H[Fix change management records]
  G -->|no| E
  E --> I[Check for hidden LVs with lvs -a, diff vs CMDB]

1. Build the archive timeline. List /etc/lvm/archive/ newest first. Each file’s description field contains the command that triggered it, and the seqno in the filename orders the changes. Read the files, not just the names: the description tells you whether it was lvcreate, vgextend, lvextend, or something else.

2. Check for gaps. Seqno increments by one per change per VG. If the archive jumps from seqno 41 to 44 with no files in between, either retention settings pruned it or someone deleted files. Retention pruning removes the oldest files, not middle ones, so holes in the middle of the sequence are suspicious.

3. Correlate with the journal. journalctl -t lvm around each archive timestamp shows the LVM subsystem’s view of the operation. The lvm.conf log/journal option (default empty) controls what is recorded in the systemd journal: set it to command, output, or debug.

4. Correlate with auditd and auth logs. If auditd has an execve rule for the LVM binaries, you get the invoking user, process, and parent process. If not, fall back to sudo logs and last to see who had root sessions at those timestamps.

5. Diff current state against records. Run pvs, vgs, and lvs -a and compare against your configuration management database or provisioning records. Pay attention to PVs on devices you do not recognize (a rogue PV on unexpected hardware is a data exfiltration vector) and to LVs that exist in metadata but are not activated or mounted anywhere.

6. If the trail is damaged, preserve everything else. Copy off the remaining archive files, the journal, audit logs, and shell histories before changing anything. Then treat the host as potentially compromised and follow your incident response process; storage forensics is one thread of a larger investigation.

Signals to monitor

Response and containment

The finding splits two ways.

If the change is authorized but undocumented: reconcile your records. Add the automation or script to your change inventory so the next audit does not page someone at 3 a.m. This is the common case, and it is still worth fixing because noise trains people to ignore the signal.

If the change is unauthorized: do not start by removing the rogue LV or VG. Preserve evidence first: copy /etc/lvm/archive/, /etc/lvm/backup/, the journal, and audit logs off the host. Then decide on containment (isolate the host, rotate credentials) per your IR runbook. Only after evidence is captured should you clean up the LVM state. lvremove is destructive and irreversible: confirm nothing legitimate is mounted from the LV and that you have captured the metadata before running it. If metadata was tampered with and the VG is now inconsistent (vgck non-zero), vgcfgrestore can restore metadata from a known-good archive or backup file. Verify the restore source predates the tampering, and treat the restore itself as a disruptive operation: it overwrites live VG metadata.

Prevention

The detection strategy rests on one principle: the local trail is not trustworthy on a compromised host, so ship the evidence off-box.

• Enable auditd execve rules for LVM binaries. Watch pvcreate, vgcreate, lvcreate, lvremove, vgextend, and vgcfgrestore so every invocation is logged with user and process context. Forward audit logs to a remote collector.
• Forward the journal off-box. journalctl -t lvm is only useful if the journal survives. Use persistent journal storage and remote syslog forwarding.
• Copy the archive directory off-host. A periodic sync of /etc/lvm/archive/ and /etc/lvm/backup/ to a central location preserves VG history even if the local copies are deleted. It also protects against the ordinary case: metadata corruption recovery needs those files, as covered in LVM Couldn’t find device with uuid scenarios where vgcfgrestore is the way back.
• Alert on seqno changes outside change windows. Poll vgs -o vg_seqno and alert when it changes without a matching change record. This is cheap and catches changes even when the archive is tampered with, because the seqno itself moves.
• Baseline your LV inventory. Keep a known-good list of PVs, VGs, and LVs per host in configuration management, and alert on drift, including PVs appearing on unexpected device types.
• Keep retention sane. The archive retention settings (retain_min, retain_days in lvm.conf) control pruning. The defaults are retain_min = 10 and retain_days = 30 in lvm.conf. Do not prune aggressively on production hosts; the files are small and the history is valuable.

How Netdata helps

• Netdata collects LVM state continuously, including VG capacity and LV layout signals, so an unexpected new LV or a change in VG extent allocation shows up in per-second charts as a visible step change, even if nobody was watching lvs.
• Because Netdata stores a time series, you can correlate the moment an LV appeared with system-level activity at the same second: process count, CPU, disk I/O on specific devices, and network connections, which helps identify what was running when the change happened.
• New block device I/O on a device with no legitimate workload is itself a detection signal: a hidden LV that gets activated and written to produces disk utilization on its backing PVs, visible in Netdata’s disk charts even if the LV is never mounted anywhere obvious.
• Alerting on Netdata’s LVM and disk metrics gives you the “something changed” trigger; the archive, journal, and auditd workflows in this guide give you attribution. Metrics detect, logs attribute.

Related guides

• LVM boot activation failure: emergency shell and missing mount points
• LVM cannot extend a logical volume: adding a PV when the VG is full
• LVM commands hang: when lvs, vgs, and pvs block on locks or dead devices
• LVM Couldn’t find device with uuid: a physical volume has gone missing
• LVM dm-N device numbers change after reboot: use /dev/mapper, not /dev/dm-N
• LVM dmeventd not running: the auto-extend safety net is offline
• LVM filesystem full while the volume group has space: the resize step everyone forgets
• LVM Found duplicate PV: multipath devices and the lvm.conf filter
• How LVM actually works in production: a mental model for operators
• LVM Insufficient free extents: the volume group is out of space
• LVM I/O hang: a suspended dm device and processes stuck in D state
• LVM lock contention: stale lock files and blocked commands