$ guides / lvm / lvm-boot-activation-failure ▌

Operations Guides

LVM boot activation failure: emergency shell and missing mount points

You boot a server and land in a dracut emergency shell or initramfs rescue prompt. Or the system boots but /mnt/data, /var/lib/postgresql, or other non-root filesystems are not mounted. The common thread: LVM volumes that should have activated during early boot did not.

The initramfs activates LVM volumes before mounting root and before systemd mounts the rest of fstab. If the initramfs lacks the right LVM tools or configuration, or if a required PV is slow to appear (iSCSI not connected, multipath not configured, SAN LUN not presented), activation fails before you have a running system to diagnose from.

What this means

The initramfs (built by dracut on RHEL/Fedora/Arch, or initramfs-tools on Debian/Ubuntu) contains a snapshot of the tools and configuration needed to bring up root. For LVM-backed root, that snapshot must include the lvm binary, the relevant udev rules, and a copy of /etc/lvm/lvm.conf matching the current storage topology.

When the initramfs runs, it triggers LVM activation. On dracut-based systems, this happens via udev rules and the lvm_scan mechanism. On initramfs-tools (Debian/Ubuntu), the script at /usr/share/initramfs-tools/scripts/local-block/lvm2 calls lvchange -aay to autoactivate volumes.

If any link breaks, the LV backing root never gets a device-mapper table loaded. The kernel cannot mount root. dracut drops you to its emergency shell. If root activated fine but a secondary VG did not, the system boots but fstab mounts fail.

Key distinction: vgchange -ay activates all LVs in a VG unconditionally. vgchange -aay (autoactivation) only activates LVs matching the auto_activation_volume_list in lvm.conf. If that list is an explicit empty array ([]), no volumes autoactivate. If it is unset (commented out), all volumes are eligible. The initramfs uses autoactivation, so this distinction matters at boot.

For degraded or partial VGs: if a PV is missing at boot, the VG is in a partial state. dracut can activate degraded VGs using --activationmode degraded after a retry timeout. initramfs-tools on Debian/Ubuntu historically did not support partial VG activation the same way after the lvm2 2.03.15/16 udev rule changes.

Note on lvmetad: the metadata caching daemon was removed in LVM 2.03.00. All currently supported distributions (RHEL 8+, Ubuntu 22.04+, Debian 12+) ship LVM 2.03.x and have never included lvmetad. If you are on a legacy system with use_lvmetad = 1 in lvm.conf, that setting is ignored.

flowchart TD
    A["Kernel boot"] --> B["initramfs loads"]
    B --> C["LVM tools in initramfs?"]
    C -->|No| X["Emergency shell: lvm not found"]
    C -->|Yes| D["Scan for PVs"]
    D --> E["All PVs present?"]
    E -->|No| F["Wait for rd.retry timeout"]
    F --> G["Degraded activation allowed?"]
    G -->|Yes, dracut| H["Activate with --activationmode degraded"]
    G -->|No, initramfs-tools| X
    E -->|Yes| I["vgchange -aay autoactivation"]
    I --> J["LV matches auto_activation_volume_list?"]
    J -->|No| K["LV stays inactive"]
    J -->|Yes| L["dm table loaded, LV active"]
    L --> M["Mount root from /dev/mapper/VG-LV"]
    M --> N["Mount succeeds?"]
    N -->|No| X
    N -->|Yes| O["systemd takes over, mounts fstab"]
    K --> O

Common causes

Cause	What it looks like	First thing to check
initramfs not rebuilt after LVM config change	Emergency shell, `lvm` works manually but root LV not found	`lsinitrd` or `lsinitramfs` for lvm.conf presence
PV slow to appear (iSCSI, multipath, SAN)	Emergency shell, but `vgchange -ay` succeeds after manual run	`journalctl -b` for iSCSI/multipath login timing vs LVM activation
lvm.conf filter excludes root PV	Shell reports root LV not found, PV not scanned	`grep filter /etc/lvm/lvm.conf` and whether root disk matches
auto_activation_volume_list set to empty	System boots but specific LVs never activate	`grep auto_activation_volume_list /etc/lvm/lvm.conf`
kernel root= parameter mismatch	Emergency shell, root device path does not resolve	`cat /proc/cmdline` vs actual LV path
Partial VG at boot (missing PV)	VG shows partial flag, root LV has extents on missing PV	`vgs -o +vg_attr` for `p` flag in attribute 4

Quick checks

Run these from the emergency shell or after manual recovery. All are read-only.

# Check whether LVM config is baked into the initramfs
lsinitrd -f /etc/lvm/lvm.conf 2>/dev/null || lsinitramfs /etc/lvm/lvm.conf 2>/dev/null

# List all PVs and their VG membership
pvs -o pv_name,vg_name,pv_attr,pv_size

# Check VG status and partial flag (position 4 of vg_attr)
vgs -o vg_name,vg_attr,pv_count

# Check LV activation state (position 5 of lv_attr: a=active, blank=inactive)
lvs -o lv_name,vg_name,lv_attr,lv_active

# Verify device-mapper devices exist in the kernel
dmsetup ls

# Check the kernel root= parameter
cat /proc/cmdline

# Look for LVM activation failures in this boot
journalctl -b | grep -iE 'lvm|activation|pvscan'

# Verify the root LV device node exists
ls -la /dev/mapper/

# Check dmesg for device discovery and I/O errors
dmesg | grep -iE 'scsi|iscsi|multipath|I/O error|device not found'

How to diagnose it

Determine whether root is affected. If you are in a dracut or initramfs emergency shell, root failed to mount. If the system booted to a login prompt but mounts are missing, root activated fine and the problem is in secondary VGs.
Manually activate VGs. From the emergency shell:
```
lvm vgscan --mknodes
lvm vgchange -ay
```
If this succeeds, the root LV should now have a /dev/mapper/ entry.
Mount root and exit the shell.
```
mount /dev/mapper/<vg>-<root_lv> /sysroot
exit
```
On dracut the target is typically /sysroot. On initramfs-tools it may be /root. Exiting the shell (or Ctrl+D) tells the initramfs to retry mounting root.
If activation fails with a partial VG. A PV is missing. Check whether the underlying device exists:
```
lsblk
pvs
```
If the PV is on iSCSI or multipath, the storage may not be ready. On dracut systems, try degraded activation:
```
vgchange -ay --activationmode degraded <VG>
```
Warning: degraded activation on a VG with extents on the missing PV means that data is inaccessible regardless of activation mode. Use only to get root mounted when root LV extents are on surviving PVs.

Once booted, find the root cause.

systemctl status lvm2-activation.service lvm2-pvscan@*.service
journalctl -b -u 'lvm2-*' --no-pager

Check lvm.conf filter. The filter and global_filter directives control which block devices LVM scans. If the filter rejects the root PV’s device, LVM never finds the VG.
```
grep -E '^\s*(global_)?filter' /etc/lvm/lvm.conf
```
The filter must accept (the a rule) the device underlying the root PV. After fixing, test with pvs to confirm the root PV is found, then rebuild the initramfs.
Check auto_activation_volume_list. If set to an explicit empty list, boot-time autoactivation skips all volumes.
```
grep auto_activation_volume_list /etc/lvm/lvm.conf
```
A commented-out (unset) value means all volumes are eligible. An explicitly empty [] means none.

Rebuild the initramfs after any configuration change so the fix persists:

dracut -f
# or for Debian/Ubuntu, rebuild for all installed kernels
update-initramfs -u -k all

Metrics and signals to monitor

Signal	Why it matters	Warning sign
LV activation state (lv_attr position 5)	Inactive LV after boot means mount failure	Expected LV shows blank or `i` instead of `a`
VG partial flag (vg_attr position 4)	Missing PV prevents full activation	`p` flag present after boot completes
PV accessibility	Missing PV blocks VG activation	PV shows `[unknown]` in `pvs` output
lvm2-activation service status	Systemd unit controls post-initramfs activation	Service failed or timed out in journal
Boot mount completeness	Missing mounts indicate secondary VG activation failure	Expected mount points absent from `/proc/mounts`
LVM command execution time	Slow scans indicate device discovery timing issues	`pvs` or `vgs` taking more than a few seconds at boot

Fixes

initramfs not rebuilt after LVM configuration change

The most common cause after adding a PV, changing a VG, or editing lvm.conf. The running system has the new configuration, but the initramfs snapshot is stale.

dracut -f          # RHEL, Fedora, Arch with dracut
update-initramfs -u -k all   # Debian, Ubuntu

Safe to run at any time. Keep the previous kernel installed as a fallback in case the new initramfs has a problem.

PV slow to appear during boot

On systems with iSCSI, FCoE, or multipath-backed PVs, storage may not be ready when the initramfs runs LVM activation. dracut provides the rd.retry kernel parameter to control how long it waits before giving up.

Increase the retry timeout on the kernel command line if devices consistently appear late:

rd.retry=300

For iSCSI boot, ensure the initiator configuration is included in the initramfs. For multipath, verify the multipath configuration and rules are copied into the initramfs.

Tradeoff: longer retry means slower boots when storage is genuinely unavailable.

lvm.conf filter excludes the root PV

If the filter rejects the device holding the root PV, LVM never scans it. This often happens after a device path change (new disk added, NVMe namespace renumbered, multipath configuration changed).

Adjust the filter to accept the root device. Prefer persistent identifiers:

filter = [ "a|/dev/disk/by-id/|", "a|/dev/mapper/|", "r|.*|" ]

Test with pvs to confirm the root PV is visible, then rebuild the initramfs.

auto_activation_volume_list misconfigured

If set to an explicit empty list, no volumes activate during boot even though the initramfs runs vgchange -aay correctly.

Fix: either unset the parameter (comment it out so all volumes are eligible) or explicitly list the VG and LV that must activate at boot. Rebuild the initramfs afterward.

Partial VG at boot (missing PV)

If a PV is genuinely missing (disk failure, SAN LUN unpresented, multipath all paths down), the VG is partial. Root LV extents on the missing PV mean root cannot mount.

On dracut systems: degraded activation via --activationmode degraded is supported. The initramfs falls back to this after the retry timeout expires.

On initramfs-tools (Debian/Ubuntu): partial VG activation is not supported the same way.

Options if initramfs-tools cannot activate a partial VG:

Restructure the root VG with vgsplit so root lives on a VG with all PVs guaranteed present at boot. Move secondary LVs to a separate VG that mounts later from the running system.
Switch to dracut on the affected host if your distribution supports it.

Tradeoff: degraded activation gets you booting on a damaged VG. For RAID or mirror LVs this may be acceptable. For linear or striped LVs with extents on the missing PV, that data is gone regardless of activation mode.

Prevention

Rebuild the initramfs after every LVM change. Any filter edit, PV addition, VG modification, or lvm.conf change requires dracut -f or update-initramfs -u -k all. This prevents the majority of boot activation failures.
Verify the initramfs contains your lvm.conf. After rebuilding, check with lsinitrd or lsinitramfs that the expected file is present and matches.
Use persistent device names in lvm.conf filters. Filter on /dev/disk/by-id/ or /dev/mapper/ rather than /dev/sd* to survive device reordering.
Document the root= kernel parameter. It must match the actual LV path (/dev/mapper/vg-root or UUID=...). Mismatches after LV or VG renames cause boot failure.
Test boot after storage stack changes. Adding multipath, changing iSCSI targets, or swapping HBA cards shifts device discovery timing and can break activation ordering.
Keep a fallback kernel installed. The previous kernel’s initramfs is your recovery path if a rebuild introduces a problem.
For network-backed root storage, verify boot timing. Ensure the rd.retry window is long enough for your SAN to present LUNs, and that iSCSI or multipath initiator config is baked into the initramfs.

How Netdata helps

Netdata collects per-second device-mapper metrics, so you can correlate the moment an LV activation fails with device discovery events in the kernel log and disk I/O patterns. The LVM collector surfaces VG free space, thin pool utilization, and PV accessibility alongside boot and service metrics, helping you distinguish a missing PV from a configuration issue.

Correlating lvm2-* service status with mount point availability and disk I/O latency in the same dashboard helps separate “LV never activated” from “LV activated but mount failed for a different reason.” Historical retention of device-mapper metrics provides a baseline for normal activation timing, making slow scans or missing PVs immediately visible after a reboot.