$ guides / lvm / lvm-monitoring-checklist ▌

Operations Guides

LVM monitoring checklist: the signals every production volume manager needs

Most LVM incidents are not exotic. They are a thin pool that hit 100% because nobody watched metadata_percent, a mirror that ran degraded for three months because nobody alerted on copy_percent, or a PV that went missing on a Friday and paged nobody because the check only ran df. The failure modes are well known. What is usually missing is a deliberate, minimal set of signals with thresholds and severities attached.

This checklist is that set, organized the way an operator should think about LVM: first the three independent capacity layers, which fail separately and must be monitored separately; then the core availability and health signals; then the supporting signals that separate a competent setup from a mature one. Each item includes the command to collect it, the threshold worth alerting on, and the severity it deserves.

Use it as an audit. Walk your current monitoring against each section and note what is missing.

The three independent capacity layers

The single most common LVM monitoring mistake is treating storage capacity as one number. It is three:

Filesystem usage (df) - how full the filesystem on each LV is.
VG free space - how many unallocated physical extents remain for extending LVs, growing thin pools, and creating snapshots.
Thin pool usage (data and metadata, separately) - how full the pool backing thin LVs actually is.

These fail independently. A thin pool can be 100% full, freezing all writes, while df on the thin LV filesystems shows 50%. A VG can be completely full while every filesystem has space remaining, which means the moment any filesystem fills, you cannot extend it. df alone sees neither condition.

flowchart TD
  FS["Filesystem usage (df)"] -->|"extends into"| LV["LV (lvextend)"]
  LV -->|"allocates from"| VG["VG free extents (vgs)"]
  THIN["Thin LVs"] -->|"allocate blocks from"| POOL["Thin pool: data_percent + metadata_percent (lvs)"]
  POOL -->|"auto-extends into"| VG
  POOL -.->|"df on thin LVs does not show pool fullness"| FS

Every capacity alert on any of these three layers should include the state of the other two in its context. “Filesystem 90% full” is only actionable if the operator can also see whether the VG and the thin pool have room to extend into.

Level 1: survival signals

The absolute minimum. If you monitor nothing else, monitor these four. They answer: can we allocate, is thin storage viable, are all disks present, is anything in a failed state.

VG free space percentage

Why: When VG free extents hit zero, every allocation operation fails: LV extension, snapshot creation, and thin pool auto-extend. The failure is a cliff edge, with no graceful degradation between 99% and 100%.
Collect: vgs -o vg_name,vg_size,vg_free,vg_free_count,vg_extent_count
Thresholds: below 20% add to capacity planning; below 10% ticket (a single growth event can exhaust what remains); below 5% critical ticket.
Gotcha: free space distribution matters. A VG showing 30% free with all of it on one PV cannot allocate new stripes for a striped LV. At the mature level, track per-PV free with pvs -o pv_name,vg_name,pv_free,pv_size.

Thin pool data usage

Why: When data_percent reaches 100%, all thin LVs in the pool fail writes simultaneously. Depending on configuration, writes either queue for a timeout and then error out, or error immediately. Databases corrupt, VMs pause, applications take I/O errors.
Collect: lvs -o lv_name,vg_name,data_percent,metadata_percent. During an active incident, prefer dmsetup status <vg>-<thinpool>; lvs takes metadata locks and performs I/O, and can hang on the very storage you are trying to diagnose.
Thresholds: above 85% ticket (extend within shift); above 95% urgent ticket. Also alert on consumption rate projecting exhaustion within 48 hours.
Gotcha: thin_pool_autoextend_threshold defaults to 100, which means auto-extend is disabled. Do not assume the safety net exists. Also note that pool usage only goes up unless discard/TRIM is configured and fstrim runs; spikes do not self-resolve.

PV accessibility

Why: A missing PV means a disk failed, a SAN path dropped, or a multipath device lost all paths. Linear and striped LVs with extents on that PV are immediately inaccessible; RAID and mirror LVs degrade silently.
Collect: pvs -o pv_name,vg_name,pv_attr and look for [unknown] devices or the m (missing) attribute. Cross-check vgs -o vg_name,vg_attr for the partial flag.
Thresholds: binary. Any PV missing with system uptime over 600 seconds is a ticket, minimum. Page when the affected LV is non-redundant with an active workload, application I/O errors are present, or a thin pool’s data or metadata LV sits on the missing PV. The uptime gate absorbs slow SAN scans during boot.
Gotcha: correlate before concluding hardware death. PV missing plus I/O errors in dmesg points at the device; PV missing plus multipath path failures points at the fabric or multipathd; PV missing with no kernel messages points at SAN presentation or udev.

LV health status (lv_attr position 9)

Why: This single character surfaces structural problems no capacity metric will: partial volumes, thin pool failure, RAID mismatches.
Collect: lvs -o lv_name,vg_name,lv_attr, reading position 9:

Value	Meaning	Severity
`-`	healthy	none
`p`	partial, a PV is missing	ticket; page if non-redundant with active workload
`D`	thin pool out of data space	ticket; page if production workloads are erroring
`F`	thin pool or cache failed	ticket; page with confirmed I/O impact
`M`	thin/cache metadata read-only	ticket, pool needs repair
`m`	RAID mismatches detected	ticket, data integrity concern
`r`	RAID refresh needed (device returned)	ticket
`c`	RAID scrub/check needed	ticket

Gotcha: m can appear transiently after an unclean shutdown until the RAID scrub completes. Do not page on it at boot; do ticket it if it persists.

Level 2: operational signals

What a competent production team adds on top of survival.

Thin pool metadata usage

Why: This is the signal most teams miss, and it is more dangerous than data exhaustion. When the metadata LV fills, the pool can become corrupted, not merely full. Recovery via lvconvert --repair is not guaranteed, and the man page warns that data may be unrecoverable. The insidious part: data_percent can sit at 50% while metadata_percent hits 100%, because metadata consumption tracks unique block writes and snapshot count, not data volume. Random I/O consumes metadata faster than sequential I/O.
Collect: same lvs or dmsetup status output as data usage; the fields come as a pair.
Thresholds: above 50% plan an extension; above 75% ticket; above 90% urgent ticket, escalating to a page when corroborated by no working auto-extend path (dmeventd down, VG full) or health status showing M or F.
Gotcha: metadata does not shrink. Discard reclaims data blocks, not metadata. Any growth rate above roughly 1% per week deserves investigation.

LV activation state (lv_attr position 5)

Why: An LV that should be active but is not is already an outage for whatever mounts it. A suspended LV (s) blocks all I/O; brief suspension during resize is normal, a suspension stuck for minutes is a kernel-level problem with D-state processes piling up.
Collect: lvs -o lv_name,vg_name,lv_attr,lv_active and dmsetup info -c -o name,suspended. Position 5 values to know: a active, s suspended, I invalid snapshot, i inactive.
Thresholds: production LV unexpectedly inactive (expected in the mount table, not present) is a page. Any dm device suspended longer than 5 minutes with no approved LVM operation in progress is a page. Active in metadata but missing from dmsetup ls is a ticket (kernel/userspace desync).
Gotcha: do not alert on intentionally inactive LVs such as templates and backup snapshots. Maintain a known-inactive list.

Mirror/RAID synchronization status

Why: The classic silent killer. A mirror that loses one leg keeps running with zero redundancy and zero visible errors until the second failure, which is total data loss. Resync operations also compete with production I/O, so you need to know they are progressing rather than stalled.
Collect: lvs -o lv_name,vg_name,lv_attr,copy_percent for sync progress, and dmsetup status <vg>-<lv> for per-device health characters: A alive and in sync, a alive but resyncing (normal during rebuild), D dead (a failure).
Thresholds: any device showing D is a ticket (same-shift device replacement). copy_percent not progressing for over an hour is a ticket. Healthy copy count below the required redundancy, or a degraded array with ongoing I/O errors on survivors, is a page.
Gotcha: a during a rebuild is not a failure; do not alert on it as one. Gate resync alerts on mirrors that existed before the current boot, since newly created mirrors legitimately start at 0%.

Snapshot usage (traditional COW snapshots)

Why: When a traditional snapshot’s exception store fills, the snapshot is immediately and irreversibly invalidated. No warning, and if a backup was reading from it, the restore point is silently gone. Usage is driven by writes to the origin, not the snapshot, so it can fill in minutes on a write-heavy origin.
Collect: lvs -o lv_name,vg_name,origin,snap_percent,lv_attr.
Thresholds: above 80% ticket (extend the snapshot or finish the backup now); above 95% urgent ticket; I flag in position 5 means the snapshot is already lost. Also ticket on snapshots older than 24 hours on high-churn origins.
Gotcha: allocation happens in chunks, so snap_percent jumps several percent at once. Do not extrapolate a smooth line to overflow time. Thin snapshots do not have this failure mode; they inherit the thin pool’s instead.

dmeventd running

Why: dmeventd is the event daemon behind thin pool auto-extend and RAID failure handling. If it is not running, auto-extend silently does nothing no matter what lvm.conf says, and you find out at 100% pool usage. Whether a pool is actually monitored is also visible per-pool via the seg_monitor field, which is worth checking because dmeventd can run without monitoring every pool.
Collect: pgrep -x dmeventd, or check the relevant systemd unit. Add lvs -o lv_name,seg_monitor for thin pools.
Thresholds: ticket if dmeventd is down on any host with thin pools. Combine with a config audit: verify thin_pool_autoextend_threshold in /etc/lvm/lvm.conf is set to something below 100 (70 to 80 is typical), and that VG free space can actually fund an extension. Auto-extend configured against a full VG fails silently.

Level 3 and 4: mature and expert signals

Once the operational set is stable and quiet, add:

Thin pool overprovisioning ratio. Sum of thin LV virtual sizes versus pool physical size. A pool at 60% data_percent with 3:1 overprovisioning is one runaway process away from an outage.
Per-PV free space distribution, to catch allocation imbalance before striped LVs fail to extend.
Auto-extend configuration audit, treated as a monitored state, not a one-time setup task.
VG metadata consistency, periodic vgck per VG. Any non-zero exit is a ticket; combined with LV activation failures, escalate.
LVM command execution time. time lvs taking over 10 seconds indicates lock contention or an unresponsive device, and means your LVM-based monitoring is going stale exactly when you need it. This is the argument for collecting via dmsetup status, which reads kernel state without LVM locks.
Snapshot count and age per origin, since multiple snapshots multiply COW overhead on origin writes.
D-state process count and kernel I/O errors, as corroboration signals. Several page-level conditions above (suspended LV, partial VG, pool failure) should only page when confirmed by workload impact, and that confirmation lives in ps state and dmesg, not in LVM output.
Thin pool metadata integrity, periodic offline thin_check runs. Any failure is a ticket.

Behaviors that will waste your time

A short calibration list, so the checklist does not generate noise:

Mirror resync after an unclean shutdown is expected; copy_percent below 100% right after boot is recovery, not failure.
Transient s (suspended) during lvextend is sub-second and normal.
data_percent dropping after fstrim is discard reclaiming pool blocks. Healthy.
Slow first activation of a large thin pool (5 to 30 seconds) is metadata loading, not a hang.
An LV larger than its filesystem after lvextend just means nobody ran the filesystem resize; use lvextend --resizefs next time.
dm-N numbers changing across reboots is normal; never reference /dev/dm-X in configs or alerts, use /dev/mapper/VG-LV.

Conversely, treat these innocent-looking states as urgent: a degraded mirror still serving I/O, a thin pool at moderate data_percent with high overprovisioning, auto-extend “configured” with a full VG, and any snapshot above 90% that nobody owns.

How Netdata helps

Netdata’s value for LVM is correlation across the layers this checklist treats as separate:

It tracks VG free space, thin pool data_percent and metadata_percent, and filesystem usage as independent per-second time series, so a “filesystem full” alert arrives with pool and VG headroom already in view.
It retains history, which LVM itself never does. Trend lines turn “pool at 83%” into “pool fills in 6 days at current rate,” which is the difference between a planned extension and a 3 a.m. page.
It collects the corroboration signals the severity gates depend on: dm device latency from /proc/diskstats, D-state processes, and kernel I/O errors, alongside the LVM layer, so “pool at 96%” and “applications erroring” show up on the same dashboard instead of in two tools.
Anomaly detection on metadata_percent growth and dm latency catches the slow, non-linear failure modes (metadata creep, resync storms) that fixed thresholds miss.
Per-device charts make silent redundancy loss visible: a degraded RAID leg or stalled copy_percent trends as a picture, not a log line nobody read.