LVM monitoring maturity model: from survival to expert

Most teams monitor LVM by accident. They alert on filesystem fullness, maybe on disk failure, and discover thin pool exhaustion or a silently degraded mirror only when applications start returning I/O errors. LVM has its own failure domains that filesystem metrics cannot see: a thin pool can be at 100% data usage while df shows 50% free, and a mirror can run on one leg for months with no visible symptom.

This article defines four monitoring maturity levels for LVM, based on which failure modes each level actually catches. Level 1 keeps you from being blindsided by the disasters that generate postmortems. Level 4 gives you the signals you need during an incident, when the standard LVM tools themselves may hang. Use it as a self-assessment: find the first level where you are missing signals, and close that gap before adding anything from a higher level.

The levels build on each other deliberately. Collecting expert-level dmsetup baselines is wasted effort if you are not yet alerting on thin pool metadata usage.

flowchart TD
  L1["Level 1: Survival
VG free, thin pool data%, PV present,
LV health and activation"]
  L2["Level 2: Operational
+ metadata%, snapshot COW%, RAID sync,
vgck, dmeventd, D-state"]
  L3["Level 3: Mature
+ overprovisioning, runway, resync rate,
autoextend audit, command latency"]
  L4["Level 4: Expert
+ dmsetup baselines, queue_if_no_space audit,
D-state stacks, backup freshness, integrity"]
  L1 --> L2 --> L3 --> L4

Level 1: survival

The absolute minimum. These five signals answer: can we allocate, is thin storage about to fail, are all disks present, and is anything already in a failed state.

Two severity notes from production experience. First, a missing PV with a redundant (RAID/mirror) layout is a ticket, not a page: the array survives on remaining legs. It becomes a page when the affected LV is non-redundant with an active workload, or when application I/O errors confirm impact. Second, thin pool data% is a ticket even at 95%, because the pool crossing a threshold is not itself the outage. The outage is write failure, and that shows up as application errors, not as a metric.

Do not page on thin pool data% alone at 85%. Auto-extend, if actually configured, may handle it. The relevant escalation is the composite: pool filling AND VG free space too low to extend it.

Level 2: operational

Everything in Level 1, plus the signals that catch the failure modes Level 1 misses. This is where most teams discover their gaps.

• Thin pool metadata usage (lvs -o metadata_percent). The silent killer. Metadata exhaustion can corrupt the pool even when data% is moderate, and recovery via lvconvert --repair is not guaranteed. Ticket at 75%, urgent ticket at 90%. Page at 90% only when corroborated: no dmeventd monitoring (seg_monitor), no VG free space to extend into, or health showing M or F.
• Snapshot COW usage (lvs -o snap_percent). Traditional snapshots invalidate instantly and irreversibly at 100%. The origin keeps running; the restore point dies silently. Ticket at 80%, urgent at 95%. An I in attr position 5 means the snapshot is already lost.
• Mirror/RAID sync status (lvs -o copy_percent, dmsetup status). Health character D (dead leg) is a ticket: you are running without redundancy. Character a (not yet in-sync) is normal during rebuild and should not alert by itself. copy_percent stuck for over an hour means the resync is stalled, usually by underlying device errors.
• VG metadata consistency (vgck <vg>). Any non-zero exit is a ticket. Escalate to page only if LV activations are failing alongside it. Note that vgck needs all PVs accessible; a missing PV causes failures unrelated to corruption.
• dmeventd running (systemctl is-active lvm2-monitor.service, pgrep -x dmeventd). Without dmeventd, thin pool auto-extend does nothing. This is the safety net, and it is frequently discovered to be offline only during the incident it was supposed to prevent. Ticket if absent on any host with thin pools, RAID, or snapshots.
• D-state processes on dm devices (ps -eo pid,stat,wchan:30,comm | grep ' D'). Accumulating D-state processes are the user-visible symptom of a hung thin pool or suspended dm device. Page when the count is growing and processes have been stuck more than 60 seconds.

Level 2 is also where you verify a widely-missed fact: thin_pool_autoextend_threshold defaults to 100, which means auto-extend is effectively disabled. If you have not explicitly set it in /etc/lvm/lvm.conf, your thin pools will fill without any extension attempt.

Level 3: mature

Level 3 turns monitoring from reactive to predictive. The signals here are leading indicators and configuration audits, not just current state.

• Overprovisioning ratio. Sum of thin LV virtual sizes versus pool physical size. A pool at 60% data% overprovisioned 3:1 is far more dangerous than the percentage suggests; one runaway writer can consume the remaining physical space quickly.
• Snapshot count and age per origin. Each traditional snapshot multiplies COW overhead on every origin write (read old, write snapshot, write new). Any snapshot older than 24 hours on a high-churn LV is a ticket: either the backup forgot to clean up, or nobody is watching the restore point fill.
• Resync progress and rate. Beyond the binary “degraded or not,” track whether copy_percent is actually moving. Resyncs compete with production I/O, and multiple simultaneous resyncs (a resync storm after a crash) can saturate the storage. Estimate time to full redundancy, not just current state.
• Per-PV free distribution (pvs -o pv_name,pv_free,pv_size). A VG showing 30% free with all free extents on one PV cannot extend a striped LV evenly. Aggregate VG free space hides allocation imbalance.
• Auto-extend configuration audit. Verify all three conditions, not just the config file: dmeventd is running, the threshold in lvm.conf is below 100, and the VG has free extents for at least two extension cycles. Auto-extend that fires but fails because the VG is full fails silently.
• Allocation rate and runway. Compute consumption rate from vgs free extents and dmsetup status used-data-block deltas over time: days_until_full = vg_free / consumption_rate. For thin pools, project (100 - data_percent) / growth_rate with a safety margin. Any metadata growth above roughly 1% per week warrants investigation, since metadata consumption is non-linear.
• LVM command latency (time pvs, time lvs). A healthy query completes in under a second. Above 10 seconds something is wrong: device scanning on an unresponsive device, lock contention, or a filter misconfiguration. This is also your early warning that the management plane itself is degrading.

Level 4: expert

These signals exist because of a specific problem: during the worst LVM incidents, pvs, vgs, and lvs hang. They take VG locks and read metadata from PVs, which requires I/O to the very devices that are failing. Level 4 instrumentation works when LVM tools do not.

• dmsetup status baselines. dmsetup reads from kernel memory, takes no LVM locks, and works during incidents. Collect dmsetup status --target thin-pool regularly so you know what normal used_data/total_data and transaction ID rates look like. Running it for the first time during an incident, with no baseline, slows diagnosis.
• queue_if_no_space audit. Know which policy each thin pool uses before the incident. With the default queue policy, writes stall when the pool fills; the visible symptom is a system freeze, not error messages. The kernel queues them for up to 60 seconds (the dm-thin no_space_timeout module parameter, default 60) before switching the pool to error_if_no_space and failing the queued writes. The policy flag is visible in dmsetup status output as error_if_no_space once that switch has happened.
• D-state stack analysis. Automate capture of /proc/<pid>/stack for stuck processes. The stack tells you whether the process is waiting on dm, a filesystem, or a lower storage driver, which routes the investigation to the right layer. D-state processes cannot be killed, so this is your only diagnostic.
• Seqno and backup freshness. Compare the seqno in /etc/lvm/backup/<vg> against vgs -o vg_seqno. A stale backup means your metadata recovery lifeline is out of date, often because /etc filled or a config issue blocked backups. If /etc itself lives on the VG, you also have a chicken-and-egg recovery problem: keep a copy off-volume.
• dm-integrity violation tracking. For integrity-protected LVs, watch dmsetup status --target integrity and kernel log checksum messages. Any mismatch is a data corruption event.
• Audit trail monitoring. /etc/lvm/archive/ records every metadata change. Unexpected entries on production systems outside change windows are a security signal: LVM operations require root, so unauthorized changes imply root-level access.

Common ways teams stall between levels

• Filesystem monitoring standing in for LVM monitoring. df cannot see VG free space, thin pool data%, or metadata%. These are three independent capacity dimensions; track them separately.
• Assuming the safety net works. Auto-extend requires dmeventd running, a threshold below 100, and VG free space. Most teams have verified none of the three.
• Alerting on absolute thresholds without corroboration. The severity logic above repeatedly gates pages on kernel-level evidence (I/O errors, D-state processes, health flags). If your monitoring only collects LVM command output, those gates are unimplementable and every severity decision lacks context.
• Monitoring through tools that hang during the incident. If your collector runs lvs every 10 seconds and lvs blocks behind a VG lock or a dead PV, you go blind exactly when you need data most. Prefer dmsetup-based collection for critical signals.

How Netdata helps

• Netdata collects per-second block-device I/O metrics from /proc/diskstats for every dm device, giving you dm-layer latency and inflight I/O trends that LVM itself never retains. LVM reports current state only; trending requires external time-series storage.
• Process state collection surfaces D-state processes as they accumulate, which is the corroboration signal several page conditions depend on and which lvs cannot provide.
• Collecting dmsetup status output alongside command-based metrics gives you a data path that keeps working when LVM tools hang behind locks or unresponsive PVs.
• Correlating thin pool data%, metadata%, VG free, and dm latency on one dashboard makes the composite patterns (thin pool death spiral, snapshot overflow with origin degradation) visible as patterns rather than isolated threshold crossings.
• Long retention of VG free and pool usage history turns the Level 3 runway calculations from manual spreadsheet work into a trend you can read off a chart.

Related guides

• LVM monitoring checklist: the signals every production volume manager needs
• How LVM actually works in production: a mental model for operators
• LVM Insufficient free extents: the volume group is out of space
• LVM volume group running low on free space: vg_free and runway estimation
• LVM cannot extend a logical volume: adding a PV when the VG is full
• LVM has free space but striped or mirrored allocation still fails
• LVM filesystem full while the volume group has space: the resize step everyone forgets
• LVM thin pool out of data space: every thin volume freezes at once
• LVM thin pool metadata full: the exhaustion that can corrupt the pool
• LVM reached low water mark for data device: the thin pool warning before the freeze
• LVM thin pool auto-extend not working: threshold 100 means disabled
• LVM thin pool queue_if_no_space: why a full pool hangs instead of erroring