LVM volume group running low on free space: vg_free and runway estimation

Your volume group still works: LVs are active, filesystems are mounted, applications are writing. But vgs shows vg_free shrinking week over week, and the trend says you hit zero sometime next month. VG exhaustion is a cliff edge: there is no graceful degradation between 99% and 100% full. Everything works until an allocation request cannot be satisfied, and then it fails immediately. Watching vg_free is the warning the cliff never gives you.

This article covers reading the VG free space signals, estimating runway from consumption rate, why low VG free is especially dangerous with thin pools, and what to do before the playbook in LVM Insufficient free extents becomes your incident.

What this means

A volume group is a pool of physical extents (PEs, default 4 MiB each) aggregated from one or more PVs. vg_free is the unallocated portion: the raw material for creating LVs, extending LVs, creating snapshots, and growing thin pools. When vg_free declines, nothing breaks yet. What you are losing is options.

The three ways low VG free turns into an outage:

1. Thin pool auto-extend fails. Auto-extend requires free extents in the VG. If vg_free is too small, the pool fills to 100% and every thin LV in it stops accepting writes. With the default queue_if_no_space policy, writes queue and then error rather than failing instantly. The kernel dm-thin module queues writes for up to 60 seconds by default when the pool is full (configurable via the no_space_timeout module parameter). This takes down all thin volumes in the pool simultaneously.
2. You cannot respond to the next incident. Emergency snapshots during a failure, extending an LV whose filesystem just hit 100%, growing thin pool metadata: all need VG free extents. A full VG means your standard remediation toolkit is empty.
3. Provisioning operations fail. Automated snapshot jobs and provisioning scripts that assume “we can just allocate” start failing, often without anyone noticing until a backup restore point is missing.

flowchart TD
  A[vg_free declining] --> B{Thin pools in VG?}
  B -->|yes| C[auto-extend needs VG extents]
  C -->|vg_free too small| D[pool data_percent hits 100%]
  D --> E[writes queue then fail]
  E --> F[all thin LVs in pool affected]
  B -->|no| G[next lvextend or snapshot fails]
  G --> H[no emergency headroom]
  A --> I[runway = vg_free / consumption_rate]
  I --> J[act before runway < procurement time]

Common causes

Quick checks

All read-only and safe.

# VG free space, absolute and in extents
vgs -o vg_name,vg_size,vg_free,vg_free_count,vg_extent_count

# Percentage view
vgs --noheadings -o vg_name,vg_free_percent

# Per-PV free distribution: is the free space usable where you need it?
pvs -o pv_name,vg_name,pv_size,pv_free

# Thin pool fill state: data and metadata are independent failure domains
lvs -o lv_name,vg_name,lv_size,data_percent,metadata_percent

# Is auto-extend actually armed? seg_monitor must show the pool is monitored
lvs -o lv_name,seg_monitor

# Is dmeventd running at all?
systemctl is-active lvm2-monitor.service
pgrep -x dmeventd

# What are the auto-extend settings?
grep -E 'thin_pool_autoextend' /etc/lvm/lvm.conf

# Snapshots holding space
lvs -o lv_name,vg_name,origin,snap_percent,lv_size

# If lvs hangs (stressed system, I/O problems), bypass LVM locks entirely
dmsetup status --target thin-pool

Two things to note. First, thin_pool_autoextend_threshold defaults to 100, which means auto-extend is disabled. Many operators believe their thin pools auto-grow and have never verified it. Second, even with auto-extend configured and dmeventd running, extension fails if the VG has no free extents. The safety net has three dependencies, and all three must hold.

How to diagnose it

1. Establish the current position. Run the vgs and pvs commands above. Record vg_free as both an absolute value and a percentage. If free space is concentrated on one PV, note that: striped LVs may fail to allocate even when the VG total looks adequate.

2. Identify what is consuming. Compare thin pool data_percent and metadata_percent against their last known values. List snapshots. Check whether auto-extend events explain step drops in vg_free: each event consumes roughly current_pool_size * thin_pool_autoextend_percent of VG space.

3. Measure the consumption rate. LVM keeps no history; vgs shows current state only. You need external time-series data. If you have it:

   consumption_rate = (vg_used_now - vg_used_30_days_ago) / 30   # GB/day
   days_until_full = vg_free / consumption_rate
   

   If you have no history, take a reading now and another in 24 hours as a crude first estimate. Check for known growth events (imports, batch jobs, retention changes) that make linear extrapolation wrong.

4. Check the thin pool coupling. This is the critical correlation. If vg_free is low AND thin pool data_percent is high and climbing, auto-extend cannot save the pool. Estimate both runways: days until the VG is full, and hours until the pool hits 100% at its current growth rate ((100 - data_percent) / growth_rate_per_hour). Whichever is shorter is your real deadline.

5. Verify the safety net state. Confirm dmeventd is running, seg_monitor shows the pool monitored, and thin_pool_autoextend_threshold is below 100. Then ask the harder question: is vg_free large enough for at least two auto-extend cycles plus one emergency snapshot? If not, the safety net is armed but unloaded.

6. Compare runway to procurement time. The operational question is not “when does the VG fill” but “can I get new storage presented and vgextend-ed before that.” If runway is shorter than your hardware or cloud-volume lead time, escalate now, not at 5% free.

Metrics and signals to monitor

Fixes

Add capacity to the VG

The direct fix. Present a new device (disk, SAN LUN, cloud volume), then:

# Destructive to the target device: pvcreate writes LVM headers to it.
# Verify you have the right device before running.
pvcreate /dev/<new_device>
vgextend <vg> /dev/<new_device>

This is the only fix that actually solves a growth problem. Everything else buys time.

Reclaim space inside the VG

• Remove stale snapshots. lvremove <vg>/<snapshot> frees a traditional snapshot’s COW space; removing thin snapshots frees pool space. Destructive: confirm the snapshot is not a live restore point.
• Run fstrim on thin LV filesystems. Without discard, deleted data never returns to the pool. Reclaiming pool space does not grow vg_free, but it stops the pool from demanding more of it.
• Delete unneeded thin LVs. Same effect: reduces pool pressure and future auto-extend demand.

Extend the thin pool while you still can

If the pool is approaching full and the VG still has headroom, extend proactively rather than waiting for auto-extend:

lvextend -L +<size>G <vg>/<thinpool>

This spends vg_free deliberately, on your schedule, instead of letting dmeventd spend it at 3 a.m. Size the extension so the remaining vg_free still covers at least two auto-extend cycles plus one emergency snapshot.

If the VG is effectively full already

You are in the failure state, not the warning state. Follow LVM Insufficient free extents for the allocation-failure path. Do not run vgreduce --removemissing as a space fix; it is for missing PVs and permanently rewrites metadata.

Prevention

• Set explicit headroom floors. Production VGs with thin pools or frequent snapshots: keep 20% or more free. Development: 10%. Absolute minimum: enough for two thin pool auto-extend cycles plus one emergency snapshot.
• Alert on the trend, not just the level. A VG at 40% free that lost 20 points this month is more urgent than one stable at 25% for a year. Project exhaustion within 30 days and treat it as a capacity-planning action.
• Enable and verify auto-extend. Set thin_pool_autoextend_threshold below 100 (for example 80; the minimum accepted value is 50) and thin_pool_autoextend_percent to a value large enough to matter. Verify dmeventd is running and the pool shows as monitored. Audit this configuration periodically; it drifts.
• Manage snapshot lifecycle. Alert on snapshots older than 24 hours on high-churn LVs. Automate deletion where possible.
• Keep discard working. Ensure fstrim runs regularly on thin LV filesystems, or configure online discard, so deletions actually return space to the pool.
• Plan against procurement lead time. Capacity alerts should fire with enough runway to order, receive, and commission storage, not with enough runway to write a postmortem.

How Netdata helps

Netdata’s value here is correlation across layers that LVM itself keeps separate:

• VG free space as a time series. LVM shows current state only; Netdata retains vg_free history so consumption rate and runway are computed from data, not guesswork.
• Thin pool data and metadata percent alongside VG free. The dangerous combination (pool filling, VG too empty to extend it) is visible on one screen instead of two separate checks.
• Per-PV free space distribution. Surfaces the “30% free but all on one PV” case that breaks striped allocation.
• DM device I/O rates. Identifies which LV is driving consumption, since LVM gives no per-LV growth attribution on its own.
• Kernel-level corroboration. D-state process counts and block device errors sit next to the LVM signals, so you can see when a capacity problem is becoming an availability problem.
• Threshold and trend alerts. Alert at the 20% headroom floor and on projected exhaustion, so the ticket arrives while procurement is still an option.

Related guides

• How LVM actually works in production: a mental model for operators
• LVM Insufficient free extents: the volume group is out of space
• LVM monitoring checklist: the signals every production volume manager needs
• LVM monitoring maturity model: from survival to expert