LVM thin pool overprovisioning: 60% full can still be dangerous

Your thin pool shows 60% data usage. Every dashboard is green, no alert has fired, and the pool has sat around that number for weeks. Then a single process on one thin volume starts writing aggressively, and eleven minutes later every thin LV in the pool is frozen: databases crash, VMs pause, and writes across the entire pool queue for 60 seconds and then fail with I/O errors.

This is one of the classic LVM states that looks normal but is silently dangerous. The number you are watching, data_percent, measures physical pool occupancy. It says nothing about how much data the thin volumes are allowed to write. That second number is the overprovisioning ratio, and it is the risk multiplier that data_percent hides.

This article covers how to compute the ratio, how to convert a comforting percentage into actual GiB of headroom, and how to decide whether your pool is safe or one writer away from an outage.

What this means

A thin pool is a fixed amount of physical storage (the data LV) shared by thin LVs whose virtual sizes can add up to far more than the pool itself. LVM only allocates physical blocks when data is actually written. data_percent is used data blocks divided by total data blocks in the pool. It is not “used divided by the sum of thin LV virtual sizes.”

The overprovisioning ratio is:

ratio = sum of thin LV virtual sizes / pool data size

Worked example. A 100 GiB pool holds three thin LVs of 100 GiB each. Virtual sum is 300 GiB, ratio 3:1. At data_percent of 60%, the pool has 40 GiB physically free. One of those LVs currently uses 20 GiB of its 100 GiB virtual size, so it can legally write another 80 GiB of unique blocks. That single LV can consume twice the remaining physical space on its own. When the pool hits 100%, every thin LV in the pool fails at once, not just the writer. With default settings, writes queue for up to 60 seconds (the kernel no_space_timeout parameter) and then return errors. The LV health attribute (position 9 of lv_attr) flips to D for out of data space.

flowchart TD
  A[Thin LVs: virtual sum 300 GiB] --> B[Pool: 100 GiB physical]
  B --> C[data_percent 60%: 40 GiB free]
  D[Runaway writer on one thin LV] --> E[New unique blocks exceed 40 GiB]
  E --> F[Pool hits 100%]
  F --> G[Writes queue 60s then error]
  G --> H[Every thin LV in the pool freezes]

The percentage is not lying. It is just answering a different question than the one you care about. “How full is the pool” and “how much can the tenants still demand” are independent, and the ratio connects them.

Common causes

Quick checks

All of these are read-only. Prefer dmsetup status over lvs during an active incident: lvs takes metadata locks and reads from disk, and it can hang on exactly the systems you most need to observe. dmsetup reads kernel memory.

# Pool data and metadata occupancy
lvs -o lv_name,vg_name,data_percent,metadata_percent,lv_size

# Sum thin LV virtual sizes per pool (divide the result by pool size for the ratio)
lvs --separator=, --units=m --noheadings -o lv_size,pool_lv | \
  awk -F, '{ s[$2] += $1 } END { for (k in s) print k, s[k] }'

# Pool block counts straight from device-mapper (safe when lvs hangs).
# The thin-pool target lives on the -tpool device, not the LV name itself.
dmsetup status <vg>-<pool>-tpool

# Can the pool even grow? VG free space
vgs -o vg_name,vg_size,vg_free

# Is auto-extend configured? Threshold 100 means disabled
grep -E 'thin_pool_autoextend' /etc/lvm/lvm.conf

# Is dmeventd actually watching the pool?
systemctl is-active lvm2-monitor.service
lvs -o+seg_monitor <vg>/<pool>

# Any pool already in a failed state? Check position 9 of lv_attr
lvs -o lv_name,vg_name,lv_attr

Thin pool usage is updated by the kernel periodically and can lag tens of seconds behind reality. During a fast fill, treat the numbers as a lower bound.

How to diagnose it

1. Compute the ratio. Sum the virtual sizes of all thin LVs in the pool (the awk one-liner above) and divide by the pool’s data size. Anything at or above roughly 2:1 deserves a headroom policy; 3:1 and above is where a single tenant can sink the pool.
2. Convert the percentage to GiB. Free physical space is pool_size x (1 - data_percent/100). “60% used” on a 100 GiB pool means 40 GiB. On a 2 TiB pool it means 800 GiB. Percentages hide scale.
3. Compare free GiB against tenant demand. For each thin LV, its unwritten virtual space is roughly lv_size x (1 - its_data_percent/100). If the largest single LV’s unwritten space exceeds the pool’s free GiB, one workload can fill the pool. If the sum across all LVs exceeds it (it always does when overprovisioned), a coordinated burst certainly can.
4. Measure the growth rate. Take two dmsetup status samples a few minutes apart and diff the used data blocks, or use your monitoring trend. Runway is free_blocks / growth_rate. A pool growing toward exhaustion within 48 hours is a ticket even if the percentage looks modest.
5. Verify the safety net end to end. dmeventd running, the pool listed as monitored in seg_monitor, thin_pool_autoextend_threshold below 100, and enough VG free space for at least one extension. Auto-extend fails silently when the VG is full, and the default threshold of 100 means disabled.
6. Classify the risk. Safe: low ratio, bounded writers, working auto-extend with VG headroom. Dangerous: high ratio plus unbounded writers plus any break in the safety net chain. The dangerous case is an incident waiting for a trigger, so treat it as one.

Metrics and signals to monitor

Fixes

Extend the pool

# Add physical space to the pool (consumes VG free space)
lvextend -L +<size>G <vg>/<pool>

This is the immediate relief and the standard first response. Tradeoff: it buys time but does not change the ratio. If the pool was 3:1 overprovisioned before, it still is, just with more runway.

If the VG is full, extend the VG first

# Add a new physical volume, then grow the VG
pvcreate /dev/<new_device>
vgextend <vg> /dev/<new_device>

Auto-extend and manual extension both draw from VG free extents, so a full VG makes every other fix impossible. See LVM cannot extend a logical volume: adding a PV when the VG is full for the details.

Reclaim space already in the pool

Delete thin snapshots you no longer need; they share the pool’s data blocks. Then run fstrim on the filesystems living on the thin LVs. Without a working discard path, deleted files never return blocks to the pool, which is one reason data_percent only ever climbs. If data_percent does not drop after a large deletion plus fstrim, your discard chain is broken somewhere between the filesystem and the pool.

Stop the runaway writer

LVM gives you no per-process I/O attribution; dm statistics are aggregated per device. Identify the writer from application logs, backup schedules, or guest behavior, then stop or throttle it. This is the only fix that addresses the trigger rather than the capacity.

Reduce the ratio

Longer term: provision new LVs with realistic virtual sizes, move some tenants to a second pool, or grow the pool until the ratio is sane. Shrinking an existing thin LV is rarely practical: XFS cannot shrink at all, and ext4 only shrinks offline. In practice, reducing virtual sizes means migrating data to new, smaller LVs, so plan for that cost.

Tune the safety net

Set thin_pool_autoextend_threshold (80 is a common choice) and thin_pool_autoextend_percent in /etc/lvm/lvm.conf, confirm dmeventd is running, and confirm the pool shows as monitored. Auto-extend buys reaction time; it does not lower the ratio, and it cannot fire if the VG has no free extents.

Prevention

• Headroom policy by workload. Steady-state pools: keep data_percent under 75%. Fast-growing or aggressively overprovisioned pools: under 60%. Never let a production pool exceed 90% outside an active emergency.
• Ratio as a first-class metric. Record the overprovisioning ratio at provisioning time and review it whenever a new thin LV is created. The ratio drifting from 2:1 to 4:1 over a year of “just one more volume” is how this state develops.
• Runway-based alerts. Alert on projected time to 100% from the measured growth rate, not only on the percentage. A pool at 60% with 6 hours of runway is worse than one at 85% that never moves.
• VG free space reserve. Keep enough VG free for at least two auto-extend cycles plus one manual extension, so the safety net has room to operate.
• Safety net audits. Quarterly: threshold below 100, dmeventd active, every pool monitored, and evidence that at least one auto-extend event has succeeded. The first test of auto-extend should not be the incident.
• Scheduled fstrim. Reclaim discarded blocks on a schedule so the pool reflects real usage.
• Fullness policy review. Know whether each pool queues writes (default, 60 second grace then errors) or fails immediately via --errorwhenfull y. Failing fast is better for applications that handle ENOSPC cleanly; queuing is better when a short stall is preferable to an error. Decide deliberately per pool.

How Netdata helps

• LVM itself retains no history. Netdata keeps per-second time series, so you can trend data_percent and compute growth rate and runway instead of reacting to a point reading.
• Per-device dm I/O metrics from /proc/diskstats let you see the latency degradation that reclaim activity causes as the pool approaches full, before any write fails.
• Correlating pool fill with D-state process counts and kernel I/O errors pinpoints the moment writes start failing across all thin LVs at once, which is the actual outage.
• Tracking metadata_percent alongside data_percent catches the companion exhaustion that a data-only dashboard misses entirely.
• Trending VG free space next to pool usage tells you whether auto-extend still has room to save you, which is the difference between a close call and a freeze.

Related guides

• LVM thin pool out of data space: every thin volume freezes at once
• LVM reached low water mark for data device: the thin pool warning before the freeze
• LVM thin pool auto-extend not working: threshold 100 means disabled
• LVM thin pool metadata full: the exhaustion that can corrupt the pool
• LVM cannot extend a logical volume: adding a PV when the VG is full
• LVM volume group running low on free space: vg_free and runway estimation
• LVM monitoring checklist: the signals every production volume manager needs
• How LVM actually works in production: a mental model for operators