LVM thin pool out of data space: every thin volume freezes at once

The symptom usually arrives as “the server hung” rather than “the storage filled up.” Processes writing to any thin volume in the pool go into uninterruptible sleep (D state). Applications stop mid-write without returning errors at first. Databases stall, VMs pause, containers wedge. If root or swap sits on a thin LV in the pool, the whole system becomes unresponsive and SSH sessions freeze.

The underlying event is simple: the thin pool’s data device reached 100% allocation, and the kernel’s device-mapper thin-pool target cannot satisfy a new block allocation for any thin volume in that pool. Because all thin LVs share the pool, they all stop at once. This is the cliff-edge failure behind most thin provisioning postmortems.

The trap for operators is that the standard LVM tools often cannot tell you this is happening. lvs, vgs, and pvs take metadata locks and perform I/O to read VG metadata, and on a system whose storage is frozen they can hang indefinitely. The one tool that keeps working is dmsetup, which reads pool state directly from kernel memory.

What this means

A thin pool is a pair of internal LVs: a data LV that holds the actual blocks and a metadata LV that tracks which blocks belong to which thin volume. Every thin LV allocates blocks from the same shared data device on first write. When the data device is fully allocated, the next write to any thin LV has nowhere to go.

The default kernel policy is queue_if_no_space. Writes do not fail immediately. They queue in the kernel for up to 60 seconds, controlled by the no_space_timeout parameter, waiting for space to appear. If the timeout expires without space being freed, the pool switches to returning errors. During the queue window, every process doing write I/O to a thin LV sits in D state and cannot be killed, not even with SIGKILL. This is why the incident looks like a hang before it looks like an error.

If the pool was configured with error_if_no_space (the errorwhenfull property in LVM), writes fail immediately instead. That is easier on applications that handle ENOSPC gracefully and worse for ones that do not, but at least it does not take the host down with it.

Kernel log messages follow a recognizable escalation: device-mapper: thin: ... reached low water mark warnings as the pool approaches full, then messages about the pool switching to out-of-data-space queue mode, and finally to error mode. On systems with XFS on the thin LVs, write errors can trigger an XFS shutdown, which makes the failure permanent until remount even after space is restored.

flowchart TD
  A[Write to any thin LV] --> B[Pool allocates data block]
  B --> C{Pool data full?}
  C -->|No| D[Normal I/O]
  C -->|Yes| E[queue_if_no_space: write queued in kernel]
  E --> F{Space freed within no_space_timeout, default 60s?}
  F -->|Yes| D
  F -->|No| G[Pool switches to error mode]
  G --> H[D-state processes pile up, write errors, possible XFS shutdown]

Common causes

One cause to rule out early: metadata exhaustion. If dmsetup status shows metadata near 100% while data is not, you have a different and worse problem. Metadata exhaustion can leave the pool needing offline repair with lvconvert --repair, and recovery is not guaranteed. Do not confuse the two.

Quick checks

All of these are read-only and safe. Start with dmsetup, which talks to the kernel directly and does not take LVM locks or touch disk.

# Confirm pool fullness without LVM tools. Works even when lvs hangs.
dmsetup status --target thin-pool

The output contains used_metadata/total_metadata and used_data/total_data in sectors, followed by flags including queue_if_no_space or error_if_no_space. If used_data equals total_data, the pool is out of data space and that is your incident.

# Check a specific pool by name
dmsetup status /dev/mapper/vg0-pool0-tpool

# Count processes stuck in uninterruptible sleep
ps -eo pid,stat,wchan:30,comm | awk '$2 ~ /D/'

# See the kernel's version of events
dmesg | grep -i 'thin\|device-mapper' | tail -30

# Check VG free space (may hang if storage is frozen; run after dmsetup confirms)
vgs -o vg_name,vg_size,vg_free

# Check health flags once lvm tools respond: position 9 of lv_attr
# 'D' = out of data space, 'F' = failed, 'M' = metadata read-only
lvs -o lv_name,vg_name,lv_attr,data_percent,metadata_percent

During the incident, treat dmsetup status as your only reliable window: no locks, no disk I/O. Save lvs and vgs for after the pool has headroom again.

How to diagnose it

1. Confirm the pool is the cause. Run dmsetup status --target thin-pool. If used_data/total_data is at or near 1:1 and the flags show queue_if_no_space or an out-of-data-space state, you have the diagnosis. If data is fine but metadata is at 100%, stop and switch to the metadata exhaustion recovery path instead; the fixes below assume data exhaustion.

2. Confirm the blast radius. Every thin LV in this pool is affected. List pool members with dmsetup ls and your knowledge of the topology, or with lvs if it responds. Identify which applications write to those LVs, because they are all stalled or erroring.

3. Check for space to extend into. Run vgs -o vg_name,vg_free. This decides your fix: free extents in the VG mean a one-command recovery; zero free extents mean you need to free pool space or add a PV.

4. Check why auto-extend did not save you. Verify dmeventd is running (systemctl status lvm2-monitor ), the pool is monitored (lvs -o+seg_monitor vg/pool shows a monitor string, not blank), and thin_pool_autoextend_threshold in /etc/lvm/lvm.conf is below 100. In most incidents at least one of these three is false, because the default threshold of 100 means auto-extend is disabled.

5. Find the consumer. Once the pool has headroom again, check which thin LVs and snapshots hold the space (lvs -o lv_name,origin,data_percent,lv_size) and whether growth was a burst (runaway job) or a trend (capacity problem). This drives prevention, not recovery.

Metrics and signals to monitor

Fixes

VG has free space: extend the pool

This is the fast path and the first thing to try.

# Extend the thin pool's data device. Extending the pool LV resumes writes.
lvextend -L +20G vg0/pool0

Extending the pool gives queued writes somewhere to land, and the pool transitions out of out-of-data-space mode. D-state processes unblock as their writes complete. If the pool does not return to normal promptly after extension, lvchange --refresh vg0/pool0 clears stale state flags.

Tradeoff: this consumes VG free extents, which is fine if this was a one-off burst. If growth is a trend, you have bought time, not solved the problem.

VG is full: free space inside the pool

If there is nowhere to extend into, reclaim from within the pool:

# Delete thin snapshots you no longer need (frees the shared blocks they pin).
# Destructive: the snapshot's data is gone. Confirm it is not your only backup.
lvremove vg0/old_snapshot

# Reclaim discarded blocks from mounted filesystems on thin LVs
fstrim /mnt/thinlv_mount

Deleting snapshots frees pool blocks immediately. fstrim only helps if the filesystems were not already issuing discards. One caution: freeing space alone does not always unfreeze a pool that has already entered out-of-data-space error mode; extending the pool is the more reliable unfreeze action, so prefer adding even a small extension after cleanup if the VG has any room at all.

VG is full and nothing can be deleted: add a PV

# Add a new device to the VG, then extend the pool
pvcreate /dev/sdX
vgextend vg0 /dev/sdX
lvextend -L +20G vg0/pool0

This requires a spare disk, LUN, or cloud volume. It is the slowest recovery but the only one that works when the VG is genuinely exhausted. pvcreate is destructive to whatever is on the device; verify the device identity with lsblk and serial numbers before running it.

Make the failure mode survivable next time

Decide deliberately between queue and error behavior:

# Fail writes immediately instead of queueing them for 60s
lvchange --errorwhenfull y vg0/pool0

Queue mode (the default) turns a capacity problem into a host hang, but gives you a 60-second grace window where a fast extension can recover with zero application errors. Error mode keeps the host alive and returns ENOSPC to applications, which is better for stateless workloads and worse for software that corrupts on write failure. There is no universally right answer; the wrong answer is not knowing which policy your pools run. Check with lvs -o whenfull vg/pool.

Prevention

• Enable auto-extend explicitly. Set thin_pool_autoextend_threshold to 80 and thin_pool_autoextend_percent to 20 in /etc/lvm/lvm.conf, ensure dmeventd runs, and confirm the pool shows a monitor string in lvs -o+seg_monitor. The default threshold of 100 means disabled, and most pools that freeze were never configured otherwise.
• Keep VG headroom for the extension. Auto-extend consumes VG free extents and fails silently when there are none. Maintain enough free space for at least two extension cycles.
• Alert on data_percent before the cliff. Ticket above 85%, urgent above 95%. The gap between “working fine at 99%” and “total write freeze at 100%” is zero; your only warning is the trend.
• Alert on metadata_percent separately. It exhausts independently, and the recovery is offline repair, not a simple extension.
• Manage snapshot lifecycle. Thin snapshots pin pool blocks indefinitely. Delete them when the backup completes; alert on snapshots older than your backup window.
• Configure discard or scheduled fstrim. Without it, deleted data never returns to the pool and data_percent is a ratchet.
• Know your whenfull policy per pool. Record whether each pool queues or errors, before the incident, so the failure mode matches what your applications tolerate.

How Netdata helps

• Netdata tracks thin pool data_percent and metadata_percent over time, so the ticket at 85% fires on a trend days before the freeze instead of during it.
• VG free space is charted alongside pool usage, which makes the “auto-extend configured but VG full” failure visible before it fires silently.
• D-state process counts and per-device I/O wait are collected from the kernel, giving you the corroboration that turns “pool at 100%” into “confirmed active write freeze affecting these workloads.”
• Disk latency on dm devices can show pre-freeze degradation, giving a second signal alongside data_percent growth.
• Because Netdata samples every second and stores history locally, you keep visibility during the incident window when LVM command-line tools themselves are hanging.

Related guides

• How LVM actually works in production: a mental model for operators
• LVM volume group running low on free space: vg_free and runway estimation
• LVM Insufficient free extents: the volume group is out of space
• LVM cannot extend a logical volume: adding a PV when the VG is full
• LVM has free space but striped or mirrored allocation still fails
• LVM monitoring checklist: the signals every production volume manager needs
• LVM monitoring maturity model: from survival to expert