LVM thin pool queue_if_no_space: why a full pool hangs instead of erroring

The server looks dead. Processes pile up in uninterruptible sleep, shells that touch certain filesystems never return, and even lvs hangs. There are no I/O errors in the application logs, no kernel oops, nothing in dmesg that screams hardware. Teams burn the first hour of this incident investigating a kernel or storage fault, when the actual cause is an LVM thin pool that ran out of data space.

The reason there are no errors is the pool’s when-full policy. The default for LVM thin pools is queue_if_no_space: when the pool fills, the kernel queues write I/O instead of failing it. The alternative, error_if_no_space, returns write errors immediately, which applications can see, log, and handle. Most operators have never checked which policy their pools use, because the default is silent until the pool fills.

This article covers how to confirm a hang is a full thin pool, how to recover, and how to audit and change the policy before it happens again.

What this means

A thin pool is a pair of internal LVs (data and metadata) that backs overcommitted thin volumes. When the data LV fills to 100%, the device-mapper thin-pool target cannot allocate blocks for new writes. What happens next is a policy decision baked into the dm table:

• queue_if_no_space (the default): writes are queued in the kernel. Every process doing write I/O to any thin LV in the pool goes into D-state. Because all thin LVs share the pool, they all freeze at once. If the root filesystem, a journal, or swap sits on the pool, the whole system progressively locks up.
• error_if_no_space: writes fail immediately with ENOSPC. Databases, VMs, and applications see real errors and can retry, abort, or alert.

There is a kernel-level safety valve: the no_space_timeout module parameter for dm_thin_pool. Queued writes that wait longer than the timeout are failed with errors. Even when a timeout is configured and fires, it does not save you: tens of seconds of frozen I/O already trips application timeouts, new writes keep re-queueing behind the failed ones, and the lvmthin documentation warns that tuning the timeout can lead to memory exhaustion, hung tasks, and deadlocks. Operationally, queue_if_no_space means “the system hangs.”

flowchart TD
  A[Thin pool data reaches 100%] --> B{When-full policy?}
  B -->|queue_if_no_space - default| C[Writes queue in kernel]
  C --> D[Processes enter D-state]
  D --> E[lvm tools hang on pool I/O]
  E --> F[Looks like kernel or hardware hang]
  B -->|error_if_no_space| G[Writes fail with ENOSPC]
  G --> H[Applications see errors and react]

The cruelest part is the observability gap. lvs, vgs, and pvs read metadata from the PVs and take LVM locks. When the pool is wedged, those commands hang too, so the standard diagnostic toolkit goes blind at exactly the wrong moment. dmsetup talks directly to the kernel and still works. That asymmetry is the key to fast diagnosis.

Common causes

Quick checks

All of these are read-only. Lead with dmsetup; it does not take LVM locks or read from disk.

# 1. Thin pool fullness and policy, straight from the kernel
dmsetup status --target thin-pool
# Fields: ... <used_meta>/<total_meta> <used_data>/<total_data> ... queue_if_no_space|error_if_no_space
# used_data/total_data at or near 1:1 confirms a full pool.

# 2. D-state processes, the visible symptom of queued I/O
ps -eo pid,stat,wchan:30,comm | awk '$2 ~ /D/'

# 3. Confirm a stuck process is waiting on dm/thin
cat /proc/<pid>/stack
# Look for device-mapper or thin-pool functions in the stack.

# 4. Any suspended dm devices
dmsetup info -c --noheadings -o name,suspended

# 5. Pool usage via LVM (may hang during an active incident)
lvs -o lv_name,vg_name,lv_attr,data_percent,metadata_percent
# 'D' in position 9 of lv_attr means the pool is out of data space.

# 6. Which when-full policy each pool uses
lvs -o lv_name,vg_name,lv_when_full

# 7. Recovery headroom: can the pool be extended at all
vgs -o vg_name,vg_size,vg_free

# 8. Rule out hardware as the cause of the hang
dmesg | grep -i 'I/O error' | tail -20

How to diagnose it

1. Establish that the system is I/O-hung, not dead. If you still have a working shell, run ps and count D-state processes. A growing count of processes stuck for more than 60 seconds each is an active I/O blockage, not a CPU or memory event.

2. Run dmsetup status --target thin-pool. If this returns immediately while lvs hangs, you are already looking at the answer: the LVM management plane is blocked on the same storage that is hung. Compare used_data to total_data. At or near 1:1, the pool is full.

3. Read the policy flag in the same output. queue_if_no_space at the end of the status line explains the absence of errors. This is the moment the “server hung” ticket becomes an LVM capacity incident.

4. Confirm the wait channel. For one or two D-state processes, read /proc/<pid>/stack and check for device-mapper or thin-pool frames. This distinguishes a thin pool freeze from an NFS hang or a dying disk, which can look identical from the ps output alone.

5. Check dmesg for competing explanations. Block I/O errors, device resets, or path failures point at hardware or multipath instead. A full thin pool is quiet in the kernel log; a failing disk usually is not.

6. Check recovery headroom before touching anything. vgs -o vg_free tells you whether lvextend on the pool can succeed. This determines which recovery path you take.

Metrics and signals to monitor

Thin pool usage is updated by the kernel periodically and can lag reality by tens of seconds. Treat these as trend signals, not tripwires.

Fixes

Recover the hung pool

The only clean recovery is giving the pool more space. Everything else is a workaround.

# Extend the pool if the VG has free space
lvextend -L +50G vg0/thinpool

As soon as the pool has free data blocks, queued writes complete and D-state processes unblock on their own. D-state processes cannot be killed, not even with SIGKILL; freeing pool space is what releases them.

Caveats that matter during the incident:

• LVM commands may hang because they read metadata from PVs involved in the freeze. Run recovery commands from a shell that has not touched the hung filesystems. If lvextend blocks, your remaining options narrow quickly.
• If the VG is full, you must add capacity first: pvcreate a new device and vgextend the VG, then extend the pool. If no spare device exists, deleting an unneeded thin LV or old thin snapshots frees pool blocks directly.
• Last resort: forced reboot (for example via echo b > /proc/sysrq-trigger or out-of-band power control). This is disruptive and risks filesystem corruption on every thin LV that had queued writes, and the pool will still be full after boot. Extend the pool before workloads restart and refill it.
• After recovery, reclaim space: run fstrim on the thin LV filesystems so deleted data returns blocks to the pool, and review snapshots sharing the pool.

Switch the pool to error_if_no_space

For workloads that handle write errors sanely (most databases, queue systems, and anything with its own retry logic), failing fast is far better than hanging:

# Change the policy on an existing pool
lvchange --errorwhenfull y vg0/thinpool

# Verify
lvs -o lv_name,lv_when_full vg0/thinpool
dmsetup status --target thin-pool

To make new pools default to erroring, set activation/error_when_full = 1 in /etc/lvm/lvm.conf. The lvchange --errorwhenfull option exists in LVM since lvm2 2.02.114 (RHEL 7.1 era); on older releases the policy can only be set through dmsetup table manipulation.

Understand the tradeoff before flipping the policy globally. With error_if_no_space, applications that do not handle ENOSPC well will crash or corrupt instead of hanging. A hang preserves the option of extending the pool and continuing as if nothing happened; an error forces the application to cope. Choose per pool, per workload.

There is also a runtime switch at the device-mapper level via dmsetup message to toggle error_if_no_space without reloading the table.

Consider the timeout, cautiously

no_space_timeout bounds how long queued writes wait before failing. Raising it gives you more time to extend the pool before errors hit applications; setting it to 0 disables the timeout and the queue becomes truly indefinite. The lvmthin documentation warns that disabling timeouts can cause memory exhaustion, hung tasks, and deadlocks. Treat this as a deliberate tuning decision, not a safety feature, and leave the default unless you have a specific reason.

Prevention

• Audit the policy now, not during the incident. Run lvs -o lv_name,vg_name,lv_when_full on every host with thin pools and record the answer. This is a Level 4 maturity item in the LVM monitoring maturity model for a reason: most teams discover their default the hard way.
• Alert on pool fill, not on the hang. data_percent above 85% is a same-shift ticket; above 95% is urgent. By the time D-state processes accumulate, you are in the incident.
• Do not trust auto-extend blindly. The default thin_pool_autoextend_threshold is 100, which means disabled. See LVM thin pool auto-extend not working: threshold 100 means disabled. Even when configured, auto-extend fails silently if dmeventd is down or the VG is full.
• Keep VG headroom. The pool can only be extended into free VG space. Track vg_free and its runway; see LVM volume group running low on free space.
• Monitor D-state processes as a corroborating signal. A rising count of processes stuck on dm devices turns “pool at 97%” into “pool freezing workloads now,” which is the difference between a ticket and a page.
• Use dmsetup status in your monitoring path. LVM tools take locks and perform I/O; they hang during the exact incidents you most need to observe. Kernel-side status does not.

How Netdata helps

• Netdata collects per-dm-device I/O from the kernel block layer, so inflight I/O and latency on thin pool devices are visible even when LVM user-space tools are hung.
• Process state tracking surfaces D-state accumulation, letting you correlate “processes stuck on I/O” with the pool filling rather than chasing a phantom kernel bug.
• Disk space and block device trends give you the data_percent trajectory and growth rate, so you can alert on runway (hours to full) instead of a static threshold.
• Because collection is per-second and local, the moments right before the freeze (latency spikes from reclaim activity, inflight I/O climbing) are captured rather than averaged away.
• Correlating dm device saturation, D-state count, and pool fill on one dashboard compresses the diagnosis from “why is the server hung” to “pool full, queue policy, extend now” in one screen.

Related guides

• LVM thin pool out of data space: every thin volume freezes at once
• LVM reached low water mark for data device: the thin pool warning before the freeze
• LVM thin pool auto-extend not working: threshold 100 means disabled
• LVM thin pool metadata full: the exhaustion that can corrupt the pool
• LVM volume group running low on free space: vg_free and runway estimation
• LVM Insufficient free extents: the volume group is out of space
• LVM monitoring checklist: the signals every production volume manager needs
• LVM monitoring maturity model: from survival to expert
• How LVM actually works in production: a mental model for operators