You deleted 200 GB of files from a filesystem on a thin LV. df shows the space as free, but lvs still reports the thin pool data_percent exactly where it was. Nothing is broken in the reporting: this is how thin provisioning works. Deleting a file only updates filesystem metadata. The pool blocks that held the data stay allocated until a discard (TRIM) operation travels from the filesystem, through the thin LV, into the pool.

Without a working discard path, data_percent is a one-way ratchet. Operators usually discover this when the pool hits 85 or 95 percent and the first reaction, “just delete some files,” does nothing.

This guide covers how to confirm the discard path is missing or broken, how to reclaim space now with fstrim, and how to choose between periodic fstrim and the discard mount option for ongoing reclamation.

What this means

A thin pool hands out blocks to thin LVs on first write. The filesystem on the thin LV tracks which of its blocks are in use; the pool tracks which pool blocks back them. These are independent bookkeeping layers. When a file is deleted, the filesystem marks its blocks free but tells the pool nothing. The pool still considers those backing blocks allocated, so data_percent is unchanged.

A discard closes that gap. When the filesystem issues a discard for freed blocks, the device-mapper thin target unmaps the corresponding pool blocks and returns them to the pool’s free list, and data_percent drops. What happens next depends on the pool discard mode:

  • passdown (the default): the discard is also forwarded to the underlying device, which matters when that device is itself thin (an SSD, a SAN LUN, or another virtual layer).
  • nopassdown: pool blocks are freed but nothing is sent downward.
  • ignore: discards are dropped entirely and the pool never reclaims anything.

Two consequences matter operationally:

  • A pool can fill up and freeze every thin LV in it even while all the filesystems report plenty of free space. Pool exhaustion and filesystem usage are separate failure domains.
  • A sudden drop in data_percent after an fstrim run is healthy behaviour, not a glitch. It is the reclaim working.
flowchart TD
  A[File deleted on thin LV filesystem] --> B{Discard issued?}
  B -->|fstrim or discard mount option| C{Pool discard mode}
  B -->|never| D[Pool blocks stay allocated]
  C -->|ignore| D
  C -->|nopassdown| E[Pool blocks freed, not passed down]
  C -->|passdown| F[Pool blocks freed and passed to underlying device]
  E --> G[data_percent drops]
  F --> G
  D --> H[data_percent only climbs]

Common causes

CauseWhat it looks likeFirst thing to check
No discard ever issueddata_percent climbs monotonically for weeks, never dipsIs fstrim.timer enabled? Is any filesystem mounted with discard?
Pool discard mode is ignorefstrim completes and reports trimmed bytes, but data_percent does not movelvs -o+discards vg/pool
Thin snapshots pin the blocksfstrim runs but data_percent barely dropslvs for thin snapshots sharing the pool
discard_passdown silently disabledPool reclaims, but the underlying device (SSD, SAN, VDO) never sees discardsdmsetup table for no_discard_passdown
issue_discards = 1 misconceptionOperator set it in lvm.conf expecting runtime reclamationIt only applies to LVM operations like lvremove and lvreduce, not file deletion
Underlying device does not support discardPassdown disabled by the kernel at pool activationlsblk -D on the data device; kernel log at activation time

Quick checks

All read-only and safe to run during production hours. Note that lvs takes LVM metadata locks and does disk I/O; if the pool is already full and I/O is hung, prefer the dmsetup variants, which read from kernel memory.

# Pool usage and health
lvs -o lv_name,vg_name,lv_attr,data_percent,metadata_percent

# Current discard mode of the pool (ignore / nopassdown / passdown)
lvs -o+discards vg/pool

# Kernel view of the pool, no LVM locks: table line shows no_discard_passdown if disabled
dmsetup status --target thin-pool
dmsetup table vg-pool-tpool 2>/dev/null || dmsetup table | grep thin-pool

# Is periodic trim scheduled?
systemctl status fstrim.timer
systemctl list-timers fstrim.timer

# Is any filesystem mounted with the discard option?
findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS | grep discard

# Does the underlying device support discard at all?
lsblk -D

# Kernel log evidence of passdown being disabled
dmesg | grep -i "discard"

Compare df on each thin LV filesystem against pool data_percent. A large and growing gap between “filesystem says free” and “pool says used” is the signature of a missing discard path.

How to diagnose it

  1. Confirm the symptom is unreclaimed space, not real growth. Check df on every filesystem in the pool. If filesystems are at 40 percent and the pool is at 90 percent, reclamation is the problem. If filesystems are also at 90 percent, the data is real and you need capacity, not TRIM.
  2. Check the pool discard mode. lvs -o+discards vg/pool. If it shows ignore, discards are dropped at the pool and no amount of fstrim will reclaim anything until you change the mode.
  3. Check for snapshots. Thin snapshots share pool blocks by design. Any block still referenced by a snapshot will not be freed by a discard, even after fstrim succeeds. List thin volumes and their origins with lvs -o lv_name,origin,lv_attr.
  4. Check whether passdown reached the underlying device. Look for no_discard_passdown in the dmsetup table output for the pool. If the underlying device’s maximum discard granularity is smaller than the thin pool block size, the kernel disables passdown at activation and logs a message of the form “device-mapper: thin: Data device (dm-N) max discard sectors smaller than a block: Disabling discard passdown.” This affects only the downward leg; pool blocks are still reclaimed as long as the mode is not ignore.
  5. Check the schedule. If you rely on periodic trim, verify fstrim.timer is enabled and has actually run recently (systemctl list-timers fstrim.timer shows the last run). A timer that exists but is disabled reclaims nothing.
  6. Rule out the issue_discards red herring. issue_discards = 1 in lvm.conf makes LVM issue discards when space is freed by LVM operations such as lvremove, lvreduce, or pvmove. It does nothing for files deleted inside a mounted filesystem. Setting it and expecting data_percent to drop after deletions is a common misdiagnosis.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
data_percent per thin poolThe actual pool fill level; exhaustion freezes every thin LVAbove 85 percent; above 95 percent is urgent
data_percent trend shapeA healthy pool with working discard oscillates as data is written and trimmedMonotonic increase over weeks with filesystem usage flat
data_percent after fstrimConfirms the reclaim path works end to endfstrim reports trimmed bytes but the metric does not move
metadata_percentIndependent exhaustion domain; not reclaimable by discardAbove 75 percent
VG free spaceHeadroom to extend the pool if reclaim is not enoughBelow 10 percent with a pool above 85 percent
Pool discard mode (lvs -o+discards)Configuration drift can set it to ignoreAnything other than the intended mode

Thin pool usage is updated periodically by the kernel and can lag reality by tens of seconds, so do not alert on sub-minute changes. Trend over hours instead.

Fixes

Reclaim space right now: run fstrim manually

# Trim one mounted filesystem on a thin LV
fstrim -v /mountpoint

-v prints the number of bytes trimmed, which tells you the filesystem issued the discards. Watch data_percent afterwards; a visible drop confirms the path works. If trimmed bytes are reported but data_percent does not move, the pool discard mode is ignore or snapshots are pinning the blocks.

On a nearly full pool, fstrim is often the fastest way to get headroom back without adding storage. It is safe to run online, though it does generate I/O against the underlying device in passdown mode, so on busy systems run it during a quieter window.

Fix the pool discard mode

# Allow the pool to process discards and pass them to the underlying device
lvchange --discards passdown vg/pool

Modes:

  • passdown (default): process in the pool and forward to the underlying device. Correct for most setups, and required if the underlying device is itself thin.
  • nopassdown: reclaim pool blocks but do not forward. Use when the underlying device handles discard badly or slowly, and you only care about pool-level reclamation.
  • ignore: drop all discards. Almost never what you want on a pool that holds real filesystems.

The default for newly created pools is controlled by allocation/thin_pool_discards in lvm.conf when not specified on the command line.

Set up ongoing reclamation: periodic fstrim (preferred)

Enable the systemd timer so reclamation happens on a schedule instead of relying on operator memory:

systemctl enable --now fstrim.timer
systemctl list-timers fstrim.timer

Timer defaults vary by distribution, including whether fstrim.timer is enabled out of the box and the default RandomizedDelaySec. Check the timer unit on your system rather than assuming.

Alternative: the discard mount option

Mounting the filesystem with discard issues a discard synchronously on every file deletion. This keeps the pool tight without a scheduled job, but each delete now stalls while TRIM is processed, which shows up as latency spikes under delete-heavy workloads. Both Red Hat documentation and the XFS man page recommend periodic fstrim over the discard mount option for production XFS workloads.

Choose discard only when the workload deletes files infrequently, or when the underlying device handles queued TRIM cheaply and you have measured the impact. Otherwise, use the timer.

When reclaim is not enough

If fstrim recovers little because the data is genuinely live, the pool needs capacity, not TRIM:

  • Extend the pool: lvextend -L +<size>G vg/pool (requires VG free space).
  • If the VG is also full, add a PV first. See the related guide on extending LVs with a full VG.
  • Delete thin snapshots that no longer serve a purpose. Their blocks pin pool space that no discard can free.

If the pool is already at or near 100 percent, follow the pool-exhaustion runbook rather than this one; the freeze mechanics and first response are different.

Prevention

  • Scheduled trim on every thin LV filesystem. Enable fstrim.timer and verify it runs. Without any discard path, every pool eventually fills regardless of how much filesystem space is free.
  • Alert on the trend, not just the level. A monotonic data_percent climb while filesystem usage is flat is the earliest signal that reclamation is broken. Catching it at 60 percent beats catching it at 95.
  • Intended discard mode documented per pool. Record whether each pool should be passdown or nopassdown and audit with lvs -o+discards. Treat ignore on a production pool as a misconfiguration.
  • Snapshot lifecycle policy. Thin snapshots pin pool blocks indefinitely. Alert on old snapshots so they are deleted before they silently defeat reclamation.
  • Capacity headroom independent of reclaim. Keep data_percent below 75 percent steady-state and VG free space sufficient for at least one pool extension, so a reclaim failure is a ticket, not an incident.
  • Verify after VG metadata restores. After any vgcfgrestore, confirm discards still reclaim space with a manual fstrim. Operators have reported thin pool discard state breaking after metadata restores.

How Netdata helps

  • Tracks data_percent and metadata_percent per pool over time, so the monotonic-climb signature of a missing discard path is visible in the trend long before the pool is full.
  • Makes fstrim verification trivial: correlate the timestamp of a timer run or manual trim with the data_percent drop. No drop means the path is broken.
  • Alerts on pool fill thresholds (85 percent ticket, 95 percent urgent) while showing filesystem usage alongside, so you can tell “needs TRIM” apart from “needs capacity” from the same screen.
  • Correlates pool growth rate with write throughput on the dm devices, helping you estimate runway when reclaim is not keeping up.
  • Surfaces VG free space in the same view, so you know whether pool extension is possible before you are forced to choose between trimming, extending, and deleting snapshots.