LVM thin pool metadata full: the exhaustion that can corrupt the pool

Your thin pool’s metadata_percent has hit 100. Writes to every thin LV in the pool are failing or hanging, and yet lvs shows data_percent at 40%. That is not a contradiction. It is the defining feature of thin pool metadata exhaustion, and it is the LVM failure mode most likely to cost you data.

Metadata exhaustion is more dangerous than data exhaustion for one reason: a full data area stops new allocations, but a full metadata area can leave the pool structurally corrupted. The metadata LV holds the block mapping tables for every thin LV and snapshot in the pool. When the kernel can no longer allocate a metadata block mid-transaction, it aborts the transaction and switches the pool to read-only mode. From that point, recovery via lvconvert --repair is possible but not guaranteed, and the lvmthin(7) man page warns plainly that data from thin LVs may ultimately be unrecoverable.

This guide covers confirming the condition safely, why it happened, repair with realistic expectations, and prevention.

What this means

A thin pool is two internal LVs managed as one: a data LV that holds actual blocks, and a much smaller metadata LV that tracks which pool blocks belong to which thin LV. Every unique block allocation creates or updates metadata entries. The two spaces exhaust independently.

Metadata does not grow with bytes written. It grows with:

• Unique blocks written. Random I/O across a large address space consumes metadata far faster than sequential writes, because each new mapping needs an entry.
• Thin LV count. Each thin LV adds its own mapping tree.
• Snapshots. Every thin snapshot multiplies metadata entries by sharing and diverging mappings.

The result: a pool doing heavy random-write workloads, or carrying many snapshots, can reach metadata_percent 100 while data_percent sits below 50. Teams watching only data usage, or only df output, miss it until the pool is read-only.

When the metadata area fills, the kernel logs messages like these and switches the pool to read-only mode:

device-mapper: space map metadata: unable to allocate new metadata block
device-mapper: thin: ... aborting current metadata transaction
device-mapper: thin: ... switching pool to read-only mode

flowchart TD
  A[Random writes, many thin LVs, snapshots] --> B[metadata_percent climbs]
  B --> C[Metadata 100%: new metadata block allocation fails]
  C --> D[Kernel aborts metadata transaction]
  D --> E[Pool switches to read-only]
  E --> F{Was the transaction aborted cleanly?}
  F -->|Yes| G[Pool degraded but repairable: extend metadata]
  F -->|No| H[Metadata structurally inconsistent]
  H --> I[lvconvert --repair: not guaranteed]
  I -->|Fails or worsens| J[Restore thin LVs from backup]

One more trap: thin_check runs on pool activation. If the metadata fails the check, the pool will not activate. A reboot during this incident can turn a degraded-but-running pool into a full outage.

Common causes

Quick checks

These are read-only. Run them before touching anything.

# Pool state: data vs metadata usage, health flag, monitoring state
lvs -a -o lv_name,vg_name,lv_attr,lv_size,data_percent,metadata_percent,seg_monitor

# Health flag is position 9 of lv_attr: M = metadata read-only, F = failed, D = out of data space
lvs -o lv_name,vg_name,lv_health_status

# Kernel memory view. Use this if lvs hangs, which it can when the pool is wedged.
# dmsetup takes no LVM locks and does no disk I/O.
dmsetup status --target thin-pool
# Output includes used_metadata/total_metadata and used_data/total_data block counts

# Confirm the kernel-side story
dmesg | grep -iE 'thin|metadata|device-mapper' | tail -50

# Can the pool even be extended? Check VG headroom and the autoextend safety net
vgs -o vg_name,vg_free
grep -E 'thin_pool_autoextend_threshold|thin_pool_autoextend_percent' /etc/lvm/lvm.conf
pgrep -x dmeventd

# Processes blocked on I/O (corroboration that the pool is actually stalling work)
ps -eo pid,stat,wchan:30,comm | awk '$2 ~ /D/'

How to diagnose it

1. Confirm which space is full. Look at lvs output (or dmsetup status if lvs hangs). If metadata_percent is at or near 100 while data_percent is well below it, you are in the metadata exhaustion pattern, not the data exhaustion pattern. The response is different, so get this right first.

2. Read the health flag. Position 9 of lv_attr, or the lv_health_status field: M means the pool metadata has gone read-only, F means the pool has failed. M with recent kernel messages about an aborted transaction means the abort already happened on this running pool. That distinction matters for step 4.

3. Check the kernel log for the abort sequence. The “unable to allocate new metadata block” / “aborting current metadata transaction” / “switching pool to read-only mode” sequence tells you the kernel already hit the wall. If those messages are absent and metadata is merely at 98%, you may still have a clean pool and a much easier fix: extend metadata before the abort happens.

4. Assess corruption before attempting repair. This is the judgment call. If the pool went read-only due to the abort, the metadata may already be inconsistent. Red Hat guidance warns that once metadata fullness has caused corruption, running thin_repair (which is what lvconvert --repair invokes) into a similarly sized metadata LV may be futile or make things worse. If you have current backups of the thin LVs, treat them as your primary recovery path and repair as the optimistic one.

5. Check integrity explicitly if you can. With the pool inactive, thin_check /dev/mapper/<vg>-<pool>_tmeta validates the metadata structures. A clean check is good evidence repair will go well. If the pool will not activate at all, it is likely because thin_check already failed during activation.

6. Inventory what is at stake. List the thin LVs and snapshots in the pool with lvs -o lv_name,vg_name,origin,lv_size. Every one of them shares this metadata. A failed repair is not a one-LV problem.

Metrics and signals to monitor

Fixes

If metadata is high but the pool has not aborted yet

This is the good outcome. Extend the metadata LV now:

# Extend pool metadata. Size generously; the max supported is ~16 GiB.
lvextend --poolmetadatasize +4G <vg>/<pool>

The metadata LV can range from 2 MiB to roughly 16 GiB, and 1 GiB is the size upstream recommends starting with, which is frequently too small for snapshot-heavy or random-write pools. Metadata space is cheap: the tradeoff is only VG free extents, and that tradeoff is never close.

Current LVM2 extends pool metadata online while the pool is active (the dmeventd auto-extend path relies on it). On an older release that refuses the extension on an active pool, fall back to lvchange -an, lvextend --poolmetadatasize, then lvchange -ay.

Also enable autoextend (see Prevention) so this does not depend on a human being awake.

If the pool aborted and went read-only

Stop writes at the application layer if you can, and verify your backups of the thin LVs before running anything. The repair path is disruptive (full pool downtime) and not guaranteed to succeed.

# 1. Unmount filesystems and deactivate the pool (all thin LVs on it must be inactive first)
lvchange -an <vg>/<pool>

# 2. Attempt repair. Uses thin_repair and the pmspare LV; NOT guaranteed to succeed.
lvconvert --repair <vg>/<pool>

# 3. Only if repair succeeds: enlarge metadata so it does not refill immediately
lvextend --poolmetadatasize +4G <vg>/<pool>

# 4. Reactivate, then check every filesystem before returning to service
lvchange -ay <vg>/<pool>
# fsck each thin LV's filesystem before mounting read-write

Be clear about what you are accepting. lvconvert --repair reconstructs metadata into the spare metadata LV (pmspare). If the metadata was already structurally inconsistent from the abort, repair can fail, and per Red Hat’s guidance it can also make the situation worse, because you are repairing into a metadata LV the same size as the one that just proved too small. Do not attempt to resize metadata on a pool that is already in a corrupted state; the operation can hang. If repair fails, the realistic path is restoring the thin LVs from backup and recreating the pool with a much larger metadata LV.

Freeing metadata pressure without repair

Metadata is not reclaimable via discard or fstrim; those only return data blocks. The levers that reduce metadata consumption are:

• Delete old thin snapshots. Snapshots are a direct metadata multiplier. Removing them reduces entry count going forward.
• Consolidate thin LVs. Fewer thin LVs means fewer mapping trees.
• Extend the metadata LV. This is the primary fix, not a workaround.

If lvs hangs during the incident

LVM commands read metadata from disk and take VG locks, so they can block behind the same I/O they are trying to observe. Fall back to dmsetup status --target thin-pool, which reads from kernel memory with no locks and no disk I/O. This is your primary diagnostic tool during the incident, and the reason to collect dmsetup baselines before one.

Prevention

• Size metadata aggressively at pool creation. Use --poolmetadatasize and go well above the default. 1 GiB is a floor for trivial pools, not a target. The cost of 4-8 GiB of metadata is negligible next to one failed repair.
• Enable autoextend for real. Set thin_pool_autoextend_threshold to 70-80 and thin_pool_autoextend_percent to at least 20 in /etc/lvm/lvm.conf, and confirm dmeventd is running and the pool has an active monitor. The dmeventd policy extension covers both data and metadata. Then verify VG free space can actually fund the extension; a configured autoextend against a full VG fails without extending anything.
• Alert on metadata_percent independently. Ticket at 75%, urgent at 90%. Never gate metadata alerting on data_percent; they exhaust on different curves.
• Budget snapshots. Thin snapshots are cheap in data terms and expensive in metadata terms. Track count and age, and automate cleanup.
• Match the workload to the pool. Random-write-heavy workloads (databases, VM images, container storage) belong on pools with oversized metadata. Sequential archival workloads tolerate smaller metadata.
• Keep backups current enough to survive a failed repair. Repair is a coin you do not get to flip twice. For any pool you cannot rebuild from backup, metadata headroom below 50% used is the only acceptable steady state.

How Netdata helps

• Tracks data_percent and metadata_percent as separate series per pool, so the low-data/full-metadata divergence is visible on one chart instead of discovered in a postmortem.
• Trends metadata consumption over time, which matters because metadata growth is non-linear and tied to write patterns, not volume. A slow climb from 30% to 60% over a month is the early warning this article is about.
• Correlates pool usage with LV health flags and dmeventd state, so an alert can distinguish “metadata at 85% with a working autoextend” from “metadata at 85% with no safety net,” which are very different pages.
• Surfaces kernel-level corroboration (block I/O errors, D-state processes) next to LVM metrics, so you can see the moment degradation becomes an actual outage.
• Retains history LVM itself does not keep. LVM reports current state only; trending, rate-of-change, and runway estimation require an external time series.

Related guides

• How LVM actually works in production: a mental model for operators
• LVM Insufficient free extents: the volume group is out of space
• LVM volume group running low on free space: vg_free and runway estimation
• LVM cannot extend a logical volume: adding a PV when the VG is full
• LVM filesystem full while the volume group has space: the resize step everyone forgets
• LVM has free space but striped or mirrored allocation still fails
• LVM monitoring checklist: the signals every production volume manager needs
• LVM monitoring maturity model: from survival to expert