$ guides / lvm / lvm-thin-pool-metadata-full ▌

Operations Guides

LVM thin pool metadata full: the exhaustion that can corrupt the pool

Your thin pool’s metadata_percent has hit 100. Writes to every thin LV in the pool are failing or hanging, and yet lvs shows data_percent at 40%. That is not a contradiction. It is the defining feature of thin pool metadata exhaustion, and it is the LVM failure mode most likely to cost you data.

Metadata exhaustion is more dangerous than data exhaustion for one reason: a full data area stops new allocations, but a full metadata area can leave the pool structurally corrupted. The metadata LV holds the block mapping tables for every thin LV and snapshot in the pool. When the kernel can no longer allocate a metadata block mid-transaction, it aborts the transaction and switches the pool to read-only mode. From that point, recovery via lvconvert --repair is possible but not guaranteed, and the lvmthin(7) man page warns plainly that data from thin LVs may ultimately be unrecoverable.

This guide covers confirming the condition safely, why it happened, repair with realistic expectations, and prevention.

What this means

A thin pool is two internal LVs managed as one: a data LV that holds actual blocks, and a much smaller metadata LV that tracks which pool blocks belong to which thin LV. Every unique block allocation creates or updates metadata entries. The two spaces exhaust independently.

Metadata does not grow with bytes written. It grows with:

Unique blocks written. Random I/O across a large address space consumes metadata far faster than sequential writes, because each new mapping needs an entry.
Thin LV count. Each thin LV adds its own mapping tree.
Snapshots. Every thin snapshot multiplies metadata entries by sharing and diverging mappings.

The result: a pool doing heavy random-write workloads, or carrying many snapshots, can reach metadata_percent 100 while data_percent sits below 50. Teams watching only data usage, or only df output, miss it until the pool is read-only.

When the metadata area fills, the kernel logs messages like these and switches the pool to read-only mode:

device-mapper: space map metadata: unable to allocate new metadata block
device-mapper: thin: ... aborting current metadata transaction
device-mapper: thin: ... switching pool to read-only mode

flowchart TD
  A[Random writes, many thin LVs, snapshots] --> B[metadata_percent climbs]
  B --> C[Metadata 100%: new metadata block allocation fails]
  C --> D[Kernel aborts metadata transaction]
  D --> E[Pool switches to read-only]
  E --> F{Was the transaction aborted cleanly?}
  F -->|Yes| G[Pool degraded but repairable: extend metadata]
  F -->|No| H[Metadata structurally inconsistent]
  H --> I[lvconvert --repair: not guaranteed]
  I -->|Fails or worsens| J[Restore thin LVs from backup]

One more trap: thin_check runs on pool activation. If the metadata fails the check, the pool will not activate. A reboot during this incident can turn a degraded-but-running pool into a full outage.

Common causes

Cause	What it looks like	First thing to check
Metadata LV undersized for the workload	metadata_percent climbs steadily for weeks, data_percent modest	`lvs -o lv_name,data_percent,metadata_percent,lv_size` on the pool
Heavy random-write workload	Metadata grows much faster than data; databases, VM disks, container overlays	Compare metadata growth rate vs data growth rate over days
Snapshot accumulation	Many thin snapshots on the pool; metadata jumps with each new snapshot	`lvs -o lv_name,origin` and count snapshots per origin
Autoextend never enabled	Pool at 100% with plenty of VG free space; no extension ever fired	`grep thin_pool_autoextend /etc/lvm/lvm.conf` (default threshold is 100 = disabled)
dmeventd not running	Autoextend configured but nothing happened	`systemctl is-active lvm2-monitor.service` or `pgrep -x dmeventd`
Autoextend fired but VG full	dmeventd logs extension attempts that fail	`vgs -o vg_name,vg_free`

Quick checks

These are read-only. Run them before touching anything.

# Pool state: data vs metadata usage, health flag, monitoring state
lvs -a -o lv_name,vg_name,lv_attr,lv_size,data_percent,metadata_percent,seg_monitor

# Health flag is position 9 of lv_attr: M = metadata read-only, F = failed, D = out of data space
lvs -o lv_name,vg_name,lv_health_status

# Kernel memory view. Use this if lvs hangs, which it can when the pool is wedged.
# dmsetup takes no LVM locks and does no disk I/O.
dmsetup status --target thin-pool
# Output includes used_metadata/total_metadata and used_data/total_data block counts

# Confirm the kernel-side story
dmesg | grep -iE 'thin|metadata|device-mapper' | tail -50

# Can the pool even be extended? Check VG headroom and the autoextend safety net
vgs -o vg_name,vg_free
grep -E 'thin_pool_autoextend_threshold|thin_pool_autoextend_percent' /etc/lvm/lvm.conf
pgrep -x dmeventd

# Processes blocked on I/O (corroboration that the pool is actually stalling work)
ps -eo pid,stat,wchan:30,comm | awk '$2 ~ /D/'

How to diagnose it

Confirm which space is full. Look at lvs output (or dmsetup status if lvs hangs). If metadata_percent is at or near 100 while data_percent is well below it, you are in the metadata exhaustion pattern, not the data exhaustion pattern. The response is different, so get this right first.
Read the health flag. Position 9 of lv_attr, or the lv_health_status field: M means the pool metadata has gone read-only, F means the pool has failed. M with recent kernel messages about an aborted transaction means the abort already happened on this running pool. That distinction matters for step 4.
Check the kernel log for the abort sequence. The “unable to allocate new metadata block” / “aborting current metadata transaction” / “switching pool to read-only mode” sequence tells you the kernel already hit the wall. If those messages are absent and metadata is merely at 98%, you may still have a clean pool and a much easier fix: extend metadata before the abort happens.
Assess corruption before attempting repair. This is the judgment call. If the pool went read-only due to the abort, the metadata may already be inconsistent. Red Hat guidance warns that once metadata fullness has caused corruption, running thin_repair (which is what lvconvert --repair invokes) into a similarly sized metadata LV may be futile or make things worse. If you have current backups of the thin LVs, treat them as your primary recovery path and repair as the optimistic one.
Check integrity explicitly if you can. With the pool inactive, thin_check /dev/mapper/<vg>-<pool>_tmeta validates the metadata structures. A clean check is good evidence repair will go well. If the pool will not activate at all, it is likely because thin_check already failed during activation.
Inventory what is at stake. List the thin LVs and snapshots in the pool with lvs -o lv_name,vg_name,origin,lv_size. Every one of them shares this metadata. A failed repair is not a one-LV problem.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`metadata_percent` per thin pool	The countdown to this incident; grows with unique block writes, LV count, snapshots	Above 75%, or any sudden jump
`data_percent` alongside metadata	Divergence (low data, high metadata) reveals the random-I/O or snapshot pattern	Metadata growing much faster than data
`lv_health_status` / attr position 9	`M` and `F` are the pool telling you it is already degraded or failed	Any non-`-` value
dmeventd running and `seg_monitor` active	Without it, autoextend cannot fire; the safety net is silently absent	dmeventd down on any host with thin pools
`thin_pool_autoextend_threshold` in lvm.conf	Default is 100, meaning disabled; autoextend covers metadata AND data	Threshold at 100
VG free space	Metadata extension and repair both need VG extents	Free space below one extension cycle
Kernel dm-thin messages	The abort and read-only transition show up here first	“unable to allocate new metadata block”
D-state process count on dm devices	Confirms actual I/O impact, separates metric noise from outage	Growing count of stuck processes

Fixes

If metadata is high but the pool has not aborted yet

This is the good outcome. Extend the metadata LV now:

# Extend pool metadata. Size generously; the max supported is ~16 GiB.
lvextend --poolmetadatasize +4G <vg>/<pool>

The metadata LV can range from 2 MiB to roughly 16 GiB, and 1 GiB is a common creation-time default, which is frequently too small for snapshot-heavy or random-write pools. Metadata space is cheap: the tradeoff is only VG free extents, and that tradeoff is never close.

Also enable autoextend (see Prevention) so this does not depend on a human being awake.

If the pool aborted and went read-only

Stop writes at the application layer if you can, and verify your backups of the thin LVs before running anything. The repair path is disruptive (full pool downtime) and not guaranteed to succeed.

# 1. Unmount filesystems and deactivate the pool (all thin LVs on it must be inactive first)
lvchange -an <vg>/<pool>

# 2. Attempt repair. Uses thin_repair and the pmspare LV; NOT guaranteed to succeed.
lvconvert --repair <vg>/<pool>

# 3. Only if repair succeeds: enlarge metadata so it does not refill immediately
lvextend --poolmetadatasize +4G <vg>/<pool>

# 4. Reactivate, then check every filesystem before returning to service
lvchange -ay <vg>/<pool>
# fsck each thin LV's filesystem before mounting read-write

Be clear about what you are accepting. lvconvert --repair reconstructs metadata into the spare metadata LV (pmspare). If the metadata was already structurally inconsistent from the abort, repair can fail, and per Red Hat’s guidance it can also make the situation worse, because you are repairing into a metadata LV the same size as the one that just proved too small. Do not attempt to resize metadata on a pool that is already in a corrupted state; the operation can hang. If repair fails, the realistic path is restoring the thin LVs from backup and recreating the pool with a much larger metadata LV.

Freeing metadata pressure without repair

Metadata is not reclaimable via discard or fstrim; those only return data blocks. The levers that reduce metadata consumption are:

Delete old thin snapshots. Snapshots are a direct metadata multiplier. Removing them reduces entry count going forward.
Consolidate thin LVs. Fewer thin LVs means fewer mapping trees.
Extend the metadata LV. This is the primary fix, not a workaround.

If lvs hangs during the incident

LVM commands read metadata from disk and take VG locks, so they can block behind the same I/O they are trying to observe. Fall back to dmsetup status --target thin-pool, which reads from kernel memory with no locks and no disk I/O. This is your primary diagnostic tool during the incident, and the reason to collect dmsetup baselines before one.

Prevention

Size metadata aggressively at pool creation. Use --poolmetadatasize and go well above the default. 1 GiB is a floor for trivial pools, not a target. The cost of 4-8 GiB of metadata is negligible next to one failed repair.
Enable autoextend for real. Set thin_pool_autoextend_threshold to 70-80 and thin_pool_autoextend_percent to at least 20 in /etc/lvm/lvm.conf, and confirm dmeventd is running and the pool has an active monitor. The dmeventd policy extension covers both data and metadata. Then verify VG free space can actually fund the extension; a configured autoextend against a full VG fails without extending anything.
Alert on metadata_percent independently. Ticket at 75%, urgent at 90%. Never gate metadata alerting on data_percent; they exhaust on different curves.
Budget snapshots. Thin snapshots are cheap in data terms and expensive in metadata terms. Track count and age, and automate cleanup.
Match the workload to the pool. Random-write-heavy workloads (databases, VM images, container storage) belong on pools with oversized metadata. Sequential archival workloads tolerate smaller metadata.
Keep backups current enough to survive a failed repair. Repair is a coin you do not get to flip twice. For any pool you cannot rebuild from backup, metadata headroom below 50% used is the only acceptable steady state.

How Netdata helps

Tracks data_percent and metadata_percent as separate series per pool, so the low-data/full-metadata divergence is visible on one chart instead of discovered in a postmortem.
Trends metadata consumption over time, which matters because metadata growth is non-linear and tied to write patterns, not volume. A slow climb from 30% to 60% over a month is the early warning this article is about.
Correlates pool usage with LV health flags and dmeventd state, so an alert can distinguish “metadata at 85% with a working autoextend” from “metadata at 85% with no safety net,” which are very different pages.
Surfaces kernel-level corroboration (block I/O errors, D-state processes) next to LVM metrics, so you can see the moment degradation becomes an actual outage.
Retains history LVM itself does not keep. LVM reports current state only; trending, rate-of-change, and runway estimation require an external time series.