$ guides / lvm / lvm-reached-low-water-mark-data-device ▌

Operations Guides

LVM reached low water mark for data device: the thin pool warning before the freeze

You found this line in dmesg or the journal:

device-mapper: thin: 253:4: reached low water mark for data device: sending event

This is not an error. It is the dm-thin kernel target reporting that a thin pool’s data device has crossed its low water mark and that a device-mapper event has been sent to userspace. dmeventd listens for that event and can auto-extend the pool. On a correctly configured system, you may see this message once, dmeventd extends the pool, and usage drops back below the threshold.

On a misconfigured system, this message is the last warning before data usage reaches 100%. With the default queue_if_no_space behavior, writes to every thin LV in the pool can then queue in the kernel until space becomes available. Processes pile up in D state, latency climbs, and applications stop making progress. If the pool is configured to error instead, writes fail directly. If the root filesystem or a database journal lives on that pool, the machine progressively freezes.

The message matters because thin pool auto-extend is disabled by default. thin_pool_autoextend_threshold in lvm.conf defaults to 100, which means “never extend.” If the low water mark message repeats and the pool never grows, the automatic handoff is broken and the pool is still filling.

What this means

A thin pool consists of two internal LVs:

A data LV that stores allocated blocks.
A metadata LV that tracks which blocks belong to which thin volume and snapshot.

Thin LVs are overprovisioned against the data LV. Their virtual sizes do not consume space; only written and retained blocks do. Snapshots increase retention because blocks shared with an origin cannot be freed until every snapshot that references them is removed.

When pool data usage crosses the low water mark, dm-thin emits the kernel message and raises a device-mapper event. If dmeventd is running and monitoring the pool, it receives the event and performs the configured policy extension, effectively lvextend --use-policies. The pool grows by thin_pool_autoextend_percent of its current size if the VG has enough free extents.

The failure cascade looks like this:

flowchart TD
  A["Pool data usage crosses low water mark"] --> B["Kernel: reached low water mark, event sent"]
  B --> C{"dmeventd running, threshold below 100, VG has free space?"}
  C -- yes --> D["Auto-extend succeeds, pool grows, warning clears"]
  C -- no --> E["No extension, usage keeps climbing"]
  E --> F["data_percent reaches 100%, lv_attr shows D"]
  F --> G["Writes queue under queue_if_no_space, D-state processes grow"]
  G --> H["Writes stall or fail, applications hang or crash"]

The message itself is informational. The operational question is whether anything acted on the event. Answer that by correlating the message with the data_percent trend.

Common causes

Cause	What it looks like	First thing to check
Auto-extend disabled by the default threshold	Warning repeats, pool never grows, `data_percent` climbs	`lvmconfig activation/thin_pool_autoextend_threshold`
dmeventd not running	Event is sent, but no listener acts on it	`systemctl is-active lvm2-monitor.service` and `pgrep -x dmeventd`
Pool not monitored by dmeventd	dmeventd is alive, but this pool has no monitored segment	`lvs -o lv_name,vg_name,seg_monitor`
VG has no free extents	Extension cannot allocate space, so usage continues rising	`vgs -o vg_name,vg_free`
Runaway writer or bulk import	`data_percent` jumps quickly after the warning	Repeated `dmsetup status` samples
Thin snapshots accumulating	Old snapshots retain blocks and increase metadata use	`lvs -o lv_name,origin,data_percent,metadata_percent`
No working discard/TRIM path	Usage stays high after large filesystem deletions	Run `fstrim` on supported thin LV filesystems, then re-check

Quick checks

These checks are read-only and safe during an incident.

# 1. Identify the pool and current usage.
lvs -o lv_name,vg_name,lv_attr,data_percent,metadata_percent

# 2. Read kernel pool state without going through the higher-level LVM tools.
#    This is useful when the system is already slow.
dmsetup status --target thin-pool
# Look for used and total metadata/data fields and the no-space policy.

# 3. Check lv_attr position 9 for pool health.
#    D = out of data space, F = failed, metadata flags indicate metadata trouble.
lvs -o lv_name,vg_name,lv_attr

# 4. Verify auto-extend configuration.
lvmconfig activation/thin_pool_autoextend_threshold activation/thin_pool_autoextend_percent
# A threshold of 100 means auto-extend is disabled.

# 5. Verify dmeventd is alive.
systemctl is-active lvm2-monitor.service
pgrep -x dmeventd

# 6. Check whether dmeventd is monitoring the pool.
#    Use seg_monitor; do not infer monitoring from lv_attr.
lvs -o lv_name,vg_name,lv_attr,seg_monitor

# 7. Check VG headroom for automatic or manual extension.
vgs -o vg_name,vg_size,vg_free

# 8. Look for processes stuck in uninterruptible I/O.
ps -eo pid,stat,wchan:30,comm | awk '$2 ~ /D/'

# 9. Check whether the warning is repeating.
journalctl -k | grep "low water mark" | tail -20

A single data_percent reading does not show burst rate. Take multiple readings and use the delta before deciding whether you have hours or minutes.

How to diagnose it

Measure the trajectory, not just the value. Take three dmsetup status --target thin-pool samples one minute apart and calculate the change in used data blocks. A pool at 88% growing 0.5% per hour is a capacity ticket. A pool at 88% growing 5% per hour is an incident.
Determine whether auto-extend was supposed to fire. A threshold of 100 disables it. A lower threshold still does nothing if dmeventd is stopped or the pool is not monitored. If both are configured correctly, check VG free space. Repeated warnings with no pool growth are the signature of a broken handoff.
Check the pool health flags. If lv_attr position 9 already shows D, the pool is out of data space. Skip further trend analysis and extend the pool immediately.
Check the no-space policy. The dmsetup status output shows whether the pool queues or errors when full. Queueing turns exhaustion into application hangs; erroring turns it into write failures.
Check metadata independently. Data exhaustion stops new allocations. Metadata exhaustion is more dangerous and can make recovery significantly harder. Treat metadata usage above 90% as urgent.
Identify what consumes space. Compare total thin LV virtual size with pool size, list snapshots, and check whether deletions return space. A pool that remains full after large file deletions usually lacks a working discard path. fstrim can reclaim space on supported filesystems, but it can add noticeable I/O load during an incident.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Thin pool `data_percent`	Countdown to write stalls across every thin LV	Above 85%, or any growth rate projecting exhaustion within 48 hours
Thin pool `metadata_percent`	Metadata exhaustion can require pool repair	Above 75%; urgent above 90%
`lv_attr` position 9	Kernel-reported pool health, including `D` and failed states	Any health flag other than normal
No-space policy	Determines whether exhaustion causes hangs or immediate I/O errors	Policy does not match application failure tolerance
VG free space	Determines whether automatic or manual extension can succeed	Less than two auto-extend cycles of headroom
dmeventd state and pool monitoring	The safety net for the low water mark event	Daemon absent or pool unmonitored
Allocation rate from repeated samples	Converts a percentage into a runway estimate	Projected exhaustion within hours
D-state process count	Shows that queued writes are already stalling applications	Count growing or a process stuck for more than 120 seconds

Fixes

If the pool has not hit 100% yet

Extend the thin pool from VG free space:

# Grow the pool LV, not an individual thin LV.
# Choose the increment from the measured allocation rate and available VG space.
lvextend -L +50G <vg>/<thinpool>

This is normally an online operation. After the immediate risk clears, fix the automatic extension path so the next event does not require manual intervention.

If the VG is full

Create VG headroom or free space inside the pool:

Add a new PV, extend the VG, and then extend the pool:

# pvcreate writes LVM metadata to the device. Verify that <new_disk> is the
# intended, unused device before running it.
pvcreate /dev/<new_disk>
vgextend <vg> /dev/<new_disk>
lvextend -L +50G <vg>/<thinpool>

Delete unneeded thin snapshots. This frees only blocks no longer referenced by another snapshot or origin. Deletion is destructive to the snapshot.
Run fstrim on supported thin LV filesystems if discarded blocks were not previously returned to the pool. Expect additional storage load while it runs.
Remove a sacrificial thin LV only after the service owner confirms that its data can be destroyed.

If the pool already hit 100%

Extend the pool first. The extension often still succeeds when data space is exhausted, and queued writes resume when space becomes available. If metadata is also exhausted or lv_attr shows F, repair may be required.

# DISRUPTIVE AND DATA-RISKING:
# Stop applications and unmount filesystems on the affected thin LVs first.
# This takes the pool out of service and the repair is not guaranteed.
lvchange -an <vg>/<pool>
lvconvert --repair <vg>/<pool>

lvconvert --repair uses the thin repair tooling and spare metadata. Treat it as best-effort recovery, not as a substitute for restoring from a verified backup.

If XFS shut itself down after I/O errors, restore pool space first, then unmount and remount the filesystem if it can be unmounted cleanly. If errors remain, follow the normal filesystem recovery process before returning it to production. If the root filesystem was on the pool, a reboot may be unavoidable.

Prevention

Enable auto-extend. Set thin_pool_autoextend_threshold to 70-80 and thin_pool_autoextend_percent to at least 20 in /etc/lvm/lvm.conf. Verify the effective configuration with lvmconfig, confirm dmeventd is running, and check seg_monitor for the pool.
Reserve VG headroom. Auto-extend consumes VG extents every time it fires. Keep enough free VG space for at least two extension cycles.
Alert on usage and rate. Create a ticket alert above 85%, an urgent alert above 95%, and an earlier alert when the projected exhaustion window is under 48 hours. Lower those thresholds when dmeventd is absent or the VG has little free space.
Monitor metadata separately. Metadata grows with mapping churn and snapshot complexity, not only with data volume. It can become the bottleneck while data usage still looks acceptable.
Manage snapshot lifecycles. Assign owners and expiration dates to thin snapshots. A forgotten snapshot can retain data indefinitely.
Keep discard working. Ensure the filesystem, thin LV, and pool discard behavior are compatible so deletions can return blocks to the pool.

How Netdata helps

Thin pool data_percent and metadata_percent trends show whether the low water mark event was followed by an extension or ignored.
Allocation-rate history turns the kernel message into a runway estimate instead of a one-off log entry.
Device latency and inflight I/O reveal the slowdown that often appears before writes fail.
D-state process counts corroborate that queued thin-pool writes are already stalling applications.
VG free-space trends answer the immediate follow-up question: if dmeventd fires, can an extension actually allocate space?
Alerting on both absolute usage and projected exhaustion catches slow leaks and runaway writers without waiting for the 100% freeze.