LVM snapshot invalid: the COW exception store filled and the snapshot is gone

You run lvs and see it: a snapshot with snap_percent at 100.00 and an I in the fifth position of lv_attr. In the kernel log there is a line like Invalidating snapshot: Unable to allocate exception. The snapshot is dead. It did not degrade, it did not warn you at the application layer, and it cannot be brought back.

This article covers traditional (non-thin) LVM snapshots only. Thin snapshots live in a thin pool and fail differently; if your snapshots are thin, the relevant failure is pool exhaustion, not this one. See LVM thin pool out of data space for that case.

The good news, such as it is: the origin volume is fine. The kernel skips invalid snapshots, and origin I/O continues normally. The bad news: whatever the snapshot existed for, usually a backup or a rollback point, is silently lost. If a backup job was reading from this snapshot when it invalidated, that backup is incomplete or corrupt, and nothing in the backup tool’s exit code necessarily tells you so.

What this means

A traditional LVM snapshot is not a copy of the origin. It is a fixed-size exception store (the COW area) plus a mapping table. When a block on the origin is about to be overwritten, the original contents of that block are first copied into the exception store. Reads from the snapshot resolve to either unchanged origin blocks or saved blocks in the exception store, reconstructing the point-in-time view.

Two operational consequences follow:

1. The exception store is filled by writes to the origin, not by writes to the snapshot. A snapshot of a quiet volume can sit at 0% for weeks. A snapshot of an active database can fill in minutes. Sizing is about the origin’s write volume during the snapshot’s lifetime, nothing else.
2. The store has a hard ceiling. Allocation happens in chunks, so snap_percent jumps rather than rising smoothly. At 100%, the device-mapper snapshot target cannot allocate the next exception, logs the invalidation message, and marks the snapshot invalid. This is immediate and irreversible. The snapshot stays visible in lvs output with the I flag, but it cannot be read, merged, or repaired. lvconvert --merge requires a valid snapshot.

flowchart TD
  A[Writes to origin LV] --> B[COW: old blocks copied to exception store]
  B --> C{Exception store full?}
  C -->|No| D[snap_percent climbs in chunks]
  D --> A
  C -->|Yes| E[dm logs: Invalidating snapshot]
  E --> F[lv_attr position 5 = I, snapshot unusable]
  F --> G[Origin I/O continues normally]
  F --> H[Backup or rollback point lost]

Common causes

The default deserves emphasis: snapshot_autoextend_threshold = 100 in /etc/lvm/lvm.conf disables automatic extension of snapshots. Unless someone explicitly changed it, dmeventd watches the snapshot fill and does nothing.

Quick checks

All of these are read-only.

# Snapshot usage, origin, and validity flag
lvs -o lv_name,vg_name,origin,snap_percent,lv_attr

# Find invalid snapshots: position 5 of lv_attr is 'I'
lvs -o lv_name,vg_name,lv_attr | awk 'NR>1 && substr($3,5,1)=="I"'

# Kernel-side confirmation of invalidation
dmesg | grep -i 'invalidat\|snapshot' | tail -20

# dm state without taking LVM locks (safe even when lvs is slow)
dmsetup status

# Is auto-extend configured at all?
grep -E 'snapshot_autoextend' /etc/lvm/lvm.conf

# Is dmeventd running (required for auto-extend to do anything)?
systemctl is-active lvm2-monitor.service
pgrep -x dmeventd

Two caveats on the tools themselves. First, an invalidated snapshot returns I/O errors on read, and operators have reported lvs/vgs printing errors like read failed after 0 of 4096 while an invalid snapshot exists. The command output is still usable; the errors are about the dead snapshot device, not your metadata. Second, LVM commands take VG locks and read metadata from the PVs, so on a stressed system dmsetup status is the more reliable probe.

How to diagnose it

1. Confirm the invalidation. lvs shows snap_percent 100.00 and I in position 5 of lv_attr. dmesg shows the invalidation line with a timestamp. The timestamp matters: it tells you which backup job or write burst killed the snapshot.
2. Verify the origin is healthy. The origin LV should show active state (a in position 5) and normal I/O. If the origin also shows problems, you are dealing with something beyond snapshot overflow (missing PV, metadata issue) and should diagnose that first.
3. Assess the blast radius. What was this snapshot for? If a backup was in flight, treat that backup as untrusted and re-run it from a fresh snapshot. If it was a pre-change rollback point, the rollback option is gone; proceed with the change freeze accordingly.
4. Reconstruct why it filled. Compare snapshot size against origin write volume during its lifetime. Check for a write burst in the window before the dmesg timestamp: batch jobs, package updates, database maintenance, log storms. Check whether other snapshots on the same origin were compounding COW pressure.
5. Check the safety net that wasn’t there. Confirm snapshot_autoextend_threshold is 100 (disabled) or, if someone set it lower, whether dmeventd was actually running and whether the VG had free space to extend into. Auto-extend fails silently when the VG is full.

Metrics and signals to monitor

Note the instrumentation limit: exception store allocation is chunked, so snap_percent jumps several percent at a time. A monitoring interval of minutes is too coarse for a snapshot on a busy origin; it can go from 70% to invalid between scrapes.

Fixes

The snapshot is already invalid

There is no recovery. Remove it and recreate if still needed.

# Remove the dead snapshot (destructive: discards the snapshot device)
lvremove <vg>/<snapshot>

# If removal complains about the device, force it (no confirmation prompt)
lvremove -f <vg>/<snapshot>

# Recreate with a realistic size if the snapshot is still needed
lvcreate -s -n <snapshot> -L <size>G <vg>/<origin>

Removal immediately frees the exception store space back to the VG. Origin latency, if it was elevated from COW overhead, returns to normal at once.

The snapshot is still valid but filling

Act before 100%. You have two options, and waiting is not one of them.

# Extend the exception store (requires VG free space)
lvextend -L +<size>G <vg>/<snapshot>

# Or, if the backup is done, remove it now
lvremove <vg>/<snapshot>

Extension is online and safe. If lvextend fails with insufficient free extents, the VG is out of headroom; see LVM Insufficient free extents and LVM cannot extend a logical volume.

Enable auto-extend for future snapshots

In /etc/lvm/lvm.conf:

# Extend when the snapshot crosses 80%, growing it by 20% each time
snapshot_autoextend_threshold = 80
snapshot_autoextend_percent = 20

This only works if dmeventd (lvm2-monitor.service) is running and the VG has free extents. Auto-extend is a safety net, not a substitute for sizing: on a very busy origin, the snapshot can jump past the threshold between dmeventd poll cycles.

Move recurring snapshot workloads to thin snapshots

Thin snapshots share the thin pool’s data space instead of a private fixed-size exception store, so they do not have this failure mode. They trade it for pool exhaustion failure modes, which are at least monitorable and extendable. For long-lived or write-heavy snapshot use, thin is the operationally better choice. Be aware that moving to thin means adopting the thin pool monitoring discipline (data and metadata percent, auto-extend configuration) described in the related guides.

Prevention

• Size for the origin’s write volume, not its size. Estimate total writes during the snapshot’s intended lifetime and allocate at least 2x that.
• Keep snapshots short-lived. Create, back up, remove. Automate the removal; zombie snapshots are the most common path to this incident.
• Alert on snap_percent at 80% and on any I flag. Invalidation is irreversible, so the only useful alert fires before it, and the post-hoc alert exists to tell you a backup is now untrusted.
• Enable and verify auto-extend. Set the threshold below 100, confirm dmeventd is running, and keep VG free space available so extension can actually happen.
• Watch origin write rate during snapshot lifetime. A planned bulk load or database maintenance during a backup window is a classic invalidation trigger.
• Minimize concurrent snapshots per origin. Each one adds COW overhead to every origin write and is an independent invalidation risk.

How Netdata helps

• Per-second disk I/O metrics on dm devices show origin write throughput and latency, so you can see the COW penalty appear at snapshot creation and the write bursts that consume the exception store.
• Correlating origin write rate against snapshot age turns “a snapshot exists” into a runway estimate, which is the only way to act before a chunked snap_percent jump reaches 100%.
• Alerting on block device latency spikes during backup windows surfaces the 3x COW write amplification before it becomes an application complaint.
• Long retention of I/O history lets you reconstruct what wrote to the origin in the minutes before the dmesg invalidation timestamp, which is the post-incident question everyone asks.
• Because invalidation is instant and irreversible, the operational value is entirely in leading indicators: write-rate trends and headroom, not the failure event itself.

Related guides

• LVM monitoring checklist: the signals every production volume manager needs
• How LVM actually works in production: a mental model for operators
• LVM Insufficient free extents: the volume group is out of space
• LVM cannot extend a logical volume: adding a PV when the VG is full
• LVM thin pool out of data space: every thin volume freezes at once
• LVM thin pool auto-extend not working: threshold 100 means disabled
• LVM reached low water mark for data device: the thin pool warning before the freeze