LVM snapshot COW usage climbing: extend or remove before it overflows

lvs shows a traditional snapshot with snap_percent at 78%. An hour ago it was 61%. You have a fixed-size COW exception store filling up from writes to the origin volume, and when it reaches 100% the snapshot is invalidated instantly and permanently. There is no warning state, no graceful degradation, and no recovery. If a backup is reading from that snapshot, the backup dies with it.

This is a countdown, not a trend. The right response is a quick calculation (how fast is it filling, how much time is left) followed by one of three actions: finish the backup faster, extend the snapshot, or remove it.

The origin volume is not at risk from the overflow itself. After invalidation, the kernel skips the dead snapshot and origin I/O continues. But while the snapshot is alive and filling, every origin write pays COW overhead, and the restore point you thought you had may already be gone.

What this means

A traditional (non-thin) LVM snapshot reserves a fixed-size exception store at creation time. Every write to a block on the origin that has not yet been copied triggers a copy-on-write sequence: read the original block, copy it into the exception store, then write the new data to the origin. The exception store fills with pre-change data, one chunk at a time. Writes to the snapshot volume itself also consume the store.

Two properties of this mechanism shape everything below:

1. Fill rate is proportional to origin write rate (more precisely, to writes touching blocks not yet saved in the exception store). A snapshot of a quiet volume can sit at 3% for weeks. A snapshot of an active database can go from 50% to 100% in minutes.
2. Allocation happens in chunks, so snap_percent jumps. The exception store allocates in chunk-sized increments (set at creation with --chunksize), so the percentage climbs in steps, not smoothly. You cannot predict the exact second of overflow from the percentage alone.

At 100%, the kernel invalidates the snapshot. You will typically see a message like device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception. in the kernel log, and the snapshot’s lv_attr position 5 flips to I. The LV stays listed in lvs output, looking almost normal, which is exactly how invalidated snapshots go unnoticed for days.

Before invalidation, there is a quieter cost: origin write latency. The COW sequence turns one write into a read plus two writes, and multiple snapshots on the same origin multiply that overhead. Origin latency commonly runs 2-5x baseline while a heavily written snapshot exists, and recovers immediately when the snapshot is removed or invalidated. If your application got slow during the backup window, this is why.

Common causes

Quick checks

# Snapshot usage, origin, and validity flag for all snapshots
lvs -o lv_name,vg_name,origin,snap_percent,lv_size,lv_attr

# Find already-invalidated snapshots (position 5 of lv_attr = 'I')
lvs -o lv_name,vg_name,lv_attr --noheadings | awk 'substr($3,5,1) == "I"'

# Check VG free space: you cannot extend the snapshot without it
vgs -o vg_name,vg_size,vg_free

# Measure the fill rate: two samples, 10 minutes apart
lvs --noheadings --nosuffix --units g -o lv_name,snap_percent
# ... wait 10 minutes, repeat, compute delta(snap_percent) / delta(hours)

# Origin write activity (cumulative counters; take deltas)
grep ' dm-' /proc/diskstats

# Kernel log for invalidation events
dmesg | grep -i 'snapshot'

All of these are read-only and safe. One caveat: lvs takes LVM metadata locks and reads PV metadata, so on a heavily loaded system it can be slow. dmsetup status reads from kernel memory and does not take LVM locks; prefer it during an active incident if lvs hangs.

How to diagnose it

1. Confirm the snapshot is still valid. Check lv_attr position 5. If it shows I, the snapshot is already dead. Skip to removal; there is nothing to save.

2. Compute the fill rate. Sample snap_percent twice with a known interval. growth_rate = delta(snap_percent) / delta(hours), then hours_until_overflow = (100 - current_percent) / growth_rate. Remember the chunked allocation: treat this as an estimate with real error bars, not a countdown timer.

3. Decide whether the snapshot is still needed. Is a backup actively reading from it? Is it a rollback point for an in-progress change? Is it a leftover from a backup that finished three days ago?

4. Check VG free space. Extending the snapshot allocates more extents from the VG. If the VG has no free extents, extension is not an option until you add capacity or free space elsewhere. See LVM Insufficient free extents if that is your situation.

5. Check origin latency. If applications are complaining, compare current dm device latency against baseline. Elevated origin latency that correlates with snapshot lifetime is COW overhead, and it argues for removing the snapshot sooner rather than extending it.

flowchart TD
  A[snap_percent rising] --> B{Snapshot still valid?}
  B -- "attr shows I" --> C[Remove it, recreate if still needed]
  B -- valid --> D{Still needed?}
  D -- "no / backup done" --> E[lvremove the snapshot]
  D -- "backup running" --> F[Prioritize finishing the backup]
  D -- "needed longer" --> G{VG free space?}
  G -- yes --> H[lvextend the snapshot]
  G -- no --> I[Free VG space or add a PV first]
  F --> J{On track before ~90%?}
  J -- no --> G
  J -- yes --> K[Monitor until backup completes, then remove]

Metrics and signals to monitor

Fixes

The backup is still running: finish it first

If a backup is mid-flight, the cheapest fix is usually to let it complete and then remove the snapshot immediately. Prioritize the backup: pause competing I/O on the origin if you can, and watch the runway estimate against the backup’s remaining time. If the math says the snapshot overflows before the backup finishes, extend the snapshot (below) to buy time. Do not plan to reach 100%; treat 90% as the real ceiling.

Keep the snapshot longer: extend it

# Extend the snapshot's exception store by 20G
lvextend -L +20G <vg>/<snapshot>

# Verify
lvs -o lv_name,origin,snap_percent,lv_size

This is online and safe; the origin and snapshot stay available. The space comes from VG free extents, so confirm vgs shows enough headroom first. Tradeoff: the exception store now occupies more of your VG, and the COW performance tax on the origin continues for as long as the snapshot lives. Extending buys time, it does not fix the underlying problem of a long-lived snapshot on a busy origin.

LVM can also auto-extend snapshots via dmeventd, controlled by snapshot_autoextend_threshold and snapshot_autoextend_percent in /etc/lvm/lvm.conf. The threshold default of 100 means disabled. If you rely on this, verify dmeventd is actually running and that the VG has free space, because auto-extend fails silently when it cannot allocate. On a fast-writing origin it can also lose the race: the daemon polls periodically, and a burst can fill the store between polls.

The snapshot is done or dead: remove it

# Remove a snapshot (destructive: the restore point is gone)
lvremove <vg>/<snapshot>

Removal immediately frees the exception store back to the VG and eliminates the COW overhead on the origin. Latency recovery on the origin should be visible right away. For an invalidated snapshot (attr I), removal is the only option; if the LV is still active, deactivate it first with lvchange -an <vg>/<snapshot>, then remove it. If you still need a restore point, create a fresh snapshot afterward with a realistic size.

Be deliberate: lvremove on a valid snapshot destroys the restore point. If you intended to roll back with lvconvert --merge, note that merge requires a valid snapshot, so an overflowed one has already foreclosed that path.

Prevention

• Size the exception store for the workload, not the origin size. Estimate expected write volume over the snapshot’s intended lifetime and provision the COW area above it, with margin. A snapshot meant to survive a two-hour backup of a busy database needs a very different size than one held for a ten-minute config change.
• Alert at 80%, act before 90%. snap_percent above 80% is a ticket: extend or complete the backup same-shift. Above 95% is urgent. Chunked allocation means the last few percent can arrive in one jump.
• Alert on snapshot age. Any snapshot older than 24 hours on a high-churn origin deserves a review. Zombie snapshots from finished backups are the most common way overflows happen unattended.
• Remove snapshots as part of the backup job, not as a separate step. The snapshot lifecycle should be create, back up, remove, in one automated flow with alerting on failure.
• Limit snapshots per origin. Each active snapshot adds COW amplification to every origin write. If you need long-term, low-overhead point-in-time copies, evaluate thin snapshots instead; they share the thin pool rather than holding a fixed private store (with their own failure modes, covered in LVM thin pool out of data space).
• Track the trend, not just the current value. LVM reports current state only. Without history you cannot compute fill rate or runway, which are the numbers that actually drive the extend-or-remove decision. The LVM monitoring checklist covers the full signal set.

How Netdata helps

• Netdata tracks per-snapshot snap_percent over time, so fill rate and runway are visible as a trend rather than requiring manual sampling during an incident.
• Alerting on snapshot usage thresholds (80%, 95%) fires before overflow, while an invalidated snapshot (attr I) shows up as a state change you can alert on directly.
• Correlating snapshot growth with origin dm device I/O and latency shows both sides of the problem: how fast the store is filling and how much the snapshot is costing the origin in write performance.
• VG free space on the same dashboard answers the follow-up question immediately: is there room to extend, or is removal the only option?
• Snapshot age and count per origin become visible over time, which is what makes zombie snapshots detectable before they overflow at 3 a.m.

Related guides

• LVM cannot extend a logical volume: adding a PV when the VG is full
• LVM filesystem full while the volume group has space: the resize step everyone forgets
• How LVM actually works in production: a mental model for operators
• LVM Insufficient free extents: the volume group is out of space
• LVM monitoring checklist: the signals every production volume manager needs
• LVM monitoring maturity model: from survival to expert
• LVM reached low water mark for data device: the thin pool warning before the freeze
• LVM thin pool auto-extend not working: threshold 100 means disabled
• LVM thin pool out of data space: every thin volume freezes at once
• LVM thin pool metadata full: the exhaustion that can corrupt the pool
• LVM thin pool check needed: thin_check and lvconvert –repair
• LVM thin pool overprovisioning: 60% full can still be dangerous