vSphere forgotten snapshot growing: the delta VMDK time bomb

A VM has a snapshot that should have been deleted days ago. The delta VMDK grows with every guest write. The datastore slowly fills. The guest has no idea anything is wrong. By the time someone notices, consolidation is a multi-hour, I/O-intensive operation that stuns the VM, and the datastore is hours from full. Forgotten snapshots are one of the most common preventable vSphere incidents.

Snapshots are a point-in-time capture, not a backup. They are designed to live for hours. VMware recommends 2 to 3 snapshots in a chain and a maximum of 24 to 72 hours in production. A snapshot that is 0 bytes today is the beginning of the problem, not its absence. Every guest write to the base disk is redirected to the delta file, and the delta grows monotonically until you delete or consolidate.

Most setups have zero snapshot monitoring. Teams discover two-week-old, 300GB deltas when a datastore hits 100% and every VM on it halts I/O. The guest OS is unaware of the snapshot. Performance degradation is invisible from inside the VM until the chain gets deep enough to add read latency on top of the space problem.

This guide covers how to find forgotten snapshots before they detonate, how to consolidate safely, and what to monitor to prevent recurrence. It assumes familiarity with the broader vSphere failure model covered in the production mental model guide.

What this means

A vSphere snapshot is a delta file (*-delta.vmdk, or on VMFS6 an SEsparse delta) plus a small descriptor and a snapshot database entry. When a snapshot is active, the base VMDK becomes read-only. Every write the guest issues lands in the delta file. Reads must walk the chain from the top delta down to the base, picking up the newest version of each block. This is why snapshot chains cost both space and read latency.

A delta can grow up to the size of the base disk. A 200GB base disk with a forgotten snapshot can produce a 200GB delta file on the same datastore. Reserve additional free datastore capacity beyond the total VM disk size to absorb snapshot growth .

Snapshot chain growth is the most common cause of unexpected datastore space consumption. When the datastore hits 100%, every VM on it halts I/O. Thin-provisioned VMDKs cannot expand. .vswp files cannot be created during power-on. New snapshots cannot form. Consolidation cannot complete because there is no working space. Recovery is multi-hour and disruptive.

A second, subtler failure is the orphaned delta. A backup job is interrupted, or a snapshot delete task times out. The delta file remains on disk, but Snapshot Manager shows nothing. The VM keeps writing to the delta, which keeps growing. Get-Snapshot returns nothing. Only the consolidationNeeded runtime property catches it. These orphans are more dangerous than visible snapshots because nothing in the normal UI alerts you.

flowchart TD
    A["Snapshot created
delta at 0 bytes"] --> B["Guest writes redirect
to delta VMDK"]
    B --> C["Delta grows
day over day"]
    C --> D["Datastore free
space declines"]
    C --> E["VM read latency rises
from chain traversal"]
    D --> F["Datastore hits 100%
all VMs halt I/O"]
    E --> G["Consolidation needed:
multi-hour, VM stuns"]
    F --> G

Common causes

CauseWhat it looks likeFirst thing to check
Forgotten manual snapshotA snapshot named “before change” or “temp” older than 72 hours, visible in Snapshot ManagerGet-VM | Get-Snapshot | Select VM,Name,Created,SizeGB sorted by Created
Failed backup job cleanupOrphaned delta files, no snapshot visible in Snapshot Manager, consolidationNeeded = trueGet-VM | Where-Object {$_.ExtensionData.Runtime.ConsolidationNeeded}
Backup tool lost cleanup stateMultiple snapshots in the chain, some created at backup window timesChain depth and Created timestamps
Interrupted snapshot deleteConsolidation task stuck near completion or timed out; delta still presentRecent vCenter tasks for ConsolidateDisks or RemoveSnapshot
Deep chain from repeated snapshotsChain depth greater than 3, latency elevated on the affected VMChain depth against the 2 to 3 recommended

Quick checks

Safe, read-only commands.

# PowerCLI: list all snapshots with age and size
Get-VM | Get-Snapshot | Select VM, Name, Created, SizeGB | Sort Created

# PowerCLI: snapshots older than 72 hours (the VMware limit)
Get-VM | Get-Snapshot | Where-Object {$_.Created -lt (Get-Date).AddDays(-3)} | Select VM, Name, Created, SizeMB

# PowerCLI: VMs with orphaned deltas that need consolidation
Get-VM | Where-Object {$_.ExtensionData.Runtime.ConsolidationNeeded} | Select Name

# ESXi SSH: look for delta files in a VM folder
ls -la /vmfs/volumes/<datastore>/<vm-folder>/*-delta.vmdk

# ESXi SSH: snapshot info for a single VM
vim-cmd vmsvc/get.snapshotinfo <vmid>

# ESXi SSH: datastore free space
esxcli storage filesystem list

The PowerShell snippet filtering on Created is the pattern for any age threshold.

How to diagnose it

  1. Find the offending snapshot. Run the PowerCLI list across all VMs and sort by Created descending. Anything older than 72 hours is past the VMware limit. Anything older than 24 hours outside a maintenance window warrants investigation.
  2. Confirm it is not an orphan. Run the consolidationNeeded query. If a VM returns true, there are delta files that Snapshot Manager does not display. These grow silently. Creating a new snapshot and deleting it sometimes clears the entire hierarchy, but do not rely on that as the primary fix.
  3. Check datastore free space headroom. If free space is below 5% and below 500GB, or below 10GB on small datastores, treat it as critical. Consolidation needs working space roughly equal to the largest delta. If free space is already tight, do not start consolidation without freeing space or adding capacity first.
  4. Check VM disk latency. VMs with deep snapshot chains experience elevated read latency from chain traversal. If GAVG is elevated only on the affected VM while DAVG is normal on the datastore, the snapshot chain is the cause, not the array.
  5. Identify the source. Check recent vCenter events for backup job failures, snapshot create and delete tasks, and any consolidation tasks that ended in error. Backup products that use snapshots in-band are the usual source of orphaned deltas.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Snapshot age per VMSnapshots are meant for hours, not weeksAny snapshot older than 72 hours
Snapshot chain depth per VMVMware recommends 2 to 3Chain deeper than 3, or approaching the hard limit in some versions
Snapshot delta size per VMDelta can approach the size of the base diskDelta growing day over day
Datastore free spaceSnapshot growth consumes space until the datastore is fullLess than 15% free, or trending down
consolidationNeeded per VMCatches orphaned deltas invisible in Snapshot ManagerAny VM with the flag set
VM disk latency (GAVG) per VMChain traversal adds read latencyElevated GAVG on a VM with snapshots, while peers are normal
Backup job success and failureFailed backups are the top source of orphaned snapshotsRepeated failures or retries on the same VM

Snapshot age and chain depth are not performance counters. They must be queried through the vCenter API (VirtualMachine.snapshot property) or PowerCLI. The standard ESXi performance counter stream does not surface them, which is why most teams do not monitor them.

Fixes

Fixes group by cause and by how much time you have before the datastore fills.

Visible snapshot, plenty of datastore headroom

Delete the snapshot through Snapshot Manager. If there are multiple snapshots in the chain, use “Delete All” to consolidate the entire chain into the base disk in one operation. Schedule a maintenance window. Consolidation is I/O-intensive, and for large deltas it is multi-hour. Expect a brief VM stun during the final commit.

Visible snapshot, datastore critically low

Do not start consolidation if free space is below the delta size plus a working margin. Free space first by relocating other VMs off the datastore, deleting other snapshots, or expanding the datastore. Only then consolidate. Consolidation cannot complete when there is no working space.

Orphaned delta (consolidationNeeded = true)

Trigger consolidation through the vSphere Client (right-click the VM, Snapshots, Consolidate). This commits the orphaned deltas into the base disk. If consolidation fails or hangs near completion, the usual causes are locked files held by a third-party backup product, insufficient datastore space, or vCenter to ESXi connectivity issues. Resolve the lock before retrying. Do not cancel a running consolidation task; interrupting it can corrupt the chain.

Cannot free enough space

If the datastore cannot be expanded and there is no room to consolidate, storage vMotion the VM to a datastore with headroom and consolidate there. This is disruptive and slow on a large VM, but it is safer than attempting consolidation with no working space.

Things you cannot do

You cannot increase the VMDK size while snapshots exist. This is unsupported and can corrupt the snapshot chain. You cannot cold-migrate a VM with a broken chain. You cannot rely on a guest-level backup to capture a consistent state if the chain is already corrupt.

Prevention

  • Monitor snapshot age and size daily. Daily is the minimum cadence. Hourly is better for environments with active backup jobs.
  • Treat 0 bytes as the start of the problem. A newly-created snapshot is 0 bytes today and grows from the first guest write.
  • Alert on consolidationNeeded. Orphaned deltas are invisible in Snapshot Manager. The runtime property is the only reliable signal.
  • Alert on chain depth greater than 3. Beyond the VMware recommendation, read latency compounds with each level of chain traversal.
  • Track backup job success. Failed or retried backup jobs are the top source of orphaned snapshots. Correlate backup events with snapshot creation and deletion events.
  • Reserve datastore headroom for snapshot operations. Keep enough free space to absorb at least 24 hours of snapshot growth for all VMs simultaneously.
  • Avoid manual snapshots outside a change window. When you take one, set a calendar reminder to delete it the same day.
  • On vSphere 8.0.3 and later, use scheduled snapshot deletion. This lets you set a hard TTL at creation time.

How Netdata helps

  • Snapshot age and chain depth are not perf counters. Netdata queries the vCenter API on a schedule and surfaces snapshot age, size, and chain depth as first-class metrics alongside ESXi performance counters, so a forgotten snapshot does not sit invisible for weeks.
  • Correlate snapshot growth with datastore free space. The time bomb is the combination of a growing delta and a shrinking datastore. Plotting both on one timeline shows the collision course days before the datastore fills.
  • Correlate snapshot presence with per-VM disk latency. Chain traversal raises GAVG on the snapshotted VM while DAVG stays normal. Seeing both together pinpoints the snapshot as the latency source instead of the array.
  • Surface consolidationNeeded as an alert, not a buried property. Orphaned deltas are the most dangerous variant because nothing in the UI flags them. Treating the boolean as a ticket-level alert catches them the same day they appear.
  • Trend datastore free space against snapshot growth rate. Extrapolating the delta growth rate against remaining free space gives a concrete runway estimate in days, which is more actionable than a static percentage threshold.