vSphere datastore full: ‘No space left on device’, paused VMs, and power-on failures

A vSphere datastore hitting 100% is a cliff-edge failure. Below 100%, VM performance is unaffected. At 100%, every VM that needs to write to the datastore stops: running VMs pause with the “There is no more space for virtual disk” dialog, thin-provisioned VMDKs cannot extend, snapshot deltas cannot grow, and power-on operations fail because the per-VM .vswp swap file cannot be created.

The vmkernel log records the kernel’s view with No space left on device. The most common root cause is unattended snapshot growth: a delta VMDK consumes the same datastore as the base disk and grows with every guest write. A VM with a 200 GB base disk and an active snapshot can grow its delta to 200 GB.

What this means

Three classes of operation break when the datastore is full:

• Running VM I/O pauses. ESXi stops the VM at the world layer rather than corrupting the disk. Memory state stays in RAM; nothing is written to a .vmss suspend file. The vSphere Client shows a dialog offering Retry or Cancel. Cancel powers the VM off and loses in-memory state. Retry resumes I/O once space is available.
• Power-on fails. Each VM has a .vswp file sized to the gap between configured memory and the memory reservation. If the datastore cannot hold the new .vswp, the power-on task fails with No space left on device.
• Snapshot and consolidation operations fail. Delta VMDKs cannot grow. In-flight consolidations can leave the VM with locked delta files even after the snapshot entry is removed from the snapshot manager, surfacing later as consolidationNeeded = true.

ESXi needs roughly 100 MB of free space for VMFS journal and metadata writes. Below that, even space-freeing operations can error out.

NFS datastores add a second trap. The vSphere Client reports free space from the NFS server’s view, which itself may be thin-provisioned. The UI can show ample free space while the backing array is physically out of blocks. The vmkernel log distinguishes this with an NFS-volume-specific warning rather than a VMFS Fil6: error.

flowchart TD
    A[Forgotten snapshot] --> B[Delta VMDK grows per guest write]
    B --> C[Datastore free space declines over days]
    C --> D[Datastore hits 100% full]
    D --> E[Running VMs pause]
    D --> F[Power-on fails: no .vswp space]
    D --> G[Consolidation fails: deltas locked]
    E --> H[Retry or Cancel dialog]
    H --> I[Cancel: lose RAM state]
    H --> J[Retry: resume once space is freed]

Common causes

Quick checks

# Datastore free space from the ESXi shell
esxcli storage filesystem list

# Same data via df on the VMFS volume
df -h /vmfs/volumes/<datastore-name>

# List snapshot delta files consuming the most space
ls -lahS /vmfs/volumes/<datastore-name>/*/*-delta.vmdk 2>/dev/null | head -20

# VMFS6 block pool health (look for numNotAvailable counters)
vmkfstools -P -v10 /vmfs/volumes/<datastore-name> | grep -E "File Block|numNotAvailable"

# vmkernel evidence of allocation failures
grep -i "no space" /var/log/vmkernel.log | tail -50

# Per-VM snapshot view via vim-cmd (no vCenter required)
vim-cmd vmsvc/get.snapshotinfo <vmid>

With PowerCLI:

# All snapshots across the inventory, largest first
Get-VM | Get-Snapshot | Select VM, Name, Created, SizeGB |
  Sort-Object SizeGB -Descending

# VMs needing consolidation (orphaned deltas not visible in snapshot manager)
Get-VM | Where-Object {$_.ExtensionData.Runtime.ConsolidationNeeded} |
  Select Name, PowerState

# Datastore capacity and free space
Get-Datastore | Select Name,
  @{N='CapacityGB';E={[math]::Round($_.CapacityMB/1024,1)}},
  @{N='FreeGB';E={[math]::Round($_.FreeSpaceMB/1024,1)}},
  @{N='FreePct';E={[math]::Round($_.FreeSpaceMB/$_.CapacityMB*100,1)}}

How to diagnose it

1. Confirm which datastore is full. esxcli storage filesystem list gives a host-side view. With vCenter, the Get-Datastore one-liner above with the FreePct property is faster across many datastores. Multiple datastores filling at once suggests a backup storm or a shared log writer.
2. Find the largest consumers on that datastore. SSH to the ESXi host with the datastore mounted and run ls -lahS against the *-delta.vmdk pattern. If the largest files are deltas, snapshots are the cause. If the largest files are *-flat.vmdk, thin-provisioned VMDK growth is the cause.
3. Check for orphaned snapshots that Get-Snapshot misses. Run the ConsolidationNeeded query. A backup job that crashed mid-snapshot leaves delta files locked by VMFS metadata but no entry in the snapshot tree. These are invisible to Get-Snapshot and visible only via consolidationNeeded or by direct ls.
4. Read the vmkernel log for the allocation path. VMFS errors are tagged Fil6: and reference the device ID. NFS errors reference the NFS volume name. This tells you whether you are debugging a VMFS block accounting issue or a backend storage issue.
5. If free space exists in the UI but writes still fail, check VMFS6 block pools. vmkfstools -P -v10 shows Small File Block and Large File Block counts. Thin disks consume SFBs (1 MB each); thick disks consume LFBs (512 MB each). One pool can exhaust while the other still has capacity. There was a known bug in ESXi 7.0.2 and 7.0.3 where freshly converted blocks were not immediately available for allocation; the fix shipped in 7.0 U3c.

Metrics and signals to monitor

Use a dual threshold for paging: page when free space drops below both 5% and 500 GB (the AND prevents false alarms on very large datastores where 5% may still be terabytes), or below an absolute floor of 10 GB for small datastores.

Fixes

The fastest path to recovery is freeing enough space for VMs to write again. Free at least a few GB before retrying paused VMs, not just enough to drop the percentage below a threshold. Below roughly 100 MB of additional free space, VMFS metadata operations may still fail.

Snapshot deltas (most common case)

1. Identify the largest deltas with Get-VM | Get-Snapshot | Select VM, Name, Created, SizeGB | Sort-Object SizeGB -Descending.
2. Confirm with the VM owner that the snapshot can be deleted. Snapshots older than 72 hours outside a maintenance window should not exist in production without a documented reason.
3. Delete the snapshot through the snapshot manager or Remove-Snapshot. If consolidation is needed, trigger it from the vSphere Client or via ExtensionData.ConsolidateVMDisks_Task().
4. Large consolidations stun the VM briefly and are I/O intensive. Plan a maintenance window if the delta is a meaningful fraction of the base disk.

Orphaned deltas with locked files

If consolidation fails with a lock error and delta files remain locked after the snapshot is gone from the manager:

1. Identify which ESXi host holds the lock: vmkfstools -D /vmfs/volumes/<datastore>/<vm>/<file>-delta.vmdk. Read the MAC address in the output to map to the host.
2. Either reboot the lock-holding host or unregister and re-register the VM. Both are disruptive. Open a VMware support case before taking either action in production.

Thin-provisioned VMDK growth

If the largest files are flat base disks, the guest is genuinely consuming the space. Options:

• Shrink the disk from inside the guest. OS-dependent; requires the guest filesystem to support shrink and the VMDK to not be eager-zeroed thick.
• Storage vMotion the VM to a datastore with more capacity.
• Add a second VMDK on another datastore and move data inside the guest.

Freeing space inside the guest does not reclaim VMFS space unless UNMAP is enabled. VMFS6 supports automatic background UNMAP. VMFS5 requires manual esxcli storage vmfs unmap. On thin-provisioned backend arrays, even VMFS-level UNMAP may not reach the physical backend if the array does not propagate it.

NFS backend exhaustion

If the UI shows free space but writes fail, the backing array is the problem. Engage the storage team to either thick-provision the volume on the backend or expand the LUN. Until the array frees space, file deletion on the NFS-visible layer will not help.

.vswp power-on failures

If VMs are running but new power-ons fail, check whether previously powered-off VMs left orphaned .vswp files. Deleting these frees space immediately because they are not tied to running VM state. Do not delete .vswp files for running VMs.

What does not work

• Rebooting ESXi. The filesystem state persists across reboots. This costs management availability without freeing a byte.
• Restarting management services. Same reason.
• Deleting files inside a guest. No VMFS reclaim without UNMAP.
• Freeing tiny amounts of space. A few MB may not be enough to resume VMFS metadata operations. Free GB, not KB, before retrying anything.

Prevention

• Alert on both percentage and absolute free space. Page at <5% AND <500 GB, or <10 GB on small datastores. Ticket at <15%. Plan capacity work at <25%.
• Monitor snapshot age daily. Any snapshot older than 24 hours outside a maintenance window warrants investigation. Any snapshot older than 72 hours is an incident.
• Monitor consolidationNeeded daily. Orphaned deltas are invisible to Get-Snapshot and grow silently.
• Track free-space rate of change, not just current level. A datastore losing 1 GB per day will fill. The interesting signal is the slope.
• Reserve headroom for snapshot operations. At least 20-25% free, or enough to absorb 24 hours of snapshot growth across all VMs simultaneously. Backup jobs can create snapshots across hundreds of VMs at once.
• Verify UNMAP is configured on VMFS6 datastores. Without it, thin-provisioned VMDKs grow monotonically even when guests delete data.
• Treat backup job snapshot hygiene as part of the backup SLA. Failed backup jobs are the most common source of orphaned snapshots. Monitor backup success alongside snapshot count and age.

How Netdata helps

Netdata’s value for this failure mode is correlation across the signals that precede the cliff:

• Per-second datastore free-space metrics. You see the rate of consumption, not just the current level. A datastore losing space at 5 GB per hour is visible hours before it hits 100%.
• Anomaly detection on free-space rate of change. A snapshot delta suddenly growing after weeks of flat consumption surfaces as an anomaly even when the absolute percentage is still healthy.

• Snapshot age and consolidationNeeded tracking. These are the leading indicators for the most common cause. Alerting on snapshot age >72 hours catches the time bomb before it triggers the datastore signal.
• Correlated vmkernel log evidence. When free space crosses a threshold, the same dashboard shows whether No space left on device entries have started in the vmkernel log, confirming the failure mode without a separate SSH session.
• Per-datastore breakdown. The cluster average is misleading. Netdata shows each datastore individually so the one filling is obvious.

Related guides

• vSphere active vs consumed vs granted memory: why the percentage lies
• vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty
• vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM
• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• vSphere datastore latency high: reading GAVG, DAVG, and KAVG
• vSphere storage latency cliff: the ’everything is slow’ incident that hits every VM at once
• vSphere host swapping (SWCUR/SWW/s): hypervisor swap and the memory death spiral
• How vSphere actually works in production: a mental model for operators
• vSphere memory ballooning (MCTLSZ): the host is reclaiming guest RAM
• vSphere memory compression: the reclamation tier between balloon and swap
• vSphere memory reclamation cascade: balloon to compress to swap in minutes
• vSphere monitoring checklist: the signals every host, VM, and vCenter needs