vSphere storage latency cliff: the ’everything is slow’ incident that hits every VM at once

Every VM on a datastore degrades at the same moment. Application pages fire across unrelated services. The root cause is one shared resource below ESXi. This is the storage latency cliff.

The pattern is distinctive once you have seen it. Latency on the affected datastore jumps from a few milliseconds to tens or hundreds of milliseconds. Queue depth climbs above zero. Every workload on that datastore suffers, regardless of which host it runs on. Guest operating systems report high I/O wait, but CPU and memory are fine. Applications report timeouts, but the application itself is healthy.

The diagnostic question is not “what is slow” but “where in the I/O path is the slowdown.” vSphere exposes three latency counters that split the path into segments. Reading them in the right order tells you whether to escalate to the storage team, the network team, or nobody. The leading indicator is DAVG. When DAVG spikes, the array is the cause. When only KAVG spikes, the VMkernel is the cause.

What this means

vSphere splits storage latency into three measurements along the I/O path:

  • DAVG (Device Average). Latency at the physical storage device, including network transit for IP-based storage. When DAVG rises, the problem is below ESXi.
  • KAVG (Kernel Average). Latency added by the VMkernel: queuing time plus virtualization overhead. Should be near zero. Sustained above 2ms means the VMkernel is queuing I/O.
  • GAVG (Guest Average). Total latency as seen by the VM. GAVG = DAVG + KAVG. This is the user-impact metric.

The cascade looks like this:

flowchart TD
    A[Array slows: failover, rebuild, noisy neighbor] --> B[DAVG spikes from ~5ms to 50-500ms]
    B --> C[Device queue fills: ACTV near queue depth limit]
    C --> D[I/O backs up in VMkernel: QUED greater than 0 sustained]
    D --> E[KAVG rises as queued I/O waits]
    E --> F[GAVG spikes on every VM on the datastore]
    F --> G[Guest I/O wait, application timeouts]
    G --> H[Possible SCSI reservation timeouts, heartbeat loss]

The pattern affects every VM on the datastore simultaneously. That is the tell that separates it from CPU or memory contention, which is per-host. If only one VM is slow on an otherwise healthy datastore, the problem is that VM’s snapshot chain, not the array.

Healthy baselines depend on the storage tier. All-flash arrays should show DAVG below 1ms; anything above 5ms indicates a problem. HDD-based arrays run 5-15ms baseline, with degradation above 20ms. KAVG should stay below 2ms. QUED should be zero in steady state.

Common causes

CauseWhat it looks likeFirst thing to check
Array controller failoverDAVG spikes for 10-60 seconds, then recovers. Often scheduled or triggered by a path event.Storage array management console for failover events in the same window.
RAID rebuildDAVG climbs and stays elevated for hours or days. Rebuild consumes I/O bandwidth.Array management for active rebuild operations on the pool or disk group.
Noisy neighbor on shared LUNDAVG rises during specific workload windows. One VM or host shows disproportionate IOPS.esxtop per-VM disk view sorted by IOPS to find the heavy consumer.
Snapshot chain depthGAVG elevated on specific VMs with deep snapshot chains. DAVG may be normal.Snapshot inventory on the affected VMs. Single-VM latency is almost always snapshots.
NFS server exhaustion or network congestionDAVG high on NFS datastores. May correlate with uplink utilization.Physical uplink utilization and NFS server load.

Quick checks

These are safe, read-only checks. Run them on a host connected to the affected datastore.

# esxtop storage device view: look at DAVG, KAVG, GAVG, ACTV, QUED per device
# Press 'u' for device view, 'v' for per-VM view
esxtop
# Inside esxtop: press 'u', then look for the affected naa.<id> device

# Check datastore free space and type
esxcli storage filesystem list

# List snapshots on a specific VM folder on the datastore
ls /vmfs/volumes/<datastore-name>/<vm-folder>/*-delta.vmdk
# Note: VMFS-6 may use *-sesparse.vmdk for large disk snapshots

# Check storage path state for the affected device
esxcli storage core path list | grep -E "Path|State"
<!-- TODO: verify whether 'esxcli storage core path stats list' exists as a subcommand on ESXi 7/8 -->

For inventory-wide snapshot checks from a machine with PowerCLI:

# Find all snapshots with age and size
Get-VM | Get-Snapshot | Select VM, Name, Created, SizeGB

# Find VMs with orphaned delta disks requiring consolidation
Get-VM | Where-Object {$_.ExtensionData.Runtime.ConsolidationNeeded} | Select Name

How to diagnose it

  1. Confirm the scope. Is every VM on the datastore slow, or just one? Open esxtop, press v for the per-VM disk view, and look at GAVG across VMs on the datastore. If only one VM shows elevated latency, skip to step 6. The problem is local to that VM.

  2. Read DAVG versus KAVG. In esxtop device view (press u), find the affected device and compare DAVG and KAVG.

    • High DAVG, normal KAVG: the array or storage network is the bottleneck. Problem is below ESXi.
    • High KAVG, normal DAVG: the VMkernel is queuing I/O. Look for queue depth saturation, SCSI reservation conflicts, or VMFS metadata locking.
    • Both high: usually a cascade. Array slowness fills the queue, which raises KAVG.

  3. Check queue depth. In the same esxtop view, look at ACTV and QUED for the device. ACTV near the device queue depth limit means the device is at maximum concurrent I/O. QUED above zero sustained means I/O is backing up in the VMkernel. Sustained QUED above zero is the direct cause of high KAVG.

  4. Correlate with IOPS. High latency with low IOPS points to a device problem. High latency with high IOPS points to saturation. Compare current IOPS against what the array is rated for.

  5. Check the array. If DAVG is the leading indicator, the answer is in the storage layer. Look for controller failover, RAID rebuild, replication jobs, cache miss storms, or a noisy neighbor on the shared pool. Hand the timestamp and device ID to the storage team.

  6. If single VM: check snapshots. A deep snapshot chain inflates read latency because every read may traverse the chain. Use Get-Snapshot or look for *-delta.vmdk (or *-sesparse.vmdk) files in the VM’s folder. Also check consolidationNeeded, which catches orphaned deltas that the snapshot manager may not display.

  7. Check path health. A dead path can force all I/O through one remaining link. esxcli storage core path list shows path state. Path thrashing, where paths flap up and down, produces latency spikes visible in DAVG.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
DAVG per deviceLeading indicator for array-side problems. Rises before KAVG.Sustained above 5ms on all-flash, above 20ms on HDD.
KAVG per deviceShows VMkernel queuing. Should be near zero.Sustained above 2ms.
GAVG per deviceTotal latency the guest sees. The user-impact metric.Sustained above 30ms with QUED above zero is a page.
QUED per deviceConfirms active queue saturation, not transient load.Any sustained non-zero value.
ACTV per deviceShows how close the device is to its queue depth limit.Regularly above 70% of the configured device queue depth.
Snapshot age and chain depthSingle-VM latency is almost always snapshots.Any snapshot older than 72 hours or chain deeper than 3.
Datastore free spaceA full datastore halts all VMs on it. Snapshot growth is the usual cause.Below 15% free, or declining rate that correlates with a specific VM.
Storage path stateDead paths reduce bandwidth and cause failover latency.Any path in dead state, or path state changes more than 5 per hour.

Fixes

Array-side slowdown (high DAVG)

The fix is in the storage layer. Coordinate with the storage team. Common actions: wait out a controller failover, throttle or reschedule a rebuild, move a noisy neighbor to a different pool, or add cache. Do not mask array slowness by increasing ESXi queue depth. If the array cannot service the I/O, larger queues just hold more waiting I/O and raise KAVG further.

VMkernel queuing (high KAVG, normal DAVG)

Check the device queue depth limit. Some HBA driver defaults are low for modern flash arrays, and the per-device queue depth (Disk.SchedNumReqOutstanding) can be tuned. Verify against your storage vendor’s guidance before changing defaults. Also look for SCSI reservation conflicts or VMFS metadata locking, which show up as KAVG spikes during operations that take metadata locks, such as snapshot consolidation or LUN extension.

Snapshot chain depth

Consolidate snapshots during a maintenance window. Deleting a large snapshot delta is I/O-intensive and briefly stuns the VM. Plan for the stun. If a backup job left orphaned snapshots, fix the backup job cleanup logic before consolidating, or the problem returns on the next run. Use consolidationNeeded to catch deltas that the snapshot manager does not display.

Path failure

If a path is dead, investigate the physical layer: cable, HBA, switch port, array port. A single dead path on a multipath config may be silent to VMs but you have lost redundancy. If paths are thrashing, check the multipath policy. Round Robin, Fixed, and MRU behave differently during failures.

Datastore nearing full

Free space on the datastore immediately. The fastest action is usually consolidating the largest snapshots. Thin-provisioned VMDKs only grow; VMFS does not reclaim space when the guest deletes files unless UNMAP is configured. On VMFS-6, automatic UNMAP runs in the background at low priority. On VMFS-5, UNMAP is manual.

Prevention

  • Monitor DAVG and KAVG independently. Aggregate “datastore latency” hides which layer is the problem. A team that sees high latency and blames the array will miss KAVG-driven queuing.
  • Monitor QUED alongside latency. High latency with low queue depth suggests a different root cause than high latency with full queues.
  • Track snapshot age daily. Snapshots are the most common preventable vSphere incident. Alert on any snapshot older than 72 hours or chain deeper than 3. Track consolidationNeeded separately because it catches orphaned deltas.
  • Track datastore free space as both percentage and absolute. A terabyte of free space on a large datastore may still be 5%. Use both conditions.
  • Baseline your storage tiers. All-flash DAVG should be under 1ms. Know what normal looks like before an incident so the deviation is obvious.
  • Review multipath policy with your storage vendor. Defaults are not always optimal for modern arrays. Document the agreed policy so path events are interpreted correctly.

How Netdata helps

  • Per-second DAVG, KAVG, and GAVG per device let you see which latency layer moves first. DAVG rising before KAVG confirms an array-side problem within seconds, not minutes.
  • QUED and ACTV alongside latency turn a latency spike into a confirmed queue saturation event. A composite page condition (GAVG above 30ms and QUED above zero) filters out transient load that does not indicate a real incident.
  • Per-VM disk latency surfaces the single-VM case. When only one VM shows elevated GAVG on an otherwise healthy datastore, the correlation points directly at that VM’s snapshot chain.
  • Snapshot age and consolidationNeeded as separate signals catch both old snapshots and orphaned deltas before they fill the datastore.
  • Datastore free space with both percentage and absolute thresholds prevents false positives on large datastores while still catching small ones.
  • Anomaly detection on DAVG baselines catches slow drift and sudden cliffs against the per-device normal, which matters because healthy baselines differ between all-flash and HDD tiers.