vSphere datastore latency high: reading GAVG, DAVG, and KAVG

High datastore latency is the single most common cause of “everything is slow” in vSphere. Applications time out, guest iowait climbs, and in severe cases VMs lose heartbeats. Storage I/O traverses guest OS, virtual SCSI adapter, VMkernel SCSI stack, storage driver, fabric, and array. A single “latency is high” reading does not tell you where the time is going.

Three counters slice that path into layers: GAVG is what the guest sees, DAVG is what the array reports, KAVG is what the VMkernel adds in between. The relationship among the three is the diagnostic.

What this means

The three counters are additive: GAVG is approximately DAVG + KAVG.

• GAVG (Guest Average) is the total latency the VM experiences. It is the number the guest OS and the application see.
• DAVG (Device Average) is the response time the physical storage device reports back to ESXi. For block storage this is the array or fabric response. For NFS the “device” is the NFS client inside the VMkernel, so DAVG includes network transit time to the NFS server.
• KAVG (Kernel Average) is the time the I/O spends inside the VMkernel storage stack: queuing before the HBA, virtualization overhead, VMFS metadata processing, NFS client processing. KAVG includes queue time, so it is not separate from GAVG.

Source counters in vCenter:

In esxtop, press u for the disk device view or v for the disk VM view, and read the DAVG/cmd, KAVG/cmd, and GAVG/cmd columns. Values are milliseconds per I/O.

Thresholds most operators use:

• All-flash arrays: DAVG under 1ms is healthy. Above 5ms indicates a problem.
• HDD arrays: DAVG baseline of 5 to 15ms. Above 20ms indicates degradation.
• KAVG: under 2ms is healthy. Above 2ms indicates VMkernel-level queuing.
• GAVG: under 10ms for general workloads, under 5ms for databases. Sustained above 30ms is severe.

Broadcom KB 344099 states that DAVG/cmd, KAVG/cmd, and GAVG/cmd should not exceed 10ms for sustained periods. Older VMware guidance was looser: GAVG above 20 to 30ms is a concern, KAVG should be near zero. The flash-era threshold is tighter.

Common causes

Quick checks

Run these on the affected ESXi host during the incident. All are read-only.

# Open esxtop in disk device view and watch DAVG/KAVG/GAVG
# Press 'u' for per-device, 'v' for per-VM
# Key columns: DAVG/cmd, KAVG/cmd, GAVG/cmd, QUED, ACTV
esxtop

# Confirm datastore type - DAVG means different things on VMFS vs NFS
esxcli storage filesystem list

# Storage path health: dead or flapping paths cause latency spikes
esxcli storage core path list | grep -E "Path|State"

# VMkernel storage errors and reservation conflicts
grep -i "scsi\|reservation\|APD\|PDL" /var/log/vmkernel.log | tail -50

# PowerCLI: snapshots on affected VMs (run from a machine with PowerCLI)
Get-VM | Get-Snapshot | Select VM, Name, Created, SizeGB

How to diagnose it

flowchart TD
    A["GAVG high
sustained above 10ms"] --> B{"DAVG high?"}
    B -->|"Yes, KAVG normal"| C["Array or network slow
problem below ESXi"]
    B -->|"No, KAVG high"| D["VMkernel queuing
queue depth or locks"]
    B -->|"Both high"| E["Cascade: array slowness
fills VMkernel queue"]
    C --> F["All VMs on datastore hit?
Contact storage team"]
    D --> G["QUED above 0?
Check queue depth, paths,
SCSI reservations"]
    E --> H["Treat array problem first
queue drains when DAVG returns"]

1. Confirm scope. Are all VMs on the datastore affected, or just one? Storage-array problems hit every VM on the datastore simultaneously. A single slow VM points at that VM’s snapshot chain, virtual disk configuration, or a local VMFS lock.

2. Read DAVG versus KAVG. High DAVG with normal KAVG means the time is being lost below ESXi: at the array, the fabric, or (for NFS) the network and NFS server. High KAVG with normal DAVG means the VMkernel itself is holding the I/O: queue saturation, VMFS metadata locks, or VAAI lock contention. When both are high, it is usually a cascade where array slowness backs up the VMkernel queue.

3. If DAVG is high, check the array and fabric. Look for controller failover (a 10 to 60 second I/O pause), RAID rebuild consuming bandwidth, a noisy neighbor on a shared LUN, or array-side throttling. Confirm with the storage team before assuming the host is at fault. Check storage path health with esxcli storage core path list: a dead path funnels all I/O through the remaining paths and produces latency indistinguishable from array congestion.

4. If KAVG is high, check the queue. In esxtop, look at QUED and ACTV. QUED above zero sustained means I/Os are backing up because the device queue is full. ACTV at or near the device queue depth limit means the device is at maximum concurrent I/O capacity. The device queue depth is configurable per HBA driver and per device (DSNRO on most drivers); defaults are often too low for modern flash arrays. Also watch for SCSI reservation conflicts in /var/log/vmkernel.log, which indicate VMFS metadata lock contention.

5. Handle NFS datastores specially. On NFS, the “device” in DAVG is the NFS client inside the VMkernel, so DAVG includes network transit time to the NFS server. A slow NFS server or congested network raises both KAVG and DAVG together, which can look like a cascade but is actually a single NFS-layer problem. Check uplink utilization on the vmnic carrying NFS traffic and the NFS server’s own load.

6. Rule out snapshot chains for single-VM cases. A deep snapshot chain forces the VMkernel to traverse multiple delta VMDKs on read, which inflates KAVG without necessarily raising DAVG. The guest OS is unaware of snapshots, so this degradation is invisible from inside the VM. Use Get-VM | Get-Snapshot to find old or deep chains. VMware has historically recommended no more than 2 to 3 snapshots in a chain and no snapshot older than 24 to 72 hours in production.

7. Beware the averaging artifact. DAVG/cmd is a simple average over the sample window. A single slow VAAI XCOPY command mixed with many fast commands can pull the average up dramatically even when 95 percent of I/Os completed quickly. If guests feel fine but DAVG looks alarming, look for VAAI offload commands in the mix and sample over a shorter window. Conversely, a backup job running VAAI-assisted clones can produce a brief, benign DAVG spike.

8. Check statistics level. At statistics level 1 (the default), per-VM disk latency counters may not be collected. If vCenter performance charts are missing per-VM latency, elevate the statistics level for the affected datastore or use esxtop for realtime visibility. Level 3 and 4 dramatically increase vCenter database growth.

Metrics and signals to monitor

Fixes

Array-side congestion (high DAVG, normal KAVG)

Coordinate with the storage team. Controller failover and RAID rebuild are usually transient and resolve on their own, but they can last minutes. For persistent congestion, move noisy-neighbor VMs to other datastores, spread IOPS across additional LUNs, or upgrade array tiering. Storage I/O Control and Storage DRS I/O Load Balancer are deprecated as of ESXi 8.0 U3, so plan for alternative approaches if you have relied on them for latency management.

VMkernel queue saturation (high KAVG, normal DAVG)

Increase the device queue depth. The setting is per HBA driver and per device (DSNRO on most drivers). Defaults are often tuned for older arrays and are too low for modern flash. Verify multipathing first: a single dead path funnels all I/O through the remaining paths and saturates their queues. Fix path health before tuning queue depth. If KAVG spikes correlate with specific operations, check for VMFS metadata lock contention (SCSI reservation conflicts in /var/log/vmkernel.log) and confirm VAAI ATS is enabled on the array.

NFS server or network slow (both KAVG and DAVG elevated)

Because DAVG on NFS includes network transit, treat this as an NFS-layer problem. Check uplink utilization on the NFS vmnic, confirm jumbo frames are consistent end-to-end, and check the NFS server’s own CPU, memory, and disk load. If the NFS server is shared, a noisy tenant on the server side can produce this pattern.

Snapshot chains (single VM, high KAVG, normal DAVG)

Consolidate the snapshot chain. Warning: consolidating large snapshots causes brief VM stuns and is I/O intensive. Schedule a maintenance window. A deep chain on a busy VM can take hours to consolidate. Prevent recurrence by monitoring snapshot age daily and investigating any snapshot older than 24 hours.

Averaging artifact (DAVG looks high, guests feel fine)

Do not chase a single high DAVG reading in isolation. Sample over a shorter window, look for VAAI XCOPY or ATS commands in the mix, and correlate with guest-observed latency. If GAVG is normal and guests are responsive, the elevated DAVG is likely an averaging artifact, not a real problem.

Prevention

• Monitor all three counters, not just GAVG. A GAVG-only alert cannot distinguish array congestion from VMkernel queuing, so the on-call engineer has no starting point.
• Alert on queue state alongside latency. A good composite alert is GAVG above 30ms sustained for more than 5 minutes AND QUED above 0 sustained. The dual condition filters out transient load that briefly spikes latency without saturating the queue.
• Baseline DAVG per storage tier. Flash and HDD have very different healthy baselines. A single threshold across tiers produces false positives on HDD and false negatives on flash.
• Track snapshot age daily. Snapshot chains are the most common preventable cause of single-VM latency degradation. Alert on any snapshot older than 72 hours or chain deeper than 3.
• Check statistics level. Level 1 may not collect per-VM disk latency counters. Level 3 and 4 collect more but bloat the vCenter database. Pick the level that actually captures the counters you alert on.
• Monitor storage path health. A single dead path halves bandwidth and can saturate the remaining path’s queue, producing high KAVG that looks like VMkernel congestion.

How Netdata helps

• Per-second latency breakdown. If you collect DAVG, KAVG, and GAVG via the vSphere integration, per-second resolution lets you see the leading indicator (DAVG rising before KAVG) instead of averaging it away in a 5-minute rollup.
• Correlate latency with queue state. Pairing GAVG with QUED and ACTV in a single view confirms whether high latency is genuine saturation or transient load.
• Tier-aware baselines. Anomaly detection learns the normal DAVG baseline per datastore, so a flash LUN drifting from 0.5ms to 3ms surfaces as anomalous even though both values sit under a naive 10ms threshold.
• Snapshot and free-space context. Correlating latency spikes with snapshot age and datastore free-space trends catches the slow-burn snapshot chain problem before it becomes an outage.
• Cross-layer correlation. Storage latency often cascades from memory pressure: host swapping writes .vswp files to the datastore, competing with VM I/O. Seeing swap rate and datastore latency together shortens diagnosis.

Related guides

• How vSphere actually works in production: a mental model for operators
• vSphere host swapping (SWCUR/SWW/s): hypervisor swap and the memory death spiral
• vSphere memory ballooning (MCTLSZ): the host is reclaiming guest RAM
• vSphere monitoring checklist: the signals every host, VM, and vCenter needs
• vSphere monitoring maturity model: from survival to expert