vSphere storage queue depth: QUED, ACTV, and DSNRO saturation

Storage latency on an otherwise healthy all-flash array is usually a queue problem, not an array problem. The VMkernel stacks I/O requests in queues at every layer of the path between the guest and the physical device. When those queues fill, requests wait, and the latency the guest sees climbs independently of what the array is actually doing.

The two numbers that tell you whether you are in this state are ACTV (in-flight I/O at the device) and QUED (I/O waiting in the VMkernel). Sustained QUED greater than zero with KAVG climbing is the signature of device queue saturation. The fix is rarely “make the array faster”; it is almost always “give the device more outstanding-I/O capacity, or stop funneling everything through a single path.”

What this means

The VMkernel storage path has a configurable queue at each layer. From the guest downward:

  1. Guest and virtual adapter queue. Per virtual disk, governed by the SCSI controller type. PVSCSI gives a much larger adapter queue than LSI Logic and is the right choice for I/O-heavy VMs.
  2. VMkernel device queue (DQLEN). Per LUN or device. This is the queue the HBA draws from.
  3. HBA or adapter queue (AQLEN). Per physical HBA. Bounded by the HBA driver’s per-LUN limit (for example, ql2xmaxqdepth on QLogic).

ACTV is the count of I/Os the VMkernel has handed to the device and is waiting on. QUED is the count of I/Os sitting in the VMkernel because the device queue is full and cannot accept more. The relationships that matter:

  • ACTV can never exceed DQLEN.
  • When ACTV equals DQLEN and there is more demand, additional I/O backs up into QUED.
  • QUED greater than zero sustained drives KAVG up, because requests are waiting in the kernel before they ever reach the device.

GAVG (what the VM sees) equals DAVG (device and array latency) plus KAVG (kernel latency, including queue wait time). If the array reports sub-millisecond latency but the VM sees tens of milliseconds, and QUED is non-zero, the bottleneck is in the ESXi stack, not below it. That split is the single most useful diagnostic when an all-flash array “feels slow.”

flowchart TD
  A[VM I/O] --> B[PVSCSI adapter queue]
  B --> C[VMkernel device queue - DQLEN]
  C --> D[ACTV: in-flight at device]
  C --> E[QUED: waiting in VMkernel]
  D --> F[HBA / physical path]
  F --> G[Storage array]
  E -. queued because ACTV == DQLEN .-> D
  E -. raises KAVG .-> H[VM sees GAVG = DAVG + KAVG]

Common causes

CauseWhat it looks likeFirst thing to check
DSNRO throttling shared LUNsQUED rises the moment a second VM issues I/O to the same LUN; per-world outstanding I/O capped at DSNRO (default 32); KAVG climbs; DAVG stays lowesxtop device view: ACTV versus QUED while adding load from a second VM on the same LUN
Dead path funnelingQUED spikes after a path failure; ACTV pegged on one path; AQLEN exhausted on the surviving adapteresxcli storage core path list for dead paths; vmkernel.log
Device queue too low for the arrayAll-flash array reports under 1 ms; VMs see 10 to 50 ms; KAVG dominant; QUED sustained above zeroCompare DQLEN to the array’s expected outstanding I/O; check HBA driver default
Slow array or controller failoverDAVG dominant; KAVG rises secondarily as the queue backs up behind slow device responsesDAVG versus KAVG split; array-side alerts
Snapshot chain read amplificationSingle VM slow, not the whole datastore; latency on reads onlySnapshot inventory on the affected VM

Quick checks

Read-only and safe during production.

# esxtop -> press 'u' -> press 'f' -> enable Queue Stats (DQLEN, ACTV, QUED, %USD)
esxtop

# DSNRO is only visible under competing-worlds load.
# Probe with at least two VMs issuing I/O to the same LUN.

# Path state - look for any path that is not 'active'
esxcli storage core path list | grep -E "Path|State"

# Per-path statistics and error counts
esxcli storage core path stats list

# Multipath policy per device (Fixed, MRU, Round Robin)
esxcli storage nmp device list

# vmkernel log for path, HBA, NMP, APD, and PDL events
grep -iE "path|NMP|HBA|APD|PDL" /var/log/vmkernel.log | tail -50

How to diagnose it

  1. Confirm the queue is the bottleneck, not the array. In esxtop device view, compare DAVG and KAVG. KAVG above 2 ms sustained with DAVG low points at the ESXi stack. KAVG near zero with high DAVG points below ESXi.
  2. Look at QUED and ACTV together. QUED greater than zero for more than about 30 seconds with ACTV pinned at DQLEN means the device queue is saturated. If QUED is zero and latency is high, the problem is elsewhere (array, IP storage network, snapshot chain).
  3. Check for DSNRO contention. DSNRO only kicks in when more than one world (VM or virtual disk) is issuing I/O to the same LUN. A single-VM LUN can use the full device queue depth; the moment a second VM touches it, each world’s outstanding I/O is capped at DSNRO, which defaults to 32. If a previously fast VM starts queueing when a second VM spins up on the same datastore, DSNRO is the throttle. Compare ACTV per world during the contention window.
  4. Check path state. esxcli storage core path list should show all expected paths as active. Any dead path means the surviving paths carry the full load, and their per-path queue depth can be exhausted even when the aggregate queue depth looks fine.
  5. Check the multipathing policy. Fixed and MRU pin a LUN to one path; Round Robin spreads I/O across active paths. A LUN on Fixed policy with a single active path is one path failure away from a queue blow-up.
  6. Confirm with the array. If DAVG is the dominant latency and KAVG is low, the queue is full because the device is slow, not because the queue is small. Raising queue depth will not help and can make the problem worse by stacking more load on a device that is already behind.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
QUED (per device)Direct measure of VMkernel queueingSustained above zero for more than 30 s
ACTV divided by DQLENHow close the device is to max concurrencyACTV regularly above 70% of DQLEN
KAVG (disk.kernelLatency.average)Latency added by the VMkernel stackAbove 2 ms indicates queueing
DAVG (disk.deviceLatency.average)Array plus transit latencyAll-flash above 5 ms is a problem
GAVG (disk.totalLatency.average)What the VM actually seesAbove 30 ms sustained, application timeouts
Path stateRedundancy and per-path saturationAny path in dead state
DQLEN valueEffective device queue depth in useDQLEN well below HBA default suggests SIOC capping or driver limits

Fixes

Raise DSNRO for shared LUNs

The most common fix on all-flash arrays with multiple VMs per datastore. DSNRO is per-LUN and online-changeable, unlike HBA queue depth, which needs a reboot.

# Set DSNRO per LUN (online, no reboot, takes effect immediately)
esxcli storage core device set -d naa.6005076300810186f00000000000001 -O 64

Caveats:

  • On vSphere 6.5 and later, DSNRO cannot exceed the device’s max queue depth; attempting to set it higher returns an error. Match the HBA default: 64 for QLogic on ESXi 5.0 through 8.0, 128 for software iSCSI.
  • The per-host Disk.SchedNumReqOutstanding advanced option still exists as a global default, but the per-LUN esxcli form above is the recommended method for granular control.
  • DSNRO has no effect when only one world is issuing I/O to the LUN. It only matters when there is contention, which is exactly when you want it higher.

Raise PVSCSI adapter queue depth for I/O-heavy VMs

For a single VM that is saturating its virtual adapter queue, the PVSCSI controller is the right lever. Default per-virtual-disk queue depth is 64 and can be raised to 254, with a corresponding increase to the request ring pages from the default of 8 to 32. Both must be set together; raising only the queue depth does not deliver the full benefit because the ring buffer becomes the limiter. This is a VM-level change that requires a guest power cycle to apply. It helps when the bottleneck is the per-virtual-disk queue, not the VMkernel device queue. If QUED is the problem, raise DSNRO first.

Fix the pathing

If dead paths are the cause, restoring them redistributes I/O and relieves the surviving paths immediately. Verify the multipathing policy: Round Robin spreads I/O across active paths; Fixed and MRU do not aggregate bandwidth on a single LUN. Pathing fixes often resolve queue saturation without any queue-depth tuning, and they should always be the first thing you check when QUED spikes correlate with a path event.

Do not raise the HBA queue depth casually

HBA driver parameters such as ql2xmaxqdepth, lpfc_lun_queue_depth, and fnic_max_qdepth require a host reboot to change, and the defaults are vendor- and driver-version-specific. Raise them only when you have measured that the HBA-level queue is the limiter and DSNRO plus pathing are already correct. Changing these values also changes behaviour for every LUN on that adapter, so the blast radius is wider than a per-LUN DSNRO change.

Prevention

  • Standardize DSNRO on shared all-flash datastores. The default 32 is a legacy spinning-disk value. For modern flash arrays shared across many VMs, set DSNRO to match the HBA default as part of the host build.
  • Monitor path state continuously, not just on alert. A single dead path on a Fixed-policy LUN is a latent queue-saturation incident.
  • Prefer Round Robin for arrays that support it. It uses available paths and reduces the blast radius of a path failure.
  • Track the ACTV-to-DQLEN ratio as a leading indicator. Sustained ACTV above 70% of DQLEN during peak is the early warning before QUED ever appears.
  • Right-size the PVSCSI ring for known I/O-heavy VMs (databases, build agents, search nodes) at provisioning time rather than reactively.
  • Watch for SIOC queue-depth changes. When Storage I/O Control is enabled it can dynamically lower the effective queue depth based on detected congestion. If you see DQLEN moving on its own, SIOC is why.

How Netdata helps

Per-second collection catches queue saturation before it becomes an incident. The signals worth correlating:

  • QUED and ACTV per device, sampled every second. Sustained QUED above zero is the direct signal; ACTV pinned at DQLEN is the precursor. Per-second resolution catches the 30-second bursts that 5-minute vCenter rollups average away.
  • KAVG versus DAVG split. Correlating kernel latency against device latency tells you whether to tune the ESXi stack (KAVG dominant) or engage the storage team (DAVG dominant).
  • Path state alongside latency. A latency spike that coincides with a path going dead is a different incident than a spike with all paths active, and the fix is different.
  • GAVG per virtual disk, joined with guest-side I/O wait. When the guest’s I/O wait tracks GAVG, you have end-to-end confirmation that the queue is the user-visible bottleneck.
  • DQLEN drift detection. If DQLEN changes without an operator action, SIOC or a path event is reshaping the effective queue. Trending DQLEN over time surfaces this before QUED does.