vSphere SCSI sense codes and path failover: intermittent fabric faults in vmkernel.log

When storage connectivity degrades, the first hard evidence usually lands in /var/log/vmkernel.log as SCSI sense codes. Latency counters (DAVG, KAVG, GAVG) tell you I/O is slow; the sense codes tell you why. ESXi logs a status tuple and optional sense data on every command that fails or is retried, and the Native Multipathing Plugin (NMP) records the resulting path state transitions.

The recurring shape of the problem is path flapping: a path cycles active, dead, active, sometimes within seconds. Each transition aborts in-flight commands, drains the retry queue on a surviving path, and surfaces as a latency spike on every VM sharing that datastore. Because the path recovers, the symptom looks transient and the latency counters average out over the 5-minute rollup window. The vmkernel log is where the pattern is actually visible.

This is a field reference for reading those lines: what the H:/D:/P: tuple means, which sense keys point at the fabric versus the array versus a lock, and how to correlate them with the multipathing and latency signals you already collect.

What this means

Every failing or retried I/O is logged in a form similar to:

H:0xNN D:0xNN P:0xNN [Valid|Possible] sense data: 0xKK 0xASC 0xASCQ

• H: (host status) comes from the VMkernel storage stack. The value that matters most is H:0x1 (NO_CONNECT): the command was sent down a path that never completed. The host side could not reach the target.
• D: (device status) is the SCSI status returned by the target. 0x0 is GOOD. Non-zero values (CHECK CONDITION, BUSY, TASK SET FULL, RESERVATION CONFLICT) come from the array or the LUN.
• P: (plugin status) reflects what NMP concluded, for example that it failed the command over to another path or queued it for retry.
• sense data appears when D: is CHECK CONDITION. Three bytes: sense key (0xKK), Additional Sense Code (0xASC), Additional Sense Code Qualifier (0xASCQ). Together they name the failure.

NMP drives each path through a small state machine: active, a policy-dependent standby or unused state, and dead. A path goes to dead when the host cannot reach the target, typically logged as H:0x1. The Path Selection Policy (PSP) then activates another working path, in-flight commands are aborted and placed on the retry queue, and if a SCSI reservation is outstanding NMP issues a LUN reset before draining the queue on the new path.

The flapping signal is repetition: the same path moving active to dead to active more than a handful of times per hour. The playbook threshold is path state changes greater than 5 per hour equals path thrashing at ticket severity. When you see that pattern, the fabric is the prime suspect, not the array.

NMP also rate-limits identical messages via nmp_ThrottleLogForDevice, so a single log line can stand in for many suppressed events. Treat any throttled line as a proxy for a burst, not a single occurrence.

flowchart TD
    A["Sense line in vmkernel.log"] --> B{"H:0x1 NO_CONNECT?"}
    B -- yes --> C["Fabric / HBA / cable fault"]
    C --> D{"Recovers, then fails again?"}
    D -- yes --> E["Intermittent fabric flap"]
    D -- no --> F["APD if all paths to LUN lost"]
    B -- no --> G{"SK 0x05 ASC 0x25?"}
    G -- yes --> H["PDL: LUN removed, stop retrying"]
    G -- no --> I{"Reservation or NOT READY?"}
    I -- reservation --> J["VMFS lock contention / PR"]
    I -- NOT READY --> K["Array overloaded / LUN busy"]

How to read a sense line

Sense keys are T10-standard and stable across ESXi 7.x and 8.x. ASC/ASCQ are mostly standard, with vendor-specific extensions. The entries below are the ones that drive most fabric and failover incidents.

Two distinctions decide the response. APD versus PDL: with APD there is no PDL sense code, so ESXi retries indefinitely; with PDL the array returns SK 0x05 ASC 0x25 and ESXi stops retrying. Some iSCSI arrays never return a proper PDL sense code, so PDL cannot be detected on them and you are stuck in APD-like retry loops. And sense key versus device status: reservation conflicts surface in the D: field, not as a sense key, which is why a “reservation conflict” line may not have a sense-key triple attached.

Common causes

Quick checks

These are read-only. Run them on the affected host over SSH or ESXi Shell.

# Recent NMP and HBA activity in vmkernel.log
grep -iE "path|NMP|HBA|sense" /var/log/vmkernel.log | tail -50

# Current path states and per-path error counts
esxcli storage core path list | grep -E "^Path|State|Errors"

# Per-path IO statistics (errors, active time)
esxcli storage core path stats list

# NMP path list per device with policy
esxcli storage nmp path list            # <!-- TODO: verify exact flags on ESXi 8.0 U3 -->
esxcli storage nmp device list

# esxtop storage view: ERR/s, ACTV, QUED, GAVG/KAVG/DAVG per device
# press 'u' for device view, 'v' for per-VM; watch ERR/s and QUED
esxtop

# ESXi 8.0 U2+: FPIN heap usage (Fabric Performance Impact Notifications)
esxcfg-info -a | grep -A3 storageFPINHeap

Interpret ERR/s as the rate of I/O errors the device path is reporting. In a healthy steady state it is zero. Any sustained non-zero value on a path means commands are failing, and the vmkernel log will explain which sense code they returned.

How to diagnose it

1. Confirm it is a path problem, not just slow storage. Pull the sense lines and sort by host status. If H:0x1 dominates, the host could not reach the target: fabric, HBA, or array port. If sense keys dominate with H:0x0, the array was reached but rejected or deferred the command: array-side issue.
2. Quantify the flapping rate. Count active -> dead transitions per path per hour. Above 5 per hour is thrashing. Below that and recovering cleanly, treat as degraded redundancy.
3. Check the blast radius. Are the sense codes limited to one LUN, one array port, one HBA, or one fabric? Single LUN points at array/PDL/reservation. Single fabric or HBA points at the fabric. All LUNs on one target port point at the array port or controller.
4. Correlate with latency and queue depth. Pull GAVG/KAVG/DAVG and QUED for the affected datastore around the same timestamps. High DAVG with sense codes means the array is the source. High KAVG with H:0x1 means the kernel is queuing because paths are dying. QUED greater than 0 sustained confirms active saturation, not transient load.
5. Check the fabric side. FC: switch port CRC errors, ISL saturation, zoning changes, SFP faults. iSCSI: session reconnection rate, switch drops, NIC ring buffer overruns. Look for the change that correlates with the first sense line timestamp.
6. On ESXi 8.0 U2+, check FPIN. FPIN events (congestion, link integrity, delivery, peer congestion) are logged in vmkernel.log. NMP can mark affected paths as flaky with a timeout. If you see StorageFPIN: 521: Failed to allocate memory or the FPIN heap near exhaustion, see the FPIN fix below.

Metrics and signals to monitor

Path health and failover events are mature baseline signals. Per-VM SCSI sense code tracking plus path-flapping detection are the expert signals most teams add only after a bad incident.

Fixes

Grouped by cause, with tradeoffs.

Intermittent fabric fault

Fix the physical layer: faulty SFP/GBIC, dirty or bent fibre, marginal switch port, oversubscribed ISL, or a zoning change that left one path alive only part-time. Verify both fabrics carry independent, working paths to every production LUN. If you run Fixed PSP, verify the preferred path is sane; Round Robin spreads load but will still ride a flapping link unless the path is administratively disabled.

FPIN heap exhaustion (ESXi 8.0 U2/U3, fixed in U3e)

When FPIN cannot allocate memory it holds reference counts on old paths, blocking the FC HBA driver from creating new paths after a fabric outage. Paths stay dead even after the fabric recovers. The supported fix is to upgrade to ESXi 8.0 U3e. Do not assume a path is permanently failed if the only error is StorageFPIN: 521.

“Awake but blind” HBA

After an abrupt FC switch crash or restart, the HBA physical link comes up but the software driver fails to re-register with the fabric. Paths remain dead despite a green link light. Recovery requires an HBA reset or a host reboot. This is disruptive: plan it, evacuate the host first.

Permanent Device Loss (PDL)

If the array confirms the LUN is gone (SK 0x05 ASC 0x25), stop retrying. Detach the datastore, unregister affected VMs cleanly, and resolve the array-side removal. Do not hold hosts in APD-style retry loops hoping the LUN returns.

Reservation conflict (VMFS lock contention)

Identify which host or operation holds the lock: backup jobs creating snapshot storms, mass consolidation, or an AV scan hitting VMDKs in parallel. Clear the contention at the source. If a Persistent Reservation is stuck, follow VMware’s PR clearing procedure rather than force-unmounting the datastore.

LUN busy / array overloaded

This is the array talking back, not the fabric. Check for a rebuild, controller failover, thin-provision exhaustion, or a noisy neighbor on the shared LUN. Reduce load (pause backups, throttle the noisy VM) and let the array catch up. Do not change multipathing policy to mask an array-side problem.

Prevention

• Forward vmkernel.log off the host into centralized logging and alert on sense-code rate, not just on the word “error.” Rate matters more than presence.
• Track path state change rate continuously. A clean failover is one transition; flapping is many.
• On ESXi 8.0 U2+, monitor FPIN heap and events. Treat FPIN congestion reports as fabric telemetry, not noise.
• Patch to U3e or later if you are on 8.0 U2/U3 to eliminate the FPIN heap recovery bug.
• Validate fabric redundancy in change reviews: dual fabrics, no single ISL, no shared single switch for both paths of any LUN.
• Review PSP policy per array vendor guidance. Round Robin can hide a marginal path by spreading load; Fixed makes a dead preferred path obvious.
• Watch the rollup cliff. Per-second capture of path state and latency catches micro-flap that 5-minute averages erase.

How Netdata helps

The value here is correlating the log-derived signal with per-second performance telemetry, which is exactly where intermittent flap hides from rollup-based tools.

• Parse vmkernel.log sense-code and FPIN events into a rate signal so H:0x1 bursts and PDL transitions surface as a metric, not a buried log line.
• Correlate path state changes with per-second DAVG/KAVG/GAVG and QUED/ACTV so a flap shows up as a synchronized latency and queue spike rather than an averaged-out blip.
• Anomaly detection on path state change rate surfaces thrashing that has not yet crossed a static 5-per-hour threshold.
• Per-second resolution is what makes the active-to-dead-to-active pattern visible; 5-minute rollups erase it by design.
• Bring storage path health into the same view as CPU ready, memory balloon, and datastore latency so a fabric flap that triggers guest-visible latency can be traced end to end.

Related guides

• vSphere datastore latency high: reading GAVG, DAVG, and KAVG
• vSphere storage latency cliff: the ’everything is slow’ incident that hits every VM at once
• How vSphere actually works in production: a mental model for operators
• vSphere monitoring checklist: the signals every host, VM, and vCenter needs
• vSphere active vs consumed vs granted memory: why the percentage lies
• vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty
• vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM
• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• vSphere host swapping (SWCUR/SWW/s): hypervisor swap and the memory death spiral
• vSphere memory ballooning (MCTLSZ): the host is reclaiming guest RAM
• vSphere memory compression: the reclamation tier between balloon and swap
• vSphere memory reclamation cascade: balloon to compress to swap in minutes