vSphere HA host isolation and split-brain: when isolation response goes wrong

vCenter shows one or more ESXi hosts as “Not Responding.” VMs on those hosts may have been restarted on surviving hosts by HA. Or they may still be running on the unreachable host with no way to manage them. In the worst case, the same VM is now running in two places at once, and nobody noticed until storage corruption or application errors surfaced.

vSphere HA uses two independent heartbeat mechanisms, network and datastore, to decide whether a host is alive. When those signals disagree, the isolation response policy determines what happens next. If the policy and the actual host state diverge, you get split-brain: two running copies of the same VM. VMFS file locking is the only guardrail against concurrent writes. On NFS, that guardrail is advisory at best.

This article covers how to tell isolation from actual host failure, what each isolation response option does, how to detect and recover from split-brain, and how to prevent the configuration mistakes that cause it.

What this means

HA classifies hosts into three states during a network event. The classification depends on which heartbeat mechanisms succeed.

flowchart TD
    A[Host loses mgmt network] --> B{Datastore heartbeat alive?}
    B -- yes --> C[Host ISOLATED]
    B -- no --> D[Host declared DEAD]
    C --> E{Isolation response?}
    E -- Leave powered on --> F[VMs stay running, unmanageable]
    E -- Shut down or Power off --> G[VMs stopped, HA restarts]
    D --> H{Host truly down?}
    H -- Yes --> I[Clean failover]
    H -- No --> J[SPLIT-BRAIN: dual VM instances]

Isolated means the host has lost network heartbeats to HA peers but is still exchanging datastore heartbeats. The host is running, its VMs are running, but it cannot be reached over the management network. HA knows the host is probably alive but cannot manage it.

Dead means both network and datastore heartbeats have stopped. HA assumes the host has failed and proceeds to restart its VMs on surviving hosts.

The isolation response policy applies only to hosts classified as isolated, not dead. The three options in vSphere 7.0 and 8.0:

  • Leave powered on (historically labeled “Disabled” in the VM isolation response dropdown): VMs on the isolated host keep running. HA does not attempt to restart them elsewhere. No duplicate VMs are created. The tradeoff is that VMs on the isolated host are unmanageable until the network is restored, and if the host is actually hung rather than network-isolated, those VMs are stuck with no recovery path.

  • Shut down: HA attempts a graceful guest shutdown of VMs on the isolated host, then restarts them on surviving hosts. The advanced option das.isolationShutdownTimeout (default 300 seconds) controls how long HA waits before forcing power off.

  • Power off: HA forcibly powers off VMs on the isolated host, then restarts them on surviving hosts. Faster than shut down but risks guest filesystem corruption.

Split-brain happens when HA declares a host dead (both heartbeats lost) and restarts its VMs, but the host is still running. This occurs when datastore connectivity is lost alongside network connectivity, or when heartbeat datastores are not properly configured. VMFS file locking prevents the restarted VM from powering on if the original instance still holds the lock on its .vmx file. NFS does not enforce equivalent mandatory locking, so both instances can write to the same VMDK simultaneously.

Common causes

CauseWhat it looks likeFirst thing to check
Management network switch failureMultiple hosts go Not Responding simultaneouslyPhysical switch health and uplink status
Management VLAN misconfigurationHosts isolated after VLAN or trunk reconfigurationVLAN tagging on management vmkernel adapter
Single management NIC, no redundancySingle host isolated, one vmnic link downesxcli network nic list for link state
Firewall blocking HA heartbeatsHosts appear isolated after firewall changeHA inter-host communication ports
NFS storage with “Leave powered on” responseVMs running on isolated host and restarted elsewhereStorage type and VMFS lock state
Missing or misconfigured heartbeat datastoresHost declared dead instead of isolatedHA datastore heartbeat configuration

Quick checks

# Check HA cluster configuration and failover level (PowerCLI)
Get-Cluster | Select Name, HAEnabled, HAAdmissionControlEnabled, @{N='CurrentFailover';E={$_.ExtensionData.Summary.CurrentFailoverLevel}}

# Check host connection states (PowerCLI)
Get-VMHost | Select Name, ConnectionState

# Check FDM (HA agent) status on a specific ESXi host (ESXi SSH)
/etc/init.d/fdm status
# <!-- TODO: verify correct FDM status command for ESXi 7.0 and 8.0 -->

# Search HA agent log for isolation or partition events (ESXi SSH)
grep -i "isolation\|partition\|split" /var/log/fdm.log | tail -50

# Check VMFS lock holder on a VM during suspected split-brain (ESXi SSH)
# Run on the .vmx file itself, not the lock file
vmkfstools -D /vmfs/volumes/<datastore>/<vm-folder>/<vm>.vmx

# List VM power state from ESXi local shell (ESXi SSH)
vim-cmd vmsvc/getallvms
vim-cmd vmsvc/power.getstate <vmid>

# Check physical NIC link state and error counters (ESXi SSH)
esxcli network nic list
esxcli network nic stats get -n vmnic0

vmkfstools -D and vim-cmd vmsvc/power.getstate are read-only and safe. Do not run vim-cmd vmsvc/power.off until you have confirmed which host holds the VMFS lock.

How to diagnose it

  1. Verify actual host state via BMC before trusting vCenter. Use the out-of-band controller (iLO, iDRAC, IMM, XCC) to check whether the isolated host is powered on and responsive at the hardware level. If the BMC shows the host running and healthy, the problem is network isolation, not host failure. Acting on vCenter’s view without BMC verification leads to wrong decisions.

  2. Distinguish isolation from failure using datastore heartbeats. If the HA heartbeat datastores show recent timestamp updates from the affected host, the host is isolated, not dead. If heartbeat files have stopped updating, the host may have genuinely failed or lost storage connectivity.

  3. Check the FDM agent log on the affected host. /var/log/fdm.log records the HA agent’s view of events. Look for explicit isolation declarations, partition events, and the reasoning behind host state transitions. This log is authoritative for what HA decided and why.

  4. Identify the network root cause. Check physical switches, VLAN configuration, and vmnic health. A single uplink failure on a host with redundant management vmnics should not cause isolation. If it does, the redundancy is misconfigured or the failover policy is wrong.

  5. If HA already restarted VMs, check for split-brain. Compare the VM power state reported by vCenter (VMs running on surviving hosts) against the actual power state on the isolated host, accessible via ESXi local shell through BMC console redirect or a separate management network. If VMs are powered on in both places, you have split-brain.

  6. Determine VMFS lock ownership. For each affected VM, check which host holds the lock on its configuration file using vmkfstools -D on the .vmx file. The host holding the lock is running the legitimate instance. The other host is running the stale duplicate.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
ESXi host connection stateDistinguishes isolated from disconnected from downnotResponding for more than 10 minutes with other hosts connected
HA cluster health and host statusShows isolation and partition events directlyHost marked isolated or partitioned for more than 5 minutes
Datastore heartbeat freshnessProves host is alive via secondary pathHeartbeat file not updating, suggesting host is dead not isolated
Management network dropped packetsEarly warning of network degradationSustained non-zero drop rate on management vmnic
VM heartbeat statusIndicates guest OS responsivenessRed heartbeat on VM that vCenter shows as running
VMFS lock stateIdentifies lock holder during split-brain recoveryLock owner differs from where vCenter believes the VM runs

Fixes

Host is isolated but still running

If the BMC confirms the host is alive and the issue is network-only, fix the network before HA takes irreversible action. With isolation response set to “Leave powered on,” VMs continue running on the isolated host but are unmanageable through vCenter until connectivity is restored. No duplicate VMs, no unnecessary restarts, no data risk.

If the isolation response is “Shut down” or “Power off” and the network blip was transient, VMs may have been needlessly restarted. For most production environments, transient management network blips are more common than genuine host failures, making “Leave powered on” the better default.

HA already restarted VMs, potential split-brain

If HA declared the host dead and restarted its VMs on surviving hosts, but the host is actually still running, follow this recovery sequence:

  1. Do not reconnect the isolated host to the management network yet. Reconnecting while both VM instances are powered on can cause storage corruption, especially on NFS.

  2. Identify which instance to kill. On VMFS, use vmkfstools -D to find the lock holder. The host that does not hold the lock has the stale copy and should be powered off first. On NFS, there is no lock-based determination. Power off the instance on the host that was incorrectly declared dead (the one that lost network heartbeats).

  3. Power off the duplicate VMs on the original host. Access the host via BMC console redirect or SSH on a separate network path. Use vim-cmd vmsvc/power.off <vmid> to force power off each duplicate instance.

  4. Verify the surviving instances are healthy before reconnecting the original host to the management network. Check guest OS state, application health, and storage I/O.

  5. Reconnect the host and verify HA agent health. After reconnection, confirm the FDM agent returns to a healthy state and the host rejoins the HA cluster properly.

Adjusting isolation response configuration

Match the isolation response to your storage type and downtime tolerance:

  • VMFS with properly configured heartbeat datastores: “Leave powered on” is safe. VMFS file locking prevents split-brain even if HA wrongly declares the host dead and attempts restarts. The SCSI reservation on the .vmx file blocks the new instance from powering on.

  • NFS: “Leave powered on” is risky. NFS does not enforce the same mandatory file locking as VMFS. If HA declares the host dead and restarts VMs while the original host is still running, both instances can write to the same VMDK simultaneously. Consider “Shut down” for NFS-backed environments, accepting the restart downtime as the cost of split-brain prevention.

  • vSAN: vSAN clusters typically rely on vSAN storage for heartbeat signals rather than traditional heartbeat datastores. This changes the isolation detection dynamic. Consult VMware documentation for vSAN-specific HA configuration guidance.

Prevention

  • Redundant management network. Use at least two vmnics for the management vmkernel adapter, connected to separate physical switches configured with active/standby or LACP teaming. A single uplink failure should never cause isolation.

  • Configure multiple isolation addresses. By default, HA pings the management network default gateway to determine isolation. If the gateway is unreachable but the rest of the network is healthy, all hosts may declare themselves isolated simultaneously. Add peer host management IPs as additional isolation addresses using the advanced settings das.isolationaddress0 through das.isolationaddress9. If your default gateway is not a reliable reachability target, set das.usedefaultisolationaddress to false.

  • Configure heartbeat datastores. Ensure HA has at least two heartbeat datastores on separate storage arrays or LUNs. Without heartbeat datastores, any network loss looks like host death.

  • Match isolation response to storage type. Use “Leave powered on” for VMFS with proper heartbeat datastores. Consider “Shut down” for NFS-backed environments where split-brain risk is higher. Document the choice and the reasoning so future operators understand the tradeoff.

  • Test HA isolation behavior. Periodically simulate a management network failure in a non-production cluster by disconnecting the management uplink. Verify that isolation is detected correctly, the response matches expectations, and VMs behave as configured.

  • Keep HA admission control enabled. If admission control is disabled, the cluster may lack resources to restart VMs after a genuine failure. A cluster showing VMs as “protected” without admission control enabled is aspirational, not guaranteed.

How Netdata helps

  • Host connection state correlation at per-second resolution. Netdata surfaces individual host availability changes alongside HA events, letting you distinguish a single-host isolation event from a vCenter-side issue affecting all hosts at once. Whether all hosts went Not Responding simultaneously (vCenter problem) or sequentially (cascading network failure) changes the entire response.

  • Management network metrics before isolation triggers. Per-vmnic packet drops, error counters, and link state changes appear at second-level granularity. Sustained drops or error rate increases on the management vmnic precede isolation declarations. Correlating these with the actual HA event in the same timeline shortens root cause identification.

  • Datastore connectivity and latency signals. Storage path health and datastore latency help determine whether heartbeat datastores remain genuinely accessible during a network event, or whether the host is losing both heartbeat paths simultaneously. This distinction determines whether HA treats the host as isolated or dead.

  • VM power state and heartbeat tracking. Unexpected VM power-off events and heartbeat status changes, overlaid with host connection state transitions, reveal whether HA restarts are occurring and whether split-brain may be in progress. A VM showing red heartbeat on a host vCenter believes is disconnected is a strong split-brain indicator.

  • Composite event timeline. Overlaying host disconnection events, network error rates, storage path state changes, and VM restart operations on a single timeline turns a confusing “vCenter says hosts are down but VMs are still running” report into a diagnosable incident with a clear sequence of events.