vCenter shows all hosts Disconnected / Not Responding: vCenter-side vs host-side

The vSphere Client shows many or all ESXi hosts as “Not Responding” or “Disconnected”. Do not start debugging individual hosts yet. In most mass-disconnect events, the hosts are fine and vCenter is the problem.

The decisive signal is HostSystem.runtime.connectionState across the inventory. When many hosts flip to notResponding at the same time, the cause is almost always vCenter-side: vpxd overload, a VCSA network problem, a vpxd restart, an expired certificate breaking trust, or STS/auth intermittency. Individual host failures do not fan out across the inventory in seconds.

VMs keep running while hosts are in notResponding or disconnected state. They cannot be managed, migrated, or reconfigured through vCenter, but their CPU, memory, storage, and network I/O continue independently. HA, via the FDM agents on the ESXi hosts, also continues independently of vCenter and can still restart VMs after a real host failure. The urgent task is to determine whether you have lost vCenter itself, the management network, or a single host.

What this means

Two distinct states show up in vCenter, and the difference matters operationally.

notResponding: vCenter is actively trying to reach the host and failing. The host’s heartbeat has stopped arriving within the timeout window. The host may still be up and its VMs running. If the underlying issue resolves, the host can return to connected automatically.

disconnected: vCenter or an administrator has marked the host cleanly disconnected. The host suspends vCenter-side management. A disconnected host does not return on its own; you must right-click and select Connect.

The first question in any mass-disconnect event is not “why are the hosts down” but “is this vCenter-side or host-side”.

• vCenter-side: many hosts flip simultaneously, often the entire cluster or inventory. vpxd, vpostgres, STS, the VCSA network, certificates, or NTP are the usual suspects. The hosts themselves are fine.
• Host-side: a single host, or a small subset sharing a management switch or VLAN, drops. The hosts themselves, their management NICs, hostd, or local network are the suspects.
• Flapping: hosts cycling connected to notResponding to connected repeatedly. This almost always points to STS/auth intermittency, clock skew, DNS instability, or network quality issues, not a hard host failure.

flowchart TD
  A[Many hosts Not Responding] --> B{vpxd restarted in last 15 min?}
  B -- yes --> C[Expected: wait 5-15 min for reconnect]
  B -- no --> D{vCenter SDK responds and login works?}
  D -- no --> E[vCenter-side: vpxd, vpostgres, STS, or disk full]
  D -- yes --> F{Hosts flapping or steady?}
  F -- flapping --> G[vCenter-side: STS, clock skew, DNS, cert]
  F -- steady --> H{Single host or shared switch only?}
  H -- yes --> I[Host-side: mgmt NIC, hostd, hardware]
  H -- no --> J[Network partition or VCSA-side network]

Common causes

Quick checks

# List host connection states across the inventory
Get-VMHost | Select Name, ConnectionState, PowerState

# Count hosts by state - many notResponding at once points to vCenter-side
Get-VMHost | Group-Object ConnectionState | Select Name, Count

# Check VCSA uptime and service health over SSH
uptime
/usr/lib/vmware-vmon/vmon-cli --list

# Is vCenter itself responsive? Healthy vCenter responds in well under a second
time curl -sk -o /dev/null -w "%{http_code}\n" https://localhost/sdk

# Disk pressure on the partitions that kill vpxd/vpostgres
df -h /storage/db /storage/log

# NTP / clock skew - skew over 5 minutes breaks auth and heartbeats
chronyc tracking
timedatectl status

# Machine SSL certificate dates
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT | openssl x509 -noout -dates

# Recent host connection events in vpxd.log
grep -iE "host.*connection.*state|host.*disconnect|host.*not responding" /var/log/vmware/vpxd/vpxd.log | tail -50

# On a single suspect host, check local state independent of vCenter
vim-cmd hostsvc/runtimeinfo | grep connectionState

How to diagnose it

1. Establish scope first. Count hosts by connectionState. If many or all are notResponding, your default hypothesis is vCenter-side. If exactly one host, or hosts on one physical switch, dropped, default to host-side or network-side.

2. Rule out expected behavior. Did vpxd restart, did the VCSA reboot, or is someone performing known maintenance? A vpxd restart makes every host briefly notResponding while connections are re-established. In large environments (500+ hosts) this can take 5-15 minutes. Suppress alerts during known maintenance windows and wait before acting.

3. Probe vCenter itself. Hit /sdk from the VCSA shell and time the response. A healthy vCenter responds in well under a second. Then make an authenticated request. If unauthenticated /sdk returns 200 but authenticated requests fail, the problem is STS/auth, not vpxd. If both fail, vpxd or vpostgres is down.

4. Check core services. Run vmon-cli --list and service-control --status vpxd vmware-vpostgres vmware-stsd.

vpxd down means no management. vpostgres down means complete outage. STS down means no authentication, which cascades to host disconnections as trust breaks.

5. Check VCSA disk. Run df -h on /storage/db and /storage/log. A full /storage/db crashes vpostgres. A full /storage/log crashes services that cannot write logs and triggers restart loops.

6. Check certificates. Use VECS CLI to inspect the machine SSL cert and, critically, the STS signing cert. The STS signing cert is not visible in a browser and has a separate lifecycle from the machine SSL cert. Its expiry causes bulk host disconnections because vCenter can no longer authenticate host reconnections.

7. Check clock skew. Run chronyc tracking from the VCSA. Skew greater than 5 minutes breaks Kerberos and SAML token validation. Even sub-minute skew contributes to intermittent STS failures and host flapping. Cross-check with esxcli system time get on a host.

8. Check DNS. From the VCSA, run nslookup against each affected host’s FQDN. Multiple A records for one hostname cause vCenter to resolve different IPs over time and mark hosts notResponding intermittently.

9. Use datastore heartbeats to triage. If a host’s connection is lost but its datastore heartbeats continue, the host is alive and the issue is the management network. If datastore heartbeats have also stopped, the host is likely failed and HA should be restarting its VMs.

10. Decide host-side vs network-side. If only a subset of hosts drop and they share a management switch or VLAN, suspect a network partition. SSH or use the DCUI on an affected host and vmkping the VCSA management IP. If the host can reach vCenter but vCenter cannot reach the host, check the host’s hostd service and firewall rules.

11. Read vpxd.log for the smoking gun. Mass host disconnection events generate distinct log signatures. Look for SSL or certificate errors (trust broken), OOM errors (vpxd memory exhaustion), database connectivity errors (vpostgres unreachable), or heartbeat timeout messages (network or host-side).

Metrics and signals to monitor

Fixes

vpxd restart or vCenter reboot (expected)

Wait. Do not page on mass host disconnection events that correlate with a vpxd restart, vCenter patch, or VCSA reboot. Hosts return to connected on their own within 5-15 minutes as vpxd re-establishes connections. In large environments, give it the full 15 minutes before escalating. Suppress host-connection alerts during known maintenance windows.

vpxd overload

Identify the source of load: backup jobs creating or deleting snapshots across hundreds of VMs, misbehaving SDK clients opening excessive sessions, or DRS at aggressive levels driving migration storms. Throttle or disconnect the offending client. Pause the backup job. Temporarily set DRS to manual. Do not restart vpxd unless necessary; a restart drops all in-flight tasks and requires 5-15 minutes to rebuild the inventory cache.

STS signing certificate expired

The STS signing cert is the most critical certificate in the VCSA. It is not visible in a browser and has a separate lifecycle from the machine SSL cert. Use VMware’s certificate-management tooling to renew it. Renewal typically requires service restarts and a maintenance window. Begin renewal at 60 days lead time because the process is non-trivial. The traditional checksts.py helper is referenced in older VMware KBs; verify the current recommended tool for your vCenter version before starting.

STS silent degradation (flapping hosts)

Check STS Java heap usage and GC behavior, NTP offset, and AD or LDAP connectivity. Intermittent STS failures often come from memory pressure causing GC pauses that time out some token validations, a flaky AD connection, or clock skew between VCSA and clients or hosts. Fix NTP first; it is the cheapest and most common root cause.

Clock skew

Synchronize time. Verify the VCSA’s NTP sources are reachable and the offset is under 1 second. If the VCSA syncs from AD domain controllers and AD is down, both NTP and auth break together. Cross-check ESXi host time with esxcli system time get. VCSA as a VM can fight with VMware Tools time sync if both are enabled; prefer NTP directly.

VCSA disk full

Identify which partition is full with df -h. For /storage/log, find the offending log file and truncate it. Do not delete; truncate with > so the file handle stays valid. Then address the underlying issue generating the logs. For /storage/db, check VCHA replication status (WAL accumulation), event and stats retention settings, and dead tuple bloat. Do not manually tune vPostgres settings without VMware support guidance.

Management network partition

If a subset of hosts on one switch or VLAN dropped but datastore heartbeats continue, the hosts are alive and isolated, not failed. Fix the network before HA acts on the isolation. Check switch logs, uplink status, and VLAN configuration. If HA has already restarted VMs elsewhere, verify the original host’s VMs are powered off before reconnecting to avoid split-brain.

DNS instability

Ensure each ESXi host FQDN resolves to exactly one IP from the VCSA. Multiple A records cause intermittent notResponding states as vCenter resolves different IPs over time. Clean up DNS and flush caches.

vpxd heartbeat port misconfiguration

If the configured heartbeat port in vpxd is set to anything other than the standard heartbeat port, vCenter pushes the wrong port to all hosts and every host enters notResponding on roughly a 60-second cadence. Fix the value on the VCSA, not on individual hosts. This is one of the few mass-flapping causes that sits squarely on the vCenter side.

Single host failure

For one host dropping while others stay connected, treat it as a host-side problem. Check BMC, iLO, or iDRAC for hardware status, hostd service health on the host, management NIC status, and firewall rules. A notResponding host with continuing datastore heartbeats is isolated, not dead. A notResponding host with stopped datastore heartbeats is likely failed, and HA should be restarting its VMs.

Prevention

• Track every certificate, especially STS. STS signing cert expiry is the single most common preventable vCenter outage. Alert at 30 days, plan renewal at 60.
• Monitor all VCSA partitions, not just root. /storage/db and /storage/log fill while / looks healthy.
• Suppress host-connection alerts during vpxd maintenance. Mass notResponding during a vpxd restart is expected behavior.
• Alert on flapping, not just steady disconnects. Hosts cycling connected to notResponding to connected indicate STS, clock, or DNS problems that steady-state alerts miss.
• Keep NTP healthy and redundant. A single NTP source is a single point of failure for auth and heartbeats.
• Watch vpxd CPU and task queue depth. Overload delays heartbeat processing and produces false host-disconnect symptoms.
• Correlate host disconnection with VCSA-side signals. Host connectionState is downstream of vpxd, STS, network, and cert health.

How Netdata helps

• Netdata surfaces HostSystem.runtime.connectionState across the inventory at per-second resolution, so the “many at once” signature of a vCenter-side event is immediately visible against the “one at a time” signature of a host-side failure.
• Correlating host connection state with VCSA service health (vpxd, vpostgres, STS) and VCSA disk utilization lets you confirm a vCenter-side root cause within seconds instead of jumping from host to host.
• ML anomaly detection on host connection flapping distinguishes a true mass-disconnect event from routine maintenance reconnect churn.
• Pairing connection state with NTP offset and STS or auth failure rate catches the silent clock-skew and certificate-degradation patterns that produce intermittent flapping.
• Tracking vpxd CPU, task queue depth, and SDK response time together identifies vpxd overload as the cause of delayed heartbeat processing before hosts appear disconnected.

Related guides

• vSphere active vs consumed vs granted memory: why the percentage lies
• vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty
• vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM
• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• vSphere datastore full: ‘No space left on device’, paused VMs, and power-on failures
• vSphere datastore IOPS and throughput: spotting storage saturation before latency bites
• vSphere datastore latency high: reading GAVG, DAVG, and KAVG
• vSphere dropped packets (%DRPRX/%DRPTX): ring buffers, CPU, and uplink backpressure
• vSphere DRS not balancing: affinity rules and reservations blocking placement
• vSphere DRS thrashing: vMotion churn with no stable placement
• vSphere storage latency cliff: the ’everything is slow’ incident that hits every VM at once
• vSphere HA host isolation and split-brain: when isolation response goes wrong