VMware vSphere Operations Guides on Netdata

How vSphere actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How vSphere actually works in production: a mental model for operators

vSphere is a layered virtualization stack with three interdependent planes, and most production incidents cross plane boundaries. A guest that “feels slow” may be starved at the hypervisor scheduler, throttled by a forgotten CPU limit, fighting for IOPS at the storage layer, or sitting behind a vCenter whose database has bloated to the point that DRS stopped rebalancing the cluster.

vCenter '503 Service Unavailable': the vSphere Client will not load

Mon, 01 Jan 0001 00:00:00 +0000

vCenter ‘503 Service Unavailable’: the vSphere Client will not load

A 503 from the vSphere Client means the reverse HTTP proxy (rhttpproxy) accepted the TLS connection but could not reach the backend it routes to. The proxy itself is healthy. One of its dependents, typically vpxd, vmware-vapi-endpoint, vmware-stsd (STS), or the HTML5 client backend (vsphere-ui), is stopped, still starting, or crash-looping. The error string often reads “Initialization of one of the components failed.”

vCenter 'Cannot complete login due to an incorrect user name or password': SSO failures

Mon, 01 Jan 0001 00:00:00 +0000

The “Cannot complete login due to an incorrect user name or password” string is the exact message operators see in the vSphere Client, in PowerCLI sessions, and in API responses when SSO authentication fails. The text is misleading: the cause is rarely a typo. For a single user it is usually a credential or permission problem. For every account at once it is an SSO/STS infrastructure failure.

vCenter /storage/db full: vPostgres stops and the whole management plane dies

Mon, 01 Jan 0001 00:00:00 +0000

vCenter /storage/db full: vPostgres stops and the whole management plane dies

The vSphere Client returns 503 Service Unavailable. PowerCLI sessions hang and time out. DRS has stopped evaluating, vMotion orchestration is gone, and provisioning fails. Running VMs on the ESXi hosts continue to operate, but the management plane is gone.

The root cause is almost certainly the /storage/db partition on the vCenter Server Appliance (VCSA). This is where vPostgres keeps its data files. At 95% utilization on any partition, VMware automatically shuts down vmware-vpxd to protect the database from corruption. At 100%, vPostgres cannot extend a data file or write a WAL record and crashes. Once vPostgres is down, vpxd has no database and cannot restart.

vCenter /storage/log full: the log-bomb disk death spiral

Mon, 01 Jan 0001 00:00:00 +0000

vCenter /storage/log full: the log-bomb disk death spiral

You log in to the vSphere Client and get a 503, or the UI hangs mid-task. You SSH into the VCSA and run df -h: /storage/log is at 100%. The root filesystem may still have plenty of free space, which is why a generic disk-space alert missed it. The VCSA has many dedicated partitions, and they fill independently.

A single failing service can write gigabytes of logs per hour. STS authentication failures, database connection errors, alarm flapping, or a misbehaving SDK client flooding vpxd with errors will take /storage/log from 40% to 100% within hours. Once the partition is full, services that try to log crash. vmon restarts them. The restart itself generates more log lines as the service hits the same fault and tries to log it again. The loop is self-reinforcing, and clearing space temporarily makes the next iteration worse because the service can write again, refilling the partition faster.

vCenter /storage/seat full: stats, events, alarms, and tasks outgrowing their partition

Mon, 01 Jan 0001 00:00:00 +0000

vCenter /storage/seat full: stats, events, alarms, and tasks outgrowing their partition

The /storage/seat partition on the vCenter Server Appliance (VCSA) holds the vPostgres tables for Stats, Events, Alarms, and Tasks: vpx_event, vpx_event_arg, vpx_task, and the vpxd_hist_stat* rollup tables. In modern VCSA it is a dedicated mount, so it can fill while /storage/db, /storage/log, and / all show healthy utilization. Operators checking only / or the VAMI dashboard’s “VCDB” usage will miss it until vpxd refuses to start.

vCenter appliance undersized: inventory outgrowing the deployment size

Mon, 01 Jan 0001 00:00:00 +0000

vCenter appliance undersized: inventory outgrowing the deployment size

A VCSA deployed as “Small” three years ago is not the same as a fresh “Small” today if your inventory has grown. The VCSA’s resource footprint is not fixed by CPU and RAM alone. It is dominated by the size of the in-memory inventory cache that vpxd maintains, and that cache grows with managed-object count: VMs, hosts, datastores, dvSwitch portgroups, tags, alarms, permissions, and snapshots.

vCenter certificate expired: the STS signing cert outage nobody saw coming

Mon, 01 Jan 0001 00:00:00 +0000

vCenter certificate expired: the STS signing cert outage nobody saw coming

vCenter is down. Not “slow” or “degraded.” Down. The vSphere Client shows a white screen or a 503. PowerCLI sessions fail to connect. API calls return authentication errors. ESXi hosts show as disconnected in bulk. Every integration that depends on vCenter (NSX, vRA, SRM, backup products) has lost connectivity simultaneously. VMs on the hosts are still running, but you cannot manage, migrate, or orchestrate anything.

vCenter clock skew: the NTP offset that breaks tokens and disconnects hosts

Mon, 01 Jan 0001 00:00:00 +0000

vCenter clock skew: the NTP offset that breaks tokens and disconnects hosts

Sudden SSO login failures. ESXi hosts flipping to “Not Responding” without a network cause. Certificate validation errors on certificates you know are valid. All three point to clock skew between vCenter, ESXi, and the identity infrastructure they depend on.

SAML token validation, certificate validation, Kerberos, HA heartbeats, and log correlation all assume clocks agree. When they diverge, the failures look like unrelated problems instead of one root cause. Nobody thinks to check the clock.

vCenter CPU and memory pressure: vpxd heap, swap, and the undersized appliance

Mon, 01 Jan 0001 00:00:00 +0000

vCenter CPU and memory pressure: vpxd heap, swap, and the undersized appliance

vCenter slows down before it fails. The vSphere Client takes 30 seconds to load a VM summary, API calls time out from backup and monitoring tools, hosts flap to “not responding” even though they are healthy, and DRS recommendations stop appearing. Inside the appliance, top shows vpxd consuming most of the memory and a small but non-zero amount of swap. From the hypervisor, the VCSA VM either looks busy or, worse, looks idle while sitting at 50% CPU ready time.

vCenter database bloat: SEAT tables, statistics level, and the failed purge job

Mon, 01 Jan 0001 00:00:00 +0000

vCenter database bloat: SEAT tables, statistics level, and the failed purge job

vCenter Server’s embedded PostgreSQL database (vPostgres) grows continuously. Under normal conditions, internal purge jobs keep the SEAT tables (Stats, Events, Alarms, Tasks) bounded by the configured retention windows. When retention is misconfigured, the statistics level is too high, or the purge job stops running, those tables grow without bound. The result is a slow vCenter, a filling /storage/seat or /storage/db partition, and eventually a vpxd crash when the database can no longer write.

vCenter HA (VCHA) replication broken: protection that is not protecting

Mon, 01 Jan 0001 00:00:00 +0000

vCenter HA (VCHA) replication broken: protection that is not protecting

VCHA makes vCenter look protected. The VAMI dashboard shows a green cluster, the active node serves traffic, and the passive node exists as a standby. When PostgreSQL streaming replication between the active and passive node stops, the passive node holds a stale database. A failover loses every transaction written since replication broke.

VCHA’s health surface is shallow. The VAMI summary can report healthy while pg_stat_replication shows NOT_REPLICATING. The passive VM is powered on. The witness is unreachable, so automated failover cannot reach quorum. None of this surfaces until an operator triggers failover and discovers the passive is hours behind, or until WAL accumulation on the active fills /storage/db and vpxd stops.

vCenter service will not start: vmon dependency order and the max-restart wall

Mon, 01 Jan 0001 00:00:00 +0000

vCenter service will not start: vmon dependency order and the max-restart wall

When vCenter services refuse to start, the failure is rarely the service you see stuck STOPPED. vmon (the VMware service lifecycle manager) starts each child service in dependency order and supervises it with a per-service restart policy. If a low-level dependency like vPostgres or STS is down or slow, every service layered on top of it fails its health check, retries within vmon’s bounded budget, and then silently stops trying.

vCenter shows all hosts Disconnected / Not Responding: vCenter-side vs host-side

Mon, 01 Jan 0001 00:00:00 +0000

The vSphere Client shows many or all ESXi hosts as “Not Responding” or “Disconnected”. Do not start debugging individual hosts yet. In most mass-disconnect events, the hosts are fine and vCenter is the problem.

The decisive signal is HostSystem.runtime.connectionState across the inventory. When many hosts flip to notResponding at the same time, the cause is almost always vCenter-side: vpxd overload, a VCSA network problem, a vpxd restart, an expired certificate breaking trust, or STS/auth intermittency. Individual host failures do not fan out across the inventory in seconds.

vCenter slow and unresponsive: vpxd overload and the task queue backlog

Mon, 01 Jan 0001 00:00:00 +0000

vCenter slow and unresponsive: vpxd overload and the task queue backlog

vCenter is sluggish. The vSphere Client hangs on login, PowerCLI calls time out, and tasks that normally finish in seconds sit in Running for minutes. ESXi hosts start flipping to Not Responding even though the hosts themselves are healthy and VMs keep running. This is the vpxd overload cascade, and it is one of the most commonly misdiagnosed vCenter incidents.

vCenter SSO intermittently failing: STS heap, flaky AD, and near-expiry certs

Mon, 01 Jan 0001 00:00:00 +0000

vCenter SSO intermittently failing: STS heap, flaky AD, and near-expiry certs

Users report “vCenter logged me out randomly.” PowerCLI jobs fail with a token validation error, then succeed on the second attempt. One host shows “Not Responding” for 90 seconds, then comes back. Ten minutes later a different host does the same. The web client sometimes loads, sometimes returns a 503. STS itself sits at “Started/Green” in vmon the whole time.

vCenter vPostgres 'too many clients already': connection exhaustion

Mon, 01 Jan 0001 00:00:00 +0000

vCenter vPostgres ’too many clients already’: connection exhaustion

vCenter operations time out. Power-on tasks hang in “Running” state. The vSphere Client is sluggish or fails to load. PowerCLI sessions return errors. Backup jobs fail mid-run. SSH into the VCSA and tail the vpxd or vPostgres logs, and you find the string that anchors the incident: FATAL: sorry, too many clients already.

The embedded vPostgres database is rejecting new connections because it has reached max_connections. VMware tunes this value per deployment size, and a tuning script rewrites it based on the appliance’s memory allocation. At the limit, anything needing a database connection fails immediately. This is a cliff-edge failure, not graceful degradation.

vCenter vPostgres autovacuum and WAL: dead tuples and an unbounded pg_wal

Mon, 01 Jan 0001 00:00:00 +0000

The embedded PostgreSQL database in vCenter Server Appliance (vPostgres) stores every managed object, event, task, alarm, and performance stat in the environment. Two internal mechanisms keep that database from consuming its own disk: autovacuum, which reclaims space from deleted and updated rows, and WAL (Write-Ahead Log) management, which controls how transaction logs are written, checkpointed, and recycled.

When either mechanism falls behind, the symptoms are subtle at first. Queries slow down. vpxd task execution takes longer. The /storage/db partition creeps upward. By the time an outside-in probe notices (vSphere Client timing out, vpxd crashing, services refusing to start), the database has been degraded for 30-60 minutes.

vCenter vpxd crash loop: the core service that keeps restarting

Mon, 01 Jan 0001 00:00:00 +0000

vCenter vpxd crash loop: the core service that keeps restarting

vpxd is the C++ core of vCenter Server. It holds the entire managed inventory in memory, dispatches every management task to ESXi hosts via hostd, runs DRS, executes statistics rollups, and serves every SDK client (vSphere Client, PowerCLI, Veeam, NSX Manager, Aria Operations, custom automation). When vpxd dies, vCenter is functionally down: no provisioning, no vMotion orchestration, no DRS, no HA reconfiguration. VMs already running on hosts keep running, and FDM still restarts them after a host failure, because HA does not depend on vpxd.

vSphere 'Virtual machine disks consolidation is needed': clearing the warning without stunning the VM

Mon, 01 Jan 0001 00:00:00 +0000

vSphere ‘Virtual machine disks consolidation is needed’: clearing the warning without stunning the VM

The “Virtual machine disks consolidation is needed” warning means the VMkernel left delta VMDKs on the datastore after a snapshot delete that did not fully commit. The VM is still running, but its writes are going through delta files that were never meant to persist.

The warning is set by VirtualMachine.runtime.consolidationNeeded in the vCenter inventory. It is distinct from the Snapshot Manager view: a VM can have this flag set while showing zero snapshots in the manager, because the flag tracks orphaned files on the datastore, not the snapshot tree.

vSphere active vs consumed vs granted memory: why the percentage lies

Mon, 01 Jan 0001 00:00:00 +0000

vSphere active vs consumed vs granted memory: why the percentage lies

The “Memory Usage” percentage on a vSphere host summary is one of the most misread signals in infrastructure monitoring. An 85% number that pages you at 3 a.m. may represent a healthy host with no reclamation at all. The same number on a different host may mean VMs are being actively swapped to disk. The percentage alone tells you nothing useful about either state.

vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty

Mon, 01 Jan 0001 00:00:00 +0000

vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty

%CSTP in esxtop is the time a vCPU in a multi-vCPU VM sits halted because the ESXi scheduler is waiting to co-schedule the VM’s other vCPUs. In a healthy environment it is essentially zero. Sustained above a few percent on modern ESXi means a sizing or topology problem, not a performance problem you can tune away.

The classic shape: you give a database 16 vCPUs and it gets slower. The guest OS reports low CPU utilization because the vCPUs are not doing work. They are parked in COSTOP waiting for their siblings. From inside the VM this is invisible. The application runs slowly while the OS reports idle capacity.

vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM

Mon, 01 Jan 0001 00:00:00 +0000

vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM

A VM is slow. The application team reports degraded throughput. You check the usual suspects: guest CPU utilization is high, but that is expected for a busy workload. Host CPU utilization is moderate, nowhere near saturated. %RDY, the standard vSphere CPU contention signal, is low. Everything looks healthy from the hypervisor’s perspective, yet the VM is underperforming.

vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle

Mon, 01 Jan 0001 00:00:00 +0000

vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle

A database VM takes twice as long to run its nightly batch. Application latency pings fire. You SSH into the guest, run top, and CPU utilization sits at 25%. Memory is fine. Disk I/O looks normal. Nothing inside the VM explains the slowdown.

This is the classic signature of CPU ready time in vSphere. The guest OS has no visibility into hypervisor scheduling decisions. When the ESXi CPU scheduler cannot find a free physical CPU for a runnable vCPU, the vCPU waits in the READY state. The guest never learns it was descheduled, so from inside the VM everything looks idle while the hypervisor sees a starved VM.

vSphere datastore full: 'No space left on device', paused VMs, and power-on failures

Mon, 01 Jan 0001 00:00:00 +0000

vSphere datastore full: ‘No space left on device’, paused VMs, and power-on failures

A vSphere datastore hitting 100% is a cliff-edge failure. Below 100%, VM performance is unaffected. At 100%, every VM that needs to write to the datastore stops: running VMs pause with the “There is no more space for virtual disk” dialog, thin-provisioned VMDKs cannot extend, snapshot deltas cannot grow, and power-on operations fail because the per-VM .vswp swap file cannot be created.

vSphere datastore IOPS and throughput: spotting storage saturation before latency bites

Mon, 01 Jan 0001 00:00:00 +0000

vSphere datastore IOPS and throughput: spotting storage saturation before latency bites

vSphere exposes IOPS and throughput counters at the datastore and virtual-disk level, but most teams only look at storage metrics after VMs are already slow. By the time DAVG or GAVG spike, the device queue is saturated and every VM on the datastore is paying for it. IOPS and throughput are leading indicators: they tell you how hard you are pushing the backend before the backend pushes back.

vSphere datastore latency high: reading GAVG, DAVG, and KAVG

Mon, 01 Jan 0001 00:00:00 +0000

vSphere datastore latency high: reading GAVG, DAVG, and KAVG

High datastore latency is the single most common cause of “everything is slow” in vSphere. Applications time out, guest iowait climbs, and in severe cases VMs lose heartbeats. Storage I/O traverses guest OS, virtual SCSI adapter, VMkernel SCSI stack, storage driver, fabric, and array. A single “latency is high” reading does not tell you where the time is going.

Three counters slice that path into layers: GAVG is what the guest sees, DAVG is what the array reports, KAVG is what the VMkernel adds in between. The relationship among the three is the diagnostic.

vSphere dropped packets (%DRPRX/%DRPTX): ring buffers, CPU, and uplink backpressure

Mon, 01 Jan 0001 00:00:00 +0000

vSphere dropped packets (%DRPRX/%DRPTX): ring buffers, CPU, and uplink backpressure

%DRPRX and %DRPTX in esxtop are usually the first sign a VM is losing packets inside the host. They should be zero at steady state. When they are not, the guest retransmits, latency climbs, and for latency-sensitive workloads (IP-based storage, databases, replicated queues) the impact can be severe well before the drop rate looks alarming.

The distinction that matters: %DRPRX and %DRPTX count drops at the virtual switch port, between the vSwitch and the guest OS driver. They are not physical NIC drops. The uplink vmnic can report zero drops via esxcli network nic stats get while %DRPRX is non-zero on the VM attached to it. Treating them as the same counter is the most common diagnostic mistake.

vSphere DRS not balancing: affinity rules and reservations blocking placement

Mon, 01 Jan 0001 00:00:00 +0000

vSphere DRS not balancing: affinity rules and reservations blocking placement

DRS is enabled and Fully Automated, yet one host sits at 90% CPU while another idles at 30%. Vmotions are not happening, the recommendations queue is empty or full of unapplied entries, and the DRS score is poor. In most cases DRS is doing exactly what it was told: it cannot find a migration that satisfies every constraint.

DRS gates every placement decision behind a chain of compatibility and capacity checks before it compares hosts by load. A single VM-Host anti-affinity rule, a reservation that fully commits a host’s CPU or memory, or a missing vMotion network can each silently zero out the set of legal destinations. When every candidate fails one check, DRS emits no recommendation and the cluster drifts. Cluster averages hide this: a cluster “averaging 60% CPU” can mask one host at 90% and another at 30%. The per-host skew that DRS is supposed to fix is invisible in aggregate, and so is the constraint that prevents the fix.

vSphere DRS thrashing: vMotion churn with no stable placement

Mon, 01 Jan 0001 00:00:00 +0000

vSphere DRS thrashing: vMotion churn with no stable placement

DRS thrashing is when the Distributed Resource Scheduler cannot find stable placement and keeps migrating the same VMs between hosts. Each migration costs vMotion bandwidth and inflicts a brief stun on the VM. The cluster looks balanced on paper, but the migrations never stop.

This is almost always the DRS cost-benefit model oscillating because something underneath is unstable: bursty CPU or memory pressure, an aggressive migration threshold, conflicting affinity rules, or capacity too tight to ever be balanced.

vSphere ESXi hardware health: ECC errors, fan failure, and thermal throttling

Mon, 01 Jan 0001 00:00:00 +0000

vSphere ESXi hardware health: ECC errors, fan failure, and thermal throttling

Hardware faults do not surface through hypervisor performance counters. They appear either as a Purple Screen of Death (PSOD) that kills every VM on the host instantly, or as slow degradation that looks like software misconfiguration until someone checks temperature sensors. Every DIMM error, failed fan, and degraded disk is the hypervisor’s problem.

Three failure classes dominate production incidents on ESXi: ECC memory errors that progress from correctable to uncorrectable, fan failures that trigger thermal throttling and silently cap CPU frequency, and predictive disk failures that kick off RAID rebuilds consuming storage I/O. All three share a common operational trap: without the vendor CIM provider VIB installed, most sensors report “unknown” or are absent entirely, leaving you blind to the root cause.

vSphere forgotten snapshot growing: the delta VMDK time bomb

Mon, 01 Jan 0001 00:00:00 +0000

vSphere forgotten snapshot growing: the delta VMDK time bomb

A VM has a snapshot that should have been deleted days ago. The delta VMDK grows with every guest write. The datastore slowly fills. The guest has no idea anything is wrong. By the time someone notices, consolidation is a multi-hour, I/O-intensive operation that stuns the VM, and the datastore is hours from full. Forgotten snapshots are one of the most common preventable vSphere incidents.

vSphere HA 'Insufficient resources to satisfy configured failover level': admission control

Mon, 01 Jan 0001 00:00:00 +0000

vSphere HA ‘Insufficient resources to satisfy configured failover level’: admission control

The error “Insufficient resources to satisfy configured failover level for vSphere HA” is admission control refusing a VM power-on, vMotion, or reservation change because granting it would leave the cluster without enough spare capacity to honor the configured HA failover policy. Admission control is doing its job: protecting the restart guarantee after a host failure.

The cluster may physically hold more capacity than admission control lets you commit. A cluster with 500 GHz of CPU and 2 TB of RAM may only let you deploy against roughly 70% of that, with the rest held in reserve so HA can restart protected VMs after a host failure. Operators who bought hardware expecting to use all of it hit this wall during provisioning and reach for the disable switch.

vSphere HA host isolation and split-brain: when isolation response goes wrong

Mon, 01 Jan 0001 00:00:00 +0000

vSphere HA host isolation and split-brain: when isolation response goes wrong

vCenter shows one or more ESXi hosts as “Not Responding.” VMs on those hosts may have been restarted on surviving hosts by HA. Or they may still be running on the unreachable host with no way to manage them. In the worst case, the same VM is now running in two places at once, and nobody noticed until storage corruption or application errors surfaced.

vSphere host 'Not Responding': dead, isolated, or is hostd hung?

Mon, 01 Jan 0001 00:00:00 +0000

vSphere host ‘Not Responding’: dead, isolated, or is hostd hung?

When vCenter shows an ESXi host as “Not Responding”, it has stopped receiving heartbeats from that host and cannot reach it on the management plane. This is distinct from “Disconnected” (a deliberate state set by an admin, or caused by license expiry) and from “Maintenance” (an intentional state for patching). The greyed-out host icon is one of the most paged-on vSphere symptoms because the same UI state covers at least four different underlying conditions.

vSphere host swapping (SWCUR/SWW/s): hypervisor swap and the memory death spiral

Mon, 01 Jan 0001 00:00:00 +0000

vSphere host swapping (SWCUR/SWW/s): hypervisor swap and the memory death spiral

When SWR/s is sustained above zero on an ESXi host, the VMkernel is actively reading VM memory pages back from .vswp files on the datastore. That is not a warning state. It is an active performance emergency. Every swapped-in page costs roughly 100x DRAM latency, and the swap I/O itself competes with VM disk I/O on the same datastore, producing a double penalty that degrades every VM on the host simultaneously.

vSphere memory ballooning (MCTLSZ): the host is reclaiming guest RAM

Mon, 01 Jan 0001 00:00:00 +0000

vSphere memory ballooning (MCTLSZ): the host is reclaiming guest RAM

You open esxtop, switch to the memory view, and a VM’s MCTLSZ column is no longer zero. A few hundred megabytes or several gigabytes, the VMkernel has inflated the vmmemctl balloon driver inside that guest and is forcing the guest OS to hand back memory it thought it owned. From the host’s perspective this is gentle reclamation. From the guest’s and the application’s perspective, it is often the start of a silent performance decline.

vSphere memory compression: the reclamation tier between balloon and swap

Mon, 01 Jan 0001 00:00:00 +0000

vSphere memory compression: the reclamation tier between balloon and swap

Memory compression is the third tier in ESXi’s four-tier memory reclamation hierarchy. It sits between ballooning, which asks the guest OS to return pages, and host-level swapping, which writes pages to .vswp files on a datastore and destroys latency. Compression is ESXi buying time: keep the page in RAM but in a smaller form, at the cost of CPU cycles and added access latency.

vSphere memory reclamation cascade: balloon to compress to swap in minutes

Mon, 01 Jan 0001 00:00:00 +0000

vSphere memory reclamation cascade: balloon to compress to swap in minutes

Hosts can sit at 75% consumed memory for hours with zero reclamation activity. A workload spike, a large VM power-on, or a memory leak can push consumed memory past physical capacity in minutes. ESXi’s response is not graceful: balloon inflates across VMs, compression activates, .vswp files open on the production datastore, swap-in begins, and datastore latency rises as swap I/O competes with VM disk I/O. By the time swap-in rate is above zero, the host has exhausted every gentler mechanism and is reading VM memory pages back from disk. That is a production emergency, not a tuning exercise.

vSphere monitoring checklist: the signals every host, VM, and vCenter needs

Mon, 01 Jan 0001 00:00:00 +0000

vSphere monitoring checklist: the signals every host, VM, and vCenter needs

Send this to someone standing up vSphere monitoring for the first time, or rebuilding an alerting setup that pages too often and misses real incidents. It lists the signals worth collecting across the hypervisor plane (ESXi hosts and VMs) and the management plane (vCenter Server Appliance).

vSphere does not fail like a generic Linux box. CPU contention is invisible from inside the guest. Memory goes from fine to catastrophic in minutes once host swapping starts. A datastore at 99% full looks identical to one at 5% full from inside a VM, until every VM on it halts. And vCenter can degrade for weeks before anyone notices, because DRS, HA, and the API quietly keep working until they don’t. Generic CPU/disk/network dashboards miss most of this.

vSphere monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

vSphere monitoring maturity model: from survival to expert

vSphere monitoring setups cluster into four maturity levels. Where you sit determines which incidents you catch early, which ones you discover after users complain, and which ones you never see coming until the management plane is already down. Use this as an assessment tool: read the signal lists for your current level and the one above, identify the gaps, and prioritize closing them.

vSphere NUMA locality low: wide VMs paying the remote-memory tax

Mon, 01 Jan 0001 00:00:00 +0000

vSphere NUMA locality low: wide VMs paying the remote-memory tax

Low NUMA locality is a silent performance killer in vSphere. Guest CPU utilization looks normal, memory usage is healthy, and disk and network latency are within baseline. But latency-sensitive workloads - databases, in-memory caches, analytics engines - run 10-30% slower than they should. The problem is below the guest, in the physical memory topology.

When a VM spans multiple NUMA nodes (a “wide VM”), some memory accesses traverse the interconnect (QPI/UPI on Intel, Infinity Fabric on AMD) to reach memory owned by a different socket or chiplet. Each remote access costs roughly 1.5-2x the latency of a local access, adding approximately 50-100ns depending on platform and interconnect generation.

vSphere physical uplink saturation: one flow uses one NIC, and storage shares the wire

Mon, 01 Jan 0001 00:00:00 +0000

vSphere physical uplink saturation: one flow uses one NIC, and storage shares the wire

A vSphere host with two 10GbE uplinks in a NIC team looks like 20Gbps. For aggregate multi-flow traffic across many VMs, it is. But a single TCP flow between one VM and one remote host uses exactly one of those uplinks. The team does not split that flow across both links. That is why a VM doing a bulk transfer can saturate one uplink while the other sits idle.

vSphere PSOD (purple screen of death): diagnosing an ESXi host crash

Mon, 01 Jan 0001 00:00:00 +0000

vSphere PSOD (purple screen of death): diagnosing an ESXi host crash

A purple screen of death (PSOD) is the ESXi VMkernel’s deliberate halt. When the kernel detects an unrecoverable condition, an uncorrectable machine check, a driver panic, or a corrupted data structure, it stops the host on purpose, paints the purple diagnostic screen, and writes a core dump if a target is configured. Every VM on that host dies instantly. There is no graceful shutdown and no live migration off the host.

vSphere SCSI sense codes and path failover: intermittent fabric faults in vmkernel.log

Mon, 01 Jan 0001 00:00:00 +0000

vSphere SCSI sense codes and path failover: intermittent fabric faults in vmkernel.log

When storage connectivity degrades, the first hard evidence usually lands in /var/log/vmkernel.log as SCSI sense codes. Latency counters (DAVG, KAVG, GAVG) tell you I/O is slow; the sense codes tell you why. ESXi logs a status tuple and optional sense data on every command that fails or is retried, and the Native Multipathing Plugin (NMP) records the resulting path state transitions.

vSphere storage latency cliff: the 'everything is slow' incident that hits every VM at once

Mon, 01 Jan 0001 00:00:00 +0000

vSphere storage latency cliff: the ’everything is slow’ incident that hits every VM at once

Every VM on a datastore degrades at the same moment. Application pages fire across unrelated services. The root cause is one shared resource below ESXi. This is the storage latency cliff.

The pattern is distinctive once you have seen it. Latency on the affected datastore jumps from a few milliseconds to tens or hundreds of milliseconds. Queue depth climbs above zero. Every workload on that datastore suffers, regardless of which host it runs on. Guest operating systems report high I/O wait, but CPU and memory are fine. Applications report timeouts, but the application itself is healthy.

vSphere storage queue depth: QUED, ACTV, and DSNRO saturation

Mon, 01 Jan 0001 00:00:00 +0000

vSphere storage queue depth: QUED, ACTV, and DSNRO saturation

Storage latency on an otherwise healthy all-flash array is usually a queue problem, not an array problem. The VMkernel stacks I/O requests in queues at every layer of the path between the guest and the physical device. When those queues fill, requests wait, and the latency the guest sees climbs independently of what the array is actually doing.

The two numbers that tell you whether you are in this state are ACTV (in-flight I/O at the device) and QUED (I/O waiting in the VMkernel). Sustained QUED greater than zero with KAVG climbing is the signature of device queue saturation. The fix is rarely “make the array faster”; it is almost always “give the device more outstanding-I/O capacity, or stop funneling everything through a single path.”

vSphere thin-provisioned VMDK growth: space that never comes back without UNMAP

Mon, 01 Jan 0001 00:00:00 +0000

vSphere thin-provisioned VMDK growth: space that never comes back without UNMAP

A thin-provisioned VMDK starts small and grows as the guest writes data. It never shrinks on its own. When a guest deletes a 100 GB database dump, the space inside the guest filesystem becomes free, but the VMDK file on the datastore stays at its high-water mark. Over months, the datastore fills with blocks the guest considers empty. The symptom is a datastore that creeps toward full while the guests report plenty of free space inside.

vSphere vCPU oversizing: why adding vCPUs made the VM slower

Mon, 01 Jan 0001 00:00:00 +0000

vSphere vCPU oversizing: why adding vCPUs made the VM slower

You gave the database VM 16 vCPUs because it was slow at 8. Now it is slower. Host CPU sits at 65%. Guest OS reports 20% utilization. Nothing in the usual dashboards explains the degradation.

This is the vCPU oversizing spiral. Adding vCPUs to a VM that does not need them does not add capacity. It adds scheduling overhead. The ESXi CPU scheduler uses relaxed co-scheduling for multi-vCPU VMs, which means it must find enough simultaneously available physical CPUs to co-schedule a VM’s vCPUs together. The more vCPUs you give a VM, the harder that search becomes, and the longer each vCPU spends in the READY state before it can execute.

vSphere vMotion slow or failing: memory dirty rate, bandwidth, and convergence

Mon, 01 Jan 0001 00:00:00 +0000

vSphere vMotion slow or failing: memory dirty rate, bandwidth, and convergence

A vMotion task that should take minutes instead crawls for ten, twenty, or thirty minutes. The vSphere Client progress bar hangs in the high nineties. Sometimes the migration completes with a multi-second stun. Sometimes it fails outright: “The migration was cancelled because the amount of changing memory for the VM was greater than the available network bandwidth.”

Almost every slow or failing vMotion traces back to one relationship: the VM is dirtying memory faster than the vMotion network can transmit it. The iterative pre-copy loop cannot converge. ESXi either stuns the vCPUs to force convergence, hurting guest performance, or cancels the migration.

vSphere vMotion stun time: the switchover pause that drops connections

Mon, 01 Jan 0001 00:00:00 +0000

vSphere vMotion stun time: the switchover pause that drops connections

vMotion promises live migration with no disruption. For most VMs, the final switchover pause is short enough that in-guest applications and network clients never notice. But “live” is not “instantaneous.” During the final switchover, the VM is stunned: its vCPUs stop executing while the last set of dirty memory pages and device state transfers from the source host to the destination. When that pause stretches past a second, TCP stacks reset, databases miss heartbeats, and clustered applications fail over.

vSphere VMware Tools heartbeat red: guest crash vs extreme CPU starvation

Mon, 01 Jan 0001 00:00:00 +0000

vSphere VMware Tools heartbeat red: guest crash vs extreme CPU starvation

A red guestHeartbeatStatus on a powered-on vSphere VM means VMware Tools has stopped sending heartbeats to the VMkernel. The common assumption is a guest OS crash. The actual cause may be extreme CPU starvation, a hung guest, a crashed Tools process, or a transient vCenter API flap.

The risk is the automation attached to the signal. With vSphere HA VM Monitoring enabled, a sustained red heartbeat plus no I/O causes HA to reset the VM. If the root cause is CPU starvation, the reset does not fix it: the VM returns, gets descheduled again, loses heartbeats again, and HA resets again. A misread cascades into restart loops.

VMware vSphere Operations Guides on Netdata

How vSphere actually works in production: a mental model for operators

How vSphere actually works in production: a mental model for operators

vCenter '503 Service Unavailable': the vSphere Client will not load

vCenter ‘503 Service Unavailable’: the vSphere Client will not load

vCenter 'Cannot complete login due to an incorrect user name or password': SSO failures

vCenter ‘Cannot complete login due to an incorrect user name or password’: SSO failures

vCenter /storage/db full: vPostgres stops and the whole management plane dies

vCenter /storage/db full: vPostgres stops and the whole management plane dies

vCenter /storage/log full: the log-bomb disk death spiral

vCenter /storage/log full: the log-bomb disk death spiral

vCenter /storage/seat full: stats, events, alarms, and tasks outgrowing their partition

vCenter /storage/seat full: stats, events, alarms, and tasks outgrowing their partition

vCenter appliance undersized: inventory outgrowing the deployment size

vCenter appliance undersized: inventory outgrowing the deployment size

vCenter certificate expired: the STS signing cert outage nobody saw coming

vCenter certificate expired: the STS signing cert outage nobody saw coming

vCenter clock skew: the NTP offset that breaks tokens and disconnects hosts

vCenter clock skew: the NTP offset that breaks tokens and disconnects hosts

vCenter CPU and memory pressure: vpxd heap, swap, and the undersized appliance

vCenter CPU and memory pressure: vpxd heap, swap, and the undersized appliance

vCenter database bloat: SEAT tables, statistics level, and the failed purge job

vCenter database bloat: SEAT tables, statistics level, and the failed purge job

vCenter HA (VCHA) replication broken: protection that is not protecting

vCenter HA (VCHA) replication broken: protection that is not protecting

vCenter service will not start: vmon dependency order and the max-restart wall

vCenter service will not start: vmon dependency order and the max-restart wall

vCenter shows all hosts Disconnected / Not Responding: vCenter-side vs host-side

vCenter slow and unresponsive: vpxd overload and the task queue backlog

vCenter slow and unresponsive: vpxd overload and the task queue backlog

vCenter SSO intermittently failing: STS heap, flaky AD, and near-expiry certs

vCenter SSO intermittently failing: STS heap, flaky AD, and near-expiry certs

vCenter vPostgres 'too many clients already': connection exhaustion

vCenter vPostgres ’too many clients already’: connection exhaustion

vCenter vPostgres autovacuum and WAL: dead tuples and an unbounded pg_wal

vCenter vpxd crash loop: the core service that keeps restarting

vCenter vpxd crash loop: the core service that keeps restarting

vSphere 'Virtual machine disks consolidation is needed': clearing the warning without stunning the VM

vSphere ‘Virtual machine disks consolidation is needed’: clearing the warning without stunning the VM

vSphere active vs consumed vs granted memory: why the percentage lies

vSphere active vs consumed vs granted memory: why the percentage lies

vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty

vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty

vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM

vSphere CPU limit hit (%MLMTD): the forgotten MHz cap that silently throttles a VM

vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle

vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle

vSphere datastore full: 'No space left on device', paused VMs, and power-on failures

vSphere datastore full: ‘No space left on device’, paused VMs, and power-on failures

vSphere datastore IOPS and throughput: spotting storage saturation before latency bites

vSphere datastore IOPS and throughput: spotting storage saturation before latency bites

vSphere datastore latency high: reading GAVG, DAVG, and KAVG

vSphere datastore latency high: reading GAVG, DAVG, and KAVG

vSphere dropped packets (%DRPRX/%DRPTX): ring buffers, CPU, and uplink backpressure

vSphere dropped packets (%DRPRX/%DRPTX): ring buffers, CPU, and uplink backpressure

vSphere DRS not balancing: affinity rules and reservations blocking placement

vSphere DRS not balancing: affinity rules and reservations blocking placement

vSphere DRS thrashing: vMotion churn with no stable placement

vSphere DRS thrashing: vMotion churn with no stable placement

vSphere ESXi hardware health: ECC errors, fan failure, and thermal throttling

vSphere ESXi hardware health: ECC errors, fan failure, and thermal throttling

vSphere forgotten snapshot growing: the delta VMDK time bomb

vSphere forgotten snapshot growing: the delta VMDK time bomb

vSphere HA 'Insufficient resources to satisfy configured failover level': admission control

vSphere HA ‘Insufficient resources to satisfy configured failover level’: admission control

vSphere HA host isolation and split-brain: when isolation response goes wrong

vSphere HA host isolation and split-brain: when isolation response goes wrong

vSphere host 'Not Responding': dead, isolated, or is hostd hung?

vSphere host ‘Not Responding’: dead, isolated, or is hostd hung?

vSphere host swapping (SWCUR/SWW/s): hypervisor swap and the memory death spiral

vSphere host swapping (SWCUR/SWW/s): hypervisor swap and the memory death spiral

vSphere memory ballooning (MCTLSZ): the host is reclaiming guest RAM

vSphere memory ballooning (MCTLSZ): the host is reclaiming guest RAM

vSphere memory compression: the reclamation tier between balloon and swap

vSphere memory compression: the reclamation tier between balloon and swap

vSphere memory reclamation cascade: balloon to compress to swap in minutes

vSphere memory reclamation cascade: balloon to compress to swap in minutes

vSphere monitoring checklist: the signals every host, VM, and vCenter needs

vSphere monitoring checklist: the signals every host, VM, and vCenter needs

vSphere monitoring maturity model: from survival to expert