$ guides / vmware-vsphere ▌

VMWARE VSPHERE · OPERATIONS PLAYBOOK

vSphere's invisible taxes: ready time the guest can't see, a memory cliff, and a vCenter that goes dark all at once

A two-plane stack where the ESXi scheduler deschedules vCPUs behind the guest's back, memory reclamation escalates from balloon to swap in minutes, one datastore stalls every VM on it, and the vCenter appliance — a constellation of 30-odd services on an embedded database — fails hard the moment a certificate expires or a /storage partition fills. We trace how that design behaves under load, where degradation turns into an outage, and what to do when it does.

> Start with the monitoring checklist → # Jump to the full guide list

vSphere's defaults get you virtualized quickly, then hand you a set of cliff-edges that most teams only meet during an incident — and half of them live in vCenter, not on the hosts.

The defaults work. Until a VM is sized with more vCPUs than the scheduler can co-schedule, ready time climbs, and the guest reports low CPU while everything runs slow. Until a host crosses its memory line and reclamation cascades from ballooning to compression to .vswp swap, whose disk I/O then drags down every VM on the datastore. Until a forgotten snapshot's delta disk fills the datastore and power-ons fail with No space left on device. Until a host PSODs and HA races to restart its VMs. And until the vCenter side breaks: an STS signing certificate expires and every login fails at once, or /storage/db fills and vPostgres stops, taking the whole management plane with it.

These guides are written for engineers who already run vSphere, not for people learning what a hypervisor is. The goal is the mental model of how ESXi and vCenter actually behave under load, the failure patterns that keep recurring on both planes, the monitoring story that catches them before they page anyone, and the runbooks you wish someone had handed you before your last incident.

How vSphere actually runs in production

vSphere is two planes, not one. The ESXi data plane schedules vCPUs onto pCPUs, reclaims memory in tiers, and moves every I/O through the VMkernel; the vCenter management plane orchestrates HA, DRS, and vMotion on top of an appliance database. Most production failures live between these layers — a symptom in the guest can originate at the scheduler, the datastore, or a certificate three layers away.

VMs + VMware Tools

Each VM is a sandbox with emulated hardware. VMware Tools (or open-vm-tools) provides the balloon driver, the guest heartbeat, and time sync. Without Tools, the hypervisor is partly blind to the guest — and reclamation skips ballooning straight to swap.

GUEST

CPU scheduler + NUMA

The VMkernel maps vCPUs to pCPUs with a proportional-share scheduler. A vCPU that is runnable but waiting shows as <code>ready time</code>; multi-vCPU VMs pay <code>co-stop</code> to stay co-scheduled. The NUMA scheduler tries to keep each VM's memory local to its vCPUs.

SCHED

memory reclamation hierarchy

Four tiers, escalating in desperation: page sharing, then <code>ballooning</code> (guest pages internally), then <code>compression</code> into a cache, then host-level <code>swap</code> to a <code>.vswp</code> file. Each tier is an order of magnitude worse than the last. There is no graceful slope near the edge.

MEM

storage I/O path + datastores

Guest to virtual SCSI to the VMkernel queue to HBA/NFS client to the array. Latency decomposes into <code>KAVG</code> (VMkernel/queue) and <code>DAVG</code> (device/array). Datastores hold VMDKs, snapshot deltas, and swap files — and fill silently as thin disks and snapshots grow.

STORAGE

network I/O path

Guest to vNIC to port group on a vSwitch/dvSwitch to a physical uplink. A NIC team distributes flows but never aggregates one flow's bandwidth. VM, vMotion, storage, and management traffic often share the same wire unless separated.

NET

ESXi host (VMkernel)

The microkernel arbitrating all hardware. Its own health — hardware sensors, thermal state, driver stability — is the floor everything stands on. A PSOD kills every VM on the host instantly and triggers HA restarts elsewhere.

ESXi

cluster services: HA + DRS + vMotion

HA restarts VMs from failed hosts using network and datastore heartbeats; DRS rebalances placement every few minutes; vMotion live-migrates memory with a brief switchover stun. Misconfigured isolation response, admission control, or affinity rules turn these safety nets into incidents.

CLUSTER

vCenter management plane (vCSA)

The vCenter Server Appliance: <code>vpxd</code> holding an in-memory inventory cache, an embedded <code>vPostgres</code> database on dedicated <code>/storage/*</code> partitions, and <code>STS</code>/SSO issuing the tokens every login and integration depends on. Running VMs survive its outage, but DRS, HA reconfiguration, vMotion, and all management stop.

VCENTER

Why this matters: 'the VM is slow' can come from ready time the guest can't see, a memory reclamation cascade, one saturated datastore, a wide VM straddling NUMA nodes, a snapshot chain, or a vCenter that can't authenticate. The symptom rhymes but each layer has a different signal — and a different fix. And when vCenter itself is the problem, the VMs are usually fine; you've lost the ability to manage them.

The failures you'll actually see

Most vSphere incidents fall into a small set of recurring patterns, split across the ESXi data plane and the vCenter management plane. Recognise the shape, and triage gets dramatically faster.

CRITICAL

The storage latency cliff

The array slows — a controller failover, a RAID rebuild, a noisy neighbour — and DAVG jumps from a few milliseconds to hundreds. The VMkernel queue backs up, KAVG follows, and every VM on the datastore stalls at once. Databases time out, guests report I/O errors, and heartbeats can drop. This is the number-one 'everything is slow' incident in vSphere, and it hits the whole datastore, not one VM.

DAVG spiking from under 5ms to 50-500ms on a datastore
KAVG rising and QUED sustained above zero (VMkernel queuing)
Every VM on the same datastore degraded simultaneously
Guest I/O wait high, application timeouts, possible heartbeat loss

Investigate →

CRITICAL

The memory reclamation cascade

A host runs short of physical memory and reclamation escalates: ballooning forces guests to page internally, compression fills its cache, then the VMkernel swaps VM memory to .vswp files on the datastore. That swap I/O competes with real VM I/O, so latency spikes feed more memory pressure — the death spiral. It can go from fine to catastrophic in minutes, and a single node's pressure degrades every VM on it.

Balloon (MCTLSZ) inflating, then compression, then swap-out
Any sustained swap-IN rate (SWR/s) — active disk-speed memory access
Balloon at zero with swap active — VMware Tools not reclaiming
Datastore latency rising in step with swap file I/O

Investigate →

IMMINENT

The snapshot time bomb

A snapshot is created — manually before a change, or by a backup job — and never removed. Its delta VMDK grows with every write, day after day, until the datastore fills. Then thin disks can't grow, swap files can't be created, power-ons fail, and consolidating the accumulated chain becomes a multi-hour, VM-stunning operation. The guest is completely unaware; performance just slowly gets worse.

Snapshot older than 24-72 hours and still growing
Datastore free space declining in step with one VM's write rate
Large *-delta.vmdk files in the VM folder
No space left on device / consolidation-needed warnings appearing

Investigate →

ACTIVE

CPU starvation the guest can't see

A host is overcommitted, or a VM is oversized with more vCPUs than the scheduler can place at once. The vCPUs wait for physical CPUs — ready time and co-stop climb — but the guest OS has no idea it was descheduled, so in-guest CPU looks moderate. The result is the classic mystery: the VM is slow, guest CPU looks fine, and nobody can explain it. Adding vCPUs makes it worse.

CPU ready (%RDY) above 5-10% while guest CPU looks moderate
Co-stop (%CSTP) elevated on multi-vCPU VMs
Host CPU only moderate (60-80%), yet specific VMs are starved
Larger-vCPU VMs performing worse than smaller ones on the same host

Investigate →

CRITICAL

The vCenter certificate expiry cascade

An internal certificate expires — most devastatingly the STS signing certificate, which is separate from the machine SSL cert and invisible in a browser. STS can no longer issue valid tokens, so every authentication fails at once: the vSphere Client won't log in, PowerCLI and every integration drop, and ESXi hosts disconnect in bulk. There is no gradual warning — everything works perfectly right up to expiry, then breaks together. VMCA certs default to a two-year life.

All logins failing simultaneously (not one account)
ESXi hosts disconnecting in bulk rather than one by one
SSL/TLS handshake errors across multiple services in the logs
checksts.py / vecs-cli showing a certificate at or past expiry

Investigate →

CRITICAL

The vCenter disk death spiral

A dedicated vCSA partition fills — commonly /storage/db or /storage/log — while root looks healthy. When /storage/db fills, vPostgres can't write its WAL and crashes; vpxd has no database and the whole management plane goes down. When /storage/log fills, services that can't log crash, vmon restarts them, they log the failure — a self-reinforcing loop. The trigger is often a different issue (an SSO error storm, a failed purge) that flooded a partition.

df -h shows one /storage/* partition at 100% while / is fine
vPostgres or vpxd crash-looping (check vmon status)
Log generation rate paradoxically rising as services fail
SDK/API returning 503 or refusing connections

Investigate →

The Netdata solution

VMware vSphere monitoring with Netdata

Netdata auto-discovers vCenter, ESXi hosts, VMs, and datastores through the vSphere API and collects them per second with ML-powered anomaly detection. Correlate CPU ready and co-stop, ballooning and host swap, datastore latency, and snapshot growth against the host and guest signals behind them, so you catch the incidents in these runbooks before they page anyone.

See VMware monitoring → Start monitoring free

vSphere monitoring maturity levels

vSphere observability works in four practical levels, and each spans both the ESXi hosts and the vCenter appliance. Each level is a complete operation, not a stepping stone. Pick the one that matches how much your environment matters. Most production clusters should land at the second level.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor. With these signals you can answer one question: are the hosts and VMs alive, is there space to run, and is vCenter reachable? You won't learn what broke, but you'll learn that something broke before users do. Survival is enough for lab and non-critical clusters.

ESXi host up / VM power state Are the hosts reachable and are the VMs powered on?
Datastore free space (% and absolute) A full datastore halts every VM writing to it.
Host CPU and memory (aggregate) Blunt, but catches a host that is clearly saturated.
vCenter SDK / API reachable Can consumers actually talk to vCenter right now?
vCSA /storage/db and /storage/log The two partitions whose filling takes vCenter down.
Certificate expiry (machine SSL + STS) Expiry is a silent, total, cliff-edge outage.

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production environments should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident alone: CPU starvation, memory pressure, storage latency, snapshots, HA/DRS state, and the vCenter side.

CPU ready time per VM The starvation the guest can't see; the top CPU signal.
Memory balloon and swap (host + VM) Read the reclamation tiers, not one blended percentage.
Datastore latency (GAVG/DAVG/KAVG) Decompose it: array-slow vs VMkernel-queuing.
Snapshot age and count per VM Check daily — the #1 preventable vSphere incident.
HA cluster health + host connection Isolation, admission control, hosts Not Responding.
Per-partition vCSA disk + NTP offset All /storage/* mounts; clock skew breaks auth.
vpxd / vPostgres service status The two services whose loss is a full outage.
Certificate expiry for ALL cert types Not just the browser cert — the STS cert too.

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone up. Co-stop creeping on an oversized VM, NUMA locality drifting, a datastore queue filling, the vPostgres database bloating, stats rollup falling behind. None of these page you on day one. They become page-out incidents on day thirty.

CPU co-stop and max-limited per VM Oversizing and forgotten CPU limits, invisible in %RDY.
NUMA locality per VM Wide VMs paying the remote-memory tax.
Memory compression + queue depth The tier before swap; QUED/ACTV before latency bites.
Per-vdisk latency and datastore IOPS Localise the noisy VM and the saturated LUN.
Dropped packets + vMotion stun time Ring/uplink pressure and connection-dropping pauses.
vPostgres database + SEAT table size Retention and statistics-level bloat, weeks early.
Task queue depth + stats rollup lag vpxd overload and rollup falling behind its interval.
vCSA VM seen from the hypervisor Ready time / balloon on vCenter itself — half the picture.

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. SCSI sense codes, VMkernel world CPU, STS heap and GC, vmon restart counts, vPostgres autovacuum and WAL, VCHA replication. Most teams never need every signal here. Add the ones your incident history says you do.

SCSI sense codes + path state changes Classify storage errors; catch a flapping fabric path.
VMkernel world / system-time CPU NSX firewall and storage-driver overhead, not VM CPU.
STS Java heap + GC frequency The intermittent-SSO-failure root cause.
vmon service restart counts Silent crash loops that brief probes never catch.
vPostgres autovacuum + pg_wal size Dead-tuple bloat and unbounded WAL, 30-60 min early.
SDK session count by client IP The misbehaving backup job or script overloading vpxd.
VCHA / vmdir replication health Protection that silently isn't; ELM sites diverging.
Guest time drift + host power state NTP offset under contention; thermal throttling.

Operating mistakes worth avoiding

The traps vSphere teams keep falling into, on both the host and the vCenter side. Each has a clear, well-known fix. Most teams only learn it after an incident.

Monitoring guest CPU instead of hypervisor ready time

The single most common vSphere monitoring mistake. The in-guest agent reports utilisation of the time the VM got, not the time it wanted — so a VM at 30% guest CPU with 15% <code>ready time</code> looks healthy while running at 85% speed. The guest cannot see that it was descheduled. You must monitor CPU ready from the hypervisor level, per VM.

Treating memory as a single utilisation percentage

vSphere memory is a multi-tier reclamation system, not a number. A host at 85% 'memory utilisation' might be fine (large gap between consumed and active, no balloon) or actively swapping into a death spiral. The percentage tells you nothing on its own. Monitor the reclamation indicators independently: <code>balloon</code>, <code>compression</code>, and <code>swap</code>.

Not monitoring snapshots

Most vSphere setups have zero snapshot monitoring, yet forgotten snapshots are the number-one cause of preventable incidents. Teams discover two-week-old, 300GB snapshots when the datastore fills and VMs crash. Snapshot count and age must be checked daily — and a failed backup job is the classic source of an orphaned, ever-growing delta.

Reading datastore latency at the wrong layer

Teams that do watch storage latency often see 'high latency' and blame the array. But if <code>KAVG</code> is high and <code>DAVG</code> is normal, the problem is in the VMkernel (queue depth, VAAI locks, driver), not the array — and the diagnosis goes down the wrong path. Decompose latency into <code>DAVG</code> and <code>KAVG</code> before pointing anywhere.

Not monitoring vCenter as infrastructure

Many teams monitor thousands of VMs but never watch the vCSA's own disk partitions, database size, or service health. vCenter is the control plane: if it degrades, DRS stops, HA reconfiguration stops, provisioning stops, and during a host failure you may find HA can't be managed. Monitor vCenter like the critical infrastructure it is — including from the hypervisor layer.

Watching only the machine SSL certificate

The STS signing certificate is the most critical certificate in the vCSA ecosystem and has caused more outages than any other. It is separate from the machine SSL cert, invisible in a browser, and needs specific tooling (<code>checksts.py</code>, <code>vecs-cli</code>) to inspect. When it expires, all authentication fails at once. Track every certificate type, and alert 90 and 30 days out — not 7.

Monitoring disk on / but not the /storage/* partitions

The vCSA partition layout means root <code>/</code> can be perfectly healthy while <code>/storage/db</code>, <code>/storage/log</code>, or <code>/storage/seat</code> sits at 100%. Standard Linux monitoring that only checks <code>/</code> misses the most common vCenter disk failures entirely. Watch every <code>/storage/*</code> mount separately, and check inodes (<code>df -i</code>) on the log partition too.

Not monitoring CPU limits (max-limited)

A VM (or resource pool) with a forgotten MHz limit is throttled with low <code>ready time</code> but high <code>max-limited</code> time — so teams that only watch ready miss it. Limits get set during testing or inherited from a template and then silently cap production VMs for years. Monitor <code>cpu.maxlimited.summation</code> alongside <code>cpu.ready.summation</code>.

vSphere and vCenter runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

CPU scheduling, limits, and NUMA

▸

Memory reclamation: balloon, compression, swap

▸

Datastore latency, queues, and IOPS

▸

Snapshots and datastore space

▸

Networking, uplinks, and vMotion

▸

ESXi host health, PSOD, and hardware

▸

HA, DRS, and cluster availability

▸

vCenter services and availability (vCSA)

▸

vCenter disk partitions and vPostgres

▸

vCenter certificates, SSO, and time

▸

vCenter resources, sizing, and VCHA

WHERE TO GO NEXT

Setting up vSphere monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing — on the hosts or in vCenter.

> Start with the checklist > Back to Operations Guides