vSphere monitoring maturity model: from survival to expert

vSphere monitoring setups cluster into four maturity levels. Where you sit determines which incidents you catch early, which ones you discover after users complain, and which ones you never see coming until the management plane is already down. Use this as an assessment tool: read the signal lists for your current level and the one above, identify the gaps, and prioritize closing them.

Levels are cumulative. A team at Operational monitors everything in Survival plus operational signals. The progression tracks which failure modes you can detect before they become outages, not how sophisticated your tooling is.

flowchart TD
    L1["Level 1: Survival
Is it alive?"]
    L2["Level 2: Operational
Is it degraded?"]
    L3["Level 3: Mature
Why is it degraded?"]
    L4["Level 4: Expert
What will break next?"]
    L1 --> L2 --> L3 --> L4

Level 1: survival

Survival monitoring answers one question: is vSphere running? You know when a host is disconnected, when a VM is powered off, when a datastore is full, or when vCenter is unreachable. You do not know why performance is degrading, and you cannot detect resource contention until it becomes an outage.

What you can detect:

  • Host and VM liveness. ESXi host connection state (connected, disconnected, notResponding) and VM power state (poweredOn, poweredOff).
  • Datastore free space. Percentage and absolute bytes remaining.
  • vCenter reachability. Service status (vpxd up or down) and SDK/API availability via HTTPS probe to /sdk.
  • Certificate expiry dates. Machine SSL and STS signing certificate validity periods. You can catch the most common preventable vCenter outage before it happens.
  • Aggregate host utilization. CPU and memory consumed at the host level. You know when a host is loaded but not whether individual VMs are being starved.

What you cannot detect:

  • CPU ready time or co-stop. A VM can be starving for pCPU time while aggregate host CPU looks moderate. The guest OS has no idea it is being descheduled.
  • Memory reclamation cascade. Ballooning, compression, and host-level swap are invisible if you only watch aggregate memory percentages. A host at 85% memory might be fine or it might be actively swapping.
  • Storage latency breakdown. You know the datastore is not full, but you do not know whether device latency (DAVG) is spiking or kernel latency (KAVG) is queuing.
  • Snapshot accumulation. The datastore fills before you know a two-week-old snapshot exists.

The survival trap is reactivity. Every alert is either a page-worthy outage or noise. The first sign of a memory pressure cascade is the page that fires when swap-in has already started.

Level 2: operational

Operational monitoring adds the signals needed to diagnose the most common production incidents: CPU contention, memory pressure, storage latency, snapshot growth, and cluster health problems.

What you add:

SignalWhat it catches
CPU ready time per VM (cpu.ready.summation)Hypervisor-level CPU starvation invisible to the guest OS
Memory balloon and swap (per host and per VM)The memory reclamation cascade before it becomes catastrophic
Datastore latency: GAVG, DAVG, KAVGWhere in the I/O path latency occurs (array, VMkernel, or both)
Snapshot age and count per VMForgotten snapshots before they fill the datastore
HA cluster health and host statusHA protection gaps, isolation events, admission control violations
DRS balance and migration countCluster imbalance, DRS thrashing, constraint conflicts
Per-partition disk usage on VCSASpecific partition filling (/storage/db, /storage/log) while root looks fine
NTP synchronization offsetTime skew before it breaks SSO authentication and certificate validation

The operational advantage is diagnosis. You can tell whether an “everything is slow” incident is storage latency (DAVG high, KAVG normal means the array is the problem) or VMkernel queuing (KAVG high, DAVG normal means the problem is in the ESXi I/O stack). You can detect ballooning before it escalates to compression and swap.

What you still miss at operational:

  • Co-stop. Multi-vCPU VMs suffer co-scheduling delays that ready time alone does not reveal. A 16-vCPU database with high ready might actually be a co-stop problem.
  • NUMA locality. A large VM spanning NUMA nodes silently loses memory throughput. The guest OS cannot see this.
  • Queue depth saturation. High latency with no queue context tells you there is a problem but not whether the device queue is full.
  • vPostgres internals. Database growth and bloat are visible at the filesystem level, but connection exhaustion, dead tuple accumulation, and WAL growth are not.
  • CPU limits (max-limited). A VM with a forgotten CPU limit shows low ready time but is being throttled. If you only check ready, you miss this entirely.

Level 3: mature

Mature monitoring covers the signals that a professional SRE team needs for full incident diagnosis and proactive capacity planning.

What you add:

SignalWhat it catches
CPU co-stop (cpu.costop.summation)Multi-vCPU VM scheduling overhead, usually from oversizing
CPU max-limited (cpu.maxlimited.summation)Forgotten CPU limits silently throttling production VMs
NUMA locality percentage (numa.local vs numa.remote)Memory accesses crossing the interconnect, adding latency per access
Memory compression rate (mem.compressionRate.average)The danger zone between ballooning and host-level swapping
Outstanding I/Os and queue depth (QUED, ACTV)Queue saturation as the direct cause of elevated KAVG
Per-virtual-disk latency (read and write separately)Per-vdisk problems vs datastore-wide problems
vPostgres database size and table-level growthDatabase bloat from stats retention or failed purge jobs
vpxd task queue depth and response timevCenter overload before it becomes unresponsive
VMkernel log error rateEarliest signal of hardware or driver failure
Capacity trending and runway estimationWhen each resource will hit its cliff, expressed in days

The mature advantage is precision. You detect the vCPU oversizing spiral: a 16-vCPU database VM has high co-stop because the scheduler cannot find 16 simultaneously-free pCPUs, and you know the fix is to reduce vCPU count, not add more. You catch NUMA locality violations before someone files a ticket about “the database being slow since the migration.” You monitor queue depth alongside latency, so you know whether high GAVG is a device problem (low queue, slow array) or a saturation problem (full queue, too much demand).

What you still miss at mature:

  • SCSI sense codes. When storage fails, the specific sense key and ASC/ASCQ tell you exactly what happened. But these live in vmkernel.log and most teams never parse them systematically.
  • STS internals. You monitor SSO failure rate, but you do not see the Java heap pressure or garbage collection pauses that cause intermittent token validation failures.
  • vmon restart counts. A service that crashes and restarts within seconds may be invisible to external probes, but the restart count reveals a crash loop.

Level 4: expert

Expert monitoring adds the deep signals that operators typically instrument after a major incident taught them what they were missing. These are the signals that turn multi-hour investigations into minutes.

What you add:

SignalWhat it catches
Per-VM SCSI sense code trackingSpecific storage failure modes via sense key, ASC/ASCQ, and command opcode
VMkernel world CPU utilizationHelper and system worlds (vMotion helper, NFS client, iSCSI initiator) consuming pCPU
STS Java heap utilization and GC frequencyIntermittent SSO failures caused by GC pauses, not certificate expiry
vmon restart counts per serviceSilent crash loops invisible to external availability probes
VCHA replication state and lagWhether failover would actually work and how much data would be lost
vmdir replication healthSSO topology divergence between vCenter nodes in Enhanced Linked Mode
vPostgres checkpoint frequency and durationI/O storms from checkpoints competing with query I/O
vPostgres autovacuum effectiveness per tableTables where vacuum is falling behind, causing bloat and slow queries
Storage path state change ratePath flapping, indicating unstable fabric or misconfigured multipathing

The expert advantage is speed. You read SCSI sense codes in vmkernel.log and can distinguish medium errors from hardware errors from unit attention conditions based on the sense key, without waiting for a storage team to interpret array logs. You catch STS heap pressure before it causes the intermittent authentication failures that are notoriously hard to diagnose. You know whether vmon has restarted vpxd three times in the last hour, even if each restart was brief enough that the SDK probe never failed.

The cost of staying at each level

LevelWhat kills youTypical detection time
SurvivalCertificate expiry cascade, snapshot growth, memory death spiralHours to days after user impact
OperationalNUMA penalties, co-stop from oversizing, queue saturation, vPostgres bloatMinutes to hours after impact
MatureSCSI sense code misinterpretation, STS heap pressure, silent crash loopsRoot cause takes hours despite detection
ExpertProactively caught before user impactMinutes

The gap between Survival and Operational is the largest single improvement a team can make. Moving from Operational to Mature catches the “mystery slow VM” problems that ready time and aggregate latency cannot explain. Moving from Mature to Expert collapses investigation time for storage failures, SSO degradation, and silent service crashes.

How Netdata helps

Netdata collects vSphere performance counters at per-second resolution and applies anomaly detection across the signals at each maturity level:

  • CPU contention correlation. CPU ready, co-stop, and max-limited collected per VM. Anomaly detection flags the VM where ready time spikes while host CPU is moderate.
  • Memory cascade detection. Balloon, compression, and swap rates tracked together. The ML layer detects the cascade pattern (balloon rising, then compression, then swap) before swap-in begins.
  • Storage latency breakdown. DAVG, KAVG, and GAVG collected per device and per VM, with queue depth context.
  • vCenter health. Per-partition disk usage on VCSA and certificate expiry tracked alongside host and VM performance metrics.
  • Composite pattern detection. The memory reclamation cascade, storage latency cliff, and CPU oversizing spiral are multi-signal patterns. Per-second resolution makes these patterns visible in the anomaly timeline without manual dashboard correlation.