vSphere CPU co-stop high (%CSTP): the SMP vCPU co-scheduling penalty

%CSTP in esxtop is the time a vCPU in a multi-vCPU VM sits halted because the ESXi scheduler is waiting to co-schedule the VM’s other vCPUs. In a healthy environment it is essentially zero. Sustained above a few percent on modern ESXi means a sizing or topology problem, not a performance problem you can tune away.

The classic shape: you give a database 16 vCPUs and it gets slower. The guest OS reports low CPU utilization because the vCPUs are not doing work. They are parked in COSTOP waiting for their siblings. From inside the VM this is invisible. The application runs slowly while the OS reports idle capacity.

What this means

The ESXi CPU scheduler co-schedules multi-vCPU VMs to keep the SMP group coherent: vCPUs the guest kernel assumes are progressing together must not run far out of step. The scheduler tracks per-vCPU skew and deliberately parks leading vCPUs so slower siblings catch up. This is relaxed co-scheduling, and it significantly reduces co-stop compared with older strict scheduling.

%CSTP is the time charged to that deliberate hold. The four VM runtime states account for 100% of VM time:

%RUN + %RDY + %CSTP + %WAIT = 100%

When %CSTP is non-trivial, some vCPUs are runnable but the scheduler is parking them to keep the SMP group coherent. That time is neither work nor idle. It is coordination overhead, and the guest cannot see it.

Important properties:

• Only meaningful for VMs with 2 or more vCPUs. A single-vCPU VM always shows 0% because it has no siblings to wait for.
• Should be near zero in healthy environments. Sustained above 3% on modern ESXi is a strong signal that the VM has more vCPUs than the scheduler can efficiently co-schedule. Above 1% is worth investigating.
• Relaxed co-scheduling already suppresses this counter almost entirely. Any visible co-stop is more severe than it looks.

The counter is the smoking gun behind oversized SMP VMs, but it is easy to misread. The vCenter chart value is a summation in milliseconds, not a percentage. Convert it the same way you convert CPU ready time: divide by both the sample interval and the vCPU count to get a per-vCPU percentage.

flowchart TD
  A["Multi-vCPU VM
N runnable vCPUs"] --> B["Scheduler seeks
N simultaneously-free pCPUs"]
  B --> C{"Co-scheduling
feasible?"}
  C -->|"Yes"| D["vCPUs run together
%RUN accrues"]
  C -->|"Some vCPUs blocked
or NUMA-bound"| E["Leading vCPUs parked
to limit skew"]
  E --> F["%CSTP accrues
VM time lost to coordination"]
  F --> G["Guest sees low CPU%
but app runs slowly"]

Common causes

Quick checks

These are all read-only and safe to run during normal operation.

# Check %CSTP per multi-vCPU VM (interactive esxtop on the host)
esxtop
#   press 'c' for CPU view
#   press 'V' to filter to VM worlds only
#   look at the %CSTP column per VM world

# Pull co-stop from vCenter (realtime, 20s samples) via PowerCLI
Get-Stat -Entity (Get-VM "myvm") -Stat cpu.costop.summation -Realtime -MaxSamples 6
# Convert: value_ms / (20000 * vCPU_count) * 100 = %CSTP per vCPU

# Pull CPU ready alongside, since %RDY and %CSTP almost always move together
Get-Stat -Entity (Get-VM "myvm") -Stat cpu.ready.summation -Realtime -MaxSamples 6

# Rule out a CPU limit: if %MLMTD is high and %RDY is low, a limit is the real cause
# esxtop -> 'c' -> %MLMTD column per VM

# Check host CPU pressure from vCenter
Get-VMHost | Select Name, @{N='CpuPct';E={[math]::Round($_.Summary.QuickStats.OverallCpuUsage / ($_.Summary.Hardware.CpuMhz * $_.Summary.Hardware.NumCpuCores) * 100, 2)}}

# Check NUMA locality per VM (esxtop)
#   press 'm'
#   press 'f' to enable NUMA stats
#   look at N%L per VM (should be >90% for properly sized VMs)

# Check for snapshots that may be inducing co-stop via storage I/O
Get-VM | Get-Snapshot | Select VM, Name, Created, SizeGB

How to diagnose it

1. Confirm the VM has 2 or more vCPUs. %CSTP is meaningless for single-vCPU VMs. Non-zero co-stop on a 1-vCPU VM means you are reading the wrong row in esxtop.

2. Convert the vCenter chart value to a percentage. The raw cpu.costop.summation counter is in milliseconds accumulated over the sample interval. Apply the ready-time formula: (value_ms / (interval_ms * num_vCPUs)) * 100. A value of 300 in a 20-second realtime interval on a 4-vCPU VM is 300 / (20000 * 4) * 100 = 0.375%, not 300%.

3. Look at host CPU utilization alongside %CSTP. This single correlation splits the diagnosis:

   • High %CSTP and host CPU above 85%: genuine host contention. The VM is collateral damage from a saturated host.
   • High %CSTP and host CPU in the 60-80% band: vCPU oversizing or NUMA misalignment. The host is not the bottleneck; the scheduler’s coordination overhead is.
   • High %CSTP and host CPU below 60%: almost certainly oversizing. The VM has more vCPUs than the workload can use.

4. Check in-guest CPU utilization. The key comparison: how many vCPUs does the guest actually use at peak? An 8-vCPU VM that peaks at 25% in-guest CPU is consuming roughly 2 vCPUs of work. It does not need 8.

5. Check NUMA locality. If the VM’s vCPU count exceeds the number of cores in one physical NUMA node, it is a “wide VM” by definition and the scheduler must coordinate across NUMA domains. Locality below 80% confirms cross-node traffic is amplifying the penalty.

6. Check %MLMTD to rule out a CPU limit. A limit-constrained VM can show elevated co-stop because the scheduler hits the limit and then has to re-co-schedule the group. The fix is different: remove the limit, not the vCPU.

7. Check for active snapshots and datastore latency if the spike correlates with a backup window or recent snapshot creation. Snapshot delta I/O can stall vCPUs in a way that shows up as elevated co-stop on I/O-heavy workloads.

8. Check %RDY at the same time. Ready and co-stop usually move together on oversized SMP VMs, but their ratio tells you which mechanism dominates. High ready with low co-stop points to host contention or placement-layer oversizing. High co-stop with moderate ready points to SMP co-scheduling specifically.

Metrics and signals to monitor

Fixes

Right-size the vCPU count

The most common fix: remove vCPUs. Reduce the VM’s vCPU count to match actual in-guest utilization. If a 16-vCPU database peaks at 25% guest CPU, drop it to 4 or 6 vCPUs. This requires a VM power cycle, so plan a maintenance window.

More vCPUs is more scheduling liability. The scheduler must find that many simultaneously-free pCPUs to make progress, and on a moderately-loaded host that is harder, not easier. Removing vCPUs reduces the co-scheduling problem and frequently makes the VM faster.

Fix NUMA alignment

If the VM is a wide VM (vCPUs exceed cores per NUMA node), either reduce vCPUs to fit inside a single node or accept the cross-node penalty and size accordingly. Size the vCPU count so the VM splits evenly across the smallest possible number of NUMA nodes. Odd vCPU counts that cross NUMA boundaries are the worst case.

Also verify the guest sees a sensible vNUMA topology. Cores-per-socket affects guest OS topology and licensing; a nonsensical cores-per-socket value can confuse the guest’s own scheduler even when ESXi placement is correct.

Reduce host contention

If host CPU is the bottleneck, the VM is collateral damage. Options, in order of preference:

• vMotion the oversized VM to a less-loaded host with sufficient free pCPUs in a single NUMA node.
• vMotion other VMs off the host to give the oversized VM more scheduling room.
• Add compute capacity (more hosts, more cores per host).
• Review whether DRS is actually balancing, or is constrained by affinity rules.

Do not try to fix host contention by adding vCPUs to the affected VM. That makes the co-scheduling problem worse.

Remove the CPU limit

If %MLMTD is non-zero, there is a CPU limit on the VM, a resource pool, or inherited from a template. Limits look identical to contention from inside the guest. Remove the limit (set to Unlimited / -1 in the API) and re-measure before concluding that co-stop is the problem.

Consolidate snapshots

If the co-stop spike correlates with active snapshots or a backup window, consolidate the snapshots during a maintenance window. Snapshot consolidation is I/O-intensive and stuns the VM briefly. Then review the backup job to confirm it is cleaning up its own snapshots.

Prevention

• Right-size at provisioning time. Default templates with 8 or 16 vCPUs are the most common source of preventable co-stop. Start small and grow based on observed in-guest utilization, not application vendor recommendations that assume dedicated hardware.
• Know the cores-per-NUMA-node on your hosts. Treat any VM sized above that as a wide VM requiring extra scrutiny.
• Monitor %CSTP and %RDY together for every multi-vCPU VM. Ready without co-stop means one problem; co-stop without ready means another.
• Audit for forgotten CPU limits. They survive template clones, resource pool moves, and operator turnover.

How Netdata helps

Netdata’s per-second collection shortens the window between “the database got slow” and “the host cannot co-schedule it” from hours of esxtop watching to a few correlated charts:

• Per-VM cpu.ready.summation and cpu.costop.summation side by side, so the ratio between ready and co-stop is visible without manual conversion.
• Host CPU utilization correlated against per-VM co-stop, which splits the diagnosis between host contention and per-VM oversizing immediately.
• NUMA home node locality alongside vCPU count, so wide VMs are obvious rather than discovered during a performance investigation.
• cpu.maxlimited.summation tracked alongside ready and co-stop, so forgotten CPU limits cannot hide behind scheduling metrics.
• Snapshot age and datastore latency charts, so storage-induced co-stop during backup windows is visible in the same view as the CPU signals.

The co-stop value itself is most reliably read from esxtop on the host. Netdata’s value here is correlation: when you see elevated co-stop on a multi-vCPU VM, the adjacent signals that distinguish oversizing, NUMA issues, contention, a limit, or a snapshot are already in the same dashboard.

Related guides

• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• How vSphere actually works in production: a mental model for operators
• vSphere monitoring checklist: the signals every host, VM, and vCenter needs
• vSphere monitoring maturity model: from survival to expert