vSphere NUMA locality low: wide VMs paying the remote-memory tax

Low NUMA locality is a silent performance killer in vSphere. Guest CPU utilization looks normal, memory usage is healthy, and disk and network latency are within baseline. But latency-sensitive workloads - databases, in-memory caches, analytics engines - run 10-30% slower than they should. The problem is below the guest, in the physical memory topology.

When a VM spans multiple NUMA nodes (a “wide VM”), some memory accesses traverse the interconnect (QPI/UPI on Intel, Infinity Fabric on AMD) to reach memory owned by a different socket or chiplet. Each remote access costs roughly 1.5-2x the latency of a local access, adding approximately 50-100ns depending on platform and interconnect generation.

Standard monitoring misses this. VM-level CPU and memory metrics look normal. The signal lives in the numa.local and numa.remote counters, which require statistics level 3+ or esxtop for collection. Most production environments run at level 1-2, so the counters are never stored.

What this means

Modern multi-socket servers use Non-Uniform Memory Access (NUMA). Each CPU socket (and on AMD EPYC, each chiplet) owns a region of local DRAM. Accessing local memory is fast. Accessing memory owned by another NUMA node requires crossing an interconnect, adding latency.

ESXi’s NUMA scheduler tries to keep each VM’s vCPUs and memory within a single NUMA node. When a VM is small enough to fit in one node, this works well and locality stays above 90%. When a VM is “wide” - its vCPU count or configured memory exceeds what a single NUMA node provides - the scheduler must spread it across multiple nodes. The VM’s memory gets distributed across those nodes, and some accesses inevitably cross the interconnect.

The N%L counter in esxtop (memory view) reports the percentage of a VM’s memory accesses satisfied by local memory:

locality % = numa.local / (numa.local + numa.remote) * 100

Below 80%, the VM is paying a meaningful remote-memory tax. Below 70% for a database or in-memory cache, the impact shows up in application latency and throughput. The 80% threshold is operational consensus from practitioners, not a hard number in VMware’s official documentation. Treat it as a trigger for investigation, not a binary pass/fail grade.

A stateless web tier doing mostly CPU work barely notices remote memory. A database with a large buffer cache touching millions of pages per second feels every nanosecond.

flowchart TD
    A["N%L below 80% on a VM"] --> B{"VM wider than one NUMA node?"}
    B -- "Yes" --> C{"vCPU count exceeds cores per node?"}
    B -- "No" --> D["Check DRS migration history"]
    C -- "Yes" --> E["Right-size or accept structural remote tax"]
    C -- "No" --> F{"Memory limit set below configured?"}
    F -- "Yes" --> G["Remove memory limit"]
    F -- "No" --> H{"vNUMA exposed to guest?"}
    H -- "No" --> I["Enable vNUMA or upgrade HW version"]
    H -- "Yes" --> J["Wait for NUMA scheduler rebalance"]
    D --> K{"Moved to differently-shaped host?"}
    K -- "Yes" --> J
    K -- "No" --> L["Investigate host NUMA imbalance"]

Common causes

Quick checks

These are safe, read-only commands for initial diagnosis. Run them on the ESXi host or via PowerCLI against vCenter.

# Check N%L per VM in esxtop (interactive, on the ESXi host)
# Press 'm' for memory view, then 'f' to toggle NUMA stats columns on
# Look at the N%L column per VM world
esxtop

# Check host platform and memory topology
esxcli hardware platform get
esxcli hardware memory get

# Check VM hardware version, vCPU count, and memory (PowerCLI)
Get-VM "myvm" | Select-Object Name, HardwareVersion, NumCpu, MemoryMB

# Check for memory limit (-1 means unlimited; any positive value is a limit)
Get-VM "myvm" | Select-Object Name, `
  @{N='MemLimitMB';E={$_.ExtensionData.ResourceConfig.MemoryAllocation.Limit}}

# Check whether CPU Hot-Add is enabled
Get-VM "myvm" | Select-Object Name, ExtensionData.Config.CpuHotAddEnabled

# Correlate N%L drops with DRS migrations in the last 24 hours
Get-VIEvent -Entity (Get-VM "myvm") -Types "DrsVmMigratedEvent" `
  -Start (Get-Date).AddHours(-24)

How to diagnose it

1. Identify the affected VM and its N%L. In esxtop memory view with NUMA stats enabled, sort by N%L ascending. Any VM consistently below 80% is a candidate. Prioritize latency-sensitive workloads first: databases, in-memory caches, and analytics engines feel the remote-memory tax most.

2. Compare VM size to NUMA node size. Determine the host’s cores-per-NUMA-node and memory-per-NUMA-node. If the VM’s vCPU count exceeds cores-per-node, or its configured memory exceeds memory-per-node, it is a wide VM by definition and some remote access is structural.

3. Verify vNUMA exposure inside the guest. On Linux, run lscpu or numactl --hardware. On Windows, check Task Manager (Performance > CPU, switch graph to NUMA) or run the following PowerShell:

   # Returns one Win32_Processor object per NUMA node exposed to the guest.
   # A VM with more than 8 vCPUs showing only one object means vNUMA is not exposed.
   Get-WmiObject Win32_Processor | Select-Object DeviceID, NumberOfCores, NumberOfLogicalProcessors
   

   The most common cause of suppressed vNUMA is CPU Hot-Add enabled on a VM running hardware version older than 20.

4. Check for memory limits. A memory limit set below configured memory forces the VMkernel to satisfy some allocations from remote nodes. The limit does not show up in standard VM summary views. Check resource settings directly.

5. Review recent DRS migrations. If N%L dropped at a specific time, check whether a vMotion moved the VM to a host with a different NUMA topology. Locality recovers as the NUMA scheduler rebalances, but this takes time. For latency-sensitive VMs, the recovery window itself may be a problem.

6. On AMD EPYC hosts, verify BIOS NUMA settings. The recommended configuration for ESXi 8.x is NPS-1 (one NUMA node per socket) with CCX-as-NUMA disabled. Enabling CCX-as-NUMA splits each socket into chiplet-level NUMA domains, which can skew the hypervisor’s assumptions about physical memory architecture and degrade performance.

Metrics and signals to monitor

The numa.local, numa.remote, NRMEM, and NLMEM counters require statistics level 3+ for collection through vCenter historical stats, or use esxtop for real-time investigation. At the default level 1-2, these counters are not stored.

Fixes

Right-size the VM to fit within a NUMA node

The most effective fix is to size the VM so its vCPU count and configured memory both fit within a single NUMA node. If your host has 24 cores per socket and 128GB per NUMA node, a VM with 16 vCPUs and 96GB RAM fits cleanly. A VM with 32 vCPUs on the same host is wide by definition.

Some workloads genuinely need more resources than one node provides. For those, accept that some remote access is structural and focus on minimizing it. The critical step is ensuring vNUMA is exposed so the guest OS can make NUMA-aware scheduling and memory allocation decisions.

If the workload does not actually need all the vCPUs it has been assigned, reducing the count helps in two ways: better locality and easier CPU scheduling (lower co-stop and ready time). Check in-guest CPU utilization. A 16-vCPU VM averaging 20% guest CPU utilization is oversized and likely does not need to span nodes.

Enable or verify vNUMA exposure

vNUMA is activated by default when a VM has more than 8 vCPUs. This threshold has been stable across vSphere 7.x and 8.x . Once active, the guest OS sees the virtual NUMA topology and can place threads and allocate memory accordingly.

Two conditions can prevent vNUMA from being exposed:

CPU Hot-Add on older hardware versions. On hardware versions before 20 (pre-vSphere 8.0), enabling CPU Hot-Add forces the VM into a single vNUMA node regardless of vCPU count. This is a well-known gotcha for large database VMs. The fix is either to disable CPU Hot-Add (requires a power cycle) or to upgrade the VM to hardware version 20 or later. On HW20+, CPU Hot-Add and vNUMA can coexist, but you may need to set the advanced parameter numa.allowHotadd=TRUE or the API property exposeVnumaOnCpuHotadd to enable it.

Cores per Socket misconception. Since vSphere 6.5, the Cores per Socket setting no longer influences vNUMA topology. It only affects how CPUs are presented to the guest OS (relevant for per-socket licensing). Manually setting Cores per Socket to control NUMA placement is no longer effective. ESXi determines vNUMA topology automatically based on the underlying host.

Remove memory limits

If a memory limit is set below the VM’s configured memory, remove it or set it to unlimited (-1). The VMkernel will then allocate from the local NUMA node as the scheduler intends. Memory limits are one of the most commonly overlooked configuration mistakes in vSphere and do not appear in standard VM summary views.

Handle DRS migration disruption

When DRS moves a VM to a host with a different NUMA topology, locality degrades. The NUMA scheduler rebalances over time, but recovery is not instantaneous. Options:

• Use VM-Host affinity rules to keep wide VMs on hosts with consistent NUMA topology.
• Accept transient locality loss for non-latency-sensitive VMs where the tax is negligible.
• For vSphere 8.0 U3 and later, vNUMA topology is re-evaluated at every power-on by default. A VM that was correctly sized on one host may receive a different vNUMA layout after moving to a differently-shaped host and rebooting. The cluster-level advanced setting VPXD_PersistVnuma reverts to the older behavior of retaining vNUMA topology across moves.

AMD EPYC BIOS configuration

On AMD EPYC hosts, BIOS NUMA settings significantly affect locality and performance. The recommended configuration is NPS-1 (Nodes Per Socket = 1) with CCX-as-NUMA disabled. NPS-1 presents one NUMA node per socket to ESXi, aligning with the hypervisor’s memory placement assumptions.

Enabling CCX-as-NUMA splits each socket into multiple chiplet-level NUMA domains. While this can theoretically improve L3 cache locality for specific workloads, it often degrades overall performance because it changes the memory topology ESXi expects. If you see unexplained locality issues or performance regressions on EPYC hosts, check this BIOS setting first.

Prevention

• Size VMs to fit within a NUMA node whenever possible. Know your host’s cores-per-node and memory-per-node. Treat them as hard sizing constraints for latency-sensitive workloads.
• Monitor N%L for databases and in-memory caches. Add NUMA locality to your monitoring for latency-sensitive VMs. If you run statistics level 1-2, use esxtop for spot checks or selectively raise the level for the relevant counters.
• Audit for CPU Hot-Add on large VMs. VMs with more than 8 vCPUs and CPU Hot-Add enabled on hardware versions before 20 may have vNUMA suppressed silently. Either disable Hot-Add or upgrade to HW20+.
• Audit for memory limits. Memory limits break locality without any visible signal in standard monitoring. Check resource pools and individual VM settings during regular reviews.
• Standardize host NUMA topology within clusters. Mixed-topology clusters cause DRS to move VMs between differently-shaped hosts, repeatedly disrupting locality. Where possible, keep wide VMs in clusters with homogeneous NUMA layouts.
• Verify AMD EPYC BIOS before production deployment. Confirm NPS and CCX-as-NUMA settings match VMware guidance for your ESXi version. Correcting this after VMs are deployed requires host reboots.

How Netdata helps

Netdata’s per-second metric collection and anomaly detection bring several signals together for NUMA diagnosis:

• Correlate N%L drops with DRS migrations and host changes. If locality degrades after a vMotion, timeline alignment makes the root cause immediately visible instead of a multi-hour investigation.
• Pair NUMA locality with CPU ready time and co-stop. A wide VM that also has scheduling contention compounds its performance penalty. Seeing both signals together tells you whether to right-size for NUMA, for CPU scheduling, or both.
• Detect gradual locality degradation. Anomaly detection flags a slow N%L decline that threshold-based alerting would miss, especially when a workload slowly grows past a NUMA node boundary over weeks.
• Track memory reclamation alongside locality. If the host is ballooning or compressing, memory pressure can push allocations to remote nodes. Correlating these signals distinguishes a VM sizing problem from a host-level memory pressure problem.
• Baseline per-VM locality. Knowing what normal looks like for each VM makes it obvious when a configuration change, migration, or host event shifts the baseline.

Related guides

• vSphere CPU ready time high (%RDY): VMs starved while the guest looks idle
• How vSphere actually works in production: a mental model for operators
• vSphere monitoring checklist: the signals every host, VM, and vCenter needs
• vSphere monitoring maturity model: from survival to expert