Reading NVIDIA GPU memory correctly: nvidia-smi vs the PyTorch caching allocator

An on-call classic: nvidia-smi shows 38 of 40 GiB used on a training node, the line is flat, and someone declares a memory leak. The team running the job insists the model only needs 12 GiB of tensors and nothing is growing. Both sides are reading real numbers. They are reading different layers of the stack, and neither tool tells you that on its own.

PyTorch (and TensorFlow, and most CUDA frameworks) does not allocate GPU memory directly per tensor. It reserves large blocks from the CUDA driver once, caches them, and suballocates internally. nvidia-smi and DCGM report what the framework has reserved from the driver, not what live tensors are actually using. Usage looks high and flat even as tensors are created and freed, and capacity planning or leak hunts based on nvidia-smi alone go wrong in predictable ways.

This article covers what each number measures, how to get tensor-level truth, and how to distinguish the three situations that all look like “high memory” from the outside: normal caching, fragmentation, and an actual leak.

Why nvidia-smi and PyTorch disagree

There are three distinct quantities, measured at three different layers:

1. Framebuffer used (nvidia-smi, DCGM FB_USED). Everything the driver has handed out on that device: the CUDA context and driver overhead, any non-PyTorch CUDA allocations (NCCL, cuDNN workspaces), and every block PyTorch has obtained via cudaMalloc and not explicitly returned. This is the number memory.used reports.
2. Reserved (torch.cuda.memory_reserved()). The portion of framebuffer the PyTorch caching allocator currently holds: blocks in use plus blocks cached for reuse.
3. Allocated (torch.cuda.memory_allocated()). The portion of reserved memory actually occupied by live tensors right now.

The relationship is roughly:

nvidia-smi memory.used ≈ CUDA context + driver reserved + non-PyTorch CUDA allocs + PyTorch reserved
PyTorch reserved       = allocated (live tensors) + cached free blocks + fragmented remnants

A healthy workload can show memory.used at 38 GiB, reserved at 36 GiB, and allocated at 12 GiB with nothing leaking. The 24 GiB gap between reserved and allocated is cached blocks the allocator holds for reuse, and it will reuse them without calling back into the driver.

Note the fixed overhead at the bottom: the CUDA context itself consumes framebuffer that shows up in nvidia-smi but is invisible to PyTorch’s allocator, and the driver and firmware reserve memory that is not available to user allocations at all (typically 200-500 MiB; the context overhead varies by driver and CUDA version, on the order of a few hundred MiB). On a small GPU this baseline can look like “a process using 1 GiB doing nothing.”

How the caching allocator works

cudaMalloc is expensive and synchronizing. Calling it per tensor, per training step, would wreck throughput. So PyTorch’s caching allocator front-loads the cost:

• On first use it requests large blocks from the driver via cudaMalloc and caches them.
• Small tensor allocations are carved out of larger cached blocks (splitting). When tensors are freed, adjacent free pieces are merged back together.
• Freeing a tensor returns its block to the allocator’s cache, not to the driver. nvidia-smi does not move.
• Cached blocks are only returned to the driver when you call torch.cuda.empty_cache(), or when the allocator is retrying after a failed allocation during an OOM event.
• The allocator maintains per-stream pools, which matters for multi-stream code: a tensor freed on one stream but still in use on another can look like it is leaking until the stream syncs (this is what record_stream() exists for).

flowchart TD
  A[GPU framebuffer - physical DRAM] --> B[Driver and firmware reserved]
  A --> C[CUDA context + NCCL and cuDNN allocs]
  A --> D[PyTorch reserved pool via cudaMalloc]
  D --> E[Allocated - live tensors]
  D --> F[Cached free blocks]
  D --> G[Fragmented inactive splits]
  S[nvidia-smi / DCGM FB_USED] -. measures .-> B
  S -. measures .-> C
  S -. measures .-> D
  R[memory_reserved] -. measures .-> D
  T[memory_allocated] -. measures .-> E

Two consequences fall out of this design. First, the “leak after the first training step” that alarms new GPU operators is the allocator building its pool: nvidia-smi jumps to a high plateau on the first step and stays there for the life of the process. That is designed behavior, not a fault. Second, a flat nvidia-smi line tells you nothing about what is happening inside the pool. Internal fragmentation can grow invisibly under a perfectly flat FB_USED curve.

The commands and APIs that tell the truth

Driver level, safe to run any time:

# Framebuffer usage per GPU
nvidia-smi --query-gpu=memory.used,memory.total,memory.free,memory.reserved --format=csv,noheader,nounits

# Per-process usage: who is holding what
nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv,noheader

memory.reserved here is driver/firmware overhead, not PyTorch’s reserved pool; do not confuse the two uses of the word. In containerized setups the PID shown is the host PID, so translate through the container runtime before killing anything.

Allocator level, from inside the process (or a debugger attached to it):

import torch

torch.cuda.memory_allocated()   # live tensor bytes
torch.cuda.memory_reserved()    # allocator pool size
torch.cuda.memory_summary()     # human-readable breakdown, prints to stdout

stats = torch.cuda.memory_stats()
stats["allocated_bytes.all.current"]
stats["reserved_bytes.all.current"]
stats["inactive_split_bytes.all.current"]   # fragmentation indicator
stats["num_alloc_retries"]                  # allocator had to retry/reclaim to satisfy an alloc
stats["num_ooms"]                           # OOM events seen by the allocator

memory_stats() is the diagnostic goldmine. The two counters worth watching over time are inactive_split_bytes and num_alloc_retries:

• inactive_split_bytes are free pieces carved off larger blocks that cannot currently be merged or reused for a different size request. High and growing inactive splits with flat reserved memory is fragmentation, and it is exactly the state that produces “CUDA out of memory” while nvidia-smi shows gigabytes free.
• num_alloc_retries climbing means the allocator is regularly failing to find a suitable block and having to free cached memory and retry. That is early-stage memory pressure, before OOMs start.
• num_ooms is the allocator’s own count of out-of-memory events, useful for confirming that application-level OOM errors actually came from allocation failure.

If you use backend:cudaMallocAsync (see tuning below), some memory_stats() values are not meaningful under that backend and are reported as zero.

The three situations that all look like “high memory”

From nvidia-smi alone, these are indistinguishable. From the allocator stats, they are easy to separate:

The discriminator is the trend, not the level. A flat plateau at 95% for days is normal for a well-tuned training job and should not page anyone. A slow, steady climb with no plateau, even from a lower baseline, is the leak signature and deserves investigation long before OOM. For the leak case, see NVIDIA GPU memory leak: framebuffer usage climbing without a plateau. For the OOM event itself, see CUDA out of memory: diagnosing NVIDIA GPU framebuffer exhaustion.

One more trap: PyTorch can report OOM while nvidia-smi shows free memory. The free memory exists in aggregate, but not as a contiguous block large enough for the request. This is fragmentation made visible, and it is more common in inference serving (dynamic batch sizes, model switching, frequent small allocations) than in steady training.

Reading memory correctly during an incident

When someone reports “the GPU is out of memory” or “memory looks wrong,” work down the layers:

1. Establish the driver-level picture. nvidia-smi --query-gpu=memory.used,memory.total,memory.free and --query-compute-apps to see total usage and which process holds it. On MIG-enabled GPUs, check per-instance metrics, not the physical card aggregate; one slice can be OOM while the card-level number looks fine.
2. Compare against the workload’s baseline. Is this a plateau (normal caching) or a climb (leak)? A single snapshot cannot answer this. You need history.
3. Get allocator-level truth. Pull memory_allocated(), memory_reserved(), and the inactive_split_bytes, num_alloc_retries, and num_ooms counters from the process. If reserved minus allocated is large and stable, that is cache. If inactive splits are large and growing, that is fragmentation.
4. Correlate with application errors. If the job is throwing CUDA OOM while FB_USED is below total, you are looking at fragmentation, not capacity exhaustion. If num_ooms is zero and the error is something else (illegal memory access, launch failure), memory is a red herring; look at application bugs or Xid events instead.
5. Only then act. The actions differ completely: cache needs nothing, fragmentation needs allocator tuning or allocation-pattern changes, a leak needs a code fix, and true capacity exhaustion needs a smaller model, smaller batches, or a bigger GPU.

Tuning knobs, with tradeoffs

These change allocator behavior. All are set before process start; none are free.

• torch.cuda.empty_cache(): releases cached (unused) blocks back to the driver, dropping nvidia-smi FB_USED. It does not fix fragmentation of in-use blocks, and the next allocations pay the cudaMalloc cost again. Useful as a diagnostic (“does FB_USED drop when I empty the cache? then it was cache”), marginal as a production remedy.
• PYTORCH_CUDA_ALLOC_CONF / PYTORCH_ALLOC_CONF: environment variable controlling the allocator. PYTORCH_CUDA_ALLOC_CONF is the legacy name and is deprecated in recent PyTorch versions in favor of PYTORCH_ALLOC_CONF; both are currently accepted. Relevant options include max_split_size_mb (caps block splitting, the standard fragmentation knob), expandable_segments:True (uses CUDA virtual memory mapping to grow and shrink segments instead of fixed cudaMalloc blocks; reduces fragmentation with varying batch sizes, still marked experimental), garbage_collection_threshold, and backend:cudaMallocAsync (alternative allocator backend, requires CUDA 11.4+, changes which memory_stats are meaningful).
• Allocation patterns: the durable fix for fragmentation is usually in the workload: stable batch sizes, preallocated buffers, avoiding alternating large and small allocations on the same stream.

Do not set tuning flags blindly. Measure inactive_split_bytes first so you know whether you have the problem the knob addresses.

Signals to watch in production

Do not alert on memory percentage. Well-tuned ML workloads run at 95-99% FB_USED by design. Alert on the trend (leak), on the allocator counters (fragmentation), and on application-level OOM events (real exhaustion). Percentage belongs on capacity planning dashboards, where you should plan against peak reserved memory per workload class, not against the instantaneous nvidia-smi number.

How Netdata helps

• Netdata collects per-GPU framebuffer usage and per-process GPU memory from NVML/nvidia-smi at per-second resolution, which is what makes the plateau-versus-climb distinction visible. Minute-resolution scrapes smear both into the same shape.
• Long retention on the FB_USED series lets you establish per-workload baselines, so “stable but slightly above baseline” leak drift is caught days before OOM.
• Per-process memory attribution shows which PID’s reservation is growing, which matters on shared nodes where the aggregate line hides the culprit.
• Correlating memory curves with GPU utilization helps: high memory with zero compute and a dead process is a zombie CUDA context holding its reservation, not an active workload.
• Pairing node-level GPU metrics with application-level CUDA error logs closes the gap: OOM detection should come from the application, and log and metrics correlation puts the OOM event next to the memory curve that explains it.

Related guides

• CUDA out of memory: diagnosing NVIDIA GPU framebuffer exhaustion
• NVIDIA GPU memory leak: framebuffer usage climbing without a plateau
• How an NVIDIA GPU actually works in production: a mental model for operators
• NVIDIA GPU monitoring checklist: the signals every production GPU fleet needs
• NVIDIA GPU monitoring maturity model: from survival to expert
• NVIDIA persistence mode: why the GPU keeps re-initializing and P-state flaps