$ guides / nvidia-gpu / nvidia-gpu-cuda-out-of-memory ▌

Operations Guides

CUDA out of memory: diagnosing NVIDIA GPU framebuffer exhaustion

Your training job or inference server died with RuntimeError: CUDA out of memory. Tried to allocate X MiB. Sometimes the GPU is genuinely full. More often, nvidia-smi shows gigabytes free and the error makes no sense at first glance.

GPU framebuffer is a cliff-edge resource. Unlike CPU memory there is no swap and no graceful degradation: when a cudaMalloc cannot be satisfied, the allocation fails atomically and the process dies. It is also one of the most misdiagnosed GPU failures, because the number nvidia-smi shows is not the number your framework is working with.

What this means

A CUDA OOM means the driver or the framework’s caching allocator could not satisfy one specific allocation request. It does not necessarily mean the framebuffer is full. Four things consume VRAM, and only one of them is your model:

Driver and firmware reserved memory. Typically 200-500 MiB per GPU that is never available for user allocations.
CUDA context overhead. Every process that initializes CUDA reserves several hundred MB to over 1 GB before it allocates a single tensor. With multiprocessing (for example PyTorch DataLoader workers that touch the GPU), each worker pays this independently.
ECC overhead. On GPUs with ECC enabled, part of the framebuffer is consumed by error correction. The size depends on memory type and GPU SKU; verify against the datasheet for your hardware. NVIDIA does not publish a single uniform percentage: the reduction varies by GPU generation, memory width, and ECC scheme (SECDED for DRAM, different for HBM stacks).
Application memory. What your framework allocated, plus what its caching allocator is holding for reuse.

The last point causes most of the confusion. PyTorch’s caching allocator does not return freed memory to the driver; it keeps it in a pool for reuse. So nvidia-smi reports what PyTorch has reserved from the driver, not what live tensors are using. A process can show 30 GiB used in nvidia-smi while holding only 18 GiB of live tensors. Conversely, the allocator can fail an allocation while nvidia-smi shows free memory, because no single free block is large enough. That is fragmentation, and it is a different problem from true exhaustion.

Retired memory pages also shrink the allocatable pool permanently. Each retired page removes about 64 KB, up to a hard driver limit of 64 pages. A GPU whose memory.total quietly shrank over months may have retired pages.

Common causes

Cause	What it looks like	First thing to check
Model or batch too large	OOM on first iteration or first large allocation, reproducible at startup	Does the failing allocation size fit in `memory.free` before the job starts?
Another process on the GPU	OOM at startup on a GPU that “should” be empty	`nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv,noheader`
Memory leak	`memory.used` grows linearly over hours, OOM long after startup	Plot VRAM over time; compare against the workload baseline
Fragmentation	OOM with significant free memory, usually after long uptime or dynamic batch sizes	PyTorch error line: reserved total much larger than allocated
Zombie CUDA context	VRAM held but no process listed in nvidia-smi	Compare `memory.used` against the sum of per-process usage
Retired pages shrinking capacity	`memory.total` lower than the SKU spec	`nvidia-smi -q -d PAGE_RETIREMENT`
Context overhead multiplied by workers	OOM only when `num_workers > 0` or many processes share the GPU	Count compute apps; multiply count by context overhead

Quick checks

All read-only and safe to run during an incident.

# 1. Framebuffer state per GPU: used, free, total, and driver-reserved
nvidia-smi --query-gpu=index,memory.used,memory.free,memory.total,memory.reserved --format=csv

# 2. Who is on the GPU right now (host PIDs, not container PIDs)
nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv,noheader

# 3. Per-process utilization and memory, one snapshot
nvidia-smi pmon -s um -c 1

# 4. Retired pages: capacity permanently removed by ECC
nvidia-smi -q -d PAGE_RETIREMENT

# 5. ECC state: errors can precede capacity loss, and disabled ECC hides them
nvidia-smi --query-gpu=ecc.mode.current,ecc.errors.uncorrected.volatile.total --format=csv,noheader

# 6. Xid events: correlate OOM with driver-level faults
dmesg -T | grep -i "NVRM: Xid" | tail -20

Note: memory.reserved is not a standard nvidia-smi --query-gpu= field name; the reserved amount is exposed through NVML’s nvmlDeviceGetMemoryInfo_v2() (driver 510.39.01 and later) as nvmlMemory_v2_t.reserved, and through DCGM field DCGM_FI_DEV_FB_RESERVED (field 253). Verify the field against nvidia-smi --help-query-gpu on your driver before scripting against it.

The PyTorch OOM line itself is a diagnostic. It reports total capacity, memory already allocated, memory free, and memory reserved in total by PyTorch. Read it before running anything: if “reserved” is much larger than “allocated”, you are looking at fragmentation, not exhaustion.

If you have shell access to the process (or can reproduce), torch.cuda.memory_stats() exposes the allocator’s internal state. The keys most useful for fragmentation diagnosis are reserved_bytes.all.current versus allocated_bytes.all.current (large gap signals fragmentation), segment.all.current (total CUDA blocks held by the allocator), and oversize_allocations.all.current (requests too large for the pool).

How to diagnose it

Confirm the failure is allocation, not corruption. A clean CUDA out of memory or cudaErrorMemoryAllocation is an allocation failure. illegal memory access or unspecified launch failure are application bugs or hardware faults (often Xid 13, 31, or 43) and need a different response. Check dmesg for Xid events first so you do not spend an hour on capacity planning for what is actually a pointer bug.
Account for every byte. Take memory.used from check 1 and subtract the sum of used_gpu_memory across all compute apps from check 2. The remainder should be small: driver reserved plus context overhead. If the remainder is gigabytes with no listed process, you have a zombie CUDA context holding memory after a crash or SIGKILL. Kill the stale host PID if one still exists; if nothing is listed, a GPU reset or reboot is the reliable path (see the reset guide below). nvidia-smi --gpu-reset is disruptive: it fails while processes hold the device and resets state for the whole GPU.
Decide: true exhaustion or fragmentation. Use the PyTorch error line or memory_stats:
- Reserved approximately equals allocated, and both are near total capacity: true exhaustion. The workload genuinely does not fit, or it leaks.
- Reserved much larger than allocated, with free memory visible: fragmentation. The pool is chopped into pieces too small for the failing request.
Check for a leak. If reserved and allocated both grow steadily over hours on a workload that should be steady-state, sample memory.used every minute. A monotonic ramp with constant batch size is a leak (tensors kept alive by references, growing caches, or accumulating graphs). Estimate runway as (memory.total - memory.used) / growth_rate.
Check capacity shrinkage. Compare memory.total to the SKU spec and check retired pages (check 4). Pages retired by ECC reduce the pool permanently and persist across reboots.

flowchart TD
  A[CUDA OOM in app log] --> B{Xid 13/31/43 in dmesg?}
  B -- yes --> C[App bug or GPU fault, not capacity]
  B -- no --> D[Sum per-process memory vs memory.used]
  D --> E{Large unaccounted gap?}
  E -- yes --> F[Zombie context: kill stale PID or reset GPU]
  E -- no --> G{PyTorch reserved vs allocated}
  G -- reserved much greater --> H[Fragmentation]
  G -- roughly equal --> I[True exhaustion or leak]
  I --> J{memory.used ramps over time?}
  J -- yes --> K[Memory leak in workload]
  J -- no --> L[Model or batch too large for framebuffer]

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`memory.used` / `memory.total`	Capacity pressure and leak detection	Sustained >90%, or steady linear growth on a steady workload
Per-process `used_gpu_memory`	Attributes VRAM to tenants; catches unexpected consumers	Sum far below `memory.used` (zombie context), or an unexpected PID
PyTorch reserved vs allocated	Separates fragmentation from real usage	Reserved growing much faster than allocated
Retired page count	Permanent capacity reduction	Any increase, or `retired_pages.pending = Yes`
Uncorrected ECC errors (volatile)	Data corruption and context death that can masquerade as OOM-adjacent crashes	Any new event (delta > 0)
Xid events	Distinguishes allocation failure from hardware/driver faults	Any Xid correlated in time with the OOM
`utilization.gpu` vs memory growth	High memory with zero compute suggests a leak or zombie, not real work	VRAM high, SM utilization 0%

Do not page on memory percentage alone: caching allocators legitimately pin 95-99% of VRAM on well-tuned training jobs. OOM detection belongs at the application log level, with VRAM trends as the leading indicator.

Fixes

Workload does not fit

Reduce per-step footprint: smaller batch size, gradient accumulation, mixed precision, activation checkpointing, or a smaller model shard. Verify headroom with actual numbers: the failing allocation size in the error line must fit within memory.free after subtracting context overhead for every process that will touch the GPU, plus the 200-500 MiB driver reserve.

Another tenant on the GPU

Identify the PID with --query-compute-apps and reconcile it against your scheduler. On shared nodes this is the most common cause of “random” OOMs: someone else’s job landed on the same device. The durable fix is scheduling discipline (exclusive allocation, or MIG on A100/H100 for hard partitioning), not killing processes.

Memory leak

Fix the workload: find the retained tensors or growing cache. As an operational bridge, restarting the job on a cadence shorter than the leak’s time-to-OOM keeps service up, but treat that as debt. Trend VRAM per job so the bridge has an expiry date.

Fragmentation

For PyTorch: torch.cuda.empty_cache() releases cached blocks back to the driver between phases (it does not free live tensors). For recurring fragmentation with variable batch sizes, PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:<N> limits block splitting and keeps large blocks intact; treat it as a last-resort tuning knob. Restarting the process also resets the allocator pool, which is why fragmentation OOMs “fix themselves” on restart and return days later.

Zombie contexts

Kill the stale host PID (PIDs shown by nvidia-smi are host PIDs; translate through the container runtime). If no process is listed but memory is held, the context is wedged in the driver: nvidia-smi --gpu-reset on Linux, or a node reboot if reset fails. This is disruptive to every other job on that GPU.

Capacity shrinkage from retired pages

If retired pages are approaching the 64-page driver limit or uncorrectable errors have occurred, plan GPU replacement. There is no software fix for permanently retired memory.

Prevention

Enable persistence mode (nvidia-smi -pm 1, via a systemd unit so it survives reboots) so contexts initialize predictably and monitoring has no gaps between jobs.
Baseline per workload. Record expected peak VRAM per job class. Alerts on deviation from baseline catch slow leaks days before the OOM.
Attribute memory to schedulers. On shared nodes, continuously compare --query-compute-apps output against what the scheduler thinks is placed. Unexpected consumers are found in minutes, not after an OOM.
Track retired pages and ECC deltas, not raw counters, so hardware-driven capacity loss is visible before it causes allocation failures.
Size with overhead in mind. Budget context overhead per process and 200-500 MiB of driver reserve on top of the model footprint. “Fits on paper” is not fits.
Sample fast. Xid events and allocation failures are second-scale events; one-minute metric resolution will miss the sequence.

How Netdata helps

Per-second memory.used/memory.total per GPU, so the slow ramp of a leak is visible as a trend instead of a surprise at 3 a.m.
Per-process GPU memory and utilization attribution, so you can see which PID consumed the framebuffer when the OOM fired, including processes that have since exited.
ECC error counters and retired page state alongside VRAM, which connects “the GPU has less memory than it used to” to the hardware events that caused it.
Xid event visibility from system logs correlated on the same timeline as the application OOM, so you can immediately separate allocation failure from Xid 13/31/43-style application bugs.
Cross-GPU comparison on multi-GPU nodes, which surfaces the single device running hot on memory while its siblings are fine.