$ guides / nvidia-gpu / nvidia-gpu-cuda-oom-with-free-memory ▌

Operations Guides

CUDA out of memory with free memory available: GPU memory fragmentation

Your training run or inference server has been up for hours. Then it dies with torch.cuda.OutOfMemoryError: CUDA out of memory. You check nvidia-smi and the GPU shows gigabytes free. The error message and the driver disagree, and the driver looks right.

Both are right. The GPU has free memory in aggregate, but no single contiguous block large enough for the allocation that just failed. The memory is fragmented: free in total, unusable in practice.

The pattern is recognizable. It almost never fires on the first allocation. It fires after the process has been running a while, after many allocate/free cycles of varying sizes. It is far more common in inference serving with dynamic batch sizes and variable sequence lengths than in fixed-allocation training loops. And restarting the process “fixes” it, which is why it keeps coming back.

This article covers how to confirm fragmentation (as opposed to true exhaustion or a leak), how the PyTorch caching allocator creates it, and which fixes are durable versus which are band-aids.

What this means

PyTorch does not call cudaMalloc for every tensor. It uses a caching allocator: it reserves large segments from the driver up front, sub-allocates blocks from those segments for your tensors, and when you free a tensor the block goes back into the allocator’s pool, not back to the driver. This is why nvidia-smi shows high “used” memory for a healthy PyTorch process even between steps. The allocator is holding cached memory for reuse.

The failure is inside that cache. Over time, segments get carved into blocks of assorted sizes. A block freed between two still-live blocks cannot be merged with anything or returned to the driver, because the driver only gets memory back when an entire segment is empty. When a new allocation arrives that is larger than any single free block, the allocator tries to reserve a new segment. If the driver has no room left (because the allocator is already holding most of VRAM as cache), the allocation fails. OOM, with free memory showing everywhere.

Two numbers tell the whole story:

Allocated: memory actually holding live tensor data.
Reserved: memory the allocator has taken from the driver, including its free cache.

When reserved is much larger than allocated and allocations still fail, you have fragmentation.

flowchart TD
  A[Tensor allocation request] --> B{Free block in cache big enough?}
  B -->|yes| C[Reuse cached block]
  B -->|no| D{Driver has room for new segment?}
  D -->|yes| E[cudaMalloc new segment]
  D -->|no| F[CUDA OOM with free memory showing]
  G[Tensor freed] --> H[Block returns to allocator pool]
  H --> I{Whole segment free?}
  I -->|yes| J[Segment can return to driver]
  I -->|no| K[Free fragment stranded between live blocks]
  K -.->|accumulates over hours| B

Common causes

Cause	What it looks like	First thing to check
Allocator fragmentation from variable-size allocations	OOM after hours of uptime; reserved » allocated; serving workload with dynamic batch or sequence lengths	`torch.cuda.memory_stats()` reserved vs allocated
Large blocks split into unusable pieces	Repeated OOM on one specific large allocation size; plenty of smaller free blocks	`memory_summary()` segment and block sizes
Cached memory never returned to the driver	`nvidia-smi` used stays high even at idle between requests	`torch.cuda.empty_cache()` effect on `nvidia-smi`
Slow leak masquerading as fragmentation	Allocated (not just reserved) climbs steadily without a plateau	Track `memory_allocated()` over time; see the memory leak guide
True exhaustion, misread	`nvidia-smi` free is genuinely near zero; per-process list shows another consumer	`nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv`
Raw CUDA allocations outside a caching allocator	Custom CUDA extensions or non-PyTorch code allocating directly	Audit which components call `cudaMalloc` directly

The first three are the same root cause at different layers. The last three must be ruled out before concluding “fragmentation”, because the fixes are completely different.

Quick checks

All read-only. Run inside the affected process’s environment where noted.

# What the driver sees: used/free and which processes hold memory
nvidia-smi --query-gpu=memory.used,memory.total,memory.free --format=csv,noheader,nounits
nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv,noheader

memory.used includes PyTorch’s cached-but-free pool. nvidia-smi cannot see inside the allocator, so it will always overstate what is “really” in use. That is expected, not a fault.

# Inside the affected process: the allocator's own view
import torch
print(torch.cuda.memory_summary())

Look at two things: “Allocated” versus “Reserved” for GPU memory, and the segment/block breakdown. Reserved much larger than allocated, plus a rising count of small free blocks, is the fragmentation signature.

# The specific fragmentation signal
stats = torch.cuda.memory_stats()
print("allocated:", stats["allocated_bytes.all.current"])
print("reserved:  ", stats["reserved_bytes.all.current"])
print("inactive split:", stats["inactive_split_bytes.all.current"])

inactive_split_bytes counts memory that is free inside the pool but stranded as split fragments of larger segments, so it cannot be returned to the driver or reused for large requests. A large and growing inactive_split_bytes.all.current is the closest thing to a direct fragmentation metric the allocator exposes.

# What the driver reports as free, from inside the process
free_b, total_b = torch.cuda.mem_get_info()
print(f"driver free: {free_b / 2**30:.1f} GiB of {total_b / 2**30:.1f} GiB")

If mem_get_info shows little driver-level free space while reserved - allocated is large, the allocator is sitting on the memory and the driver cannot hand out a new segment. That confirms the mechanism.

# Full allocator state for offline visualization
import torch
torch.cuda.memory._record_memory_history()
# ... run until the OOM or a representative window ...
torch.cuda.memory._dump_snapshot("fragmentation.pickle")

Load the snapshot into PyTorch’s memory visualizer to see the segment layout directly: which segments are chopped up, which blocks are pinned alive.

How to diagnose it

Confirm it is not true exhaustion. Check nvidia-smi --query-compute-apps for other processes on the GPU, and check memory.free. If another tenant or a zombie process is holding memory, this is not fragmentation. Confirm the process is actually stale, kill it, and move on.
Confirm it is not a leak. A leak shows allocated_bytes climbing monotonically without a plateau; fragmentation shows allocated roughly flat while reserved and inactive_split_bytes grow. If allocated never plateaus, follow the GPU memory leak guide instead.
Capture the allocator state at failure time. Wrap the failing allocation site (or the serving loop) so that on torch.cuda.OutOfMemoryError you dump memory_summary() and the stats above before the process exits. The OOM itself is transient; the summary is what you diagnose from.
Identify the allocation that fails. Note its size. Fragmentation OOMs characteristically fail on a large allocation (an activation tensor for a long sequence, a KV cache growth step) while thousands of smaller free blocks exist. If small allocations also fail, you are closer to true exhaustion.
Correlate with workload shape. Does the failure follow a change in batch size distribution, a long-tail request, or a model switch? Dynamic shapes are the fragment generator. Fixed-shape training that OOMs on step one is a sizing problem, not fragmentation.
Decide fix class. Band-aid (cache clearing, allocator tuning), structural (allocation pattern changes, paged memory), or operational (periodic restart). See below.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`reserved_bytes` vs `allocated_bytes`	The gap is cached memory; a growing gap with failures is fragmentation	Gap grows over hours while allocated stays flat
`inactive_split_bytes.all.current`	Free memory stranded in split segments; the closest thing to a fragmentation gauge	Steady climb between GC events; large fraction of reserved
OOM error count in application logs	The failure itself; rate matters for severity	Recurring OOMs at similar uptime offsets after each restart
`nvidia-smi memory.used`	Driver-level view including cache; needed for capacity context	Near-total used with small `allocated` inside the process
Uptime at time of OOM	Fragmentation is time-and-churn dependent	OOMs clustering at similar hours-after-start values
Allocation size at failure	Distinguishes “no large block” from “no memory at all”	Same large size failing repeatedly

The key distinction from the broader monitoring checklist: VRAM percentage alone is misleading for this failure. OOM can fire at 85% “used” because percentage says nothing about contiguity. Alerting on application-level OOM events plus the reserved-minus-allocated gap catches what nvidia-smi cannot.

Fixes

Band-aid: clear the cache

import torch
torch.cuda.empty_cache()

empty_cache() returns fully-free segments to the driver, which lowers nvidia-smi used memory and gives the allocator room to re-reserve. Its limits matter: it cannot free split fragments (segments with any live block stay put), so for established fragmentation it often changes little. It also forces the next allocation to cudaMalloc fresh segments, adding latency. Use it at natural boundaries (between requests, between epochs), not in a hot loop, and do not treat it as a fix. Some operators schedule it periodically as a pressure-relief valve; that works until fragmentation outpaces it.

Allocator tuning: max_split_size_mb

# Limit how eagerly the allocator splits large blocks
export PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512

Without a limit, the allocator will split a large free block to satisfy a small request, permanently carving big segments into small pieces. max_split_size_mb tells it to leave blocks above that size intact and reserve a new segment for small requests instead, keeping large contiguous space available for large allocations. This is the mitigation the PyTorch OOM message itself suggests. The right value is workload-specific: set it near the size of the allocations that keep failing. Too low and you waste memory; too high and you are back to splitting.

Newer PyTorch versions also offer expandable_segments:True, which maps segments into one growable virtual address range so blocks can merge across old segment boundaries, making allocation order matter much less. It is marked experimental and has known interaction issues with max_split_size_mb (do not set both without checking current guidance).

If you run with backend:cudaMallocAsync, allocator semantics change: several memory_stats() fields report zero and max_split_size_mb is ignored. Diagnose fragmentation differently on that backend.

Structural: stop generating fragments

Bucket your shapes. In inference serving, quantize batch sizes and pad sequence lengths to a small set of fixed sizes. Ten allocation sizes fragment; three do not.
Pre-allocate the big things. Allocate the largest tensors (KV cache, activation workspace) once at startup, at maximum size, and reuse them. Memory that is never freed cannot fragment.
Keep allocation order stable. Large-then-small ordering at startup keeps big segments intact; interleaved sizes over hours is what churns the pool.
Use a serving runtime with paged memory management. Frameworks such as vLLM allocate the KV cache as one large pool divided into fixed-size pages, eliminating the variable-size allocation churn that causes this failure class in LLM serving. If you are hand-rolling a serving loop on raw PyTorch, this is the durable fix.
Avoid raw cudaMalloc in extensions. Raw CUDA allocations outside the caching allocator fragment worse and cannot participate in its pooling. Route allocations through the framework allocator where possible.

Operational: restart

A restart resets the allocator to a clean slate. For serving fleets, rolling restarts on a schedule (or on a watermark signal like inactive_split_bytes crossing a threshold) are pragmatic containment while a structural fix is worked out. It is containment, not a cure; the OOMs return on the same timescale if the workload does not change.

Prevention

Track the gap, not the percentage. Add reserved - allocated and inactive_split_bytes to your per-service metrics. Fragmentation is visible days before the first OOM if you watch the right numbers.
Fix allocation shapes at design time. Dynamic batching is good for throughput and bad for the allocator; bucket it.
Set allocator configuration deliberately in production images. Do not leave max_split_size_mb or expandable segments as per-host experiments; bake the tuned value into the service config with a comment explaining the allocation size it protects.
Load-test with realistic shape distributions. A soak test with uniform batch sizes will never reproduce this. Replay production-like request size mixes and watch fragmentation metrics over 24+ hours.
Size headroom for churn, not just peak. The framebuffer capacity guidance treats >90% sustained as at-risk; for dynamic-shape serving, keep more headroom because the usable fraction of “free” memory shrinks as fragmentation builds.

How Netdata helps

Per-second VRAM sampling catches the memory growth curve and the exact moment of the OOM, which minute-resolution monitoring smooths away. Fragmentation OOMs are often preceded by a slow reserved-memory climb that is only visible at high resolution.
Process-level attribution separates “another tenant ate the memory” from “the allocator is holding it”, the first branch in the diagnosis.
Application metric correlation: plotting the service’s own reserved, allocated, and OOM-event counters against driver-level memory.used on one dashboard makes the reserved-allocated gap visible without logging into the host.
Uptime-correlated alerting: OOM events annotated against process uptime expose the “fails N hours after every restart” signature that distinguishes fragmentation from a sizing problem.
Fleet-wide comparison shows whether one replica fragments faster than its peers, which usually means a different request mix or config drift rather than a platform problem.