$ guides / nvidia-gpu / nvidia-gpu-memory-leak ▌

Operations Guides

NVIDIA GPU memory leak: framebuffer usage climbing without a plateau

Your training job has been running for six hours. nvidia-smi showed 40 GiB used after warmup. Then 45. Then 52. The curve has not flattened, and at this rate the job dies with CUDA out of memory sometime tomorrow, taking a day of checkpoint progress with it.

GPU memory is a cliff resource: there is no swap and no graceful degradation. When an allocation fails, it fails immediately. The saving grace is that a true leak is visible hours before it kills you, as sustained linear growth in framebuffer usage. The hard part is telling that apart from the many things that look like a leak but are not, because ML frameworks deliberately fill memory and hold it.

This guide is about that discrimination problem: confirm the climb is real, attribute it to a process and a cause, compute your runway, and fix it before OOM does it for you.

What this means

For a healthy ML training workload, framebuffer usage is deterministic after the first few steps. Weights, optimizer state, and activations are all allocated, and the caching allocator (PyTorch, TensorFlow) grabs a large pool on first use and holds it until process exit. On nvidia-smi this looks like a fast ramp followed by a flat plateau, often at 90%+ of total VRAM. That is normal and is not a leak.

A leak is the opposite shape: usage that keeps rising, step after step, hour after hour, without ever plateauing. The increments are often small, a few MiB to a few hundred MiB per step or per thousand requests, which is why it goes unnoticed until someone looks at a long time window. The failure mode is delayed OOM: time_to_OOM = (memory.total - memory.used) / leak_rate. Small leak rate plus large headroom means the blast lands in the middle of next weekend instead of during business hours.

A second pattern presents identically at first glance: memory that stays held after a process exits. When a process dies with SIGKILL, the driver reclaims its CUDA context lazily, so freed memory can lag by seconds. And if the process is a zombie whose context was never destroyed, the memory is held indefinitely. From nvidia-smi both look like “memory that will not come back.”

High memory plus low compute utilization is the classic signature of both the leak-in-progress and the orphaned context: the GPU is holding memory nobody is computing on.

Common causes

Cause	What it looks like	First thing to check
Caching allocator warmup (not a leak)	Fast ramp on first steps, then a hard plateau at high usage	Does usage stop rising after a few steps? If yes, this is normal
Tensors retained in Python structures	Steady small growth per training step; framework allocated memory rises with it	Log `torch.cuda.memory_allocated()` per step; consistent per-step growth confirms it
Custom CUDA extension not freeing	`nvidia-smi` used climbs while framework-reported allocated stays flat	Compare driver-level `memory.used` against framework allocator stats
Growing KV cache in inference serving	Growth tracks request concurrency and sequence length, not steps	Correlate FB usage with live request count and context lengths
Orphaned CUDA context after SIGKILL	Memory held after the process is gone; may lag seconds, or persist	`fuser -v /dev/nvidia*` to find the PID still holding the device
Fragmentation	CUDA OOM raised while `nvidia-smi` still shows free memory	`torch.cuda.memory_stats()` for fragmentation indicators

Quick checks

All read-only and safe to run on a production node.

# Current framebuffer state (used, total, free, driver-reserved)
nvidia-smi --query-gpu=memory.used,memory.total,memory.free,memory.reserved --format=csv,noheader,nounits

# Sample usage over time to confirm the climb and estimate the rate
for i in $(seq 1 12); do
  nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits
  sleep 300
done

# Which processes hold memory, and how much
nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv,noheader

# Per-process utilization and memory, one sample
nvidia-smi pmon -s um -c 1

# PIDs holding the device nodes (catches orphaned contexts after SIGKILL)
fuser -v /dev/nvidia*

# Application-level CUDA errors and driver events
dmesg -T | grep -i "NVRM: Xid"

Two things to note when reading output. memory.used includes reserved memory and cached allocations, so it overstates what tensors are actually using. And in containers, the PID shown by nvidia-smi is the host PID, not the container PID; you need PID namespace translation before killing anything.

How to diagnose it

flowchart TD
  A[FB usage rising] --> B{Plateaus after warmup?}
  B -- yes --> C[Normal: caching allocator]
  B -- no --> D[Compute leak rate and runway]
  D --> E{Process still running?}
  E -- no --> F[Orphaned CUDA context: find holder with fuser]
  E -- yes --> G{Framework allocated rising too?}
  G -- yes --> H[App-level leak: retained tensors or growing KV cache]
  G -- no --> I[Custom CUDA not freeing or fragmentation]

Confirm the shape. Sample memory.used every few minutes over at least 30 to 60 minutes. A plateau, even a high one, is not a leak. A line with a nonzero slope is. Do not judge from two readings; training has phase transitions (evaluation, checkpointing) that move usage temporarily.
Compute the runway. Fit the slope: leak_rate = delta_used / delta_time. Then time_to_OOM = (memory.total - memory.used) / leak_rate. This turns “it’s leaking” into “it OOMs in approximately 9 hours,” which decides whether you hot-fix, let the current run finish to checkpoint, or schedule a restart.
Attribute to a process. --query-compute-apps and pmon tell you which PID holds the memory and whether it is doing any compute. A process holding 30 GiB with 0% SM utilization is either leaking or already dead inside.
Check whether the process is actually alive. If the job was SIGKILL’d (OOM killer, kill -9, scheduler preemption), the driver reclaims the context lazily. Wait a few seconds and re-check. If memory is still held, run fuser -v /dev/nvidia* and match the holder. A context that survives its process is an orphaned context and will hold memory until the holder dies or the GPU is reset.
Split driver view from framework view. In PyTorch, compare torch.cuda.memory_allocated() (memory live tensors actually use) against torch.cuda.memory_reserved() (what the allocator has claimed from the driver) and against nvidia-smi memory.used. Allocated rising per step means the application is retaining tensors: stored in a list or dict, kept alive by a graph reference, or an accumulating cache. Allocated flat while driver-level used climbs points below the framework: a custom CUDA extension or native library that allocates and never frees.
Instrument per step. The canonical leak confirmation: log torch.cuda.memory_allocated() at the start and end of each training step. Consistent growth per step is a leak. Growth only during evaluation or logging phases points at whatever those phases retain. For inference serving, plot FB usage against concurrent requests and sequence lengths; a KV cache that only grows means evictions are not happening or the cache is unbounded.
Check for fragmentation if you are already at OOM. If the application raised CUDA out of memory while nvidia-smi showed free space, the free memory is not contiguous. torch.cuda.memory_stats() exposes allocator internals to confirm. Fragmentation produces the same end state as a leak but a different fix.
Rule out hardware. Memory leaks are software, but check dmesg for Xid events anyway. If you see Xid 48 (double-bit ECC) or Xid 31 (memory page fault) correlating with the failures, you are looking at a different incident. See NVIDIA Xid 48: Double Bit ECC Error and NVIDIA Xid 31: GPU memory page fault.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
FB used rate of change	The actual leak detector	Sustained linear growth over hours against a stable workload
FB used vs baseline	Training deliberately fills memory, so absolute level is meaningless	Usage above the established plateau for this exact workload and config
Estimated time to OOM	Converts a slope into an operational deadline	Runway shorter than the remaining job duration
Per-process used_gpu_memory	Attribution and zombie detection	Memory held by a process with zero compute utilization
Framework allocated vs reserved gap	Distinguishes leak from fragmentation	Large and growing gap between reserved and allocated
Application CUDA OOM errors	Where the leak becomes user-visible	Recurring OOM at similar uptime after each restart
Compute utilization alongside memory	Leak signature is high memory, low compute	Memory climbing while SM utilization trends flat or down

One deliberate non-recommendation: do not page on framebuffer percentage. 95 to 99% used is normal for well-tuned training. The alert belongs on rate of change against the workload’s own baseline, and OOM detection itself belongs in application logs, where the allocation failure is actually reported.

Fixes

Caching allocator behavior (not a leak)

No fix needed. If the plateau is simply too high for co-located workloads, that is a capacity and scheduling decision, not a bug. Note that torch.cuda.empty_cache() only releases cached blocks that no tensor is using; it does not free memory backing live tensors, so it cannot stop a real leak.

Retained tensors in application code

This is a code fix: drop references, detach() tensors pulled out of the graph, avoid appending CUDA tensors to per-step lists, and move anything you only need for logging to CPU at capture time. There is no operational workaround that reclaims live tensors; the process must stop retaining them. Until the fix ships, a scheduled restart after checkpointing bounds the blast radius.

Custom CUDA extension not freeing

Fix the extension. Operationally, the only way to reclaim memory held below the framework is to terminate the process cleanly (SIGTERM first, so the context is destroyed in order). If the extension is third-party, pin to a known-good version and report upstream with your per-step growth numbers.

Growing KV cache in serving

Bound the cache: cap maximum sequence length, cap concurrent sequences, and verify the engine’s eviction policy is actually triggering. If the cache only ever grows, restarts are the containment until the eviction bug is fixed. Watch for this specifically after serving engine upgrades; cache management regressions are a recurring source.

Orphaned CUDA context

Find the holder with fuser -v /dev/nvidia*, translate to the host PID if containers are involved, and kill it. This is disruptive: verify the PID is genuinely a zombie and not a live job on another GPU sharing the device nodes. VRAM usually drops immediately. If the holder is gone but memory stays held, the context is stuck in the driver, and you are in GPU reset territory. See Resetting a wedged NVIDIA GPU.

Fragmentation

Release cached blocks with torch.cuda.empty_cache() at phase boundaries, and consider the allocator’s max_split_size_mb tuning to reduce split fragmentation. Longer term, stabilize allocation shapes: dynamic batch sizes and model switching are the usual fragmentation drivers in inference serving.

Prevention

Baseline per workload. Record the expected post-warmup plateau for each training config and serving model. A leak alert is “3% above baseline and rising,” never “above 90%.”
Alert on slope, not level. Rate of change of memory.used over a multi-hour window, per GPU, per workload. This catches the leak at hour two instead of at OOM.
Per-step memory logging in soak tests. Log memory_allocated() per step in CI and in a long soak run before promoting new training code. A leak that OOMs in 18 hours of production shows up in a 30-minute soak as a nonzero slope.
Track zombie contexts. Alert on processes holding GPU memory with zero compute utilization, and on memory still attributed to PIDs that have exited.
Sample fast enough. GPU failure modes evolve in seconds. Minute-resolution monitoring can miss the transitions that distinguish a leak from a phase change; use sampling at 10 seconds or better for FB usage.
Watch the edge cases that mimic leaks. First-step allocator ramp, evaluation-phase spikes, and lazy post-SIGKILL reclaim all look like growth. Automating restarts on “memory rising” without the plateau check will kill healthy jobs.

For the broader set of signals every production GPU node should carry, see the NVIDIA GPU monitoring checklist.

How Netdata helps

Per-second framebuffer usage per GPU, so the slope of a leak is visible as a slope, not as two disconnected readings an hour apart.
Long retention at high resolution, which is what makes “climbing without a plateau over 12 hours” a queryable shape instead of a hunch.
Per-process GPU memory and utilization attribution, exposing the high-memory-plus-zero-compute signature of leaks and zombie contexts without manual pmon sessions.
Correlation of FB usage against SM utilization, power draw, and temperature on one timeline, which separates a memory leak from a thermal or throttle incident that happens to coincide.
Alerting on rate of change and deviation from learned baselines, matching how leaks actually present, rather than static percentage thresholds that false-positive on healthy training jobs.
Application log and Xid event correlation, so the CUDA OOM in the job log lines up with the memory curve and any driver-level events on the same GPU.