$ guides / nvidia-gpu / nvidia-gpu-bar1-memory-exhaustion ▌

Operations Guides

NVIDIA BAR1 memory exhaustion: mapping failures with free framebuffer

Your job dies with a CUDA allocation or mapping error, but nvidia-smi shows gigabytes of framebuffer free. You check for fragmentation, restart the job, and it fails again at the same place. The resource that ran out is not VRAM. It is the BAR1 aperture, the PCIe-mapped window the CPU uses to reach GPU memory directly.

BAR1 exhaustion is a distinct failure mode from framebuffer OOM. A GPU can have most of its framebuffer free and still refuse new mappings because every process, IPC handle, and GPUDirect RDMA registration consumes space in a small, shared aperture. It is most common in multi-process, MPS, containerized, and multi-tenant environments where many processes map GPU memory at once.

This article covers how to confirm BAR1 is the bottleneck, what consumes it, and how to recover without blindly rebooting the node.

What this means

BAR1 (Base Address Register 1) is a PCIe region that maps GPU memory into the host CPU’s address space. The driver uses it for CPU-visible allocations, CUDA IPC between processes, GPUDirect RDMA registrations, and various internal mappings. Every concurrent consumer takes a slice.

The aperture is finite and often surprisingly small. On older and consumer GPUs without Resizable BAR, BAR1 can be as small as 256 MiB regardless of how much VRAM the card has. On datacenter GPUs with Resizable BAR enabled (which requires both BIOS and OS support), BAR1 can be sized to match the framebuffer, which makes exhaustion unlikely.

When BAR1 fills up, new mapping requests fail. Depending on the call path, that surfaces as a CUDA allocation error, a CUDA IPC failure, or a GPUDirect RDMA registration failure that forces a fallback to slower bounce buffers. The framebuffer itself is fine. The failure is in the mapping layer.

flowchart TD
  A["CUDA allocation or IPC fails"] --> B{"Framebuffer free?"}
  B -->|"No"| C["Real VRAM exhaustion
see CUDA OOM guide"]
  B -->|"Yes, GB free"| D["Check BAR1 used vs total"]
  D -->|"BAR1 > 80%"| E["BAR1 exhaustion:
too many mappings,
leaked mappings, or tiny BAR1"]
  D -->|"BAR1 low"| F["Different cause:
FB fragmentation,
reserved overhead, app bug"]

Common causes

Cause	What it looks like	First thing to check
Too many concurrent processes mapping one GPU	BAR1 grows with process count; failures start when a new job lands on an already shared GPU	Count compute apps per GPU and compare against BAR1 usage
Leaked or unreleased mappings	BAR1 stays high after jobs exit; climbs monotonically over days	Compare BAR1 before and after killing all GPU processes
Small BAR1 aperture (no Resizable BAR)	BAR1 total is 256 MiB on a card with tens of GB of VRAM	`nvidia-smi -q -d MEMORY` BAR1 total
CUDA IPC heavy workloads	Failures correlate with inter-process tensor sharing (for example, inference servers passing tensors between workers)	Correlate failures with IPC usage in application logs
GPUDirect RDMA registrations	BAR1 high on nodes doing InfiniBand/RoCE direct transfers; throughput degrades before outright failure	BAR1 usage on RDMA-enabled nodes; registration errors in fabric logs
Driver-branch mapping bugs	BAR1 climbs under sustained mapping churn with no workload growth; may end in GPU lockup	Driver version against known issues; dmesg for NVRM mapping errors

Quick checks

All of these are read-only and safe on a production node.

# Human-readable BAR1 usage per GPU (Total / Used / Free)
nvidia-smi -q -d MEMORY

# Machine-readable BAR1 usage
nvidia-smi --query-gpu=memory.bar1.total,memory.bar1.used,memory.bar1.free --format=csv,noheader,nounits

The exact BAR1 field names vary by driver branch and were not verified against every branch. If the query returns an error, list the names your driver supports and adjust:

# Confirm the BAR1 field names on your driver
nvidia-smi --help-query-gpu | grep -i bar1

# Live view: fb and bar1 columns (in MB), refreshing
nvidia-smi dmon -s m

# Which processes hold GPU contexts, and how much FB each uses
nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv,noheader

# Driver messages: mapping failures and Xid events
dmesg -T | grep -i "NVRM"

Two things to note when reading output. First, nvidia-smi does not attribute BAR1 per process, so attribution is indirect: count processes and watch BAR1 as they come and go. Second, in containers the PID shown is the host PID, not the container PID, so you may need namespace translation to find the real owner.

In dmesg, some driver branches log explicit mapping failures such as NVRM: dmaAllocMapping... can't alloc VA space for mapping when BAR1 virtual address space runs out. The exact message text is driver-version dependent; the absence of a message does not rule BAR1 out.

How to diagnose it

Confirm the symptom shape. Collect the application error and nvidia-smi output at failure time. If the error is an allocation or mapping failure while memory.free shows gigabytes available, BAR1 or fragmentation are the candidates. If free memory is genuinely near zero, you have a normal framebuffer OOM; see CUDA out of memory: diagnosing NVIDIA GPU framebuffer exhaustion.
Read BAR1 directly. Run nvidia-smi -q -d MEMORY on the affected GPU and compute used/total. Above 80% in a multi-process environment is the investigate line; above 90% in a GPUDirect RDMA environment is a ticket-level condition.
Check the aperture size. If BAR1 total is 256 MiB on a card with far more VRAM, Resizable BAR is off or unsupported on this host. That alone makes exhaustion plausible under modest process counts.
Correlate with process count. List compute apps. If BAR1 tracks the number of concurrent processes, you have a capacity problem: too many mappers for the aperture. If BAR1 is high with few or no processes, you have leaked mappings.
Test the leak hypothesis. Drain the GPU of workloads (or pick a quiet window) and watch BAR1. If it does not fall when processes exit, mappings are being held by stale contexts or a driver-level leak. Check dmesg for NVRM errors and note the driver version.
Correlate with Xid events. Application-side Xids (13, 31, 43) point at application bugs; mapping failures from aperture exhaustion are a driver-resource problem. Keeping the two apart changes who fixes it. See NVIDIA Xid 31: GPU memory page fault (invalid address) if Xid 31 appears.
Check for the wedged-GPU end state. Sustained mapping churn against a full aperture has, on some driver branches, ended in a GPU that stops responding. If nvidia-smi starts hanging or one GPU stops answering queries, treat it as a separate, more severe incident: see nvidia-smi hangs or is unresponsive.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`bar1.used / bar1.total` ratio	The direct measure of aperture pressure	>80% in multi-process environments; >90% in GPUDirect RDMA environments
BAR1 trend over days	Leaked mappings show up as a floor that never resets	Baseline ratcheting upward across job boundaries
Compute app count per GPU	Proxy for mapping concurrency	Count climbing toward the point where BAR1 historically saturates
`memory.free` at failure time	Distinguishes BAR1 exhaustion from real VRAM OOM	Allocation errors with large free framebuffer
Application CUDA IPC / mapping errors	The user-visible symptom	Recurring mapping failures at roughly constant BAR1 levels
NVRM messages in dmesg	Driver-side confirmation of mapping failure	`can't alloc VA space` or `NV_ERR_NO_MEMORY` style messages
GPUDirect RDMA throughput	Registration failure forces bounce-buffer fallback	Throughput drop on RDMA nodes with high BAR1

Fixes

Reduce concurrent mappings on the GPU

If BAR1 pressure tracks process count, cap how many processes share the device. Co-schedule fewer tenants per GPU, or stagger job starts so mappings do not peak simultaneously. On multi-tenant nodes this is a scheduling policy fix, not a GPU fix.

If you use CUDA IPC heavily (for example, worker processes sharing tensors), reduce the number of IPC peers per GPU or restructure so tensors move through fewer mapped handles.

Clear leaked mappings

Find and stop the processes holding stale contexts. nvidia-smi --query-compute-apps shows what the driver thinks is attached; zombie contexts from crashed processes may need a kill -9 on the host PID. Watch BAR1 as each process exits to identify the leaker.

A known historical variant: PyTorch DataLoader setups with num_workers > 0 leaked BAR1 when the main process was killed with SIGKILL/SIGTERM on certain driver branches (reported fixed in 515.65.01). If your driver is older than that and you kill training jobs routinely, upgrading the driver is the durable fix.

If processes are gone and BAR1 stays allocated, the mappings are held at the driver level. A GPU reset releases them:

# Destructive: kills all work on the GPU and fails if any process is attached
nvidia-smi --gpu-reset

Reset only after draining the device. If the reset fails or the GPU does not come back clean, the remaining option is a node reboot. See Resetting a wedged NVIDIA GPU for the full decision tree.

Enlarge the aperture: Resizable BAR

If BAR1 is 256 MiB, enabling Resizable BAR lets the aperture grow to match the framebuffer on supported GPUs. This requires both BIOS support (an above-4G decoding / Resizable BAR option) and OS/driver support; on the open kernel modules, the NVreg_EnableResizableBar=1 module parameter exists since driver 530.41.03.

Caveats before you schedule the reboot:

This is a BIOS change plus a reboot, so it needs a maintenance window.
On multi-GPU hosts, PCIe address space is finite. There are reports of systems where most GPUs get a large BAR1 but one GPU gets a smaller one because the host ran out of address space. Verify BAR1 size on every GPU after the change, not just the first.
Consumer platforms vary widely in whether the option exists and works.

Upgrade the driver for mapping-churn bugs

Recent open-kernel-module branches have had bugs where sustained mapping reuse churn exhausts BAR1 virtual address space and locks the GPU, even at low framebuffer usage. If your BAR1 climbs under a stable workload and dmesg shows NVRM mapping errors, check the open-gpu-kernel-modules issue tracker for your branch and plan a driver upgrade. Treat a GPU that has locked up once from this as suspect until the driver is changed.

Prevention

Alert on the ratio, not the absolute. Track bar1.used / bar1.total per GPU. Investigate at 80% in multi-process environments; ticket at 90% where GPUDirect RDMA is in play.
Baseline per workload class. A single-process training job should sit near zero BAR1. Multi-tenant inference nodes will have a higher normal. Alert on deviation from the workload’s own baseline.
Enforce per-GPU tenancy limits. The number of processes that can safely map one GPU is a property of the aperture size, not the framebuffer size. Encode it in the scheduler.
Track BAR1 across job boundaries. A floor that rises week over week is a leak; catch it before the cliff.
Verify BAR1 size at provisioning. Include BAR1 total in node acceptance checks, especially after BIOS changes or hardware swaps. A node that silently comes up with a 256 MiB aperture will fail later under multi-process load.
For the broader GPU signal picture, see NVIDIA GPU monitoring checklist: the signals every production GPU fleet needs and the GPU subsystem model in How an NVIDIA GPU actually works in production.

How Netdata helps

Netdata collects per-GPU memory metrics at per-second resolution, so BAR1 pressure is visible as a trend rather than a postmortem surprise; minute-resolution polling misses the ramp before failures.
Charting BAR1 used next to framebuffer used makes the signature obvious: flat or falling FB with climbing BAR1 means a mapping problem, not an OOM problem.
Per-GPU process counts alongside BAR1 usage let you see whether pressure tracks tenancy (capacity problem) or grows independently (leak).
Anomaly detection flags a BAR1 baseline that is ratcheting upward across job boundaries, the early leak signal that static thresholds miss.
Correlating BAR1 with application error timing and system log events in one view shortens the path from “CUDA error” to “aperture exhausted.”