NVIDIA GPU Operations Guides on Netdata

CUDA out of memory with free memory available: GPU memory fragmentation

Mon, 01 Jan 0001 00:00:00 +0000

Your training run or inference server has been up for hours. Then it dies with torch.cuda.OutOfMemoryError: CUDA out of memory. You check nvidia-smi and the GPU shows gigabytes free. The error message and the driver disagree, and the driver looks right.

Both are right. The GPU has free memory in aggregate, but no single contiguous block large enough for the allocation that just failed. The memory is fragmented: free in total, unusable in practice.

CUDA out of memory: diagnosing NVIDIA GPU framebuffer exhaustion

Mon, 01 Jan 0001 00:00:00 +0000

Your training job or inference server died with RuntimeError: CUDA out of memory. Tried to allocate X MiB. Sometimes the GPU is genuinely full. More often, nvidia-smi shows gigabytes free and the error makes no sense at first glance.

GPU framebuffer is a cliff-edge resource. Unlike CPU memory there is no swap and no graceful degradation: when a cudaMalloc cannot be satisfied, the allocation fails atomically and the process dies. It is also one of the most misdiagnosed GPU failures, because the number nvidia-smi shows is not the number your framework is working with.

DCGM nv-hostengine down or hung: GPU telemetry goes dark or stale

Mon, 01 Jan 0001 00:00:00 +0000

Your GPU dashboards flatline, or worse, they do not. The graphs keep rendering smooth, plausible values for temperature, utilization, and power, but nothing on the node has actually changed in twenty minutes. Both symptoms point at the same component: nv-hostengine, the DCGM daemon that sits between NVML and every monitoring tool you run.

nv-hostengine fails in two distinct ways, and only one of them is obvious. The obvious failure is a dead process: no data, gaps in every time series, alerts firing on absent metrics. The dangerous failure is an alive-but-hung daemon: the process exists, the socket accepts connections, and DCGM serves stale values from its in-memory field value cache because an underlying NVML call is blocked on the driver. Dashboards look normal. Threshold alerts stay quiet because the cached values are inside bounds. You find out when a human notices the numbers have not moved.

DCGM, nvidia-smi, and NVML: which NVIDIA GPU telemetry source to use

Mon, 01 Jan 0001 00:00:00 +0000

Every NVIDIA GPU troubleshooting session starts with a telemetry question: where do I get this number, and can I trust it? Teams routinely mix the three available sources without realizing they sit at different layers of the same stack. They grep nvidia-smi output in cron jobs, run DCGM and nvidia-smi side by side and wonder why both get slow, or alert on a DCGM field that silently returns zeros on their driver version.

How an NVIDIA GPU actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

GPU runbooks fail when they treat the GPU as a black box that emits a utilization percentage. “100% utilization but slow” makes sense only when you know what utilization measures. “nvidia-smi hangs” makes sense only when you know where nvidia-smi gets its data.

This guide covers the two pieces underneath GPU troubleshooting: the subsystems that produce production failures, and the telemetry plane through which you observe them. It applies primarily to datacenter GPUs such as V100, A100, and H100, with differences noted for consumer cards.

Monitoring MIG-partitioned NVIDIA GPUs: per-instance metrics

Mon, 01 Jan 0001 00:00:00 +0000

On a MIG-enabled GPU (A100, A30, H100 and later), the physical card is partitioned into isolated GPU instances, each with its own SMs, memory partition, and failure domain. Every monitoring habit built around whole-GPU metrics breaks here: aggregate utilization, aggregate memory, and even some health counters describe the card, not the tenant. An instance can be OOM while nvidia-smi shows healthy free memory on the card.

The second surprise: nvidia-smi and NVML do not attribute utilization to MIG devices at all. Per-instance utilization reads as N/A by design. The supported path for per-instance metrics is DCGM, queried with MIG-aware entity types, and not every DCGM field is available at instance granularity.

NVIDIA BAR1 memory exhaustion: mapping failures with free framebuffer

Mon, 01 Jan 0001 00:00:00 +0000

Your job dies with a CUDA allocation or mapping error, but nvidia-smi shows gigabytes of framebuffer free. You check for fragmentation, restart the job, and it fails again at the same place. The resource that ran out is not VRAM. It is the BAR1 aperture, the PCIe-mapped window the CPU uses to reach GPU memory directly.

BAR1 exhaustion is a distinct failure mode from framebuffer OOM. A GPU can have most of its framebuffer free and still refuse new mappings because every process, IPC handle, and GPUDirect RDMA registration consumes space in a small, shared aperture. It is most common in multi-process, MPS, containerized, and multi-tenant environments where many processes map GPU memory at once.

NVIDIA DCGM exporter on Kubernetes: mapping GPU metrics to pods

Mon, 01 Jan 0001 00:00:00 +0000

On a bare-metal GPU node, attribution is simple: nvidia-smi shows you a PID, you look up the process, done. On Kubernetes there are three layers between you and that answer. The device plugin assigns GPUs to pods, the kubelet tracks those assignments, and dcgm-exporter reads GPU state through NVML/DCGM, which has no idea what a pod is. Unless the exporter explicitly joins these two views, your GPU metrics are half-blind: you can see GPU 3 on node 17 at 98% utilization, but not which workload is responsible.

NVIDIA driver and CUDA version mismatch: subtle failures and crashes

Mon, 01 Jan 0001 00:00:00 +0000

The loud version of this problem is easy: a job dies at startup with CUDA driver version is insufficient for CUDA runtime version, and the fix is a driver upgrade. The version that ruins your week is quieter. A kernel update goes out fleet-wide, the NVIDIA module rebuilds on most nodes but not all, and now a subset of training jobs crash with kernel-launch failures or, worse, run to completion with subtly wrong results. Nothing in your dashboards looks broken: GPUs up, memory fine, temperatures normal.

NVIDIA Fabric Manager not running: NVSwitch GPUs lose NVLink

Mon, 01 Jan 0001 00:00:00 +0000

Your multi-GPU training job hangs during NCCL initialization, or fails with cudaErrorSystemNotReady (error 802) the moment a process touches CUDA. Each GPU shows up in nvidia-smi with normal temperature, memory, and utilization. You burn hours on NCCL debug logs, InfiniBand checks, and application-level bisection before someone runs systemctl status nvidia-fabricmanager and finds the service failed two days ago after a reboot.

On DGX and HGX systems, the NVSwitch fabric that gives GPUs their all-to-all NVLink connectivity is not self-configuring. A userspace daemon, nv-fabricmanager (the nvidia-fabricmanager systemd unit), programs the switch routing tables and health-checks the fabric. If that daemon crashes, never started, or aborted on a version mismatch, the GPUs are individually healthy and collectively deaf.

NVIDIA GPU clock throttle reasons: why the GPU isn't running at full speed

Mon, 01 Jan 0001 00:00:00 +0000

A training job that used to finish an epoch in 40 minutes now takes 90. nvidia-smi shows 100% GPU utilization, memory usage looks normal, and the workload is running, just slowly. This is the classic throttling symptom: the GPU is executing kernels at reduced clock speeds, so everything takes longer while utilization stays high.

The mistake most teams make here is guessing. They check temperature, see 82C, and debate whether that is “too hot.” They see power draw pinned at the limit. None of that says why the clocks are down. The GPU already knows, and it reports the answer directly through the clock event reasons bitmask.

NVIDIA GPU configuration drift: ECC, persistence, power limits, compute mode

Mon, 01 Jan 0001 00:00:00 +0000

A GPU that passes every health check can still be misconfigured. ECC turned off, persistence mode lost after a reboot, a power limit set below default, compute mode flipped to Exclusive. None of these produce an XID error or a crashed job on their own. They produce silent data corruption, unexplained throttling, scheduling failures, and monitoring gaps that surface days later as a different incident.

Configuration drift is a control-plane problem, not a hardware problem. The fix is comparing the live configuration of each GPU against a known-good per-model baseline and treating every unexplained difference as a signal. This guide covers the four knobs that matter most (ECC mode, persistence mode, power limit, compute mode), plus InfoROM integrity, and how to tell an unauthorized change from a failed automation run or a maintenance leftover.

NVIDIA GPU ECC disabled: the silent data-corruption risk

Mon, 01 Jan 0001 00:00:00 +0000

A datacenter GPU with ECC disabled does not crash when memory goes bad. It keeps running, keeps returning answers, and keeps writing checkpoints, while single-bit errors corrupt whatever it is computing. There is no error counter incrementing, no XID in dmesg, no page in the middle of the night. The usual discovery path is a training run that diverges to NaN on one specific node, or an inference service returning subtly wrong results, days or weeks after someone flipped the setting.

NVIDIA GPU ECC errors: corrected, uncorrected, volatile, and aggregate

Mon, 01 Jan 0001 00:00:00 +0000

You opened nvidia-smi -q -d ECC because something looked off, and now you are staring at four sets of counters with non-zero values and no idea which ones matter. Maybe a monitoring alert fired on “ECC errors present”. Maybe a training run produced NaN loss on one node and someone told you to check ECC. Either way, the raw output does not tell you the two things you need to know: has data been corrupted, and is this GPU dying.

NVIDIA GPU fan at 0%: fan failure on air-cooled cards

Mon, 01 Jan 0001 00:00:00 +0000

nvidia-smi shows Fan: 0% and the GPU is sitting at 70C under load. Either the fan is dead, or the card is doing exactly what its firmware told it to do. Telling those two apart quickly is the whole job: the wrong guess in one direction means a cooked GPU, the wrong guess in the other means a pointless RMA.

A healthy air-cooled card spins its fans up as the die warms and spins them down at idle. Zero RPM while the GPU is above idle temperature is a failed fan until proven otherwise. Sustained 100% with temperature still climbing is the opposite failure: the fan works but cooling capacity is exceeded.

NVIDIA GPU HBM (memory) temperature: the thermal limit most teams miss

Mon, 01 Jan 0001 00:00:00 +0000

Most GPU fleets monitor one temperature: temperature.gpu, the die temperature. Alerts, dashboards, and throttle runbooks are built around it. On datacenter GPUs with HBM (A100, H100, H200), that is only half the picture, and for memory-bound work it is often the wrong half.

The HBM stacks sit on the same package as the die but have a worse heat path. Under LLM inference, embedding-heavy workloads, and memory-bound training, HBM temperature commonly runs 10 to 20 C hotter than the die. HBM also has its own throttle point. When it trips, the GPU reduces memory bandwidth with no Xid, no error, and no log entry most teams watch. The job keeps running. It just gets slower while die temperature still looks healthy.

NVIDIA GPU HBM progressive failure: from single-bit errors to a dead GPU

Mon, 01 Jan 0001 00:00:00 +0000

HBM failure on a datacenter GPU is almost never a surprise if you are watching the right signals. It is a staged degradation: single-bit ECC errors rise over days or weeks, the GPU burns through its self-repair capacity (row remaps on Ampere and later, page retirement on pre-Ampere), the first uncorrectable double-bit error lands, and eventually the GPU can no longer repair itself or falls off the PCIe bus entirely.

NVIDIA GPU HW Power Brake Slowdown: the chassis is cutting GPU power

Mon, 01 Jan 0001 00:00:00 +0000

You are looking at a GPU that is running, reachable, not overheating, and still delivering a fraction of its normal throughput. nvidia-smi -q -d PERFORMANCE shows HW Power Brake Slowdown : Active (and usually HW Slowdown : Active alongside it), SM clocks are at half their rated speed or lower, and the temperature looks fine. The workload did not change. The GPU did not change. Something outside the GPU decided it gets less power.

NVIDIA GPU memory leak: framebuffer usage climbing without a plateau

Mon, 01 Jan 0001 00:00:00 +0000

Your training job has been running for six hours. nvidia-smi showed 40 GiB used after warmup. Then 45. Then 52. The curve has not flattened, and at this rate the job dies with CUDA out of memory sometime tomorrow, taking a day of checkpoint progress with it.

GPU memory is a cliff resource: there is no swap and no graceful degradation. When an allocation fails, it fails immediately. The saving grace is that a true leak is visible hours before it kills you, as sustained linear growth in framebuffer usage. The hard part is telling that apart from the many things that look like a leak but are not, because ML frameworks deliberately fill memory and hold it.

NVIDIA GPU memory-bandwidth bound: high DRAM activity, idle SMs

Mon, 01 Jan 0001 00:00:00 +0000

Your training job or inference service is running, nvidia-smi shows high GPU utilization, power draw looks healthy, and throughput is far below what the hardware should deliver. The profiling metrics show the pattern: the device memory interface is active nearly 100% of cycles while the streaming multiprocessors sit mostly idle. The GPU is memory-bandwidth bound. It is not short on compute. It is waiting on HBM.

This is one of the most misdiagnosed GPU performance problems in production. Time-based utilization metrics make the GPU look busy, so teams conclude they need more GPUs, higher clocks, or a bigger power limit. None of those help. The SMs are starving because data cannot move from HBM fast enough, and the only real fixes are algorithmic: fewer bytes moved per FLOP.

NVIDIA GPU monitoring checklist: the signals every production GPU fleet needs

Mon, 01 Jan 0001 00:00:00 +0000

Most GPU monitoring setups fail in one of two ways. Either they collect a handful of nvidia-smi counters and page on the wrong things (raw temperature, memory percentage, idle PCIe downgrade), or they collect everything and alert on nothing meaningful. Both failure modes come from the same root cause: no shared vocabulary for which signals matter, at what fidelity, and with what alert semantics.

This checklist organizes the signals a production GPU fleet needs into four tiers: survival, operational, mature, and expert. Each tier builds on the previous one. The intent is not that every fleet reaches expert. The intent is that you can place your current setup on the ladder and know which gaps to close next, in what order.

NVIDIA GPU monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Most GPU fleets sit at one of two extremes: nothing beyond “does nvidia-smi still respond,” or a wall of dashboards nobody wired to an alert. Neither survives a real incident. A GPU falling off the bus at 3 a.m., a training job running 5x slower because one card is thermally throttled, an accelerating ECC error rate that becomes silent data corruption next week: all detectable, but only if you collect the right signals at the right fidelity.

NVIDIA GPU NVLink down or degraded: a link that dropped in multi-GPU training

Mon, 01 Jan 0001 00:00:00 +0000

A link that drops mid-training is one of the nastiest GPU failures to catch, because the job usually does not crash. It hangs at a synchronization barrier, or it keeps running at a fraction of its normal step rate. Bandwidth between GPUs collapses from hundreds of GB/s (600 GB/s on A100, 900 GB/s on H100) to PCIe speeds, and every AllReduce now waits on the slowest path in the collective.

NVIDIA GPU NVLink errors: CRC and replay errors on GPU-to-GPU links

Mon, 01 Jan 0001 00:00:00 +0000

Your training job is running. No crashes, no CUDA errors, no OOM. But step time is up roughly 20%, and the slowdown shows up exactly during the collective phases: AllReduce, gradient sync. The team has already blamed the data loader, shuffled hyperparameters, and re-run the job twice. The actual problem is one degraded NVLink cable retransmitting corrupted flits, and nothing in the default monitoring stack is looking at it.

NVLink CRC and replay errors are the classic silent degradation on multi-GPU NVIDIA systems. The link layer detects corruption and retransmits, so data still arrives intact. What you lose is bandwidth. Because nothing fails, the symptom is a slow job, not an error message, and misdiagnosis is common. This guide covers how to read the per-link error counters, tell one bad link from a systemic problem, correlate with Xid 74, and isolate GPU versus NVSwitch faults on HGX/DGX systems.

NVIDIA GPU PCIe link running below maximum: x16 silently training at x8

Mon, 01 Jan 0001 00:00:00 +0000

The job runs. Health checks pass. nvidia-smi shows the GPU, temperature is fine, ECC is clean, and utilization looks normal. Yet training throughput is half of what it should be, or one GPU in an eight-GPU node drags every collective operation. A common root cause is a PCIe link that negotiated below its capability: a Gen4 x16 card silently running at Gen3, or worse, at x8 or x4.

Nothing errors. The link trained successfully, just at a lower speed or width. The driver is happy, the kernel is happy, and every green dashboard panel is technically true. The only symptom is bandwidth you no longer have: Gen4 to Gen3 costs roughly 50% of link bandwidth, x16 to x8 costs 50%, and x16 to x4 costs 75%.

NVIDIA GPU PCIe replay counter rising: physical-layer signal integrity

Mon, 01 Jan 0001 00:00:00 +0000

The PCIe replay counter on one of your GPUs is climbing. Not a burst at boot, but a sustained rate of hundreds or thousands per second. Nothing has crashed yet: training still runs, nvidia-smi still answers, and there are no XID errors in dmesg. That is exactly why this signal gets ignored, and exactly why it should not be.

A PCIe replay is a link-layer retransmission. The GPU (or the root complex, switch, or retimer on the path) received a packet that failed its CRC check and asked for it again. Occasional replays are normal PCIe behavior. A sustained high rate means bits are being corrupted on the wire often enough that retransmission is routine, not exceptional. That is a physical-layer signal integrity problem: a cable, connector, slot, riser, or an overheating retimer or PCIe switch somewhere between the CPU and the GPU.

NVIDIA GPU PCIe straggler: one slow link throttling multi-GPU training

Mon, 01 Jan 0001 00:00:00 +0000

Your distributed training job is running. No errors, no crashes, no XIDs. But step time jumped 30-60% and never came back, and scaling from 4 GPUs to 8 GPUs bought almost nothing. Per-GPU utilization shows one GPU a few points below the rest, and its PCIe throughput is consistently below its peers.

That is the PCIe straggler pattern. One GPU’s link negotiated at a lower generation or narrower width than it should have: Gen3 instead of Gen4, x8 instead of x16, or worse. That GPU now feeds data and exchanges gradients at half (or less) of its peers’ bandwidth. In synchronous data-parallel training, every collective waits for the slowest participant, so the whole job runs at the speed of the worst link.

NVIDIA GPU Remapping Failure Occurred: HBM spare rows exhausted (Ampere+)

Mon, 01 Jan 0001 00:00:00 +0000

You ran nvidia-smi -q -d ROW_REMAPPER (or your monitoring did) and saw Remapping Failure Occurred : Yes. This is not a driver problem, not a configuration problem, and not something a reboot clears. It is the GPU telling you that its hardware self-repair budget is spent.

On Ampere and later GPUs (A100, H100, H200, Blackwell), the memory subsystem does not retire whole pages the way Volta and Turing did. Instead, when HBM rows start failing, the GPU remaps them to a finite pool of spare rows built into each DRAM bank. While spares remain, the GPU heals itself transparently: correctable ECC events trigger remaps, XID 63 appears in the kernel log as an informational note, and workloads never notice.

NVIDIA GPU retired pages: page retirement, pending retirements, and end of life

Mon, 01 Jan 0001 00:00:00 +0000

Retired pages are the permanent record of a GPU’s memory hardware degrading. Every retirement means the driver found a framebuffer page it could no longer trust and removed it from the allocatable pool for good. The counts persist in the GPU’s InfoROM across reboots and driver reloads, which makes them one of the few genuinely cumulative hardware health signals nvidia-smi exposes.

Operators usually hit this signal one of two ways: monitoring flags retired_pages.pending = Yes and nobody knows whether to panic, or someone asks why a GPU with 40 GB of HBM shows slightly less memory than its siblings. Both require understanding the difference between SBE-retired and DBE-retired pages, what a pending retirement means for running workloads, and where the end-of-life line sits.

NVIDIA GPU single-bit ECC error rate rising: the leading indicator of HBM failure

Mon, 01 Jan 0001 00:00:00 +0000

A GPU in your fleet has started logging corrected ECC errors faster than it used to. Maybe you noticed a bump in ecc.errors.corrected.volatile.total during a routine check, or Xid 92 events started appearing in dmesg. Nothing is broken yet: jobs run, loss curves look normal, nvidia-smi reports a healthy card. This is exactly the moment most teams get wrong. They look at the absolute count, decide it is small, and move on.

NVIDIA GPU SW Power Cap: performance capped by the power limit

Mon, 01 Jan 0001 00:00:00 +0000

nvidia-smi reports SW Power Cap: Active under Clocks Event Reasons, the workload is running slower than it should, and the GPU is not hot. This is the power-limit throttle: the GPU’s power scaling algorithm is holding clocks below what the workload requested because the board is consuming as much power as it is allowed to.

Two things make this symptom confusing. First, SW Power Cap is not always a fault. A GPU at full load sitting exactly at its power limit is doing the maximum work its TDP allows, and datacenter operators often cap GPUs below the factory limit on purpose. Second, the number that actually matters is enforced.power.limit, not power.limit, and the two can disagree. Reading the wrong field sends you chasing a misconfiguration that does not exist, or missing one that does.

NVIDIA GPU temperature too high: die temperature and thermal limits

Mon, 01 Jan 0001 00:00:00 +0000

A GPU reporting 85 °C is not necessarily a GPU with a temperature problem. Under sustained load, datacenter GPUs like the A100 and H100 are designed to run hot, and 80 °C is routine under full load. The number alone tells you almost nothing.

What matters is where the temperature sits relative to the SKU’s own thermal limits, whether the GPU has started throttling, and whether the pattern is one hot GPU or a whole chassis running hot. A GPU at 85 °C with no throttle reasons active is fine. A GPU at 75 °C with hw_thermal_slowdown active has a cooling system that is already failing.

NVIDIA GPU Tensor Cores idle: mixed precision not being used

Mon, 01 Jan 0001 00:00:00 +0000

Your training job is running, nvidia-smi shows high SM utilization, power draw looks healthy, and step time is 5 to 10 times worse than the hardware spec says it should be. DCGM profiling shows Tensor Core Active pinned at or near zero. The GPU is busy, but it is doing the work on CUDA cores instead of Tensor Cores.

Nothing errors. Nothing crashes. The job just runs at a fraction of the throughput you paid for, and because SM utilization still reads 90%+, it looks healthy in every dashboard that only tracks utilization.gpu.

NVIDIA GPU thermal cascade in dense servers: one hot GPU heats its neighbours

Mon, 01 Jan 0001 00:00:00 +0000

Three of the eight GPUs in a node are thermal throttling. Training step time has doubled. The instinct is to suspect the cooling system as a whole, or the workload. But line up per-GPU temperatures and one unit stands out: GPU 5 is running 12 degrees hotter than everything else, and the GPUs downstream of it in the airflow path are the ones throttling.

This is the thermal cascade pattern, specific to dense GPU servers (DGX, HGX, and similar trays) where GPUs share a cooling path. One GPU’s exhaust air is another GPU’s inlet air. A single unit with a local cooling problem raises the intake temperature of its neighbours, pushing them past their thermal limits. A throttled GPU still draws substantial power, so it keeps generating heat while producing less useful work. The system settles into a stable, badly degraded equilibrium: several GPUs throttled, all because of one.

NVIDIA GPU thermal throttling: clocks dropping under heat

Mon, 01 Jan 0001 00:00:00 +0000

Training throughput fell 25% overnight. No errors in the application logs. nvidia-smi shows every GPU at 100% utilization. Nothing looks broken, and yet the job is measurably slower.

This is what NVIDIA GPU thermal throttling looks like in practice: the GPU keeps working, keeps reporting full utilization, and silently delivers 10-40% less real work because its SM clocks have dropped. Utilization measures time busy, not work done. A kernel running at half clock speed still occupies the SMs for the whole sampling window, so utilization.gpu stays pinned at 100% while tokens/sec or samples/sec collapse.

NVIDIA GPU utilization low during training: a data-pipeline bottleneck

Mon, 01 Jan 0001 00:00:00 +0000

Your training job is running, the loss is decreasing, and nvidia-smi shows the GPU at 20-40% utilization, or oscillating between 0% and 100% in a sawtooth. The job finishes, so it looks healthy. It is not: you are paying for a GPU that is idle most of the time because the host cannot feed it.

Low utilization during active training is almost never a GPU fault. It means the GPU is starved: waiting on CPU preprocessing, disk or network data loading, pageable-memory copies across PCIe, or a synchronization barrier. The hardware is fine; the pipeline upstream of it is the bottleneck.

NVIDIA persistence mode: why the GPU keeps re-initializing and P-state flaps

Mon, 01 Jan 0001 00:00:00 +0000

A training node accepts a job, and the first CUDA call takes two seconds before any kernel runs. The next job lands, and it happens again. Your dashboard shows the GPU bouncing between P0 and P8 all day, clocks collapsing to idle between launches, power draw dropping toward zero. It looks like throttling. It looks like the GPU is restarting constantly. It is neither: the driver is unloading GPU state every time the last CUDA context exits, and reinitializing it on the next one.

NVIDIA Xid 13: Graphics Engine Exception

Mon, 01 Jan 0001 00:00:00 +0000

You found a line like this in the kernel log:

NVRM: Xid (PCI:0000:65:00.0): 13, pid='<unknown>', name=<unknown>, Graphics Engine Exception.

Or, more likely, you found the application-side symptom first: a training job died with a generic CUDA error, a run produced NaN loss, or an inference service started returning garbage, and only after digging into dmesg did Xid 13 show up.

Xid 13 is the most ambiguous Xid code NVIDIA emits. It means the graphics engine raised an exception while executing work. The three candidate causes are a bad CUDA kernel (out-of-bounds access, illegal instruction), a driver bug, or genuine hardware degradation. Most isolated occurrences are software. The operational problem is telling which case you are in, because the correct response ranges from “file a bug against the application” to “RMA the GPU”.

NVIDIA Xid 31: GPU memory page fault (invalid address)

Mon, 01 Jan 0001 00:00:00 +0000

You found NVRM: Xid (PCI:0000:xx:00.x): 31, ... in the kernel log, usually right after a training job or inference process died with CUDA error: an illegal memory access was encountered. The GPU raised a memory page fault: some unit on the chip accessed a virtual address that was not mapped to valid GPU memory.

The important thing to know up front: Xid 31 almost always means the application is wrong, not the GPU. Accessing freed GPU memory, bad pointer arithmetic, an out-of-bounds index in a kernel. These produce exactly this fault. The exception is when the same GPU produces Xid 31 across multiple unrelated applications. Then the suspicion flips to hardware.

NVIDIA Xid 43: GPU stopped processing (the GPU hang)

Mon, 01 Jan 0001 00:00:00 +0000

You see NVRM: Xid (PCI:0000:XX:00.0): 43, ... GPU stopped processing in dmesg, and the job on that GPU has gone silent. CUDA calls do not return. nvidia-smi either hangs outright or returns a stale view of the card: utilization frozen at the last sampled value, memory allocation unchanged, power draw flat. The GPU has not disappeared from the bus the way it does with Xid 79. It is still enumerated and still answering the driver at some level, but the compute engine is wedged and nothing it was running will complete.

NVIDIA Xid 48: Double Bit ECC Error

Mon, 01 Jan 0001 00:00:00 +0000

You found NVRM: Xid (PCI:....): 48, Double Bit ECC Error in dmesg, a training job just died, or a monitoring alert fired on a new uncorrected ECC error. This error is unambiguous: data corruption has occurred in GPU memory, and whatever was running on that GPU when it happened cannot be trusted.

The CUDA context that triggered the error is typically killed, recent outputs (weights, checkpoints, inference results) may be corrupt, and the GPU needs a reset or node reboot before it returns to clean service. There is no “watch and see” path. The correct response is to drain the GPU, verify whether the faulting memory was retired or remapped, validate or discard recent work, and schedule replacement.

NVIDIA Xid 63 and 64: ECC page retirement and row-remap events

Mon, 01 Jan 0001 00:00:00 +0000

You found a line like this in the kernel log:

NVRM: Xid (PCI:0000:10:1c): 63, pid=1896, Row Remapper: New row marked for remapping, reset gpu to activate.

Or the less welcome variant, Xid 64. Both come from the GPU’s memory self-repair machinery: ECC detected a bad region of HBM or GDDR memory, and the driver tried to permanently remove it from service. Xid 63 means the repair was recorded successfully. Xid 64 means the recording failed, which is a different and more serious situation.

NVIDIA Xid 74: NVLink error

Mon, 01 Jan 0001 00:00:00 +0000

You found NVRM: Xid 74 in dmesg. The driver detected an error on an NVLink connection between this GPU and another GPU or an NVSwitch. The confusing part: nothing crashed. Training is still running. What actually happened is that the link’s error recovery machinery fired - CRC failures, packet replays, link recovery events - and the driver logged it.

Xid 74 is the classic silent straggler. Retransmissions eat NVLink bandwidth, and collective operations like AllReduce synchronize at the speed of the slowest participant. Step time creeps up, throughput drops, and nobody gets paged because nothing “failed.” The job is just slower, and the GPU-hour bill is higher.

NVIDIA Xid 79: GPU has fallen off the bus

Mon, 01 Jan 0001 00:00:00 +0000

You will usually find this one by accident: a training job crashes with an opaque CUDA error, nvidia-smi hangs or shows ERR! for one device, and when you check dmesg you find NVRM: Xid (PCI:0000:XX:00): 79, GPU has fallen off the bus. The GPU is gone. Not throttled, not wedged, not holding stale contexts. The PCIe link between the host and the GPU has failed, and the device is no longer reachable at all.

NVIDIA Xid 92: high single-bit ECC error rate

Mon, 01 Jan 0001 00:00:00 +0000

You found a line like this in the kernel log:

NVRM: Xid (PCI:0000:3b:00.0): 92, pid='<unknown>', name=<unknown>, High single-bit ECC error rate

Nothing crashed. No job failed. nvidia-smi still works, the GPU is still training or serving, and there is no Xid 48 or 79 demanding an immediate reboot. That is exactly the point of Xid 92: the driver has seen the correctable single-bit ECC error rate cross its internal threshold. Nothing is broken yet. Something will be.

NVIDIA Xid 94 and 95: contained vs uncontained ECC errors (Ampere+)

Mon, 01 Jan 0001 00:00:00 +0000

You are looking at a kernel log line like NVRM: Xid (PCI:0000:01:00): 94, pid=7194, Contained: ... or ...: 95, pid=7062, Uncontained: ... on an A100, H100, or newer GPU, and you need to know two things fast: how bad is it, and what do you drain.

Both Xid 94 and Xid 95 are uncorrectable (double-bit class) ECC error events. The distinction NVIDIA draws, starting with the A100, is containment: whether the corruption was confined to the faulting application context or may have spread across everything running on the GPU. That distinction drives the entire response.

NVIDIA Xid errors: reading NVRM Xid messages in the kernel log

Mon, 01 Jan 0001 00:00:00 +0000

You found a line like this in the kernel log, or a user reported a dead training job and you went looking:

NVRM: Xid (PCI:0000:41:00): 79, pid=1432, name=python, GPU has fallen off the bus.

Xid messages are the NVIDIA driver’s primary error-reporting channel. Every significant GPU fault, from a correctable memory event to a catastrophic PCIe link failure, is printed by the NVRM kernel module as an Xid code in the kernel log. Not in nvidia-smi output. Not in a sysfs file. In the log stream, alongside everything else the kernel has to say.

nvidia-smi hangs or is unresponsive: a wedged GPU or a stuck driver

Mon, 01 Jan 0001 00:00:00 +0000

You run nvidia-smi and nothing comes back. No output, no error, no exit code. Ctrl-C does nothing, and kill -9 does nothing either. This is a different class of failure from “NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver”, where the driver at least answers and tells you it is broken. A hang means the driver is stuck inside a kernel call, usually waiting on GPU hardware that will never respond.

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver

Mon, 01 Jan 0001 00:00:00 +0000

You run nvidia-smi and get:

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver.
Make sure that the latest NVIDIA driver is installed and running.

This error means the userspace side of the driver (NVML, the library nvidia-smi talks to) cannot reach the kernel side. Either the nvidia kernel module is not loaded, it is loaded but built for a different kernel than the one running, the kernel refused to load it, or the GPU itself is gone from the PCIe bus. nvidia-smi exits non-zero and prints the message to stderr; the error string is the reliable signal, not the exact exit code.

Reading NVIDIA GPU memory correctly: nvidia-smi vs the PyTorch caching allocator

Mon, 01 Jan 0001 00:00:00 +0000

An on-call classic: nvidia-smi shows 38 of 40 GiB used on a training node, the line is flat, and someone declares a memory leak. The team running the job insists the model only needs 12 GiB of tensors and nothing is growing. Both sides are reading real numbers. They are reading different layers of the stack, and neither tool tells you that on its own.

PyTorch (and TensorFlow, and most CUDA frameworks) does not allocate GPU memory directly per tensor. It reserves large blocks from the CUDA driver once, caches them, and suballocates internally. nvidia-smi and DCGM report what the framework has reserved from the driver, not what live tensors are actually using. Usage looks high and flat even as tensors are created and freed, and capacity planning or leak hunts based on nvidia-smi alone go wrong in predictable ways.

Resetting a wedged NVIDIA GPU: nvidia-smi --gpu-reset and when only a reboot works

Mon, 01 Jan 0001 00:00:00 +0000

A GPU is wedged: jobs on it are dead or hung, nvidia-smi is slow or throwing errors on that index, and the scheduler cannot place new work on the node. The question is whether you can recover the card in place with nvidia-smi --gpu-reset or whether you are burning time on a node that needs a reboot and possibly an RMA.

Try the reset first because it avoids workload migration, scheduler drain, boot time, and driver reinit. But it has hard prerequisites and hard limits: it clears running processes and GPU state, it requires that no other CUDA contexts exist on the GPU, and it does nothing for a card that has fallen off the PCIe bus. Classify the failure before you run the command; that is most of the battle.

Unauthorized NVIDIA GPU processes: detecting cryptomining and rogue workloads

Mon, 01 Jan 0001 00:00:00 +0000

A GPU node that should be idle shows sustained utilization. Or a scheduler reports a job cannot land because memory is held, yet no known workload is running. nvidia-smi shows a process named python3 burning 99% SM utilization on a box that has no Python workload scheduled. The question to answer fast: is this an expected workload you have lost track of, or something that should not be there at all?

Why NVIDIA GPU utilization is misleading: 100% doesn't mean saturated

Mon, 01 Jan 0001 00:00:00 +0000

A GPU dashboard pinned at 100% looks like proof that the hardware is fully used. It is not. During an incident, that distinction matters: a job can miss its throughput target while every utilization chart says the GPU is completely busy.

NVIDIA’s common GPU utilization metric is a time-domain measurement. It reports the fraction of a sampling period during which at least one kernel was executing. It does not report how many streaming multiprocessors were active, how many warps were resident, whether tensor cores were used, or how much memory bandwidth was consumed.