$ guides / nvidia-gpu ▌

NVIDIA GPU · OPERATIONS PLAYBOOK

A GPU that fails loudly with an Xid code, or fails silently by getting slower

An NVIDIA datacenter GPU is a semi-autonomous computer on the PCIe bus: its own HBM memory with ECC and row-remapping, its own power and thermal management, its own error log the driver writes as Xid codes to the kernel ring buffer. We trace how it behaves under load, why the worst failures are the silent ones, and what to do when nvidia-smi hangs, a job dies with CUDA out of memory, or dmesg fills with NVRM: Xid.

> Start with the monitoring checklist → # Jump to the full guide list

GPUs get you to a running workload quickly, then hand you two very different kinds of failure: the loud ones that log an Xid and crash a job, and the silent ones that just make everything slower until someone notices the bill.

The workload runs. Until the framebuffer fills and a job dies with CUDA out of memory. Until HBM cells degrade, single-bit ECC errors accumulate, and one uncorrectable double-bit error (Xid 48) corrupts a gradient and takes the CUDA context with it. Until the die or the HBM junction gets hot enough that clocks quietly drop and training slows 30% with no error at all. Until a PCIe link renegotiates from x16 to x8 at boot and halves your data-transfer bandwidth for months. Until the GPU stops answering the driver entirely — GPU has fallen off the bus — and nvidia-smi hangs.

These guides are written for engineers who already run GPUs, not for people learning what CUDA is. The goal is the mental model of how the GPU and its telemetry stack (the driver, NVML, DCGM, and the kernel log) actually behave; the failure patterns that keep recurring; the signals that catch them before they page anyone; and the runbooks for the exact Xid codes and error strings you end up pasting into a search box at 3am.

How an NVIDIA GPU actually runs in production

A GPU is not just a compute engine. It is a memory subsystem with its own error correction, a power-and-thermal controller that silently changes your clocks, an interconnect fabric to other GPUs, and a telemetry plane (NVML, DCGM, the kernel log) that can itself hang or go stale. Most production failures live between these layers, and the single most important error channel — Xid — does not appear in NVML at all.

host + CUDA runtime

The application launches kernels and issues memory copies through the CUDA runtime and the NVIDIA kernel driver (<code>nvidia.ko</code>). Persistence mode keeps the driver context warm; without it the first CUDA call pays 1-3s and the P-state flaps. When the driver is unhappy, <code>nvidia-smi</code> returns an error or, worse, hangs.

HOST

PCIe / host interconnect

The CPU-to-GPU path. Link generation and width are negotiated at boot and can silently degrade (x16 to x8, Gen4 to Gen3), halving bandwidth with no error. This is also where a GPU disappears when it 'falls off the bus' (<code>Xid 79</code>). NUMA affinity decides whether a transfer crosses the CPU interconnect.

PCIe

compute engine (SMs + Tensor Cores)

Dozens to hundreds of streaming multiprocessors run your kernels; Tensor Cores do the matrix math for mixed-precision ML. <code>utilization.gpu</code> only says at least one kernel was running — it can read 100% with a single SM busy. Real efficiency lives in the profiling fields (SM Active, Tensor Core Active).

COMPUTE

HBM framebuffer + L2

The GPU's own DRAM, managed by its memory controllers behind an L2 cache. This is what a batch size, a model, and a KV cache consume. There is no swap: an allocation that doesn't fit fails immediately as <code>CUDA out of memory</code>. Bandwidth here, not compute, is the real bottleneck for many workloads.

MEMORY

ECC + row-remap (reliability)

Datacenter HBM and SRAM are ECC-protected. Single-bit errors are corrected silently and their rate is the leading indicator of decline; double-bit errors are uncorrectable data corruption. Failing memory is retired page-by-page or, on Ampere+, remapped row-by-row from a finite pool of spares. Lifetime counts live in the InfoROM.

ECC

power + thermal management

A firmware power/thermal controller continuously adjusts clocks to stay inside the TDP and thermal envelope. When the die or HBM junction gets hot, or the chassis asserts a power brake, clocks drop — the throttle-reasons bitmask says exactly why. This is the mechanism behind most silent slowdowns.

POWER

NVLink / NVSwitch fabric

The high-bandwidth GPU-to-GPU path used by collective operations in multi-GPU training. CRC errors force retransmissions that quietly cut bandwidth; a degraded link makes one GPU a straggler the whole job waits on. On DGX/HGX, NVSwitch and Fabric Manager sit between the GPUs and can fail while each GPU looks healthy.

FABRIC

NVML + DCGM telemetry plane

The driver exposes NVML; <code>nvidia-smi</code> is a CLI over it; DCGM (<code>nv-hostengine</code>) wraps NVML with health checks, field IDs, and the Prometheus exporter. This plane can hang on a stuck driver and return stale data that looks fine. And Xid errors — the definitive fault channel — land in the kernel log, not here.

TELEMETRY

Why this matters: 'the GPU is slow' can be a data-starved pipeline, an inefficient kernel using one SM, thermal or power throttling, an HBM bandwidth wall, a degraded PCIe link, or an NVLink straggler. 'The GPU is broken' can be OOM, a wedged engine, ECC corruption, or a card off the bus. The symptom rhymes but each layer has a different signal — and the one that matters most, Xid, is in dmesg, not in your metrics dashboard.

The failures you'll actually see

Most GPU incidents fall into a small set of recurring patterns. Recognise the shape — and remember to read the kernel log — and triage gets dramatically faster.

CRITICAL

GPU fallen off the bus

The PCIe link fails catastrophically and the GPU stops answering the driver entirely. nvidia-smi hangs or shows ERR! / N/A, the device may vanish from lspci, and every process on it is dead. Xid 79 in dmesg is definitive. This is unrecoverable without at least a node reboot, and it often means the card, its power delivery, or the slot/riser has failed. A GPU that falls off once tends to do it again.

NVRM: Xid ...: 79, GPU has fallen off the bus in dmesg
nvidia-smi hangs, times out, or reports ERR! for the GPU
The device missing from lspci after being present
All CUDA processes on that GPU crashed or hung at once

Investigate →

IMMINENT

The silent HBM death march

GPU memory is developing physical defects. The single-bit ECC error rate climbs over days or weeks (faster when HBM runs hot), spare rows get remapped, and Xid 63 retirement events appear — all while training looks fine. Then an uncorrectable double-bit error (Xid 48, or Xid 94/95 on Ampere+) corrupts data and kills the context. Completely predictable if you watch the rate, invisible if you only watch counts.

Single-bit ECC error rate rising from a near-zero baseline
Retired-page count creeping up; row-remap spares being consumed
Xid 63/64 (retirement) then Xid 48/94/95 (uncorrectable) in dmesg
NaN training loss or garbage inference on one specific GPU

Investigate →

ACTIVE

CUDA out of memory

The framebuffer is a cliff-edge — there is no swap, so an allocation that doesn't fit fails immediately and the process dies with CUDA out of memory. Causes range from a batch size or model that is simply too large, to a slow memory leak, to fragmentation (OOM despite free memory), to another process quietly sharing the GPU. The trap: nvidia-smi shows what a caching allocator reserved, not what tensors actually use.

RuntimeError: CUDA out of memory. Tried to allocate ...
Framebuffer usage climbing without a plateau (leak)
OOM while nvidia-smi still shows free memory (fragmentation)
An unexpected second process holding framebuffer on the GPU

Investigate →

CRITICAL

The GPU hang

The compute engine wedges. CUDA calls stop returning, utilization freezes at its last value, and nvidia-smi may hang when it queries the affected GPU. Xid 43 (GPU stopped processing) or Xid 13 in dmesg names it. Unlike a card off the bus, the GPU is still present on PCIe — just non-responsive. nvidia-smi --gpu-reset often fails for Xid 43, so a full reboot is usually required; check ECC afterward, because the hang may be a symptom of failing hardware.

CUDA calls blocked; application processes stuck, not exiting
GPU utilization frozen at its last-reported value (stale)
Xid 43 or 13 in dmesg for the GPU's PCI bus ID
nvidia-smi hangs on that GPU but other GPUs respond

Investigate →

CRITICAL

The driver went dark

Every GPU on the node disappears at once and tooling reports NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. The usual cause is not the GPU but the kernel module: a kernel upgrade landed and the driver was not rebuilt (DKMS), the module was unloaded, or the driver crashed. Distinguish this from a single card off the bus — here nothing on the node can see any GPU.

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver
lsmod shows the nvidia module missing or mismatched after a kernel update
All GPUs vanish simultaneously (not just one)
dmesg shows NVRM version / API mismatch messages

Investigate →

ACTIVE

Garbage results without a crash

The workload keeps running but produces wrong answers — diverging or NaN training loss, degraded inference accuracy, intermittent generic CUDA errors. Often a Xid 13 graphics-engine exception, which is genuinely ambiguous: a bad CUDA kernel, a driver bug, or hardware degradation. Isolated events on one job point to software; the same GPU faulting across different applications points to hardware. This is the correctness failure that ECC-less consumer GPUs give you with no signal at all.

Xid 13 (Graphics Engine Exception) in dmesg
Training loss going NaN or diverging on a specific node/GPU
Intermittent generic 'CUDA error' with no obvious cause in app logs
The same GPU faulting across unrelated workloads

Investigate →

Choosing a tool

Best GPU Monitoring Tools for AI Workloads (2026)

A ranked review of the tools teams actually shortlist here, what each one is genuinely good at, and how the pricing behaves as you scale.

Read the buyer's guide →

NVIDIA GPU monitoring maturity levels

GPU observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your GPUs matter. Most production fleets should land at the second level — and every level must include parsing the kernel log for Xid errors.

Level 1: Survival

Know that a GPU is dead, cooking, out of memory, or corrupting data

Survival monitoring is the floor. With these signals you can answer: is the GPU present, is it overheating, is it out of memory, and has it faulted or corrupted data? You will not learn why, but you will not miss a hard failure. Enough for dev boxes and non-critical inference.

GPU presence + driver responsiveness nvidia-smi succeeds within a timeout for every expected GPU.
Xid errors in the kernel log The definitive fault channel; NVRM: Xid in dmesg/journald.
GPU die temperature Is the GPU cooking? Relative to the SKU's max operating temp.
Framebuffer used / total Approaching the ceiling means CUDA OOM is imminent.
Compute utilization Idle when a workload should be running, or a crashed job.
Uncorrectable ECC (DBE) count Any nonzero value means data corruption has occurred.

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production fleets should target. Survival says something is wrong; operational says what: throttling versus starvation, degrading memory, a capped power limit, a downgraded PCIe link, dark telemetry. With this coverage a team can usually run its own incident response.

Single-bit ECC rate (not just count) The leading indicator of HBM decline; watch the rate and its slope.
Clock throttle reasons The one bitmask that says exactly why clocks dropped.
Power draw vs enforced power limit Detects power capping and power-brake events.
PCIe link generation and width Silent x16-to-x8 / Gen4-to-Gen3 degradation halves bandwidth.
SM and memory clock frequencies Quantifies the impact of whatever is throttling.
HBM (memory) junction temperature Often the real thermal limiter, hotter than the die.
Retired pages + row-remap state GPU self-repair budget; 'Remapping Failure Occurred' = replace.
DCGM daemon liveness + freshness Is your telemetry actually live, or hung and stale?

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches the slow burns that never page you on day one. Efficiency waste (100% utilization doing 5% of the work), an NVLink cable costing 20% of throughput, a PCIe replay rate creeping up, an ECC trend that says a GPU has weeks to live. These become incidents on day thirty.

SM Active + Tensor Core Active Real compute efficiency; high util with low SM Active is waste.
DRAM Active (memory bandwidth) Whether the workload is memory-bandwidth bound.
NVLink error counters per link CRC/replay errors quietly cut collective bandwidth.
PCIe replay counter + throughput Physical-layer signal integrity, before the link degrades.
ECC rate trending vs baseline Deviation from this GPU's own history, not a fleet constant.
Per-process / per-pod attribution Who is using what; map host PID to container in K8s.
Per-MIG-instance metrics Per-GPU numbers are misleading once MIG is enabled.
Energy consumption Cost attribution and anomalous sustained-load detection.

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after an incident proved you needed them. NVSwitch and Fabric Manager health, straggler detection across GPUs, thermal trending over months, driver-version-to-Xid correlation, InfoROM validation. Most fleets never need all of it — add what your incident history demands.

Fabric Manager + NVSwitch health The invisible layer; FM down isolates GPUs while they look fine.
GPU-to-GPU performance symmetry The 'fast' GPU is normal; the others are degraded stragglers.
Continuous PCIe link monitoring Catch a downgrade during operation, not just at boot.
Thermal trend over months Rising baseline for the same workload = cooling degradation.
Driver / VBIOS version tracking Correlate Xid patterns with specific driver releases.
PCIe AER events from dmesg Deeper PCIe diagnostics than the replay counter alone.
InfoROM integrity Corruption makes ECC / retired-page history untrustworthy.
GPU reset frequency A GPU needing resets more often is on its way out.

Operating mistakes worth avoiding

The traps GPU teams keep falling into. Each has a clear fix. Most teams only learn it after an incident — or after an unexplained cloud bill.

Not parsing the kernel log for Xid errors

The single biggest gap. Teams watch utilization, temperature, and memory from <code>nvidia-smi</code> but never read <code>dmesg</code> / <code>journalctl</code> for <code>NVRM: Xid</code>. A GPU with perfectly normal metrics can be logging <code>Xid 63</code> events that say it is weeks from a fatal failure. Xid is the GPU's definitive error channel and it does not appear in NVML or DCGM history — DCGM field 230 reports only the most recent code. Ship kernel logs to a persistent store and alert on Xid.

Trusting GPU utilization as a health or efficiency metric

<code>utilization.gpu</code> (DCGM field 203) is a time-domain metric — it says at least one kernel was running, not how much of the GPU was used. It can read 100% with a single SM busy while 107 sit idle. Dashboards built on it show green while training crawls. Track the profiling fields (SM Active, Tensor Core Active, DRAM Active) and, above all, workload throughput (tokens/sec, samples/sec) — not a utilization percentage.

Monitoring ECC counts instead of ECC rates

Checking 'is DBE greater than zero?' is good; ignoring the single-bit error rate is not. A GPU with 100 corrected errors over a year is fine; the same 100 in a week is failing. The SBE rate and its acceleration give weeks of warning before an uncorrectable error — and it is routinely ignored until the first <code>Xid 48</code> already corrupted data.

Monitoring die temperature but not HBM temperature

On datacenter GPUs the HBM junction temperature is often the actual thermal limiter and runs 10-20C hotter than the die. Teams watch <code>temperature.gpu</code> and think thermal monitoring is done, while HBM silently throttles memory bandwidth with no Xid and no crash. Monitor both, and alert near the SKU's HBM slowdown point (~95C on H100/A100).

Checking PCIe link speed once, at provisioning

PCIe links renegotiate lower — a x16 link training at x8, a Gen4 link at Gen3 — often silently at boot, halving data-transfer bandwidth for months with zero other symptoms. In multi-GPU training this creates a straggler that throttles the whole job. Compare <code>pcie.link.*.current</code> against <code>.max</code> continuously, not just at install time.

Ignoring row-remap and retired-page trends

'Remapping Failure Occurred: Yes' is the single most critical hardware-replacement trigger on modern GPUs, and one of the least-watched signals in most fleets. A GPU with rising retired pages or consumed spare rows is a ticking clock. Audit these across the fleet on a schedule so you replace failing GPUs before they crash a job.

Not enabling persistence mode

Without persistence mode, the driver unloads between CUDA contexts: the first call of every workload pays 1-3s, the P-state drops to idle between launches (and looks like throttling in dashboards), and the GPU appears to restart constantly. Enable it at boot and monitor it as a configuration-compliance signal.

Trusting DCGM / nvidia-smi data as always fresh

When the driver is in trouble, DCGM and <code>nvidia-smi</code> can hang or return stale cached values — a failing GPU can report its last-known-good metrics for a while before timing out, so everything 'looks normal' until it very much doesn't. Verify telemetry freshness (timestamps), use a liveness probe that actually queries the GPU, and alert on deviation from each GPU's baseline, not only static thresholds.

NVIDIA GPU runbooks in this section

Each guide is a focused runbook for one symptom, Xid code, or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

Availability, driver, hangs, and resets

▸

Xid errors

▸

Framebuffer memory and OOM

▸

ECC, HBM health, and page retirement

▸

Temperature and thermal throttling

▸

Power, clocks, and throttle reasons

▸

PCIe link and bandwidth

▸

NVLink, NVSwitch, and Fabric Manager

▸

Utilization and workload efficiency

▸

DCGM, exporters, and MIG

▸

Processes, configuration, and security

WHERE TO GO NEXT

Setting up GPU monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the Xid code or error string that matches what you're seeing.

> Start with the checklist > Back to Operations Guides