$ guides / nvidia-gpu / nvidia-gpu-how-gpus-work-in-production ▌

Operations Guides

How an NVIDIA GPU actually works in production: a mental model for operators

GPU runbooks fail when they treat the GPU as a black box that emits a utilization percentage. “100% utilization but slow” makes sense only when you know what utilization measures. “nvidia-smi hangs” makes sense only when you know where nvidia-smi gets its data.

This guide covers the two pieces underneath GPU troubleshooting: the subsystems that produce production failures, and the telemetry plane through which you observe them. It applies primarily to datacenter GPUs such as V100, A100, and H100, with differences noted for consumer cards.

What it is and why it matters

An NVIDIA datacenter GPU is a semi-autonomous computer attached to the host over PCIe or NVLink. It has its own DRAM, memory controllers, caches, DMA engines, power management, thermal control, error correction, and persistent health state.

That architecture changes several CPU-system assumptions:

Device memory has no general swap fallback. When the framebuffer is full, an ordinary device allocation fails immediately. Managed or unified memory can migrate data to the host, but it is not a safety net for uncontrolled device-memory growth.
Utilization measures time, not capacity. utilization.gpu is the portion of a sampling window during which at least one kernel executed. One mostly idle kernel can make the GPU report 100% busy.
The GPU protects itself. It throttles clocks, retires memory, remaps DRAM rows, and can shut down without asking the operating system. The host learns about these actions only through telemetry.
Health history persists. Aggregate ECC counters, retired pages, and row-remap state survive reboots. Volatile counters may reset, so a clean-looking GPU can still carry permanent degradation.

How it works: the five subsystems

Most production GPU failures map to one of five subsystems.

flowchart TD
  HOST["Host CPU / applications"] -->|"CUDA calls, data (PCIe)"| GPU
  subgraph GPU["GPU (semi-autonomous computer)"]
    SM["SMs + Tensor Cores (compute)"]
    CE["Copy engines (DMA, host/device transfers)"]
    MC["Memory controllers + L2"]
    HBM["HBM framebuffer (VRAM)"]
    PMU["PMU (clocks, power, thermal)"]
    ECC["ECC + page retirement + row remap"]
    SM --> MC
    CE --> MC
    MC --> HBM
    PMU -.->|"throttles clocks"| SM
    ECC -.->|"self-heals"| HBM
  end
  GPU2["Peer GPUs"] <-->|"NVLink / NVSwitch"| SM
  NVRM["nvidia.ko (kernel driver)"] -->|"NVML API"| MON["nvidia-smi / DCGM (nv-hostengine)"]
  GPU --> NVRM
  NVRM -->|"XID events"| KLOG["kernel log (dmesg / journal)"]

Compute: SMs and Tensor Cores

The GPU contains dozens to hundreds of Streaming Multiprocessors. Each SM includes CUDA cores, tensor cores on Volta and later, registers, shared memory, and L1 cache. Work arrives as thread blocks dispatched by the GPU scheduler.

SM utilization is intentionally coarse. It tells you that a kernel was resident, not that the SMs were efficiently occupied or doing useful work. A memory-bandwidth-bound workload, such as token generation during LLM inference, can report high SM utilization while waiting on HBM. Diagnose that case with memory-controller utilization and achieved bandwidth, not by adding compute.

Memory: HBM, L2, and the memory controllers

Datacenter GPUs use HBM; consumer and many workstation cards use GDDR. Framebuffer memory sits behind GPU-local memory controllers and L2 cache. Two distinctions matter:

Bandwidth is not capacity. utilization.memory measures memory activity over a sampling window. memory.used measures how much framebuffer is allocated. The shared word “memory” hides two unrelated failure modes. A100 approaches 2 TB/s depending on SKU; V100 provides roughly 900 GB/s.
BAR1 is separately exhaustible. BAR1 is the PCIe-mapped aperture used for direct CPU access and GPUDirect paths. In multi-process containers or GPUDirect RDMA deployments, BAR1 exhaustion can make registrations fail or transfers fall back to slower paths. Check it with nvidia-smi -q -d MEMORY.

The driver and firmware also reserve framebuffer memory, typically hundreds of MiB and varying by GPU, driver, and enabled features. Do not size a workload against the full advertised capacity.

Interconnect: PCIe, copy engines, and NVLink

PCIe connects the GPU to the host. A x16 link provides roughly 16, 32, or 64 GB/s per direction at Gen3, Gen4, or Gen5 before protocol overhead. Copy engines move data independently of the SMs. A GPU can therefore be compute-idle with saturated DMA engines, or report 100% SM utilization while PCIe cannot supply input quickly enough.

NVLink provides GPU-to-GPU bandwidth well beyond PCIe: up to roughly 300 GB/s on V100, 600 GB/s on A100, and 900 GB/s on H100, depending on topology and configuration. Two operational traps are common:

NVLink traffic does not appear in PCIe throughput counters.
NCCL can fall back from NVLink to PCIe without a CUDA error. Training still runs, but collectives slow down. Use nvidia-smi topo -m, NVLink error counters, and NCCL logs to verify the actual transport.

NVSwitch systems add Fabric Manager as a dependency. Treat a stopped, failed, or driver-incompatible nvidia-fabricmanager as an incident: individually healthy GPUs do not imply an initialized, managed NVLink fabric.

Power and thermal management

Power and thermal limits are enforced on the GPU. When demand exceeds the configured budget or a thermal threshold, the power management unit reduces clocks. The thermal cascade is:

Software thermal slowdown near the GPU’s maximum operating temperature.
Hardware thermal slowdown at the model-specific slowdown threshold.
Emergency shutdown at the shutdown threshold.

The thresholds vary by GPU and board. Read them from the hardware with nvidia-smi -q -d TEMPERATURE; do not hard-code a universal number.

Throttle-reason flags are more diagnostic than temperature alone. Temperature indicates severity; flags such as sw_power_cap, hw_thermal_slowdown, and hw_power_brake_slowdown identify the enforcing mechanism. A GPU in hw_power_brake_slowdown can have acceptable temperatures while being constrained by the power-delivery path.

This produces a characteristic incident shape: temperature rises, throttle reasons activate, clocks drop, and the workload keeps running at much lower performance. Nothing necessarily crashes. Thermal incidents are therefore often reported first as unexplained slowness.

Error correction and reliability

Datacenter GPUs normally ship with ECC enabled. The reliability stack has three layers:

ECC correction. Single-bit errors are corrected transparently. Double-bit errors are uncorrectable and indicate detected, potentially corrupt data.
Page retirement. Failed memory pages are removed from the allocatable pool. Many datacenter GPUs expose a 64-page retirement budget, but verify the model-specific limit. At the limit, later uncorrectable errors cannot be isolated by retirement.
Row remapping. Ampere and later GPUs can remap individual DRAM rows to spares. When remapped_rows.failure becomes true, the relevant spare-row capacity is exhausted. The GPU may continue running, but it can no longer self-heal that memory and is RMA-eligible.

A low background rate of corrected ECC errors can be normal. An accelerating rate is the signal. Persistent retirement and remap state survives reboot; volatile ECC counters can reset. Corrupt InfoROM or other onboard health storage can also make persistent identity and health data unreliable.

ECC can be disabled, sometimes to recover memory or performance. With ECC off, memory faults become silent data corruption with no hardware detection. Verify ecc.mode.current before trusting a zero error count. GeForce cards do not provide the datacenter ECC reliability model.

The telemetry plane: where the data actually comes from

Three observation channels are often conflated. They are not interchangeable.

nvidia.ko and NVML. The NVIDIA kernel driver exposes the NVIDIA Management Library. nvidia-smi reads its state through that management path. NVML queries serialize on driver locks, so concurrent nvidia-smi polling and DCGM collection can slow each other. One failing GPU can also hang an all-GPU query; use nvidia-smi -i N for per-GPU health checks. Establish a normal query-latency baseline, and treat repeated multi-second responses as an early wedge warning.

DCGM and nv-hostengine. DCGM wraps NVML and adds health monitoring, policies, grouping, caching, and stable field IDs. For example, DCGM_FI_DEV_GPU_UTIL is field 1002, while volatile single-bit and double-bit ECC totals are fields 3100 and 3101. dcgm-exporter converts selected fields into Prometheus metrics. Its scrape interval is separate from DCGM’s internal sampling, so short power or utilization spikes can be missed. A daemon can also remain alive while field updates stall behind a wedged driver.

XID events. XID errors are emitted by NVRM into the host kernel log, not returned as ordinary nvidia-smi utilization or health fields. Monitoring that polls only NVML can miss XID 48, 74, 79, 95, and other hardware faults. DCGM can expose the latest XID, but the history belongs to the kernel log. Containers that do not collect the host journal lose that signal.

A safe read-only snapshot for GPU 0 is:

nvidia-smi -i 0 --query-gpu=index,name,utilization.gpu,utilization.memory,memory.used,memory.total,temperature.gpu,temperature.memory,power.draw,enforced.power.limit,ecc.mode.current --format=csv
nvidia-smi -i 0 -q -d PERFORMANCE,POWER,TEMPERATURE,MEMORY,ECC
journalctl -k --since "1 hour ago" | grep -E "NVRM: Xid|NVRM.*Xid"

Run the journal command on the host, or wherever host kernel logs are collected. Use dmesg -T if the journal is unavailable.

Finally, the telemetry plane depends on the driver staying initialized. Without persistence mode, the driver can unload between jobs, causing first-call latency and monitoring gaps. nvidia-smi -pm 1 enables persistence, but production systems should manage it through a systemd unit so it survives reboot.

Where this shows up in production

The model becomes useful when failure archetypes are mapped back to subsystems:

OOM cliff: memory subsystem. Allocation fails immediately. High memory.used is normal for PyTorch and similar caching allocators, so alert on application OOMs and unexplained growth rather than a high percentage alone.
Throttling cascade: power or thermal subsystem. Temperature leads, throttle reasons confirm, and clocks drop. One hot GPU suggests a local fan, airflow, or cold-plate issue; all GPUs hot points toward HVAC or chassis cooling.
Silent degradation: reliability subsystem. Corrected ECC errors accelerate, retirement or remap headroom shrinks, and an uncorrectable error eventually produces XID 48 or 95. Training can emit NaNs or write a corrupt checkpoint before it crashes.
Straggler: interconnect. One GPU with a downgraded PCIe link or flapping NVLink gates every collective. Check link generation and width under load; idle downgrade is normal power management. Cross-GPU comparison exposes the asymmetry.
Fallen off the bus or hang: driver and PCIe path. Look for XID 79, hanging nvidia-smi queries, and processes in uninterruptible sleep. Memory left visible after a process exits can also indicate a stuck CUDA context requiring process cleanup or a controlled GPU reset.

Deployment modes change attribution. MIG partitions memory and compute but not the GPU-wide power and thermal domains, so monitor both GPU and instance levels. MPS multiplexes processes through a shared server, weakening per-process attribution. NVSwitch adds Fabric Manager. Consumer cards remove ECC, NVLink, and several datacenter reliability assumptions.

Signals to watch in production

Signal	What it tells you	Act when
`utilization.gpu`	Kernel-busy time	High with low memory activity and poor throughput, or low when compute is expected
`utilization.memory`	Memory-path activity	Sustained above roughly 80% with throughput below expectation
VRAM used and total	Distance from allocation failure	Sustained above 90% with application OOMs, or steady unexplained growth
GPU and HBM temperature	Throttle risk	Rising toward model-specific thresholds; HBM temperature may be unavailable on some models
Clock throttle reasons	Why clocks dropped	Hardware thermal, power-brake, or unexpected software caps active under load
Power draw versus enforced limit	Power capping	Pinned at the cap while performance is below expectation
ECC corrected and uncorrected counts	Memory health and corruption risk	Corrected-error rate accelerates, or any new uncorrectable error appears
Retired pages and remapped rows	Permanent repair headroom	Count increases, retirement is pending, model limit is approached, or remap failure becomes true
PCIe generation and width	Host bandwidth	Current link is below maximum during active work
NVLink status and errors	Multi-GPU fabric health	A link drops or CRC, replay, or recovery counters increase
XID events	Hardware and driver faults	New XID 48, 63, 64, 74, 79, or 95 appears
Per-GPU management latency	Driver responsiveness	Queries repeatedly take multiple seconds or hang
Persistence, ECC, and power-limit settings	Configuration drift	Persistence or ECC is unexpectedly off, or the power limit differs from the approved baseline

Many GPU faults last only seconds. Sample thermal, power, and utilization at 10 seconds or faster; minute-level collection misses short throttle events.

Several health fields are latched state rather than events. Retired-page counts and remap failure are monotonic or persistent, while volatile ECC counters can reset when the driver reloads. Alert on deltas and transitions, not merely on a nonzero historical value.

How Netdata helps

Netdata’s NVIDIA GPU collector can collect NVML-derived metrics at per-second resolution, matching the cadence needed for thermal and throttle transients.
SM and memory-controller utilization are visible together, making compute-bound and memory-bound behavior easier to distinguish.
Temperature, power draw, enforced power limit, clocks, and throttle reasons can be correlated as one cascade instead of separate graphs.
ECC counters, retired pages, and row-remap state can be trended where the GPU and driver expose them, supporting delta- and transition-based alerts.
Per-GPU views expose the asymmetric straggler pattern, while host CPU, memory, disk, network, and NUMA metrics help distinguish a starved GPU from a failed one.
GPU charts do not replace XID monitoring. Pair them with host kernel-journal collection and alerting for the hardware fault channel.