NVIDIA GPU monitoring checklist: the signals every production GPU fleet needs

Most GPU monitoring setups fail in one of two ways. Either they collect a handful of nvidia-smi counters and page on the wrong things (raw temperature, memory percentage, idle PCIe downgrade), or they collect everything and alert on nothing meaningful. Both failure modes come from the same root cause: no shared vocabulary for which signals matter, at what fidelity, and with what alert semantics.

This checklist organizes the signals a production GPU fleet needs into four tiers: survival, operational, mature, and expert. Each tier builds on the previous one. The intent is not that every fleet reaches expert. The intent is that you can place your current setup on the ladder and know which gaps to close next, in what order.

One thing before the tiers: many GPU signals are persistent state, not events. Retired page counts, aggregate ECC counters, and row-remap failure flags are stored in the GPU’s InfoROM and stay nonzero forever. Alerting on the raw value means paging forever on a GPU that was remediated months ago. Where a signal is state, this checklist says so, and the correct pattern is alerting on deltas or transitions. This distinction is the single biggest source of alert fatigue in GPU fleets.

flowchart TD
  L4["Level 4: Expert - DCGM depth, NCCL, fabric, predictive"]
  L3["Level 3: Mature - retired pages, row remap, NVLink, drift"]
  L2["Level 2: Operational - power, throttle reasons, ECC, PCIe"]
  L1["Level 1: Survival - reachability, temp, VRAM, fatal XIDs"]
  L4 --> L3 --> L2 --> L1

Level 1: Survival

The minimum viable set. A fleet at Level 1 can answer two questions: is the GPU there, and is it about to die or corrupt data.

Signal	How to check	Warning sign
GPU reachability	`nvidia-smi -L` exit code, per-GPU with `-i N`	Non-zero exit, timeout, or hang on one GPU
Driver responsiveness	Time a lightweight query: `timeout 5 nvidia-smi --query-gpu=gpu_name --format=csv,noheader -i 0`	Sustained latency above 2s; healthy is under 100ms
GPU die temperature	`nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader,nounits`	Sustained within 5 degrees C of the GPU’s max operating temp
Framebuffer used / total	`nvidia-smi --query-gpu=memory.used,memory.total --format=csv,noheader,nounits`	Sustained linear growth over hours (leak), not the level itself
Fatal XIDs in kernel log	`dmesg -T \| grep -i "NVRM: Xid"` or `journalctl -k \| grep -i "NVRM: Xid"`	New XID 48 (double-bit ECC) or XID 79 (fallen off the bus)

Two Level 1 rules that prevent bad pages:

Gate reachability alerts on duration. Brief nvidia-smi failures during boot or driver reload are normal. Page only on sustained unreachability (for example, more than 60 seconds) and after an uptime gate. One stuck GPU can hang a whole multi-GPU nvidia-smi query, so always probe per-GPU with -i N.
Do not page on VRAM percentage. PyTorch and TensorFlow caching allocators deliberately hold 95 to 99 percent of framebuffer memory. High usage is normal. OOM detection belongs in application logs (CUDA out of memory), not in a memory threshold. Track the rate of change for leaks instead.

Level 2: Operational

Professional production monitoring. This tier adds the signals that explain why performance degraded, which is what Level 1 cannot do. Most fleets should be here.

Signal	How to check	Warning sign
Power draw vs limit	`nvidia-smi --query-gpu=power.draw,power.limit,power.default_limit,enforced.power.limit --format=csv,noheader,nounits`	Draw sustained above 95 percent of `enforced.power.limit` and `sw_power_cap` active
Clock throttle reasons	`nvidia-smi --query-gpu=clocks_event_reasons.hw_thermal_slowdown,clocks_event_reasons.hw_power_brake_slowdown,clocks_event_reasons.sw_power_cap,clocks_event_reasons.sw_thermal_slowdown --format=csv,noheader`	`hw_thermal_slowdown` active during production compute, sustained over 60s
Current vs max clocks	`nvidia-smi --query-gpu=clocks.current.sm,clocks.max.sm,clocks.current.memory,clocks.max.memory --format=csv,noheader,nounits`	SM clock below 80 percent of max under heavy load
ECC corrected + uncorrected counts	`nvidia-smi --query-gpu=ecc.errors.corrected.volatile.total,ecc.errors.uncorrected.volatile.total --format=csv,noheader,nounits`	Any new uncorrected error (delta); accelerating corrected rate
PCIe link gen and width	`nvidia-smi --query-gpu=pcie.link.gen.gpucurrent,pcie.link.gen.gpumax,pcie.link.width.current,pcie.link.width.max --format=csv,noheader`	Current below max during active workloads
Fan speed (where applicable)	`nvidia-smi --query-gpu=fan.speed --format=csv,noheader,nounits`	0 percent with temperature rising, or sustained 100 percent with high temp
Process list	`nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv,noheader`	Unexpected processes; zombies holding memory after job exit
Persistence mode	`nvidia-smi --query-gpu=persistence_mode --format=csv,noheader`	Anything other than Enabled on a production node

Operational-tier rules that matter:

Alert on throttle reasons, not raw temperature. 80 degrees C under full load is normal for many GPUs. hw_thermal_slowdown active is the real signal; temperature tells you severity and direction. One hot GPU means a local cooling problem; all GPUs hot means HVAC.
Power at the limit is not automatically a fault. Datacenter operators often cap power.limit below power.default_limit intentionally for rack density. Only treat it as throttling when sw_power_cap is active or clocks and throughput actually degrade. Compare against enforced.power.limit, which is the real ceiling.
Idle PCIe downgrade is by design. Drivers drop the link generation at idle to save power (a Gen5 GPU showing Gen2 at idle is normal). Only alert when the link is degraded under load. Note the field name: pcie.link.gen.current is deprecated; use pcie.link.gen.gpucurrent.
ECC counters need event semantics. Page on a new uncorrected error (delta in the volatile counter, or a new XID 48/95 in the log) on a GPU with active or recent production work. A historical nonzero aggregate on a drained, quarantined card is an urgent ticket, not a 3 a.m. page. Volatile counters reset on driver reload, so track deltas, not absolutes.
Persistence mode is a checkbox with real consequences. Without it (nvidia-smi -pm 1, requires root, persist it via a systemd unit), the driver unloads between jobs: seconds of first-call latency, monitoring gaps, and scheduler races. It resets on reboot, so verify it rather than setting it once.

Level 3: Mature

Full-spectrum instrumentation. This tier is where you stop reacting to failures and start predicting them, and where alert semantics get subtle.

Signal	How to check	Warning sign
Retired pages	`nvidia-smi --query-retired-pages=retired_pages.single_bit_ecc.count,retired_pages.double_bit_ecc.count,retired_pages.pending --format=csv,noheader`	Any increase (delta); `pending = Yes`; approaching the 64-page hard limit
Row remapping (Ampere and later)	`nvidia-smi --query-remapped-rows=remapped_rows.correctable,remapped_rows.uncorrectable,remapped_rows.pending,remapped_rows.failure --format=csv,noheader`	Uncorrectable rows, pending remaps needing reboot, transition to `failure = true`
HBM memory temperature	`nvidia-smi --query-gpu=temperature.memory --format=csv,noheader,nounits`	Approaching the GPU’s memory thermal limit (returns N/A on GDDR cards)
NVLink status and errors	`nvidia-smi nvlink -s`, `nvidia-smi nvlink -e`	Any CRC/replay errors; an expected link down during active multi-GPU work
PCIe throughput and replay errors	`nvidia-smi dmon -s t -d 1`; `nvidia-smi -q -d PCIE \| grep -i replay`	Sustained above 80 percent of practical link bandwidth; rising replay rate
BAR1 utilization	`nvidia-smi --query-gpu=memory.bar1.total,memory.bar1.used,memory.bar1.free --format=csv,noheader,nounits`	Above 90 percent in GPUDirect RDMA environments
ECC mode status	`nvidia-smi --query-gpu=ecc.mode.current --format=csv,noheader`	Disabled on a datacenter GPU processing production data
Full XID classification	Kernel log monitoring with per-XID routing	Fatal: 48, 64, 79, 95. App bugs: 13, 31, 43. Informational: 45, 63, 92
Configuration drift	Query persistence mode, compute mode, ECC mode, MIG mode, power limits, driver and VBIOS versions	Any deviation from the per-node-class baseline

Note: nvidia-smi -q -d PCIE exposes PCIe replay counters on driver versions that support it (added in nvidia-smi v340 per the changelog). DCGM exposes PCIe replays as field DCGM_FI_DEV_PCIE_REPLAY_TOTAL (field 202). Verify field availability against your installed driver version with nvidia-smi -q -d PCIE | grep -i replay before wiring this into alerting.

Mature-tier rules:

Retired pages and row remap are state, not events. Counts are monotonic and persist across reboots in InfoROM. Alert on increases, never on the raw value. remapped_rows.failure is latched: once true it stays true, so alert on the transition (or on a new XID 64), not the boolean. A GPU with failure = true is RMA-eligible but keeps running.
64 retired pages is the cliff. At the driver’s 64-page retirement limit, no further retirement is possible and the next double-bit error is unrecoverable. Start planning replacement well before that.
XID routing must be per-code. XID 79 (fallen off the bus) and new XID 48/95 events on a busy GPU are pages. XID 63 (row remap event) is ECC self-healing working correctly and should never page. XID 13/31/43 are application bugs, not hardware. Getting this mapping wrong sends operators chasing hardware when the code is at fault, or ignoring real hardware faults as noise.
Verify field names against your driver. The correct flags are --query-retired-pages and --query-remapped-rows (display flag -d ROW_REMAPPER), and the ECC fields are ecc.errors.corrected.* / ecc.errors.uncorrected.*. Field names from older docs or generated configs (ecc.errors.single_bit_total, -d REMAPPED_ROWS) do not exist and fail silently. Validate with nvidia-smi --help-query-gpu.
Zero ECC errors is not proof of health. It can mean ECC is disabled, which turns every memory error into silent data corruption. Always check ecc.mode.current before trusting error counters. Some cloud providers ship with ECC off.

Level 4: Expert

Deep specialization, mostly relevant at scale: multi-node training clusters, NVSwitch systems, MIG multi-tenancy, and predictive maintenance.

DCGM daemon liveness and enumeration. If you run DCGM, the daemon (nv-hostengine) is itself a signal: pgrep -x nv-hostengine for presence, dcgmi diag -r 1 for responsiveness. Compare dcgmi dmon -e 1001,1002 -c 1 against nvidia-smi -L and lspci -d 10de:; any enumeration mismatch means a container misconfiguration, MIG grouping issue, or a GPU on its way off the bus. A live daemon can still serve stale cached data when NVML calls block, so treat query latency as its own signal.
NCCL collective latency. Baseline-relative only: absolute thresholds are meaningless across topologies and message sizes. Latency above 2x baseline, or NCCL timeouts, ticket. NCCL can silently fall back from NVLink to PCIe; training still runs, just much slower. Zero NVLink utilization during multi-GPU training is a classic silent-catastrophe pattern.
IB/RoCE fabric health. perfquery and ibdiagnet for InfiniBand, ethtool -S error counters for RoCE. Fabric faults are invisible from nvidia-smi and surface only as slow collectives or timeouts, so they are routinely misattributed to GPUs.
Fabric Manager on NVSwitch systems. On DGX/HGX, nv-fabricmanager down means NVLink connectivity through the switch is lost even though every GPU looks individually healthy. systemctl status nvidia-fabricmanager; page only with an uptime gate and active NVSwitch-dependent jobs failing.
MIG per-instance health. On MIG-enabled A100/H100, aggregate nvidia-smi numbers lie: one instance can be OOM while the card-level view looks fine, and instances share thermal and power domains. Monitor per-instance via DCGM (dcgmi dmon -g <mig-group>), not the physical card.
Reset recurrence and host-starvation correlation. Track GPU reset frequency per device; a GPU that resets more than twice in 24 hours is on a path to permanent failure. Correlate low SM utilization with host CPU, disk, and network saturation to catch host-starved GPUs before someone buys more GPUs to fix a data-loading problem.
Predictive wear modeling. Trend ECC corrected-error rate acceleration (not counts), thermal baseline drift month over month, and retired-page runway toward the 64-page limit. This is what turns GPU replacement from an incident into a scheduled task.

The alert-semantics cheat sheet

The same signal often needs different handling depending on whether it is an event or latched state. Keep this mapping next to your alert config:

Pattern	Examples	Alert on
Binary event	XID 79, new XID 48/95, NVLink down during active work	New occurrence, with uptime and duration gates
Latched state	`remapped_rows.failure`, aggregate ECC counters, retired page counts	Transitions and deltas only
Level with context	Temperature, power draw, VRAM used	Never alone; corroborate with throttle reasons or application errors
Workload-gated	PCIe gen/width, straggler detection	Only during active workloads
Configuration	Persistence mode, ECC mode, compute mode, power limits	Drift from baseline, ticket severity

Common mistakes this checklist prevents

Paging on VRAM percentage when framework caching allocators make 95 percent usage normal.
Paging on raw temperature instead of throttle reasons.
Ignoring accelerating corrected ECC rates because “corrected means fine.” Acceleration is the leading indicator of an uncorrectable error.
Treating idle PCIe downgrade as a hardware fault.
Aggregating MIG metrics at the card level and missing per-instance OOM.
Alerting on raw retired-page and row-remap values, producing permanent re-alerts on remediated hardware.
Sampling once per minute. Temperature spikes, XID events, and PCIe errors live and die in seconds; sampling at 10 seconds or faster is the floor.

How Netdata helps

Netdata’s NVIDIA GPU collector queries NVML directly at per-second resolution, which is the sampling rate short-lived events like throttle transitions, power excursions, and temperature spikes actually require.
Die temperature, HBM temperature, power draw versus limit, clock speeds, and throttle reasons are charted together per GPU, so a thermal or power throttling cascade reads as one correlated view instead of five separate graphs.
ECC corrected and uncorrected counters are collected as time series, making rate-of-change and delta alerting (the correct semantics for these signals) straightforward rather than a custom scripting exercise.
Per-process GPU memory and utilization alongside host CPU, disk I/O, and network metrics on the same node makes host-starved-GPU and straggler diagnosis a correlation exercise instead of a guess.
Because Netdata also monitors the host, kernel logs, and systemd units, XID events, nv-fabricmanager state, and driver reloads land on the same timeline as the GPU metrics they explain.