NVIDIA GPU monitoring maturity model: from survival to expert

Most GPU fleets sit at one of two extremes: nothing beyond “does nvidia-smi still respond,” or a wall of dashboards nobody wired to an alert. Neither survives a real incident. A GPU falling off the bus at 3 a.m., a training job running 5x slower because one card is thermally throttled, an accelerating ECC error rate that becomes silent data corruption next week: all detectable, but only if you collect the right signals at the right fidelity.

This article is a self-assessment framework. It defines four monitoring maturity levels for NVIDIA GPU infrastructure, the specific signals and commands at each level, and the exit criteria that tell you a level is done rather than half-deployed. Use it as gap analysis: find the lowest level with missing pieces and fix that before reaching higher.

Two rules apply at every level. First, raw counters lie: alert on deltas and transitions (new XID events, new uncorrectable errors, increases in retired page counts), not on nonzero persistent state. Second, sample fast enough. Temperature spikes, XID events, and PCIe errors live and die in seconds; minute-resolution collection misses them.

flowchart TD
  L1["Level 1: Survival - is the GPU alive, cool, and doing work"]
  L2["Level 2: Operational - thresholds, throttling, reliability deltas"]
  L3["Level 3: Mature - profiling, interconnect, per-MIG, baselines"]
  L4["Level 4: Expert - fabric health, fleet trends, prediction"]
  L1 --> L2 --> L3 --> L4

Each level is cumulative. Level 3 without Level 1 reachability checks is decoration.

Level 1: Survival

The question this level answers: is every GPU present, not overheating, not out of memory, and not throwing fatal errors.

GPU reachability. nvidia-smi -L exit code and /dev/nvidia* device nodes. This is the precondition for everything else. A GPU that was in inventory and is now unreachable is immediate capacity loss: page on sustained unreachability (greater than 60 seconds, with an uptime gate to filter boot noise and driver reloads).
GPU die temperature. nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader,nounits. Temperature determines the clock ceiling, and thresholds are model-specific. Pull the actual limits from nvidia-smi -q on your hardware rather than hardcoding numbers.
VRAM utilization. memory.used vs memory.total. GPU memory has no swap: exhaustion is an allocation cliff, not gradual degradation. Do not page on high usage alone; well-tuned ML frameworks deliberately sit at 95-99% via caching allocators. Sustained linear growth over hours is the leak signal.
SM utilization. utilization.gpu. Coarse and time-based: 100% means at least one kernel ran during the sample window, not that the GPU is efficient. Useful here only as a sanity check that work is flowing.
XID monitoring. dmesg -T | grep -i "NVRM: Xid" or the journald equivalent. At minimum, catch XID 48 (double-bit ECC error) and XID 79 (GPU has fallen off the bus). XID 79 is definitive and requires a node reboot, not a software workaround.
Uncorrectable ECC count. ecc.errors.uncorrected.volatile.total. Any new double-bit error means data corruption has occurred. Alert on the delta, not the raw count: a GPU with a historical nonzero aggregate now running clean should not re-page forever.

Exit criteria: you get paged when a GPU disappears or falls off the bus, ticketed when a new DBE lands, and you can answer “which GPUs exist and are they busy” for every node in under a minute.

Level 2: Operational

The question this level answers: not just “is it up” but “is it degrading, and why is it slow.”

SBE rate, not SBE count. ecc.errors.corrected.volatile.total tracked as a rate of change. Corrected single-bit errors are background noise at low rates (cosmic rays, aging), but an accelerating rate is the leading indicator of memory failure that will produce uncorrectable corruption later. Prefer baseline-relative or peer-relative thresholds over absolutes.
Clock throttle reasons. nvidia-smi --query-gpu=clocks_event_reasons.active --format=csv,noheader (on older drivers the field is clocks_throttle_reasons.active). This is the diagnostic key for performance complaints. hw_thermal_slowdown active and sustained during production compute is page-worthy. sw_power_cap is often intentional in power-capped datacenters, so route it to a ticket. gpu_idle and sync_boost are informational.
Power draw vs enforced limit. Compare power.draw against enforced.power.limit, not power.limit (the enforced value is the real ceiling, the minimum of all limiters). A sustained ratio above 0.95 only matters when corroborated by sw_power_cap active or measurable clock degradation. Near-limit draw under heavy compute with normal clocks is healthy.
PCIe link speed and width. pcie.link.gen.gpucurrent vs pcie.link.gen.gpumax, pcie.link.width.current vs pcie.link.width.max. Gate every alert here on workload activity: drivers downclock PCIe at idle by design (Gen5 dropping to Gen2 at idle is normal), and paging on an idle downgrade is a classic false positive. A downgrade under active load is a silent bandwidth killer.
Clock frequencies. clocks.current.sm vs clocks.max.sm. Below 80% of max during heavy load means significant throttling. Interpret through throttle reasons, never independently: idle GPUs legitimately run at low clocks.
HBM memory temperature. temperature.memory on datacenter GPUs (returns N/A on GDDR consumer cards). HBM has an independent thermal throttle threshold and can throttle memory clocks while the die is fine, which is devastating for memory-bound workloads like LLM inference.
Retired pages. Use nvidia-smi -q -d PAGE_RETIREMENT (or DCGM fields 390/391/392). This is persistent InfoROM state: alert on new increases and on pending = Yes (a reboot is needed to apply the retirement). The driver limit is 64 retired pages; past that, future DBEs are unrecoverable, so plan replacement well before.
Row remapping (Ampere+). nvidia-smi --query-remapped-rows=remapped_rows.correctable,remapped_rows.uncorrectable,remapped_rows.pending,remapped_rows.failure --format=csv. Note the human-readable flag is -d ROW_REMAPPER, not -d REMAPPED_ROWS. failure = true is latched state and RMA-eligible: alert on the transition to true, not the raw boolean, or it will re-fire on every check cycle.
DCGM daemon liveness. pgrep -x nv-hostengine plus a responsiveness check such as dcgmi diag -r 1. A running process is not enough: the daemon can be alive while NVML calls hang and it serves stale cached data. Response latency above 5 seconds means overload or driver deadlock brewing.
Per-process attribution. nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv,noheader and nvidia-smi pmon -s um -c 1. You cannot debug a multi-tenant node or spot a zombie CUDA context holding memory without this. In containers, the PID shown is the host PID, so plan for namespace translation.

Also verify persistence mode: persistence_mode should be Enabled on every production node. Without it the driver unloads between jobs, causing seconds of first-call latency, monitoring gaps, and scheduler races. It is lost on reboot, so set it via a systemd unit, not a one-off command.

Exit criteria: throttling incidents come with a reason attached, ECC and retirement signals alert on deltas, and per-process visibility exists on shared nodes.

Level 3: Mature

The question this level answers: how efficiently is the GPU actually working, and how healthy is the interconnect.

DCGM profiling fields. SM Active, Tensor Active, and DRAM Active (the DCGM_FI_PROF_* family) measure actual pipe utilization rather than time-busy. This closes the biggest interpretive gap in utilization.gpu: a kernel can occupy SMs 100% of the time while the tensor pipes sit idle or the DRAM interface is the real ceiling. Note that some profiling groups share hardware resources and cannot be watched concurrently.
NVLink errors and bandwidth. nvidia-smi nvlink -s for link state, nvidia-smi nvlink -e for per-link error counters, nvidia-smi nvlink -gt d -i N for cumulative throughput. Any nonzero CRC or replay error over a short window is concerning. A failed link forces NCCL onto PCIe, and training keeps running at a fraction of the speed with no error raised, which is why 0% NVLink utilization during multi-GPU training is itself a red flag.
PCIe replay errors and throughput. Replay counts come from nvidia-smi -q -d PCIE (not --query-gpu); throughput comes from nvidia-smi dmon -s t -d 1 or DCGM fields 1009/1010 (DCGM_FI_PROF_PCIE_TX_BYTES/DCGM_FI_PROF_PCIE_RX_BYTES). Replays are cumulative since boot, so track the rate. Sustained growth means link instability that precedes worse failures.
Energy accounting. Integrate power.draw over time per GPU and per tenant for efficiency analysis and chargeback. This turns power from an alerting signal into a cost signal.
Baseline-deviation alerting. Almost every GPU threshold is workload-dependent: training oscillates 0-100% utilization, inference is bursty. Mature alerting compares each GPU against its own per-workload baseline (utilization pattern, step time, ECC rate, thermal profile) instead of fleet-wide static thresholds.
Per-MIG metrics. On MIG-enabled A100/H100, aggregate device metrics are actively misleading: one instance can be OOM while the card-level view looks fine. Monitor per-instance memory and utilization through DCGM, and remember instances share thermal and power domains, so keep the aggregate thermal signals too.

Exit criteria: you can say whether a slow job is compute-bound, memory-bound, or interconnect-bound from metrics alone, and MIG tenants are visible individually.

Level 4: Expert

The question this level answers: what will fail next, and what is the fleet telling you that no single node can.

SM occupancy. Beyond “are the pipes active,” occupancy measures how full the SMs are with resident work. It distinguishes launch-bound or latency-bound kernels from genuinely saturating ones, which matters for kernel-level optimization decisions.
NVSwitch and Fabric Manager health. On DGX/HGX systems, nvidia-fabricmanager manages NVSwitch configuration. If it dies, GPUs individually function but cannot communicate, and multi-GPU jobs hang at NCCL init. Monitor systemctl status nvidia-fabricmanager and its journal, and page only with guards: NVSwitch system, uptime gate, sustained outage, and active jobs failing.
Thermal trending over months. A rising temperature baseline under constant load indicates thermal paste degradation, dust, or cooling decline long before any threshold trips. Track the gap between peak-load temperature and the throttle threshold; if it shrinks month over month, schedule maintenance.
Straggler detection. In distributed training, one GPU 5% slower (thermal throttle, PCIe degradation, ECC overhead) gates every collective. Cross-GPU comparison is the only way to see it: per-GPU utilization divergence, NCCL collective latency against baseline, and step time. Beware the inverted reading: the “fast” GPU is often the healthy one and the rest are degraded.
Driver-version and XID correlation. XID semantics vary slightly by driver version, and driver updates change field behavior and defaults. Track driver_version alongside XID rates and alert behavior so a post-upgrade spike in a given XID gets attributed correctly: new hardware problem, or new driver reporting.
InfoROM validation. nvidia-smi -q -d INFOROM. InfoROM stores persistent reliability state such as retired-page/ECC records and object versions; query serial and part numbers through NVML/nvidia-smi separately. If its checksum fails, every persistent reliability signal on that GPU is unreliable and it may look healthy when it is not. Rare, but it invalidates half your Level 2 signals when it happens.

Exit criteria: hardware gets replaced on trend data before it fails, stragglers are found by metrics rather than by job owners complaining, and fabric-level failures have their own alerting path.

How Netdata helps

Netdata’s NVIDIA GPU collector surfaces the Level 1 and Level 2 signal set per GPU at per-second resolution: utilization, VRAM, temperature, power draw vs limit, clocks, throttle reasons, fan speed, and PCIe throughput, which matters because thermal and PCIe events are short-lived.
Per-process GPU memory and utilization charts give the Level 2 attribution view, so zombie contexts and unexpected consumers are visible without manual pmon runs.
ECC corrected and uncorrected counters are charted over time, making SBE rate acceleration and new DBE deltas visible as trends rather than raw numbers you have to diff by hand.
ML-based anomaly detection on per-GPU metrics supports the Level 3 baseline-deviation posture: deviations from each GPU’s own learned pattern surface without hand-tuned per-workload thresholds.
Because host CPU, disk, network, and GPU metrics share one timeline, the host-starved GPU and straggler patterns are diagnosable by correlation instead of guesswork.

NVIDIA GPU Operations Guides hub for the subsystem mental model, failure archetypes, and the full signal catalog this model is drawn from.