DCGM, nvidia-smi, and NVML: which NVIDIA GPU telemetry source to use

Every NVIDIA GPU troubleshooting session starts with a telemetry question: where do I get this number, and can I trust it? Teams routinely mix the three available sources without realizing they sit at different layers of the same stack. They grep nvidia-smi output in cron jobs, run DCGM and nvidia-smi side by side and wonder why both get slow, or alert on a DCGM field that silently returns zeros on their driver version.

This article lays out the layering: what the kernel driver exposes, what NVML adds, what nvidia-smi actually is, and where DCGM fits. It also covers the gaps all three share, most importantly XID error history, which lives in the kernel log and not in any of them.

For the underlying GPU subsystems these tools measure, see the mental model for how GPUs work in production.

The three layers

Everything starts with the kernel driver, nvidia.ko. It is the only component that talks to the hardware. On top of it:

• NVML (NVIDIA Management Library) is the C library the driver exposes for management queries. Temperature, clocks, ECC counters, memory info, process lists: all of these are NVML calls.
• nvidia-smi is a thin CLI over NVML. When you run nvidia-smi --query-gpu=temperature.gpu --format=csv, you are making one NVML call and printing the result.
• DCGM (Data Center GPU Manager) is a daemon, nv-hostengine, that wraps NVML and adds things NVML does not have: health checks, diagnostics, a policy engine, GPU grouping, named field IDs, and the Prometheus-facing dcgm-exporter.

Nothing bypasses the kernel driver. Every reading from any of these tools is serialized through the same driver mutexes, which has operational consequences covered below.

flowchart TD
    subgraph Host
      K["nvidia.ko kernel driver"]
      NVML["NVML C library"]
      SMI["nvidia-smi CLI"]
      ENG["nv-hostengine (DCGM daemon)"]
      EXP["dcgm-exporter"]
      PROM["Prometheus / monitoring stack"]
      KLOG["kernel log (dmesg, journalctl)"]
    end

    GPU["GPU hardware"] --> K
    K --> NVML
    NVML --> SMI
    NVML --> ENG
    ENG --> EXP --> PROM
    K -- "XID errors (NVRM: Xid)" --> KLOG

What each source is for

nvidia-smi: ad-hoc and interactive use

nvidia-smi ships with the driver, needs no daemon, and is the fastest way to answer a question at 3 a.m. Its strengths:

• Human-readable diagnostics. nvidia-smi -q -d TEMPERATURE,POWER,CLOCK,ECC,PAGE_RETIREMENT dumps thresholds and counters in one shot, including model-specific throttle temperatures that are not publicly documented anywhere else.
• Structured queries. --query-gpu=... --format=csv maps almost directly onto NVML fields. Validate field names against nvidia-smi --help-query-gpu on your driver; wrong field names fail silently in some tooling.
• Things --query-gpu does not cover. Retired pages (--query-retired-pages), row remapping (--query-remapped-rows on Ampere+), PCIe replay counters (-q -d PCIE), NVLink state (nvidia-smi nvlink -s, -e), and topology (nvidia-smi topo --matrix).

Its weakness is that it is a CLI: you get a point sample when you run it. Human-readable output is not a stable interface across driver releases, so do not build long-lived tooling that parses default nvidia-smi output. If you are writing code, call NVML directly; if you are writing monitoring, use DCGM.

Two operational notes:

• Persistence mode matters for telemetry. Without nvidia-smi -pm 1 (requires root), the driver can unload between jobs, causing brief unreachability and seconds of latency on the first query. Enable it on every production node, via a systemd unit so it survives reboots.
• Per-GPU queries for health checks. One wedged GPU can hang an entire multi-GPU query. Use -i N when checking driver health so one bad GPU does not mask the rest. If nvidia-smi fails entirely, that is a driver communication problem; see NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver.

NVML: custom code

NVML is what you use when you are writing your own collector, scheduler integration, or health-check binary. It gives you the same data as nvidia-smi with a stable API: device handles, nvmlDeviceGetTemperature(), nvmlDeviceGetMemoryInfo(), nvmlDeviceGetUtilizationRates(), ECC counters, throttle reasons, and so on.

Rules of thumb:

• If your tool must survive driver upgrades, NVML is the interface to build against, not nvidia-smi output.
• NVML calls block on driver mutexes. A collector that polls aggressively across many GPUs adds real serialization latency to every other NVML consumer on the host, including nvidia-smi and DCGM.
• If NVML is newer than the loaded driver, some calls return “function not found” rather than failing hard. Handle that gracefully instead of treating it as GPU loss.
• For Python, confirm whether your project should use `nvidia-ml-py` rather than the older `pynvml` package before writing new code.

DCGM: fleet monitoring and Kubernetes

DCGM exists because polling NVML ad-hoc does not scale to a fleet. The nv-hostengine daemon runs as root, polls GPUs at configurable intervals, and provides:

• Named field IDs. Instead of nvidia-smi field strings, DCGM exposes numbered fields you watch with dcgmi dmon -e <ids>. Examples used throughout this guide cluster: DCGM_FI_DEV_GPU_TEMP (150), DCGM_FI_DEV_GPU_UTIL (203), DCGM_FI_DEV_ECC_SBE_VOL_TOTAL (310), and DCGM_FI_DEV_XID_ERROR (230). Field support has minimum driver versions; requesting an unsupported field can return blank or zero values without an error, so verify a new field on your driver before alerting on it.
• Health checks and diagnostics. dcgmi health gives group-level status; dcgmi diag -r 1 through -r 3 run escalating validation (rapid, medium, heavy). Heavy diagnostics are disruptive; do not run -r 3 on a GPU with live production work.
• Policy engine. Thresholds on watched fields with asynchronous events, where you want DCGM itself to evaluate rules rather than an external alerting loop.
• Groups. Logical GPU groupings with their own watch configuration, useful for MIG instances and multi-tenant nodes.
• dcgm-exporter. The Prometheus exporter that wraps DCGM fields. This is the standard path for Kubernetes GPU monitoring.

# Check the daemon is alive and responsive
pgrep -x nv-hostengine
time dcgmi diag -r 1

# Watch temperature, SM utilization, and volatile ECC SBE counters
dcgmi dmon -e 150,203,310 -c 5 -d 1000

Deployment caveats that bite operators:

• dcgm-exporter temporal aliasing. The exporter scrapes on its own interval, independent of DCGM’s internal sampling. Fast signals (power spikes, brief throttle events, XID-adjacent transients) can be averaged away or missed. Know your actual observability resolution, not the one you configured in two different places.
• Concurrent NVML consumers interfere. DCGM and nvidia-smi both go through NVML. Polling both at high frequency increases latency for both. Pick one collection path per host for continuous monitoring.
• Container visibility. DCGM in a container may see a partial GPU set depending on runtime configuration. Compare dcgmi discovery -l output against nvidia-smi -L on the host if counts look wrong.
• Embedded mode tradeoffs. DCGM can run embedded in the application process, but that sacrifices persistent history, the policy engine, and event streaming. For fleet monitoring, run the daemon.
• NVSwitch systems. DCGM does not replace Fabric Manager; if nvidia-fabricmanager is down, NVLink connectivity through NVSwitch is gone regardless of what DCGM reports per GPU. See NVIDIA Fabric Manager not running.

What none of them give you

The most important shared gap: XID history is not in NVML or DCGM.

XID errors are emitted by the NVRM driver into the kernel log. DCGM field 230 (DCGM_FI_DEV_XID_ERROR) reports only the most recent XID, not a count and not history. If two XIDs fire between your samples, you lost one. The authoritative record is:

# XID events live here, not in NVML
dmesg -T | grep -i "NVRM: Xid"
journalctl -k | grep -i "NVRM: Xid"

Practical consequences:

• You need host-level log collection for XIDs. Container-only monitoring never sees them.
• Alert on new log entries (delta), not on the field value. Field 230 holding XID 48 for three days is one event, not a continuous emergency.
• Correlate XIDs with the counters NVML does give you: XID 48 with volatile ECC DBE counters, XID 63 with retired pages, XID 79 with GPU reachability. See NVIDIA GPU ECC errors: corrected, uncorrected, volatile, and aggregate for the counter side.

Other shared limits:

• Sampling resolution. Default polling (nvidia-smi roughly 1 second, DCGM defaults similar for critical fields) misses sub-second events. Many GPU faults start and resolve inside that window.
• Stale data during driver stress. When the driver is in trouble, NVML calls slow down or hang. All three tools degrade together because they share the path. nvidia-smi response latency above ~2 seconds sustained is itself a warning signal, often the last one before a wedge.
• No application context. None of these tools see CUDA OOMs, NCCL timeouts, or training step time. GPU telemetry must be correlated with application logs; a healthy-looking GPU can host a deadlocked job. For the memory side of that gap, see CUDA out of memory: diagnosing NVIDIA GPU framebuffer exhaustion.

Decision table

Signals to watch in production

Whichever source you pick, these are the signals worth collecting continuously. Field names given as nvidia-smi / DCGM where both exist.

How Netdata helps

• Netdata collects per-GPU utilization, memory, temperature, power, clocks, fan, and throttle-reason signals at per-second resolution, which catches the sub-second transients that default nvidia-smi or exporter intervals alias away.
• ECC volatile and aggregate counters are tracked as deltas, matching the event-versus-state semantics that uncorrected errors and retired pages require.
• Clock throttle reasons are charted alongside temperature and power, so you can see sw_thermal_slowdown versus sw_power_cap without cross-referencing three CLI outputs.
• Per-second nvidia-smi-equivalent telemetry makes management-path latency visible as collection gaps and outliers, the precursor to driver hangs.
• Correlating GPU signals with host CPU, disk, and network metrics on the same dashboard is what exposes host-starved GPUs and data-pipeline bottlenecks that pure GPU tooling cannot see.

Related guides

• How an NVIDIA GPU actually works in production: a mental model for operators
• NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver
• NVIDIA GPU ECC errors: corrected, uncorrected, volatile, and aggregate
• CUDA out of memory: diagnosing NVIDIA GPU framebuffer exhaustion
• NVIDIA GPU HBM (memory) temperature: the thermal limit most teams miss
• NVIDIA Fabric Manager not running: NVSwitch GPUs lose NVLink