NVIDIA DCGM exporter on Kubernetes: mapping GPU metrics to pods

On a bare-metal GPU node, attribution is simple: nvidia-smi shows you a PID, you look up the process, done. On Kubernetes there are three layers between you and that answer. The device plugin assigns GPUs to pods, the kubelet tracks those assignments, and dcgm-exporter reads GPU state through NVML/DCGM, which has no idea what a pod is. Unless the exporter explicitly joins these two views, your GPU metrics are half-blind: you can see GPU 3 on node 17 at 98% utilization, but not which workload is responsible.

There is a second trap. NVML and nvidia-smi report the host PID of a process, not the container PID. Inside a container namespace that PID is meaningless, so PID-to-container-to-pod mapping has to come from the orchestration layer, not the driver.

This article covers how the mapping works, how to verify it is actually working (not just configured), the failure modes that silently strip pod labels, and the checks that catch orphan GPU processes and driver version drift in GPU Operator deployments.

How the mapping works

dcgm-exporter runs as a DaemonSet on GPU nodes and exposes metrics on port 9400 at /metrics. The Kubernetes mapping is enabled with DCGM_EXPORTER_KUBERNETES=true (or the -k flag). When enabled, the exporter connects to the kubelet’s pod-resources API over the Unix socket at /var/lib/kubelet/pod-resources/kubelet.sock. That API answers one question: which device IDs (GPU UUIDs) are allocated to which pod, namespace, and container on this node.

The exporter then appends that identity to every per-GPU metric. A working metric looks like:

DCGM_FI_DEV_SM_CLOCK{gpu="0",UUID="GPU-4f3a...",container="trainer",namespace="ml-jobs",pod="resnet-train-7b9d"} 139

The pod, namespace, and container labels come from the kubelet join. The gpu, UUID, and device labels come from DCGM itself.

flowchart LR
  subgraph node[GPU node]
    podspec[Pod spec: nvidia.com/gpu request] --> dp[NVIDIA device plugin]
    dp --> kubelet[Kubelet]
    kubelet --> prs[pod-resources socket
/var/lib/kubelet/pod-resources/kubelet.sock]
    dcgm[dcgm-exporter
:9400/metrics]
    prs -->|GPU UUID to pod/namespace/container| dcgm
    gpu[GPU via NVML/DCGM] -->|fields: util, mem, temp, clocks| dcgm
  end
  dcgm --> prom[Prometheus scrape]

Two things to take from this. First, the mapping is a join between two independent data sources, and either side can fail alone. Second, the exporter scrapes DCGM on its own interval, independent of DCGM’s internal sampling, so fast transients can be aliased or missed. Know your scrape interval before interpreting spikes.

Prerequisites to verify

Before debugging missing labels, confirm the chain is intact:

Device plugin health. The NVIDIA device plugin DaemonSet must be running on the node and advertising GPUs to the kubelet. If the plugin is down, no new pods get GPUs and the pod-resources API has nothing current to report:

# Check device plugin pods are Running on GPU nodes
kubectl get pods -n gpu-operator -l app=nvidia-device-plugin-daemonset -o wide

# Confirm the node advertises GPU capacity
kubectl get node <node-name> -o jsonpath='{.status.allocatable}'

You should see nvidia.com/gpu (or your custom resource name) with the expected count. If the count is zero or missing, fix the plugin before touching the exporter.

Kubelet socket mount. The exporter’s DaemonSet must mount /var/lib/kubelet/pod-resources from the host. If this mount is missing, the exporter cannot reach the socket and pod mapping fails silently: metrics keep flowing, labels stay empty.

# Verify the pod-resources hostPath volume exists in the exporter DaemonSet
# GPU Operator names it nvidia-dcgm-exporter; standalone Helm installs differ
kubectl get ds nvidia-dcgm-exporter -n gpu-operator -o yaml | grep -A3 pod-resources

Kubernetes mode enabled. Confirm DCGM_EXPORTER_KUBERNETES=true is set in the container environment (or -k in args). If you deployed via the GPU Operator, dcgm-exporter is enabled by default (dcgmExporter.enabled=true); verify how the operator rendered the DaemonSet rather than assuming the flag.

Verifying pod labels are present

Configuration is not evidence. Check the actual output. Port-forward to the exporter pod on the specific node you are debugging, not the DaemonSet as a whole, since ds/... port-forward picks an arbitrary pod:

# Find the exporter pod on the node in question, then forward to it
kubectl get pods -n gpu-operator -o wide --field-selector spec.nodeName=<node-name>
kubectl port-forward -n gpu-operator pod/<exporter-pod-on-that-node> 9400:9400 &
curl -s localhost:9400/metrics | grep DCGM_FI_DEV_GPU_UTIL

# Look for empty pod labels, the signature of a broken join
curl -s localhost:9400/metrics | grep 'pod=""'

A healthy line has pod, namespace, and container populated for every GPU that has a workload. Two results need attention:

• Empty labels on a GPU with an active pod. The join is broken: kubelet socket unreachable, plugin restart gap, or a resource-name mismatch (see pitfalls).
• Empty labels on a GPU with no current pod, but processes on the GPU. This is the orphan case. Run nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv,noheader on the node. A process consuming GPU memory that maps to no live pod is typically a zombie CUDA context from a crashed or evicted pod. The PID shown is the host PID; match it against the host process table, not container namespaces. These zombies block memory reclamation and usually need a kill on the host PID. Confirm the PID belongs to the orphaned process first: killing a live compute process takes down whatever workload owns it.

Common pitfalls

Custom GPU resource names. Older exporter releases had a hard-coded check for nvidia.com/gpu; pods requesting GPUs under a different resource name (common with MIG profiles or time-slicing setups, e.g. nvidia.com/tesla-v100-sxm2-32gb) were skipped from the mapping, so their metrics carried no pod labels. The workaround is --kubernetes-gpu-id-type device-name; dcgm-exporter v3.3.7/3.5.0 made NVIDIA resource names configurable, which resolves the case for current releases. If you run non-default resource names, verify labels per workload type, not just once.

The Hostname label rename. In v4.6.0 the Hostname label was renamed to lowercase hostname (dcgm-exporter pull request #655). Any Prometheus recording rule, alert, or dashboard referencing Hostname breaks silently on upgrade. Grep your rules before rolling out.

Distroless images. Recent Helm chart releases default to a distroless image: the chart switched to distroless by default with the v4.5.1-4.8.0 release. There is no shell in the container, so kubectl exec ... -- sh debugging no longer works. Debug from the node or via a debug container instead.

MIG mode. Pod labels on MIG instances depend on exporter version; dcgm-exporter v4.5.3 added per-process GPU metrics for time-sharing and MIG (issue #594), which is the first release with per-MIG-instance pod attribution. Older releases report metrics per MIG instance without Kubernetes labels.

Runtime container labels are a separate feature. The --container-labels flag pulls labels from the host container runtime and is off by default. It is unrelated to Kubernetes pod mapping; enabling one does not fix the other.

Profiling fields are not in the default set. If DCGM_FI_PROF_* metrics (e.g. DCGM_FI_PROF_GR_ENGINE_ACTIVE) log “metric not enabled”, the deployed metrics configuration does not include the profiling fields. Check the metrics ConfigMap the operator or chart rendered.

Verifying the driver behind the metrics

In GPU Operator deployments the driver runs in a container. The driver version the node reports and the version actually loaded can diverge during upgrades, especially if a driver pod failed to roll. The driver.useOpenKernelModules field (deprecated in GPU Operator v25.3.0, replaced by driver.kernelModuleType) is one common upgrade breakage.

Do not trust the node’s reported driver alone. Check the image the driver container is actually running:

# Image running in the driver DaemonSet
kubectl get ds nvidia-driver-daemonset -n gpu-operator \
  -o jsonpath='{.spec.template.spec.containers[0].image}'

# Driver version the kernel module actually reports (run on the node)
nvidia-smi --query-gpu=driver_version --format=csv,noheader

If the DaemonSet rolled a new image but nodes report the old version, some driver pods did not load the new module, which usually means a node reboot or a blocked driver pod. DCGM fields also have minimum driver versions; an old driver can return blank or zero values for newer fields without any error, which looks exactly like an exporter problem.

Signals to monitor

How Netdata helps

• Netdata collects per-GPU utilization, framebuffer memory, temperature, power draw, and throttle reasons at per-second resolution, which matters because exporter scrape intervals alias short events.
• Correlating GPU memory used against the pod lifecycle on the same node surfaces orphan CUDA contexts: memory flat while the workload is gone.
• Cross-referencing GPU utilization with host CPU, disk, and network I/O on the same node distinguishes a starved GPU from a saturated one.
• XID events and ECC counters alongside utilization separate a hung kernel (100% util, no progress, no XID) from a hardware fault (XID present).
• Per-node GPU comparison on multi-GPU machines makes the straggler pattern visible: one GPU diverging in temperature, clocks, or PCIe throughput while the job still runs.

Related guides

• How an NVIDIA GPU actually works in production: a mental model for operators
• NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver
• CUDA out of memory: diagnosing NVIDIA GPU framebuffer exhaustion
• CUDA out of memory with free memory available: GPU memory fragmentation
• NVIDIA BAR1 memory exhaustion: mapping failures with free framebuffer
• NVIDIA GPU ECC errors: corrected, uncorrected, volatile, and aggregate