DCGM nv-hostengine down or hung: GPU telemetry goes dark or stale

Your GPU dashboards flatline, or worse, they do not. The graphs keep rendering smooth, plausible values for temperature, utilization, and power, but nothing on the node has actually changed in twenty minutes. Both symptoms point at the same component: nv-hostengine, the DCGM daemon that sits between NVML and every monitoring tool you run.

nv-hostengine fails in two distinct ways, and only one of them is obvious. The obvious failure is a dead process: no data, gaps in every time series, alerts firing on absent metrics. The dangerous failure is an alive-but-hung daemon: the process exists, the socket accepts connections, and DCGM serves stale values from its in-memory field value cache because an underlying NVML call is blocked on the driver. Dashboards look normal. Threshold alerts stay quiet because the cached values are inside bounds. You find out when a human notices the numbers have not moved.

This guide covers detecting both states, telling a hung DCGM apart from a hung driver, and recovering without making things worse.

What this means

nv-hostengine is the core DCGM daemon. It runs as root, polls GPU state through NVML on background health threads, and stores recent samples in an in-memory field value cache. Clients, including dcgmi, dcgm-exporter, and any custom collectors, query the daemon, not the driver directly.

That layering is why a hang is so deceptive. When an NVML call blocks, for example because the driver is deadlocked or a GPU has wedged mid-query, the health thread stalls but the daemon process keeps running. Client queries are answered from the cache with the last good sample. Every downstream consumer sees fresh-looking data that is actually frozen in time.

Two consequences follow:

• Process liveness is not proof of health. pgrep nv-hostengine succeeding, a listening socket, or an HTTP 200 from dcgm-exporter tells you nothing about whether real NVML queries are completing.
• A DCGM hang can be a symptom, not the disease. A hung NVML call is often the first visible sign of a driver deadlock or a GPU hardware fault stalling the bus. Restarting the daemon without checking the driver can leave the new process hanging on initialization.

flowchart TD
  A[GPU telemetry looks wrong] --> B{nv-hostengine process present?}
  B -- No --> C[Dead daemon: check logs, restart, find crash cause]
  B -- Yes --> D{Active probe returns fresh values?}
  D -- Yes --> E[Daemon healthy: check exporter and scrape config]
  D -- No or unchanged values --> F{nvidia-smi per-GPU query responds?}
  F -- Yes --> G[Wedged daemon: restart nv-hostengine]
  F -- No --> H[Driver/GPU fault: do NOT restart DCGM alone]
  H --> I[Check dmesg for Xid, plan driver reload or reboot]

Common causes

Quick checks

All of these are read-only and safe.

# 1. Is the process there at all, and how long has it been running?
pgrep -x nv-hostengine
ps -o pid,etime,cmd -C nv-hostengine

A very short etime on a node that has been up for weeks means the daemon recently restarted. Find out why before treating its return as recovery.

# 2. Active liveness probe: run a real query with a hard timeout
timeout 10 dcgmi dmon -e 1001,1002 -c 1
echo "exit: $?"

This is the check that matters. It forces a field query through the daemon. Exit code 124 (timeout) or a hang means the daemon is not completing queries even though the process is alive. A healthy daemon returns in well under 2 seconds.

# 3. Freshness check: take two samples and compare
dcgmi dmon -e 1002 -c 2 -d 5000

If the value is identical across samples on a node under load, treat the data as stale. A value unchanged across intervals, or older than roughly 2x the collection interval, is suspect.

# 4. Cross-check with the driver directly, per GPU, with a timeout
timeout 5 nvidia-smi --query-gpu=gpu_name --format=csv,noheader -i 0

If nvidia-smi answers quickly while DCGM hangs, the wedge is in the daemon. If nvidia-smi also hangs, the problem is the driver or the GPU, and DCGM is a victim. Always query per-GPU with -i N: one wedged GPU can hang a whole-node query.

# 5. Look for driver-level faults
dmesg -T | grep -i "NVRM: Xid" | tail -20

# 6. Rapid DCGM diagnostic as a deeper responsiveness check
timeout 30 dcgmi diag -r 1

# 7. On Kubernetes: what is the probe actually testing?
kubectl describe pod -n gpu-operator -l app=nvidia-dcgm-exporter | grep -A4 -i "readiness\|liveness"

If the probe is an HTTP GET on port 9400 /health, it proves the web server is up, not that DCGM is answering.

How to diagnose it

1. Classify the failure: dead or hung. Run check 1. No process means dead: go to step 5. Process present means you must distinguish hung from healthy with an active probe, not with liveness signals.

2. Probe with a real field query under a timeout (check 2). A timeout or hang means the daemon is wedged. Capture the output of check 3 to confirm values are frozen rather than merely quiet.

3. Determine whether the driver is also stuck (checks 4 and 5). This decides your recovery path. A wedged daemon over a healthy driver is a daemon restart. A wedged daemon over a hung driver is a driver or hardware incident. Do not restart DCGM alone against a hung driver: the new process will hang on initialization.

4. Check timing correlations. Did a driver update land recently without a daemon restart? Did the daemon’s start time coincide with a crash loop? Did metrics go stale at the same moment across all GPUs (driver-level) or just some (possibly one GPU wedging a shared query path)?

5. For a dead daemon, find the cause before restarting. Review the service logs and journal for a segfault, OOM kill, or fatal NVML error. Restarting without understanding the crash invites a repeat. After restart, verify GPU enumeration (dcgmi dmon -e 1001 -c 1 against nvidia-smi -L): a daemon that comes back seeing zero GPUs has a driver or permissions problem, not a monitoring gap.

6. In Kubernetes, verify what “healthy” means. A dcgm-exporter pod can be Ready while the engine inside is wedged, because the default probes only exercise the HTTP server. Confirm staleness by comparing a metric value across two scrape intervals before concluding anything about the node.

Metrics and signals to monitor

Fixes

Dead daemon

Restart the service after capturing logs. Then immediately verify two things: the daemon answers an active probe (check 2), and it enumerates the expected GPUs. If the crash was a segfault or OOM kill, treat it as a bug or sizing issue to track, not a one-off. Keep DCGM and dcgm-exporter versions matched; mismatched versions have been reported to crash the hostengine itself.

Hung daemon over a healthy driver

Restart nv-hostengine. This is disruptive to monitoring but not to running GPU work: clients lose telemetry during the restart, and the field value cache is rebuilt, so expect 1-2 polling intervals of empty or zero data before values stabilize. Do not alarm on that warmup window. After restart, confirm freshness with a two-sample comparison.

Hung daemon over a hung driver

Do not just bounce DCGM. A driver deadlock or wedged GPU requires a coordinated driver reload or node reboot, and that means draining or checkpointing GPU workloads first. Collect evidence before you clear the state: hung process list, dmesg Xid history, and per-GPU query results. If a single GPU is implicated (one device hangs nvidia-smi -i N while others respond), Xid 79 and friends in dmesg will usually name it. Follow the driver-incident path in NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver.

After any driver update

Restart nv-hostengine as part of the driver upgrade procedure. The daemon can hold a stale NVML reference across a driver reload and misbehave in ways that look like a new bug but are just version skew between the daemon and the kernel module.

Prevention

• Monitor liveness two ways, always. Process presence plus an active probe that executes a real DCGM field query under a timeout. Either alone is insufficient: the first misses hangs, the second is meaningless if the process is gone.
• Add a freshness check to your pipeline. Alert when a value that should move under load (utilization, power, temperature) is unchanged across multiple collection intervals, or when the newest sample is older than 2x the collection interval. This is the only reliable catch for stale-cache failure.
• Fix Kubernetes probes. A readiness probe that hits the exporter’s HTTP health port will keep a wedged pod in rotation. The probe must exercise DCGM itself.
• Use DCGM’s built-in hang detection. Since DCGM 4.4.2, nv-hostengine has hang detection enabled by default; it logs when a hang is detected. The DCGM_HANGDETECT_TERMINATE environment variable escalates that to daemon termination, which converts a silent hang into a visible, alertable death. DCGM_HANGDETECT_EXPIRY_SEC tunes the timeout (minimum 120s, in 60s steps).
• Pin and align versions. Deploy dcgm-exporter and the DCGM hostengine packages together, and restart the daemon on every driver update.
• Alert on unexplained restarts. Any nv-hostengine restart outside a deployment or maintenance window is a signal, not a recovery.

How Netdata helps

• Netdata’s NVIDIA GPU collector queries through NVML directly at per-second resolution, so a frozen DCGM layer does not silently freeze your only view of the node: you can compare Netdata’s live values against DCGM-derived dashboards to spot staleness.
• Per-second temperature, power, and utilization series make freshness anomalies obvious: a healthy GPU under load never prints identical values for minutes.
• Process monitoring of nv-hostengine (presence, uptime, restarts) turns unexplained daemon restarts into alertable events instead of silent flapping.
• Xid and NVRM error visibility from kernel logs, correlated on the same timeline as GPU metrics, lets you see whether a telemetry freeze preceded or followed a driver-level fault.
• Anomaly detection on GPU utilization and power flags the flatline pattern itself, which is the signature of the alive-but-hung failure that threshold alerts cannot catch.

Related guides

• NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver
• How an NVIDIA GPU actually works in production: a mental model for operators
• NVIDIA Fabric Manager not running: NVSwitch GPUs lose NVLink