nvidia-smi hangs or is unresponsive: a wedged GPU or a stuck driver

You run nvidia-smi and nothing comes back. No output, no error, no exit code. Ctrl-C does nothing, and kill -9 does nothing either. This is a different class of failure from “NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver”, where the driver at least answers and tells you it is broken. A hang means the driver is stuck inside a kernel call, usually waiting on GPU hardware that will never respond.

The blast radius extends past your shell. Everything that talks to the GPU through NVML shares the same management path: nvidia-smi, DCGM’s nv-hostengine, monitoring agents, and any tooling that queries GPU state. When that path blocks, observability on the node goes dark at exactly the moment you need it. Worse, collectors without a timeout on the query report “no data” instead of “collection timed out”, so the hang looks like a quiet gap in the charts rather than a fault.

This guide covers how to confirm the hang, how to tell a wedged GPU from a driver-level stall, what you can and cannot fix without a reboot, and how to instrument the management path so you see the next one coming.

What this means

nvidia-smi is a thin client over NVML, and NVML calls into the NVIDIA kernel driver. A healthy driver answers a lightweight query in well under 100 milliseconds. When the command hangs, the process has entered uninterruptible sleep (D state) inside the driver, waiting on a hardware resource or an internal driver lock. A process in D state cannot be killed, not even with SIGKILL, because the kernel does not deliver signals to a thread blocked in an uninterruptible call. It stays in the process table until the call returns or the kernel resets.

Two underlying situations produce this:

1. A wedged GPU. The device itself stopped responding: it fell off the PCIe bus (Xid 79), hit a fatal internal fault, or stalled the bus. Every subsequent management call against that GPU blocks. On multi-GPU nodes, one wedged GPU can hang an unqualified nvidia-smi query even though the other GPUs are fine, because the default query iterates all devices.

2. A stuck driver. The driver itself is deadlocked or livelocked: an internal mutex never releases, a firmware call never returns, or recovery from a previous fault never completes. Every GPU on the node is affected at once, and all NVML clients (nvidia-smi, nv-hostengine, your monitoring agent) stall together.

The operational consequence is the same either way: no software-only poke reliably clears it. A GPU reset (nvidia-smi -r) sometimes helps when exactly one GPU is wedged and nothing holds a context on it, but the common end state for a driver-level stall is a node reboot. Plan your diagnosis around gathering evidence quickly, then scheduling the reboot deliberately instead of discovering at hour three that nothing else works.

Common causes

Quick checks

All of these are read-only and safe to run during an incident.

# 1. Time-box every query. Never run a bare nvidia-smi on a suspect node.
timeout 5 nvidia-smi --query-gpu=gpu_name --format=csv,noheader -i 0
echo "exit: $?"   # 124 = timeout killed it, the management path is blocked

# 2. Query each GPU individually to find which device wedges the call.
# Count both device classes: datacenter GPUs enumerate as "3D controller",
# but some SKUs appear as "VGA compatible controller".
for i in $(seq 0 $(( $(lspci -d 10de: | grep -cE "3D controller|VGA compatible controller") - 1 ))); do
  echo -n "GPU $i: "
  timeout 5 nvidia-smi --query-gpu=gpu_name --format=csv,noheader -i $i \
    && echo "ok" || echo "HUNG or error"
done

# 3. Check kernel logs for Xid events, the primary evidence trail.
dmesg -T | grep -i "NVRM: Xid" | tail -20

# 4. Look for processes stuck in D state on the driver.
ps -eo pid,stat,comm,wchan | awk '$2 ~ /D/'

# 5. Confirm whether the PCIe devices are still enumerated.
lspci -d 10de:
ls -la /dev/nvidia*

# 6. If DCGM is deployed, test whether the daemon is alive but stalled.
time timeout 10 dcgmi diag -r 1

Two things to note. First, the timeout 5 wrapper is not optional decoration: it is what keeps your diagnostic shell from becoming another D-state process. Second, check 2 is the single most informative step, because its result splits the incident in half: one GPU hanging points at the device, all GPUs hanging points at the driver.

How to diagnose it

1. Confirm the hang is real, not slow. A healthy driver responds in under 100 ms. Between 100 ms and 1 s is elevated. Sustained responses over 2 s indicate driver stress and deserve immediate investigation. A query that exceeds your 5 s timeout should be treated as unreachability, not as a slow success.

2. Isolate the scope. Run the per-GPU loop from the quick checks. Exactly one hanging GPU means a device problem (wedge, fallen off bus, pending reset). All GPUs hanging means a driver-level problem (deadlock, firmware stall).

3. Pull the Xid history. dmesg -T | grep -i "NVRM: Xid" is your ground truth. Xid 79 means the GPU fell off the bus and the node needs a reboot. Recurring Xid 43 means an application kept faulting; the GPU hardware is usually fine, but cleanup of its contexts can wedge the driver. Xid 119 (GSP RPC timeout) points at a firmware-path stall on newer drivers. Xid 48 or 95 alongside the hang means uncorrectable memory errors are involved and recent work output should be treated as suspect.

4. Check for stuck user processes. Processes in D state holding CUDA contexts block any GPU reset attempt. List them, map PIDs to jobs or containers (the PID nvidia-smi reports is the host PID, not the container PID), and record them before doing anything disruptive.

5. Check DCGM separately if deployed. nv-hostengine makes NVML calls from its health threads; when NVML blocks, the daemon stalls and serves stale cached data while looking superficially alive. A daemon that responds to its socket but cannot complete dcgmi diag -r 1 quickly is blocked on the driver, not healthy. Do not restart nv-hostengine at this point: it will hang on initialization against the same stuck driver.

6. Decide: reset attempt or reboot. If exactly one GPU is wedged, no processes hold contexts on it, and your driver supports it, nvidia-smi -r is a reasonable single attempt. If the reset fails, if processes are stuck in D state, or if all GPUs are affected, schedule the node reboot. Do not burn hours retrying software recovery on a driver deadlock.

flowchart TD
    A[nvidia-smi hangs, no error] --> B[timeout 5 per-GPU queries]
    B --> C{Which GPUs hang?}
    C -->|One GPU| D[dmesg: Xid 79 or fatal Xid?]
    C -->|All GPUs| E[Driver deadlock or firmware stall]
    D -->|Xid 79| F[Node reboot required]
    D -->|Xid 43 storm| G[Kill faulting app, then retry query]
    E --> H[Processes in D state?]
    G --> H
    H -->|Yes| F
    H -->|No, single GPU| I[Try nvidia-smi -r once]
    I -->|Reset fails| F
    I -->|Reset works| J[Monitor for recurrence, track reset history]

Metrics and signals to monitor

Fixes

If one GPU is wedged and nothing holds a context on it

Attempt a single GPU reset: nvidia-smi -r (or -i N -r on multi-GPU systems). This is disruptive: it tears down all state on that device and fails outright if any process has an open context, which is why step 4 of the diagnosis comes first. If the reset succeeds, treat the GPU as suspect: track its reset history and watch for recurrence. A GPU that wedges once usually wedges again, and repeated resets are a replacement signal, not a solved problem.

If processes are stuck in D state

You cannot kill them; SIGKILL does not reach a thread blocked in an uninterruptible kernel call. Trying harder (kill -9 loops, cgroup freezes) changes nothing. Record the PIDs, the jobs they belong to, and the Xid history for the post-incident writeup, then proceed to the reboot. This is also why “just restart the training job” is not available as a fix: the job’s processes may not be killable.

If the driver is deadlocked across all GPUs

There is no supported software recovery short of reloading the driver, and a driver reload on a node with stuck contexts and blocked NVML clients is itself unreliable. The dependable path is: drain or cordon the node in your scheduler, capture dmesg and process state, and reboot. Do not restart DCGM or your monitoring agent first and hope; they will hang on initialization against the same driver and complicate the picture.

If an application fault storm (Xid 43) triggered it

Xid 43 means the GPU stopped processing work for an application after a software-induced fault; the hardware is typically fine. Kill the faulting application if it is still killable, then re-test per-GPU reachability. If the driver recovers, fix the application bug before rescheduling: repeated Xid 43 events can escalate into cleanup states that leave the driver in the wedged condition you started with.

If persistence mode was off

Enable it: nvidia-smi -pm 1, persisted through a systemd unit or startup script, since the setting is lost on driver reload or reboot. This removes the driver-unload latency and race window that produces intermittent slow or briefly hanging queries. It does not protect against a real hardware wedge; it removes a common false friend.

Prevention

• Time-box every collection call. Wrap NVML and nvidia-smi collection in a hard timeout (5 s is a sane default) in every monitoring agent, health check, and runbook script. NVIDIA support guidance for scripted nvidia-smi queries recommends prepending a timeout wrapper (widely repeated in NVIDIA community/support channels; the current custhelp article text is not publicly linked at review time).
• Make timeouts loud. The collector must report “collection timed out” as an error state, not as absent data. A silent gap on the GPU charts is the masked version of this incident. Alert on timeout events, and treat sustained latency above 2 s as a warning before the timeout ever fires.
• Query per GPU in health checks. An all-GPU query lets one bad device hide the health of seven good ones and can hang the whole check. Per-device queries with per-device timeouts give you isolation for free.
• Enable persistence mode everywhere in production and verify it as a configuration-drift signal after reboots and driver updates.
• Monitor Xid as events, not state. Alert on new Xid 79, 48, 95 occurrences as pages, recurring 43 as tickets, and treat 119/109-style timeout events as leading indicators rather than noise. Ensure log aggregation latency does not turn a fresh event into a delayed duplicate alert.
• Track GPU reset and recovery frequency per device so repeat offenders get scheduled for replacement before they take a node down mid-job.
• Sample fast enough. Wedges and their precursors develop in seconds. Minute-resolution collection misses the latency ramp that precedes a hang; sub-10-second sampling on the management path is what makes “>2 s for 60 s” an observable condition.

How Netdata helps

• Netdata’s NVIDIA collector queries the management path continuously, so per-GPU reachability and metric gaps surface in near real time instead of at the next manual check.
• Distinguishing “the GPU is gone” from “everything is fine” depends on collection health: correlate GPU chart gaps with node-level signals (D-state processes, kernel log Xid events) on the same dashboard to confirm a wedge rather than a network or agent issue.
• Temperature, power draw, ECC error counters, and clock throttle reasons for the days before the hang sit on the same timeline, which makes it easy to check whether hardware degradation preceded the wedge.
• Pairing dmesg evidence with the exact window where GPU metrics flatline or disappear turns a postmortem guess into a timestamped sequence.
• On multi-GPU nodes, per-device charts expose the “one GPU dark, seven healthy” pattern that points at a device fault rather than a driver deadlock.

Related guides

• NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver
• How an NVIDIA GPU actually works in production: a mental model for operators
• NVIDIA GPU monitoring checklist: the signals every production GPU fleet needs
• NVIDIA GPU monitoring maturity model: from survival to expert