Resetting a wedged NVIDIA GPU: nvidia-smi --gpu-reset and when only a reboot works

A GPU is wedged: jobs on it are dead or hung, nvidia-smi is slow or throwing errors on that index, and the scheduler cannot place new work on the node. The question is whether you can recover the card in place with nvidia-smi --gpu-reset or whether you are burning time on a node that needs a reboot and possibly an RMA.

Try the reset first because it avoids workload migration, scheduler drain, boot time, and driver reinit. But it has hard prerequisites and hard limits: it clears running processes and GPU state, it requires that no other CUDA contexts exist on the GPU, and it does nothing for a card that has fallen off the PCIe bus. Classify the failure before you run the command; that is most of the battle.

This is the decision reference: how to classify the wedge, how to drain and reset safely, how to verify recovery, and how to recognize the pattern where resets stop working.

What this means

A GPU reset is a driver-level recovery operation. nvidia-smi --gpu-reset -i <id> terminates whatever is running on the GPU and returns the device to a clean state without rebooting the host. It is disruptive to anything on that GPU, but far less disruptive than a node reboot.

Two things surprise operators:

1. It does not clear persistent health state. Aggregate ECC counters and retired pages live in the GPU’s InfoROM and survive the reset. A reset does not make a degrading card look healthy again. If anything, the persistent counters are how you prove the reset was needed.
2. It does not fix the underlying fault. A reset restores service. If the fault was hardware, the same Xid will come back. Reset frequency is itself a diagnostic signal, covered below.

One version-dependent caveat: on driver branch 570 and later, only the “GPU Reset Status” output fields in nvidia-smi -q are deprecated and removed (confirmed in the nvidia-smi v575-v580 changelog); the -r/--gpu-reset action itself remains available. The “GPU Recovery Action” fields are the replacement for the old “Reset Required” indicators. Check your driver branch’s nvidia-smi documentation before scripting around these fields.

Common causes

The last two rows matter most: Xid 79 and a full driver hang are the cases where --gpu-reset is a waste of time.

Quick checks

Run these read-only checks before touching anything. They take under a minute and tell you which path you are on.

# 1. Is the GPU still reachable at all?
nvidia-smi -L

# 2. Is the management path slow? (healthy is well under a second)
timeout 5 nvidia-smi --query-gpu=gpu_name --format=csv,noheader -i 2

# 3. What Xids fired?
dmesg -T | grep -i "NVRM: Xid" | tail -20

# 4. Who is still holding the GPU?
nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv,noheader -i 2

# 5. Persistent damage? (survives any reset)
nvidia-smi -q -d PAGE_RETIREMENT
nvidia-smi -q -d ROW_REMAPPER   # Ampere and later

If check 2 hangs or times out on every GPU, or check 3 shows Xid 79, skip ahead to “When only a reboot works.” Do not run a reset against a GPU whose management path is already unresponsive.

Before you reset: drain first

A reset forcibly clears everything on the GPU. That is the point, but it means the reset will either fail or cause collateral damage if anything else holds a context.

• The reset requires no other CUDA contexts on the GPU. If a live process is still attached, the reset refuses or fails. Drain first: stop the workloads, then verify with --query-compute-apps that the GPU is empty.
• In Kubernetes, cordon and drain the node (or at least the GPU-consuming pods) before resetting. A reset under a running pod kills its CUDA context; the pod may not notice cleanly and can hang in a bad state rather than restarting.
• Watch for infrastructure processes holding the GPU. Persistence-related daemons and monitoring agents can hold a handle on the device and cause the reset to report the GPU as in use. If you get an in-use error with an empty compute-apps list, check what has the device nodes open: fuser -v /dev/nvidia*. nvidia-persistenced holds the device files open by design, so it can block a GPU reset; stop the daemon before resetting.
• NVLink topology matters on multi-GPU systems. On Ampere and later, individual GPU reset is supported on NVLink-connected GPUs (with Fabric Manager required on Ampere NVSwitch systems, not required on Hopper and later). On pre-Ampere architectures, all peer-connected GPUs must be reset in the same command. If a single-GPU reset is rejected on an NVLink system, this is likely why.

How to run the reset

Once the GPU is drained and confirmed empty:

# Disruptive: kills any remaining state on GPU index 2
nvidia-smi --gpu-reset -i 2

Expected outcomes:

• Success: the command returns without error. The GPU re-initializes and reappears as a clean device. Any processes that were on it are gone.
• Failure with “in use”: something still holds a context. Go back to the drain step and find the process.
• Failure or hang against a genuinely wedged GPU: the reset could not complete. This is common with Xid 43 stops and universal with Xid 79. Move to the reboot path.

Persistence mode and other per-device settings can be affected by driver state changes. Verify persistence_mode is still Enabled after recovery if you rely on it.

When the reset will not work

The Xid in dmesg predicts the outcome. Build this into your runbook so nobody is improvising at 3 a.m.

flowchart TD
  A[GPU wedged] --> B{nvidia-smi responsive?}
  B -->|no / hangs| H[Driver hang or Xid 79 - reboot path]
  B -->|yes| C{dmesg Xid?}
  C -->|79| H
  C -->|48 / 64 / 74 / 95| D[Drain GPU]
  C -->|43| E[Check if GPU is actually healthy - context may have died cleanly]
  C -->|none / zombie context| D
  D --> F[nvidia-smi --gpu-reset -i N]
  E -->|GPU healthy, new contexts work| G[No reset needed - restart workload]
  E -->|GPU not accepting work| D
  F -->|success| I[Verify: ECC deltas, compute test, resume]
  F -->|fails| H
  H --> J[Reboot node - if GPU does not return, hardware fault]

Verifying recovery

A successful reset command is not proof of a healthy GPU. Verify before returning the node to service:

1. Reachability: nvidia-smi -L shows the GPU, and a per-index query responds in well under a second.
2. Fresh volatile ECC state: check ecc.errors.corrected.volatile.total and ecc.errors.uncorrected.volatile.total. Volatile counters reset with driver/GPU state changes, so establish the post-reset baseline and watch for new deltas.
3. Persistent state is unchanged but honest: retired pages and remapped rows in InfoROM should look exactly as they did before the reset. If counts jumped, the fault is progressing.
4. It accepts work: schedule a small known-good job or have the scheduler place one pod. A GPU that accepts a context and completes it is recovered. A GPU that wedges again immediately was not.

When only a reboot works

Three situations mean stop trying resets and reboot the node:

• Xid 79, fallen off the bus. The PCIe link to the GPU has failed catastrophically. The device may not even appear in lspci. No driver-level operation can reach it. Reboot the node; if the GPU does not reappear after reboot, the hardware is dead and you are in RMA territory.
• The management path itself is hung. If nvidia-smi hangs across GPUs or NVML queries time out, the driver cannot execute a reset. A driver reload or reboot is the remaining option, coordinated with whoever owns the workloads.
• Pending retirements or remappings. retired_pages.pending = Yes or remapped_rows.pending > 0 means the GPU recorded a memory repair that only takes effect on reboot. The GPU runs fine in the meantime, but the repair is queued until you reboot. Plan it; do not let pending state accumulate silently.

After any reboot for a wedged GPU, check that the card re-enumerates and re-run the verification steps above before releasing the node.

A GPU that keeps needing resets is degrading

Treat the first reset as an incident and every subsequent reset as evidence. A GPU that wedges, gets reset, and wedges again is telling you the fault is physical: memory degradation, a marginal PCIe link, power instability, or a thermal intermittent. Operators routinely treat a successful reset as “problem gone.” It is “problem deferred.”

Operationally:

• Log every reset per GPU with timestamp, preceding Xids, and the workload that was running.
• Escalate on recurrence. More than two resets in 24 hours on the same GPU should escalate from ticket to replacement planning. A GPU that resets repeatedly will eventually fail permanently, usually at a worse time.
• Correlate with the degradation signals: accelerating corrected ECC rates, increasing retired pages, climbing PCIe replay errors. These trends often show up weeks before the wedge pattern starts.

Reset frequency tracking converts a series of “we fixed it” tickets into one clear replacement decision.

Metrics and signals to monitor

Prevention

• Enable persistence mode (nvidia-smi -pm 1) via a systemd unit so driver state survives job churn and does not produce false wedge symptoms.
• Watch the leading indicators, not just the wedge. Corrected ECC acceleration, retired page deltas, and PCIe replay rates give you days of warning before the card wedges.
• Write the drain-then-reset runbook before you need it, including the Kubernetes cordon/drain step and the per-Xid decision table. The middle of an incident is the wrong time to learn that Xid 79 cannot be reset.
• Quarantine after uncorrected errors. If Xid 48 or 95 fired during production work, treat recent outputs as suspect regardless of how clean the reset looks.

How Netdata helps

• Per-second GPU metrics catch the short-lived events (utilization collapse, power anomalies, temperature spikes) that minute-resolution monitoring misses in the minutes before a wedge.
• Per-GPU visibility isolates one bad card on a multi-GPU node instead of paging on aggregate symptoms.
• Correlating ECC counter deltas, throttle reasons, and utilization on one timeline distinguishes “GPU wedged by hardware fault” from “job crashed and left a zombie context.”
• Tracking reachability, management latency, and PCIe health over time builds the recurrence history that turns repeated resets into a confident replacement decision.
• Retired pages and row remapping trends surface the permanent damage that persists across resets and reboots, so a “recovered” GPU does not quietly re-enter production with a shrinking memory pool.

Related guides

• NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver
• How an NVIDIA GPU actually works in production: a mental model for operators
• NVIDIA GPU monitoring checklist: the signals every production GPU fleet needs
• NVIDIA GPU monitoring maturity model: from survival to expert