You see NVRM: Xid (PCI:0000:XX:00.0): 43, ... GPU stopped processing in dmesg, and the job on that GPU has gone silent. CUDA calls do not return. nvidia-smi either hangs outright or returns a stale view of the card: utilization frozen at the last sampled value, memory allocation unchanged, power draw flat. The GPU has not disappeared from the bus the way it does with Xid 79. It is still enumerated and still answering the driver at some level, but the compute engine is wedged and nothing it was running will complete.
Xid 43 is classified as an application-induced error: the usual root cause is the workload itself, such as a kernel that deadlocked or an access pattern that poisoned engine state. The hardware is often fine. Operationally, though, it behaves like a hardware incident: recovery frequently requires a reboot, and a hang can also be the downstream symptom of genuine degradation such as ECC faults. Treating every Xid 43 as “just restart the app” is how teams lose GPUs to slow-burn memory failures they never investigated.
This article covers how to confirm the wedge, separate application-induced hangs from hardware-induced ones, attempt recovery in the right order, and decide when the card needs to leave the fleet. For the broader picture of how the compute engine, memory subsystem, and error correction interact, see how an NVIDIA GPU actually works in production.
What this means
The GPU’s watchdog logic determined that the compute engine stopped making forward progress. The channel feeding work to the engine stalled and the driver gave up on it. From that point:
- CUDA API calls into that GPU’s context block or fail. The application process is typically alive but stuck, often in uninterruptible sleep (D state) once it touches the driver again.
nvidia-smioutput for the affected GPU goes stale: utilization, memory, and power stop updating. On some systems the query itself hangs, because one wedged GPU can stall the whole NVML management path. Query per-GPU with-i Nto isolate.- The GPU is still on PCIe.
lspcistill shows it. This is the key distinction from Xid 79 (“GPU has fallen off the bus”), where the link itself is dead and no driver-level recovery applies.
Two things are commonly misread. First, Xid 43 is not page retirement. Page retirement and remapping events are Xid 63/64; confusing 43 with 63 sends operators down a hardware-RMA path for what may be a pure software bug. Second, “application-induced” does not mean “harmless”. The application induced the hang, but the GPU does not necessarily return to service afterward. The engine state is corrupt enough that nvidia-smi --gpu-reset frequently fails, and a full reboot is the usual recovery.
flowchart TD
A[Xid 43 in dmesg] --> B{nvidia-smi -i N responds?}
B -->|hangs| C[GPU wedged, skip to reboot path]
B -->|responds, stale values| D{Other Xids present?}
D -->|Xid 13/31 nearby| E[Application bug: fix code, drain, reset or reboot]
D -->|ECC errors, Xid 48/63/64| F[Suspect hardware: check remapped rows, quarantine]
D -->|nothing else| G[Try nvidia-smi --gpu-reset]
G -->|reset succeeds| H[Replay workload, watch for recurrence]
G -->|reset fails| I[Drain node, reboot]
H -->|Xid 43 recurs| FCommon causes
| Cause | What it looks like | First thing to check |
|---|---|---|
| Application bug (illegal access, deadlocked kernel) | Xid 43 with Xid 13 or 31 nearby in dmesg; same job or kernel reproducibly triggers it | dmesg history for preceding Xids; which PID/kernel was running |
| Underlying memory degradation | Xid 43 plus rising ECC counters, Xid 48/63/64 in the same window, row remap pending | nvidia-smi -q -d ECC and -d ROW_REMAPPER |
| Firmware/CUDA/tensor-core interaction | Hangs cluster on a specific driver/CUDA version or a specific kernel pattern; fleet-wide on same config | Correlate occurrence with driver version and workload type across nodes |
| Driver-level stress on the management path | nvidia-smi latency creeping up (>2s) in the hours before the hang | Management path latency history, per-GPU query timing |
| Progressive hardware failure | Xid 43 events recur on the same GPU with shrinking intervals, later joined by Xid 45 or 79 | Per-GPU Xid event history over weeks |
The last row matters most for fleet hygiene. A single Xid 43 on a healthy card after a buggy kernel is a code problem. A cadence of Xid 43s on the same card, escalating toward Xid 79, is a dying GPU.
Quick checks
All read-only and safe to run during an incident, with one caveat: query per-GPU (-i N) so a hung card does not stall your session.
# 1. Confirm the Xid and see what came before it
dmesg -T | grep -i "NVRM: Xid" | tail -50
# 2. Check reachability of the specific GPU, with a timeout
timeout 5 nvidia-smi --query-gpu=name,uuid,pstate --format=csv,noheader -i 0
# 3. Take two utilization readings 5s apart - identical stale values indicate a wedge
nvidia-smi --query-gpu=utilization.gpu,utilization.memory,power.draw --format=csv,noheader -i 0
sleep 5
nvidia-smi --query-gpu=utilization.gpu,utilization.memory,power.draw --format=csv,noheader -i 0
# 4. See what processes hold the GPU (stuck CUDA contexts)
nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv,noheader -i 0
# 5. Check process state on the host - D state = uninterruptible, unkillable until reboot
ps -eo pid,stat,comm | awk '$2 ~ /^D/'
# 6. Verify the GPU is still on PCIe (distinguishes Xid 43 from Xid 79)
lspci -d 10de:
# 7. Look for hardware-degradation fingerprints
nvidia-smi -q -d ECC -i 0 | grep -iE "volatile|aggregate" | head -20
nvidia-smi --query-remapped-rows=remapped_rows.correctable,remapped_rows.uncorrectable,remapped_rows.pending,remapped_rows.failure --format=csv,noheader -i 0
nvidia-smi -q -d PAGE_RETIREMENT -i 0 | head -20
If step 2 hangs or times out, the management path is wedged on that GPU and most software-level recovery is off the table. Note that nvidia-smi -q output under a wedge may show the last cached values; values that refuse to change across samples are themselves the signal. The --query-remapped-rows flag requires Ampere or newer; on older cards it will report N/A.
How to diagnose it
Establish the timeline from dmesg. Pull every NVRM/Xid line for the affected PCI address. You are looking for what preceded Xid 43: Xid 13 (graphics engine exception) or Xid 31 (memory page fault) point to an application bug as the trigger. Xid 48, 63, or 64 nearby point to memory hardware. Nothing nearby points to either a clean application deadlock or a driver/firmware interaction.
Identify the workload. Match the GPU’s compute-apps list and the stuck PIDs against scheduler records. If the same job, kernel, or framework version has produced Xid 43 on other nodes, you have a software pattern, not a hardware one.
Check the hardware fingerprints. On Ampere and newer, check
remapped_rows.pendingandremapped_rows.failure. On pre-Ampere, check retired pages and whetherretired_pages.pending = Yes. Rising volatile ECC counters in the same boot as the hang are a strong hint the “application error” classification is masking a memory fault that poisoned engine state.Check the management path history. If you collect nvidia-smi/NVML query latency, look at the hours before the hang. Sustained latency above 2s per GPU is often the last observable warning before a wedge.
Decide the recovery class. Application bug with clean ECC and no remap activity: reset or reboot, fix the code, move on. Any ECC/remap involvement, or recurrence on the same GPU: treat as suspected hardware and quarantine after recovery.
After recovery, re-verify. Post-reboot, re-check ECC counters, retired pages, and remapped rows. Confirm the GPU enumerates, passes a real workload, and does not immediately re-wedge. A card that hangs again on the next kernel launch after reset is telling you the cause is persistent, not transient.
Metrics and signals to monitor
| Signal | Why it matters | Warning sign |
|---|---|---|
| Xid events in dmesg (per-GPU, per-code) | Xid 43 is the detection event; neighbors classify it | New Xid 43; any Xid 13/31/48/63/64 in the same window |
| nvidia-smi / NVML query latency per GPU | Last early warning before a wedge | Sustained >2s per GPU |
| Utilization staleness | Frozen utilization/memory/power values mean the engine stopped | Identical values across samples while a job claims to be running |
| ECC volatile counters (corrected and uncorrected) | Distinguishes hardware-induced hangs | Delta increase in uncorrected count, or accelerating corrected rate |
| Remapped rows / retired pages | Pending or failing remaps after a hang indicate the real cause | remapped_rows.pending > 0, any new failure transition, rising retired pages |
| GPU process list | Stuck contexts hold memory and block reset | PIDs in D state that survive SIGKILL |
| Per-GPU Xid recurrence history | A GPU that hangs once and recovers may hang again | >1 Xid 43 on the same GPU in a rolling window, or 43 progressing to 45/79 |
On alert routing: Xid 43 by itself is an INFO/TICKET-class event in most taxonomies, because it is usually application-induced. It earns a PAGE when it arrives with a new uncorrected ECC event, with a remapped_rows.failure transition, or as part of a recurrence pattern on a GPU running production work.
Fixes
Application-induced hang (the common case)
Drain or kill the affected workload first. Stuck CUDA contexts hold memory and will block a GPU reset, and killing a D-state process may not be possible until reboot. Then attempt a GPU reset:
# Disruptive: kills all remaining work on this GPU. Drain other GPUs first if sharing the node.
nvidia-smi --gpu-reset -i 0
Expect this to fail frequently for Xid 43. The engine state after a wedge is often too corrupt for a driver-level reset. If it fails, the reliable path is a full node reboot: drain the other GPUs, migrate or checkpoint workloads, reboot, then re-verify the card before returning it to service.
After recovery, the real fix is in the workload: the kernel that deadlocked or the access pattern that wedged the engine will do it again. Capture the reproducing job, driver version, and CUDA version for the application owners.
Suspected hardware-induced hang
If ECC counters moved, remaps are pending, or the hang recurs on the same GPU, do not just reboot and forget. The reboot applies pending row remaps and page retirements. Then quarantine: keep the GPU out of production scheduling until it survives a soak under a known-good workload. Escalating Xid patterns (43, then 45, then 79) are an RMA conversation, not a monitoring conversation. See resetting a wedged NVIDIA GPU for the full reset-versus-reboot decision path.
When nvidia-smi itself hangs
A wedged GPU can take the whole management path down: nvidia-smi blocks for every GPU on the node, and the querying process enters D state where SIGKILL does nothing. There is no userspace fix for that state. Drain what you can and reboot. For the diagnostic side of that symptom, see nvidia-smi hangs or is unresponsive.
Prevention
- Monitor management path latency. Per-GPU nvidia-smi response time trending past 2s is your earliest wedge warning. It costs almost nothing to collect.
- Track Xid history per GPU, per code. Xid 43 only becomes actionable in context: what came before it, and how often this GPU has done it. Use delta-based log monitoring, not raw greps after the fact.
- Track ECC and remap state continuously. Accelerating corrected-error rates and
remapped_rows.pendinglet you retire a degrading GPU before it wedges under production load instead of after. - Keep persistence mode enabled. It does not prevent hangs, but it removes driver load/unload churn around jobs, which eliminates a class of reachability false alarms during incident triage. See NVIDIA persistence mode.
- Pin and record driver/CUDA versions per workload. When Xid 43 clusters on a specific firmware/CUDA/tensor-core combination, version records turn a multi-week investigation into a diff.
How Netdata helps
- Per-second per-GPU utilization, memory, and power make the wedge signature obvious: values stop moving while the job claims to run.
- Xid event collection from kernel logs, split by GPU and error code, so Xid 43 is seen alongside the Xid 13/31 or 48/63/64 that classify it.
- ECC corrected/uncorrected counters and row-remap state on the same dashboard as the hang event, so hardware involvement is confirmed or ruled out in one view.
- nvidia-smi collection latency per GPU surfaces the management-path degradation that precedes a wedge.
- Historical per-GPU views make recurrence patterns (the 43, 45, 79 escalation) visible across weeks instead of buried in rotated logs.
Related guides
- NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver
- How an NVIDIA GPU actually works in production: a mental model for operators
- NVIDIA GPU monitoring checklist: the signals every production GPU fleet needs
- NVIDIA GPU monitoring maturity model: from survival to expert
- nvidia-smi hangs or is unresponsive: a wedged GPU or a stuck driver
- NVIDIA persistence mode: why the GPU keeps re-initializing and P-state flaps
- Resetting a wedged NVIDIA GPU: nvidia-smi –gpu-reset and when only a reboot works