You see NVRM: Xid (PCI:0000:XX:00.0): 43, ... GPU stopped processing in dmesg, and the job on that GPU has gone silent. CUDA calls do not return. nvidia-smi either hangs outright or returns a stale view of the card: utilization frozen at the last sampled value, memory allocation unchanged, power draw flat. The GPU has not disappeared from the bus the way it does with Xid 79. It is still enumerated and still answering the driver at some level, but the compute engine is wedged and nothing it was running will complete.

Xid 43 is classified as an application-induced error: the usual root cause is the workload itself, such as a kernel that deadlocked or an access pattern that poisoned engine state. The hardware is often fine. Operationally, though, it behaves like a hardware incident: recovery frequently requires a reboot, and a hang can also be the downstream symptom of genuine degradation such as ECC faults. Treating every Xid 43 as “just restart the app” is how teams lose GPUs to slow-burn memory failures they never investigated.

This article covers how to confirm the wedge, separate application-induced hangs from hardware-induced ones, attempt recovery in the right order, and decide when the card needs to leave the fleet. For the broader picture of how the compute engine, memory subsystem, and error correction interact, see how an NVIDIA GPU actually works in production.

What this means

The GPU’s watchdog logic determined that the compute engine stopped making forward progress. The channel feeding work to the engine stalled and the driver gave up on it. From that point:

  • CUDA API calls into that GPU’s context block or fail. The application process is typically alive but stuck, often in uninterruptible sleep (D state) once it touches the driver again.
  • nvidia-smi output for the affected GPU goes stale: utilization, memory, and power stop updating. On some systems the query itself hangs, because one wedged GPU can stall the whole NVML management path. Query per-GPU with -i N to isolate.
  • The GPU is still on PCIe. lspci still shows it. This is the key distinction from Xid 79 (“GPU has fallen off the bus”), where the link itself is dead and no driver-level recovery applies.

Two things are commonly misread. First, Xid 43 is not page retirement. Page retirement and remapping events are Xid 63/64; confusing 43 with 63 sends operators down a hardware-RMA path for what may be a pure software bug. Second, “application-induced” does not mean “harmless”. The application induced the hang, but the GPU does not necessarily return to service afterward. The engine state is corrupt enough that nvidia-smi --gpu-reset frequently fails, and a full reboot is the usual recovery.

flowchart TD
  A[Xid 43 in dmesg] --> B{nvidia-smi -i N responds?}
  B -->|hangs| C[GPU wedged, skip to reboot path]
  B -->|responds, stale values| D{Other Xids present?}
  D -->|Xid 13/31 nearby| E[Application bug: fix code, drain, reset or reboot]
  D -->|ECC errors, Xid 48/63/64| F[Suspect hardware: check remapped rows, quarantine]
  D -->|nothing else| G[Try nvidia-smi --gpu-reset]
  G -->|reset succeeds| H[Replay workload, watch for recurrence]
  G -->|reset fails| I[Drain node, reboot]
  H -->|Xid 43 recurs| F

Common causes

CauseWhat it looks likeFirst thing to check
Application bug (illegal access, deadlocked kernel)Xid 43 with Xid 13 or 31 nearby in dmesg; same job or kernel reproducibly triggers itdmesg history for preceding Xids; which PID/kernel was running
Underlying memory degradationXid 43 plus rising ECC counters, Xid 48/63/64 in the same window, row remap pendingnvidia-smi -q -d ECC and -d ROW_REMAPPER
Firmware/CUDA/tensor-core interactionHangs cluster on a specific driver/CUDA version or a specific kernel pattern; fleet-wide on same configCorrelate occurrence with driver version and workload type across nodes
Driver-level stress on the management pathnvidia-smi latency creeping up (>2s) in the hours before the hangManagement path latency history, per-GPU query timing
Progressive hardware failureXid 43 events recur on the same GPU with shrinking intervals, later joined by Xid 45 or 79Per-GPU Xid event history over weeks

The last row matters most for fleet hygiene. A single Xid 43 on a healthy card after a buggy kernel is a code problem. A cadence of Xid 43s on the same card, escalating toward Xid 79, is a dying GPU.

Quick checks

All read-only and safe to run during an incident, with one caveat: query per-GPU (-i N) so a hung card does not stall your session.

# 1. Confirm the Xid and see what came before it
dmesg -T | grep -i "NVRM: Xid" | tail -50

# 2. Check reachability of the specific GPU, with a timeout
timeout 5 nvidia-smi --query-gpu=name,uuid,pstate --format=csv,noheader -i 0

# 3. Take two utilization readings 5s apart - identical stale values indicate a wedge
nvidia-smi --query-gpu=utilization.gpu,utilization.memory,power.draw --format=csv,noheader -i 0
sleep 5
nvidia-smi --query-gpu=utilization.gpu,utilization.memory,power.draw --format=csv,noheader -i 0

# 4. See what processes hold the GPU (stuck CUDA contexts)
nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv,noheader -i 0

# 5. Check process state on the host - D state = uninterruptible, unkillable until reboot
ps -eo pid,stat,comm | awk '$2 ~ /^D/'

# 6. Verify the GPU is still on PCIe (distinguishes Xid 43 from Xid 79)
lspci -d 10de:

# 7. Look for hardware-degradation fingerprints
nvidia-smi -q -d ECC -i 0 | grep -iE "volatile|aggregate" | head -20
nvidia-smi --query-remapped-rows=remapped_rows.correctable,remapped_rows.uncorrectable,remapped_rows.pending,remapped_rows.failure --format=csv,noheader -i 0
nvidia-smi -q -d PAGE_RETIREMENT -i 0 | head -20

If step 2 hangs or times out, the management path is wedged on that GPU and most software-level recovery is off the table. Note that nvidia-smi -q output under a wedge may show the last cached values; values that refuse to change across samples are themselves the signal. The --query-remapped-rows flag requires Ampere or newer; on older cards it will report N/A.

How to diagnose it

  1. Establish the timeline from dmesg. Pull every NVRM/Xid line for the affected PCI address. You are looking for what preceded Xid 43: Xid 13 (graphics engine exception) or Xid 31 (memory page fault) point to an application bug as the trigger. Xid 48, 63, or 64 nearby point to memory hardware. Nothing nearby points to either a clean application deadlock or a driver/firmware interaction.

  2. Identify the workload. Match the GPU’s compute-apps list and the stuck PIDs against scheduler records. If the same job, kernel, or framework version has produced Xid 43 on other nodes, you have a software pattern, not a hardware one.

  3. Check the hardware fingerprints. On Ampere and newer, check remapped_rows.pending and remapped_rows.failure. On pre-Ampere, check retired pages and whether retired_pages.pending = Yes. Rising volatile ECC counters in the same boot as the hang are a strong hint the “application error” classification is masking a memory fault that poisoned engine state.

  4. Check the management path history. If you collect nvidia-smi/NVML query latency, look at the hours before the hang. Sustained latency above 2s per GPU is often the last observable warning before a wedge.

  5. Decide the recovery class. Application bug with clean ECC and no remap activity: reset or reboot, fix the code, move on. Any ECC/remap involvement, or recurrence on the same GPU: treat as suspected hardware and quarantine after recovery.

  6. After recovery, re-verify. Post-reboot, re-check ECC counters, retired pages, and remapped rows. Confirm the GPU enumerates, passes a real workload, and does not immediately re-wedge. A card that hangs again on the next kernel launch after reset is telling you the cause is persistent, not transient.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Xid events in dmesg (per-GPU, per-code)Xid 43 is the detection event; neighbors classify itNew Xid 43; any Xid 13/31/48/63/64 in the same window
nvidia-smi / NVML query latency per GPULast early warning before a wedgeSustained >2s per GPU
Utilization stalenessFrozen utilization/memory/power values mean the engine stoppedIdentical values across samples while a job claims to be running
ECC volatile counters (corrected and uncorrected)Distinguishes hardware-induced hangsDelta increase in uncorrected count, or accelerating corrected rate
Remapped rows / retired pagesPending or failing remaps after a hang indicate the real causeremapped_rows.pending > 0, any new failure transition, rising retired pages
GPU process listStuck contexts hold memory and block resetPIDs in D state that survive SIGKILL
Per-GPU Xid recurrence historyA GPU that hangs once and recovers may hang again>1 Xid 43 on the same GPU in a rolling window, or 43 progressing to 45/79

On alert routing: Xid 43 by itself is an INFO/TICKET-class event in most taxonomies, because it is usually application-induced. It earns a PAGE when it arrives with a new uncorrected ECC event, with a remapped_rows.failure transition, or as part of a recurrence pattern on a GPU running production work.

Fixes

Application-induced hang (the common case)

Drain or kill the affected workload first. Stuck CUDA contexts hold memory and will block a GPU reset, and killing a D-state process may not be possible until reboot. Then attempt a GPU reset:

# Disruptive: kills all remaining work on this GPU. Drain other GPUs first if sharing the node.
nvidia-smi --gpu-reset -i 0

Expect this to fail frequently for Xid 43. The engine state after a wedge is often too corrupt for a driver-level reset. If it fails, the reliable path is a full node reboot: drain the other GPUs, migrate or checkpoint workloads, reboot, then re-verify the card before returning it to service.

After recovery, the real fix is in the workload: the kernel that deadlocked or the access pattern that wedged the engine will do it again. Capture the reproducing job, driver version, and CUDA version for the application owners.

Suspected hardware-induced hang

If ECC counters moved, remaps are pending, or the hang recurs on the same GPU, do not just reboot and forget. The reboot applies pending row remaps and page retirements. Then quarantine: keep the GPU out of production scheduling until it survives a soak under a known-good workload. Escalating Xid patterns (43, then 45, then 79) are an RMA conversation, not a monitoring conversation. See resetting a wedged NVIDIA GPU for the full reset-versus-reboot decision path.

When nvidia-smi itself hangs

A wedged GPU can take the whole management path down: nvidia-smi blocks for every GPU on the node, and the querying process enters D state where SIGKILL does nothing. There is no userspace fix for that state. Drain what you can and reboot. For the diagnostic side of that symptom, see nvidia-smi hangs or is unresponsive.

Prevention

  • Monitor management path latency. Per-GPU nvidia-smi response time trending past 2s is your earliest wedge warning. It costs almost nothing to collect.
  • Track Xid history per GPU, per code. Xid 43 only becomes actionable in context: what came before it, and how often this GPU has done it. Use delta-based log monitoring, not raw greps after the fact.
  • Track ECC and remap state continuously. Accelerating corrected-error rates and remapped_rows.pending let you retire a degrading GPU before it wedges under production load instead of after.
  • Keep persistence mode enabled. It does not prevent hangs, but it removes driver load/unload churn around jobs, which eliminates a class of reachability false alarms during incident triage. See NVIDIA persistence mode.
  • Pin and record driver/CUDA versions per workload. When Xid 43 clusters on a specific firmware/CUDA/tensor-core combination, version records turn a multi-week investigation into a diff.

How Netdata helps

  • Per-second per-GPU utilization, memory, and power make the wedge signature obvious: values stop moving while the job claims to run.
  • Xid event collection from kernel logs, split by GPU and error code, so Xid 43 is seen alongside the Xid 13/31 or 48/63/64 that classify it.
  • ECC corrected/uncorrected counters and row-remap state on the same dashboard as the hang event, so hardware involvement is confirmed or ruled out in one view.
  • nvidia-smi collection latency per GPU surfaces the management-path degradation that precedes a wedge.
  • Historical per-GPU views make recurrence patterns (the 43, 45, 79 escalation) visible across weeks instead of buried in rotated logs.