$ guides / nvidia-gpu / nvidia-gpu-xid-13-graphics-engine-exception ▌

Operations Guides

NVIDIA Xid 13: Graphics Engine Exception

You found a line like this in the kernel log:

NVRM: Xid (PCI:0000:65:00.0): 13, pid='<unknown>', name=<unknown>, Graphics Engine Exception.

Or, more likely, you found the application-side symptom first: a training job died with a generic CUDA error, a run produced NaN loss, or an inference service started returning garbage, and only after digging into dmesg did Xid 13 show up.

Xid 13 is the most ambiguous Xid code NVIDIA emits. It means the graphics engine raised an exception while executing work. The three candidate causes are a bad CUDA kernel (out-of-bounds access, illegal instruction), a driver bug, or genuine hardware degradation. Most isolated occurrences are software. The operational problem is telling which case you are in, because the correct response ranges from “file a bug against the application” to “RMA the GPU”.

This guide is about making that call quickly and defensibly.

What this means

The graphics engine is the unit that dispatches work to the SMs. When it hits a condition it cannot execute (an instruction it does not recognize, a descriptor that fails sanity checks, a memory access the fault handler escalates), it raises an exception, the faulting CUDA context is typically destroyed, and NVRM logs Xid 13.

Two properties matter for operations:

The GPU survives. Unlike Xid 48 (double-bit ECC) or Xid 79 (fallen off the bus), Xid 13 does not, by itself, indicate that the GPU is damaged. The application that owned the faulting context dies or errors out; other contexts and the device usually keep running.
The log line alone cannot classify the cause. NVIDIA’s own Xid documentation treats Xid 13 as “typically an application bug, rarely hardware”. The classification work happens in the correlation, not in the message.

That second point is why Xid 13 deserves a real triage procedure rather than a reflexive “ignore it, it’s an app bug”.

Common causes

Cause	What it looks like	First thing to check
Bad CUDA kernel (out-of-bounds access, illegal instruction)	Xid 13 fires when one specific application or job runs; the same job reproduces it	Run the app under Compute Sanitizer memcheck or cuda-gdb
Driver or firmware bug	Xid 13 appears after a driver upgrade, on a specific driver branch, or under a known-problematic driver variant (open vs proprietary kernel module)	Check whether the driver version changed recently and whether the issue tracks that version across nodes
Genuine hardware degradation	Same GPU throws Xid 13 across different applications; frequency increases over weeks; ECC counts or retired pages are also moving	ECC error counts, retired pages, row remapping status on that GPU
Environmental stress masquerading as a fault	Xid 13 correlates with thermal throttling or power events on the same GPU	Temperature and clock throttle reasons around the event timestamp

The single most useful discriminator: isolated occurrences are usually software; recurring occurrences, especially the same GPU across different applications, point to hardware.

Quick checks

All of these are read-only and safe to run during an incident.

# 1. Find all Xid events, with timestamps, for the affected GPU
dmesg -T | grep -i "NVRM: Xid"

# 2. Same via journalctl if dmesg has wrapped
journalctl -k | grep -i "NVRM: Xid"

# 3. ECC error counters on the GPU that logged Xid 13 (volatile = since driver load)
nvidia-smi -i <N> --query-gpu=ecc.errors.corrected.volatile.total,ecc.errors.uncorrected.volatile.total --format=csv,noheader,nounits

# 4. Retired pages (persistent hardware degradation record)
nvidia-smi -i <N> -q -d PAGE_RETIREMENT

# 5. Row remapping status (Ampere and later)
nvidia-smi -i <N> -q -d ROW_REMAPPER

# 6. Thermal and throttle state
nvidia-smi -i <N> --query-gpu=temperature.gpu,clocks_event_reasons.active --format=csv,noheader

# 7. What was running on the GPU when it faulted
nvidia-smi -i <N> --query-compute-apps=pid,process_name,used_gpu_memory --format=csv,noheader

# 8. Driver version, to correlate with recent changes
nvidia-smi --query-gpu=driver_version --format=csv,noheader

Two things to note while collecting:

Count Xid 13 occurrences per GPU per time window. The playbook heuristic is that more than roughly one per hour, or recurrence across different applications on the same GPU, moves you into hardware-suspect territory. This is an operational heuristic, not an NVIDIA-published threshold; NVIDIA’s Xid catalog classifies Xid 13 as “typically an application bug” and does not prescribe a rate threshold.
Check whether the faulting application is always the same binary. If three unrelated frameworks all trigger Xid 13 on GPU 4 and none of them trigger it on GPUs 0-3, the application-bug hypothesis is effectively dead.

How to diagnose it

The decision you are making is: software, driver, or hardware. Work it in this order because the cheap, high-signal checks come first.

Establish the recurrence pattern. Grep the kernel log history for every Xid 13 on this node. Group by GPU PCI address and by time. One event six months ago is noise. A rising cadence on one GPU is a degradation signature.
Check whether the fault follows the application or the GPU. If your scheduler can place the same job on a different GPU, do it. If the same job triggers Xid 13 on any GPU it lands on, you are looking at an application bug. If any job triggers Xid 13 only on this specific GPU, suspect the GPU.
Correlate with hardware health signals on the faulting GPU. Pull ECC counters, retired pages, row remapping status, temperature, and throttle reasons around the event timestamps. Xid 13 plus accelerating single-bit ECC errors, new retired pages, or pending row remaps on the same GPU is a hardware story. Xid 13 with clean ECC history and no thermal events is a software story.
Rule out the driver. Did the driver version change shortly before the first occurrence? Are other nodes on the same driver branch and GPU model reporting Xid 13? There are known cases where a specific driver variant (for example, the open kernel module on certain consumer/workstation cards) produces Xid 13/31 under specific workloads while the proprietary branch does not. A driver rollback or upgrade that makes the errors stop is diagnostic in itself.
If it reproduces with one application, debug the application. Run the workload under Compute Sanitizer (compute-sanitizer --tool memcheck) or attach cuda-gdb. The older cuda-memcheck was deprecated in CUDA Toolkit 11.5 and removed in later toolkit releases; Compute Sanitizer (included since CUDA 11.0) is the current tool. Out-of-bounds accesses and illegal instructions show up directly, and this closes the case fastest for the software branch. NVIDIA’s Xid catalog for Xid 13 specifically recommends: “Run the application in cuda-gdb or the Compute Sanitizer memcheck tool, or run the application with CUDA_DEVICE_WAITS_ON_EXCEPTION=1 and then attach later with cuda-gdb.”
If hardware is suspected, run diagnostics. dcgmi diag -r 3 (or -r 4 where supported) exercises the GPU under load and surfaces faults that intermittent Xid 13s hint at. Run it on a drained node; the heavy diagnostic is disruptive to anything sharing the GPU. A clean diagnostic result plus recurring Xid 13 in production is still evidence. Intermittent faults often pass short diagnostics.

flowchart TD
  A[Xid 13 in kernel log] --> B{Recurring on this GPU?}
  B -->|No, isolated| C[Likely application bug - track and move on]
  B -->|Yes| D{Same application each time?}
  D -->|Yes| E[Run app under Compute Sanitizer or cuda-gdb]
  D -->|No, different apps| F{ECC, retired pages, remaps, thermal anomalies on this GPU?}
  F -->|Yes| G[Hardware degradation - drain, diagnose, plan RMA]
  F -->|No| H{Driver changed recently or known bad branch?}
  H -->|Yes| I[Driver rollback or upgrade, then observe]
  H -->|No| J[dcgmi diag -r 3 on drained node, monitor recurrence]

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Xid 13 event frequency per GPU	The primary recurrence signal; cadence separates noise from degradation	More than ~1/hour, or an accelerating trend
Xid 13 across applications	Same GPU faulting under different workloads kills the app-bug hypothesis	Multiple unrelated jobs triggering it on one GPU
ECC corrected error rate (volatile)	Accelerating single-bit errors indicate degrading memory that can also surface as engine exceptions	Rate doubling over days, or above peer GPUs of the same model
ECC uncorrected errors	Data corruption has occurred; recent outputs may be invalid	Any new event (delta in volatile counter)
Retired pages / row remapping	Persistent record of memory hardware degradation	Any increase, pending retirements, or a row remapping failure flag
GPU die temperature and throttle reasons	Thermal stress can contribute to intermittent faults	Throttle reasons active around Xid 13 timestamps
Application CUDA errors in job logs	The user-visible manifestation of Xid 13	Generic `CUDA error`, `unspecified launch failure`, NaN loss correlated with Xid timestamps

The correlation is the classification. None of these signals alone answers “software or hardware”; together they usually do.

Fixes

Application bug confirmed (Compute Sanitizer or cuda-gdb finds the fault)

Fix the kernel. Common root causes are out-of-bounds memory access, illegal instruction from a miscompiled or wrong-arch binary, and misuse of CUDA APIs. This is the most common outcome and the cheapest. After the fix, watch the node for a few days to confirm Xid 13 does not recur.

Driver-suspect

If the errors track a driver version or variant:

Upgrade or roll back the driver on an affected node and observe. This is disruptive to running workloads, so drain first.
If you are on the open kernel module and hitting reproducible Xid 13/31 faults that the proprietary branch does not produce, switching branches is a legitimate diagnostic step, not just a workaround.
File a bug with NVIDIA with the full Xid line, driver version, GPU model, and a reproducer if you have one. Inconclusive software debugging is a stated reason to escalate in NVIDIA’s Xid guidance.

Hardware-suspect

Drain the GPU from production scheduling. Recurring Xid 13 under varied workloads means you cannot trust the outputs, and silent corruption is a worse failure mode than a crash.
Run dcgmi diag -r 3 on the drained node to gather evidence for the RMA conversation.
Validate or discard recent work products from that GPU. If the fault window overlaps a training run, treat checkpoints from that window as suspect.
Replace the GPU if diagnostics fail, ECC counters are moving, or Xid 13 keeps recurring after driver changes are ruled out. A GPU that throws recurring Xid 13s is on a trajectory; it rarely gets better.

Prevention

Log Xid events centrally with per-GPU, per-code counts. Xid 13 is only classifiable over time and across applications. Node-local dmesg wraps and loses the history you need.
Track ECC counters and retired pages as trends, not snapshots. The acceleration of corrected errors is the leading indicator; the absolute count is nearly useless.
Keep driver rollouts staged. A driver branch that introduces Xid 13 should be caught on a canary node, not across the fleet.
Run Compute Sanitizer on new or changed kernels in CI before production deployment. Most Xid 13s are application bugs, and catching them pre-production keeps them out of your kernel log entirely.
Baseline per-GPU health so that “this GPU behaves differently from its peers” is a measurement, not a hunch.

How Netdata helps

Xid events in kernel logs are collected and classified per code, so a recurring Xid 13 on one GPU shows up as a trend rather than a line you grep for after the incident.
ECC corrected and uncorrected error counts, retired pages, and row remapping status are charted per GPU, which makes the “is this GPU degrading” half of the classification answerable in seconds.
Temperature, power draw, and clock throttle reasons on the same dashboard let you check whether thermal or power events line up with Xid 13 timestamps.
Per-second GPU metrics matter here because the useful correlation window around a fault is short; minute-resolution data smears the event.
Per-GPU comparison across the node and fleet supports the strongest discriminator: does this GPU behave differently from identical peers under the same workloads.