You found NVRM: Xid (PCI:0000:xx:00.x): 31, ... in the kernel log, usually right after a training job or inference process died with CUDA error: an illegal memory access was encountered. The GPU raised a memory page fault: some unit on the chip accessed a virtual address that was not mapped to valid GPU memory.

The important thing to know up front: Xid 31 almost always means the application is wrong, not the GPU. Accessing freed GPU memory, bad pointer arithmetic, an out-of-bounds index in a kernel. These produce exactly this fault. The exception is when the same GPU produces Xid 31 across multiple unrelated applications. Then the suspicion flips to hardware.

This guide covers decoding the log line, the fast triage decision (does it follow the binary or the GPU?), and reproducing the fault under compute-sanitizer to prove it is a software bug.

What this means

The GPU has its own memory management unit. Every kernel and every CUDA allocation operates on GPU virtual addresses, which the MMU translates to physical framebuffer pages. When a kernel dereferences an address with no valid page table entry, the MMU raises a fault, the driver kills the faulting channel (the execution context for that process), and NVRM logs Xid 31 to the kernel ring buffer.

The dmesg line carries the diagnostic payload: the faulting PID and the channel number, plus the faulting address and access type. The PID is the single most useful field because it tells you which process to blame.

Consequences:

  • The faulting CUDA context is destroyed. The process usually dies on its next CUDA call with an illegal memory access error.
  • Other processes on the same GPU are normally unaffected. Xid 31 is contained to the faulting channel, unlike Xid 79 (GPU off the bus) which takes the whole device down.
  • The GPU itself keeps running. No reset is needed for a clean application-induced Xid 31.

Common causes

CauseWhat it looks likeFirst thing to check
Application bug (freed memory, bad pointer, out-of-bounds kernel index)Xid 31 always tied to the same binary or job; other workloads on the GPU are cleanFaulting PID in the dmesg line; reproduce under compute-sanitizer
Multi-allocator misuse (one framework frees memory another still references)Xid 31 in pipelines mixing frameworks or libraries with zero-copy tensor handoffsWhich process faults; whether it correlates with a recent code/dependency change
Driver bugXid 31 appearing across different applications after a driver upgrade, sometimes escalating into other XidsDriver version; whether the pattern started at the upgrade
Failing GPU hardwareXid 31 triggered by multiple unrelated applications on the same physical GPU, other GPUs in the node cleanECC counters, retired pages, and whether the fault follows the GPU across workloads

A genuine hardware fault is the minority outcome, but it is the one with fleet consequences, so the triage below is built to rule it in or out quickly.

Quick checks

All of these are read-only and safe on a production node.

# Find the Xid 31 event and read the full line (PID, channel, address)
dmesg -T | grep -i "NVRM: Xid"

# Same, via the journal if dmesg has wrapped
journalctl -k | grep -i "NVRM: Xid"

Read the line carefully. Note the PCI address (which GPU), the PID, and the process name if present. If the process is still alive, the PID maps directly. If it already exited, match the timestamp against your scheduler or job logs.

# Which processes are on the GPU right now
nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv,noheader

# Check ECC counters on the suspect GPU (hardware evidence)
nvidia-smi -i <N> --query-gpu=ecc.errors.corrected.volatile.total,ecc.errors.uncorrected.volatile.total --format=csv,noheader,nounits

# Check retired pages (accumulated memory hardware damage)
nvidia-smi -i <N> --query-retired-pages=retired_pages.single_bit_ecc.count,retired_pages.double_bit_ecc.count,retired_pages.pending --format=csv,noheader

# Driver version, for the driver-bug branch of triage
nvidia-smi --query-gpu=driver_version --format=csv,noheader

# Reproduce the fault under the memory checker (run the suspect job, not a benchmark)
compute-sanitizer --tool memcheck ./your_application <args>

compute-sanitizer replaced the older cuda-memcheck (deprecated in CUDA 11 and removed in CUDA 12). Running under the checker is slow, often 10x or more, so do it on a reproducer, not in the hot path.

How to diagnose it

The whole triage reduces to one question: does the fault follow one binary, or one GPU?

flowchart TD
  A[Xid 31 in dmesg] --> B[Identify faulting PID and GPU from the log line]
  B --> C{Same binary faults again, incl. on a different GPU?}
  C -- yes --> D[Application bug: reproduce under compute-sanitizer memcheck]
  C -- no --> E{Different applications fault on this same GPU?}
  E -- yes --> F[Suspect hardware: check ECC, retired pages, drain and RMA]
  E -- no --> G[One-off: capture reproducer, watch for recurrence]
  D --> H[Fix the code path memcheck flags]
  F --> I[Quarantine GPU, validate recent outputs]

  1. Decode the log line. Pull the PCI address, PID, and channel from the Xid 31 message. The PCI address identifies the physical GPU; map it to the nvidia-smi index with nvidia-smi --query-gpu=index,pci.bus_id --format=csv,noheader if needed.

  2. Attribute the fault to a workload. Match the PID and timestamp to a job: scheduler logs, container runtime, or shell history. If the same job has faulted repeatedly, you already have your prime suspect.

  3. Check whether it follows the binary. Has this exact application produced Xid 31 before, including on other GPUs or other nodes? If yes, this is a code bug with near certainty. GPU hardware does not conspire to fault only for one program.

  4. Check whether it follows the GPU. Look at the history of that physical GPU: have unrelated applications faulted on it? If multiple different binaries produce Xid 31 on one GPU while sibling GPUs run the same workloads cleanly, shift to the hardware branch.

  5. Prove the software case. Rerun the suspect workload under compute-sanitizer --tool memcheck. A memcheck report (invalid read/write, out-of-bounds access) at a specific kernel and line closes the case: it is an application bug, and you have the stack to fix. No memcheck finding but a reproducible Xid 31 is itself informative; it points at driver-level or race-condition territory rather than a simple bad pointer.

  6. Prove the hardware case. Check ECC error counts and retired pages on the suspect GPU (commands above). Rising corrected-error rates, any uncorrected errors, or retired pages corroborate hardware degradation. Also check for other Xids from the same GPU in the kernel log history. If the evidence accumulates, drain the GPU and treat it as an RMA candidate. Do not trust recent outputs from jobs that ran on it.

  7. Watch for escalation. An application Xid 31 that is followed by timeout-class or driver-internal Xids on the same GPU suggests the fault is destabilizing the driver, not just killing one context. If you see that cascade, check for a driver fix before assuming hardware; there are known driver releases where an MMU fault was the trigger for subsequent driver-level failures.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Xid events in dmesg (NVRM: Xid)The primary fault record; carries PID, channel, GPUNew Xid 31 events, especially recurring on one GPU or one binary
Application CUDA error logsWhere the fault becomes user-visibleillegal memory access errors correlating with Xid 31 timestamps
ECC corrected error rateLeading indicator of memory hardware degradationRate accelerating over days on one GPU
ECC uncorrected errors (volatile, delta)Data corruption has occurredAny new nonzero delta
Retired pages countPermanent memory hardware damageCount increasing, or pending retirement
GPU process listAttribution: who was on the GPU when it faultedThe faulting PID mapping back to a recurring job

Treat Xid monitoring as event-based: alert on new log lines, not on the historical presence of a code. A GPU that faulted once last month should not re-alert forever.

Fixes

If it is an application bug

  • Fix what memcheck flags. The sanitizer report gives you the kernel and access type. Common offenders: indexing past the end of a buffer in a custom kernel, dereferencing a tensor after its allocator freed it, and zero-copy handoffs between frameworks where one side releases memory the other still reads.
  • Audit allocator boundaries. If the pipeline mixes allocators (a deep learning framework plus a native library, an in-house CUDA extension, or video decode hardware feeding a compute pipeline), prefer copying across the boundary over sharing raw pointers. Zero-copy is fast until the lifetimes diverge.
  • Roll back the change that introduced it. Xid 31 that appears right after a code or dependency upgrade is a strong regression signal. Reverting restores service while the fix lands.

If it is the driver

  • Upgrade within your driver branch. If Xid 31 started after a driver update, or is followed by other timeout or crash-class Xids, check the release notes for your branch and move to the fixed version. Pin the working version in your node images.

If it is the GPU

  • Drain and quarantine. Cordon the node or remove the GPU from the scheduler pool. A GPU that faults across unrelated workloads will keep doing it.
  • Correlate with ECC and retirement evidence before RMA. Vendors will ask for exactly this data.
  • Invalidate recent outputs from jobs that ran on the suspect GPU if your workload is correctness-sensitive.

Do not reach for nvidia-smi --gpu-reset as a first response to Xid 31. The fault is contained to the dead context; the GPU does not need a reset for a clean application bug. Reset and reboot procedures belong to the wedged-GPU cases. See resetting a wedged NVIDIA GPU.

Prevention

  • Run compute-sanitizer in CI for custom kernels. Any in-house CUDA code or extension should pass memcheck before it ships. The 10x slowdown is acceptable in a test pipeline and intolerable in an incident.
  • Log and classify every Xid. Route Xid 31 as a ticket tied to the faulting workload, not a page. Page only on the fatal class (48, 79, 95). See reading NVRM Xid messages in the kernel log.
  • Keep per-GPU Xid history. The binary-vs-GPU decision in steps 3 and 4 is only fast if you can answer “has this GPU faulted before, for whom?” in one query.
  • Track ECC and retired pages as trends. Hardware-caused Xid 31 rarely arrives without ECC warning signs first. Accelerating corrected-error rates are your early signal.
  • Pin driver versions and review release notes on upgrade. Driver regressions that turn an application fault into a driver crash are a known pattern; a pinned, tested driver image limits the blast radius.

How Netdata helps

  • Per-second GPU metrics with event context. Netdata’s NVIDIA collector tracks utilization, memory, ECC counters, and throttle state per GPU, so you can line up the exact moment of a Xid 31 with what the GPU was doing.
  • Xid correlation. Because Netdata already monitors the node, the kernel log event, the per-process GPU usage, and the ECC trend live on one timeline. That makes the “follows the binary or follows the GPU” question answerable from history instead of guesswork.
  • ECC rate trending. Netdata charts corrected and uncorrected ECC errors per GPU over time, surfacing the acceleration pattern that precedes hardware-caused faults.
  • Per-process attribution. Netdata’s process-level GPU visibility helps map the faulting PID to a workload and show its memory footprint leading up to the fault.
  • Fleet-wide comparison. If one GPU in a node is degrading, its ECC, temperature, and error behavior diverges from its siblings. Netdata makes that divergence visible without per-node SSH sessions.