NVIDIA Xid errors: reading NVRM Xid messages in the kernel log

You found a line like this in the kernel log, or a user reported a dead training job and you went looking:

NVRM: Xid (PCI:0000:41:00): 79, pid=1432, name=python, GPU has fallen off the bus.

Xid messages are the NVIDIA driver’s primary error-reporting channel. Every significant GPU fault, from a correctable memory event to a catastrophic PCIe link failure, is printed by the NVRM kernel module as an Xid code in the kernel log. Not in nvidia-smi output. Not in a sysfs file. In the log stream, alongside everything else the kernel has to say.

This is the biggest GPU monitoring gap in most fleets. Teams watch utilization, temperature, and memory, but never parse the kernel log. The result: GPUs degrade for weeks through ECC events and page retirements, and the first anyone hears about it is a fallen-off-the-bus event that takes a node down mid-job.

This guide covers how to read Xid messages, which codes matter, and how to alert on them without paging on noise.

What an Xid message is

When the NVIDIA driver detects a GPU fault, the NVRM kernel module prints a structured message to the kernel ring buffer. The anatomy:

NVRM: Xid (PCI:0000:41:00): 79, pid=1432, name=python, GPU has fallen off the bus.
       |        |             |    |          |
       |        |             |    |          +- Process name (when applicable)
       |        |             |    +- Process ID
       |        |             +- Xid code
       |        +- PCI bus/device/function of the affected GPU
       +- Kernel module reporting (NVRM = NVIDIA Resource Manager)

Three things to extract every time:

• PCI address (here 0000:41:00). This is how you map the event to a physical GPU. nvidia-smi -L and nvidia-smi --query-gpu=pci.bus_id --format=csv give you the same identifier per GPU index.
• The code (here 79). The code determines severity and response. Same message text, different code, completely different incident.
• The process (pid/name), when present. This distinguishes “your workload did something illegal” from “the hardware failed under your workload.”

Not all Xids include a pid. Hardware-level events (fallen off bus, uncontained ECC) often stand alone.

Where Xid messages live

Xids go to the kernel log, which means:

# Kernel ring buffer, human-readable timestamps
dmesg -T | grep -i "NVRM: Xid"

# Same data via journald
journalctl -k | grep -i "NVRM: Xid"

# Persistent text logs on systems running rsyslog/syslog-ng
grep -i "NVRM: Xid" /var/log/kern.log /var/log/messages /var/log/syslog 2>/dev/null

Two operational facts that bite people:

The kernel ring buffer is circular. dmesg shows a fixed-size buffer. On a busy host, old Xids scroll away. If your only record of a page-retirement event was in the ring buffer, it is gone. Ship kernel logs to a persistent store: enable persistent journald, run a syslog daemon, or forward kern.* to your log aggregation stack.

Containers do not see host kernel logs. GPU workloads in containers lose Xid visibility entirely unless you forward host kernel logs. If your GPUs are only consumed by containers, host-level log collection is the only place Xids exist.

One more trap: DCGM exposes a “last Xid” field (DCGM_FI_DEV_XID_ERRORS), and dcgm-exporter surfaces it as a metric. It reports only the most recent Xid per GPU: a second event overwrites the first, and operators have reported the exporter metric can keep showing a stale code after the GPU recovers until the exporter restarts. Treat it as a tripwire, never as history. The kernel log is the source of truth.

Severity map

Xid codes are not a uniform severity ladder. The same log line format covers “your application has a pointer bug” and “this GPU needs an RMA.”

Hardware faults: page or urgent ticket

Application bugs: the GPU is fine

Informational: never page on these

Xid 109 (context switch timeout) deserves special mention: it is not fatal by itself, but it frequently precedes more severe errors by minutes to hours. Treat it as a leading indicator and correlate forward.

flowchart TD
  A[Xid found in kernel log] --> B{Which code?}
  B -->|48, 64, 79, 95| C[Hardware fault: page]
  B -->|74, 119, 61, 62| D[Hardware suspect: ticket or page by context]
  B -->|13, 31, 32, 43, 69| E[Application bug: route to job owner]
  B -->|45, 63, 65, 92| F[Informational: log and trend]
  C --> G[Correlate ECC, retired pages, remapping]
  D --> G
  E --> H[Match pid/name to scheduler records]
  F --> I[Watch rate of change over weeks]
  G --> J{New event or historical?}
  J -->|New| K[Act now]
  J -->|Old, already handled| L[No re-alert]

Quick checks

All read-only.

# All Xid events with timestamps, this boot
dmesg -T | grep -i "NVRM: Xid"

# Xid events in the last 24h via journald (works across ring-buffer rollover if journald is persistent)
journalctl -k --since "24 hours ago" | grep -i "NVRM: Xid"

# Context around an event: the lines before often name the GPU GUID or a precursor Xid
dmesg -T | grep -i -B3 -A3 "NVRM: Xid"

# Map the PCI address from the message to a GPU index
nvidia-smi --query-gpu=index,pci.bus_id,name --format=csv

# Reachability of the affected GPU (use per-GPU query; one hung GPU can stall a full query)
timeout 5 nvidia-smi -i 0 --query-gpu=gpu_name --format=csv,noheader

# ECC state on the affected GPU
nvidia-smi -i 0 -q -d ECC

# Retired pages (delta matters, not the raw count)
nvidia-smi -i 0 -q -d PAGE_RETIREMENT

# Row remapping, Ampere and later only
nvidia-smi -i 0 -q -d ROW_REMAPPER

How to diagnose an Xid event

1. Capture the full message and its neighborhood. Grep with context (-B3 -A3). The lines immediately before an Xid often carry a GPU GUID line or a precursor event (109, 45) that explains the sequence.
2. Map PCI address to GPU. Use pci.bus_id from nvidia-smi. On multi-GPU nodes never assume the event hit GPU 0.
3. Classify the code against the severity map above. This decides whether you are doing hardware triage or walking over to the ML team.
4. Decide: new event or history. Xids are events. If your log pipeline re-surfaces an old, already-handled Xid, do not re-act on it. Track what you have seen per GPU.
5. Correlate with hardware state. For 48/63/64/95: pull ECC counters, retired pages, row remapping. For 61/62/79: check PCIe link status (pcie.link.gen.gpucurrent, pcie.link.width.current under load) and PCIe AER errors in dmesg. For 74: nvidia-smi nvlink -s and nvidia-smi nvlink -e.
6. Correlate with the application. For 13/31/43/69, match the pid and name in the message against scheduler records and look for CUDA error: an illegal memory access was encountered or similar in job logs. That is a code bug, not an RMA.
7. Look for recurrence. One Xid 119 in six months is a data point. Three in a week on the same GPU is a failing component or a driver/firmware bug. Keep a per-GPU event history.
8. Escalate with evidence. Run nvidia-bug-report.sh as root before any reboot if you intend to open a vendor ticket. It collects kernel logs and driver state that a reboot destroys.

Codes everyone gets wrong

• Xid 43 is not page retirement. It means “GPU stopped processing,” an application-induced event. Page retirement is Xid 63.
• Xid 63 is not a thermal event. It is the page retirement / remapping notification: ECC self-healing working correctly. Informational.
• Xid 68 is not NVLink. It is an NVDEC0 (video decoder) exception. NVLink errors are Xid 74.
• Xid 49 is unused in current NVIDIA documentation. High single-bit ECC rate is Xid 92.
• Xid 45 is a consequence, not a cause. It reports cleanup after a prior error. Look backwards in the log for the real event.

Routing a page to the hardware team for Xid 13, or ignoring Xid 92 because “single-bit errors are corrected,” are the two failure modes these misreadings produce.

Signals to correlate

An Xid alone tells you what happened. Correlation tells you whether it will happen again.

Prevention

• Ship kernel logs off the host. Persistent journald at minimum; central aggregation for fleets. The ring buffer will erase your evidence.
• Alert per code, not per pattern match. A single “NVRM: Xid found” alert either pages on Xid 45 noise or snoozes on Xid 79. Route by the severity map.
• Alert on new events, not state. Track which Xids have been seen per GPU. Fire on the delta. A GPU with a handled Xid 64 from last month should not re-page every check cycle.
• Mind aggregation latency. Xids hit dmesg instantly but may take minutes to reach central logging. Do not let delayed delivery cause delayed or duplicate paging.
• Trend the informational codes. Xid 92 and Xid 63 are the early chapters of the memory-degradation story that ends in Xid 48. Plot their rate per GPU.
• Sample fast. Xid-adjacent signals (throttle events, PCIe errors, temperature spikes) develop in seconds. Minute-resolution polling misses them; use 10-second or faster sampling for GPU metrics.
• Forward host kernel logs into container platforms. If GPUs are consumed via Kubernetes or other container runtimes, node-level log forwarding is the only way Xids reach your alerting at all.

How Netdata helps

• Per-second GPU metrics (utilization, temperature, power, memory, clocks) give you the before-and-after picture around an Xid timestamp: thermal runaway before a 79, or clean operating conditions pointing at hardware.
• ECC error charts let you trend corrected error rate per GPU, so Xid 92 and Xid 63 events correlate with a visible acceleration curve instead of arriving as surprises.
• Systemd journal centralization means kernel logs, including NVRM Xid lines, survive ring-buffer rollover and host reboots and are searchable across the fleet.
• Throttle-reason and clock metrics provide the corroboration that separates a real hardware event from a transient: an Xid plus hw_thermal_slowdown is a different incident than an Xid alone.
• Anomaly detection on temperature, power, and ECC series flags the slow drift (rising baseline temps, growing SBE rate) that precedes the codes you actually page on.

Related guides

• NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver
• How an NVIDIA GPU actually works in production: a mental model for operators
• NVIDIA GPU monitoring checklist: the signals every production GPU fleet needs
• NVIDIA GPU monitoring maturity model: from survival to expert