NVIDIA Xid 79: GPU has fallen off the bus

You will usually find this one by accident: a training job crashes with an opaque CUDA error, nvidia-smi hangs or shows ERR! for one device, and when you check dmesg you find NVRM: Xid (PCI:0000:XX:00): 79, GPU has fallen off the bus. The GPU is gone. Not throttled, not wedged, not holding stale contexts. The PCIe link between the host and the GPU has failed, and the device is no longer reachable at all.

Xid 79 is one of the few NVIDIA error codes with an unambiguous operational meaning: this is a hardware-level event, it is fatal to everything running on that GPU, and there is no software fix. The recovery action in NVIDIA’s own Xid catalog is restart the bare metal and contact support. Everything else on this page is about confirming the diagnosis quickly, recovering the node safely, and deciding whether the card comes back into service or goes to RMA.

Do not confuse this with a hung GPU that still responds to NVML, or with a driver that has lost communication while the device stays on the bus. Those are different failure modes with different recoveries. Xid 79 is the one where lspci itself may no longer show the device.

What this means

A modern NVIDIA GPU is a semi-autonomous computer attached to the host over PCIe. Xid 79 means the link half of that attachment collapsed: the GPU became inaccessible because the PCI Express link went down, because the GPU hardware itself failed, or because the board lost stable power and dropped off the bus. From the host’s perspective, the device simply stops existing.

What you will see:

nvidia-smi hangs, times out, or shows ERR! / N-A for the affected GPU.
The device may no longer appear in lspci output at all.
Every process that held a CUDA context on that GPU is dead or stuck in D (uninterruptible sleep) state.
Temperature and other queries return the last cached reading or N/A.
Other GPUs on the same host may keep working normally, unless the failure is in a shared component (a PCIe switch, riser, or backplane) rather than the card itself.

One important scoping note: a GPU that has fallen off the bus is unrecoverable without at least a node reboot. nvidia-smi --gpu-reset targets a GPU the driver can still talk to. There is nothing to reset here. For the cases where a reset can work, see resetting a wedged NVIDIA GPU.

Common causes

Cause	What it looks like	First thing to check
GPU hardware failure (die, board, or PCIe interface on the card)	Xid 79 with no thermal or power precursor; GPU does not return after reboot	Does the GPU reappear in `lspci` after reboot?
VRM or board power delivery failure	Sudden drop under load; possibly burning smell, PSU or chassis events	Chassis/PSU logs, IPMI or BMC sensor events around the timestamp
Unstable or insufficient power under peak load	Xid 79 correlates with load spikes or power excursions; may affect GPUs intermittently	Power draw history vs. `enforced.power.limit`; PDU and PSU capacity
Severe thermal event	Temperature was trending up or throttling was active before the drop	Temperature history, `hw_thermal_slowdown` events, fan/cooling status
PCIe slot or riser failure	One slot consistently kills cards; swapping the card to another slot “fixes” it	Move the suspect card to a known-good slot and observe
Shared PCIe switch/backplane fault	Multiple GPUs on the same switch drop together	Whether the affected devices share a PCIe parent in `lspci -t`

Quick checks

All read-only. Run these before touching anything else.

# 1. Confirm the Xid and get the PCI address
dmesg -T | grep -i "NVRM: Xid"

# 2. See what the driver currently thinks is attached
nvidia-smi -L

# 3. Per-GPU query, because one fallen GPU can hang a full scan
timeout 5 nvidia-smi --query-gpu=gpu_name --format=csv,noheader -i 0

# 4. Is the device still on the PCIe bus at all?
lspci -d 10de:

# 5. Check for PCIe bus errors around the event
dmesg -T | grep -i "pcie\|aer" | tail -50

# 6. Check for processes stuck on the dead GPU
ps aux | awk '$8 ~ /D/ {print}'

# 7. Follow-up driver messages after the Xid
journalctl -k --since "1 hour ago" | grep -i "NVRM"

What you are looking for:

The Xid line itself carries the PCI address of the fallen device. Map it to your GPU inventory (slot position, serial) now, before the reboot makes the mapping harder to reconstruct.
If lspci no longer lists the device, the card has dropped off the bus entirely, not just stopped answering the driver. That is the strongest form of this failure.
If nvidia-smi -L hangs rather than returning, kill it and use per-GPU queries with a timeout. A single fallen GPU can wedge the full-device scan.
Look for a follow-on Xid after 79 in the kernel log.

How to diagnose it

Confirm the event. dmesg -T | grep -i "NVRM: Xid" must show Xid 79 for the affected PCI address. If you see 43, 13, or 31 instead, you are dealing with application faults, not this page.
Establish blast radius. Check lspci -d 10de: and nvidia-smi -L. Is it one GPU, or several? Multiple devices dropping at once points at a shared component (PCIe switch, riser, backplane, PSU rail), not coincidental independent card failures.
Capture the evidence before reboot. Kernel logs, any AER/PCIe error lines, BMC/IPMI power and thermal events around the timestamp, and the state of stuck processes. After the reboot this context gets harder to recover.
Check for precursors. Pull recent history: was the GPU thermally throttling? Was power draw hitting the enforced limit repeatedly? Were there PCIe replay errors accumulating (nvidia-smi -q -d PCIE on surviving boots)? Was the Xid under peak load or at idle? Under-load drops point at power delivery; idle drops point at link stability or firmware.
Drain and reboot the node. There is no softer recovery. Migrate or accept the loss of remaining workloads, then reboot. If the card is still missing after a warm reboot, power the node off completely before concluding the hardware is dead.
Assess after reboot. Does the GPU reappear in lspci and nvidia-smi -L? If yes, proceed to diagnostics. If no, you are in hardware-swap territory.
Decide: service or RMA. A GPU that has fallen off the bus once will likely do it again. Do not put it straight back into production. Run comprehensive diagnostics first, and treat any recurrence as grounds for replacement.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Xid events in kernel log	Xid 79 is the definitive signal; it is an event, not a state	Any new Xid 79 entry; also PCIe-related Xids accumulating beforehand
GPU reachability (`nvidia-smi -L` exit code)	Precondition for every other signal; binary was-present/now-gone	Sustained unreachability >60s on an inventoried GPU
Management path latency	Slowness (>2s per-GPU query) is often the last warning before a wedge or drop	Latency climbing from <100ms baseline toward timeouts
PCIe link gen/width	Link degradation can precede catastrophic link failure	Current below max under active load; worsening over time
PCIe replay errors	Retransmissions indicate link instability that can precede link loss	Steadily increasing counter (track rate, not absolute)
Power draw vs. enforced limit	Unstable power under peak load is a known Xid 79 cause	Repeated excursions at the limit correlating with drops
Temperature and throttle reasons	Severe thermal events can precede the card dropping	`hw_thermal_slowdown` active before an Xid 79 on the same GPU

Alert routing: Xid 79 is PAGE severity, gated like GPU reachability (sustained, uptime-gated, only for GPUs expected in inventory). Because Xids are log events, alert on new log entries, not on historical presence, or you will re-page forever on an already-handled event.

Fixes

There is exactly one fix path with branches at the end. Everything else is triage.

Recover the node. Drain workloads, reboot. Do not attempt driver reloads or GPU resets as substitutes; they assume a device the driver can reach.

If the GPU does not return after reboot. Try a full power cycle (power off, not just a warm reboot) if you have not already. Then treat it as failed hardware: reseat the card if the chassis allows it, try a known-good slot, and check power connectors and risers. If it still does not enumerate, RMA it.

If the GPU returns after reboot. Run full diagnostics before returning it to service (vendor diagnostic tooling or DCGM’s heavier diagnostic levels, per your environment). Inspect the physical layer anyway: reseat the card, inspect the slot and riser, verify power connectors are fully seated, and confirm cooling is functional. A card that fell off once is a recurrence risk; consider running it in a less critical role or flagging it for replacement at the next maintenance window.

If the cause was power. Verify PSU capacity against peak draw across all GPUs plus host load, check that per-GPU enforced.power.limit values are what you expect, and check PDU headroom. Power-capping the GPUs below their default limit is a legitimate mitigation if the supply is marginal.

If the cause was thermal. Fix the cooling problem first: fans, airflow, chassis cooling. Returning the card to the same thermal environment reproduces the event.

If multiple GPUs dropped. Suspect the shared component: PCIe switch, backplane, riser assembly, or a common power rail. Swapping individual cards will not fix that.

Prevention

Monitor the precursors. PCIe link degradation under load, rising replay error rates, and management path latency creeping up are all ticket-worthy warnings that can precede a bus drop. Catch them at the straggler stage, not the Xid 79 stage.
Alert on Xid events properly. Per-code routing, edge-triggered on new log entries, with Xid 79 paged. See the monitoring checklist for the full signal set.
Power engineering. Size PSUs and PDUs for peak simultaneous draw with headroom; unstable power under peak load is a documented cause of this failure.
Thermal hygiene. Track temperature baselines over months; a shrinking gap to the throttle threshold is a maintenance trigger, not background noise.
Post-incident discipline. Any GPU that has produced Xid 79 gets flagged, diagnosed, and tracked. Recurrence means replacement.
Persistence mode and clean baselines will not prevent Xid 79, but they remove noise from your reachability signal so a real drop is not masked. See NVIDIA persistence mode.

How Netdata helps

GPU reachability as a first-class signal: Netdata tracks whether each GPU responds, so a fallen GPU shows up as a clean was-here/now-gone event rather than a gap you discover when a job fails.
Kernel log correlation: Xid events from dmesg sit alongside per-GPU metrics, so you can see the Xid 79 entry next to the last temperature, power, and PCIe readings the card reported before it dropped.
Precursor visibility: per-second sampling on power draw, temperature, throttle reasons, and PCIe link state catches the load spikes, thermal run-ups, and link degradations that precede bus drops. Minute-resolution monitoring misses these.
Multi-GPU blast radius: per-device charts make it immediately obvious whether one card fell off or several dropped together, which is the key distinction between a card fault and a shared-component fault.
Event-vs-state alerting: alerting fires on new Xid occurrences rather than raw historical state, so a handled Xid 79 does not re-page after every check.