NVIDIA Xid 74: NVLink error

You found NVRM: Xid 74 in dmesg. The driver detected an error on an NVLink connection between this GPU and another GPU or an NVSwitch. The confusing part: nothing crashed. Training is still running. What actually happened is that the link’s error recovery machinery fired - CRC failures, packet replays, link recovery events - and the driver logged it.

Xid 74 is the classic silent straggler. Retransmissions eat NVLink bandwidth, and collective operations like AllReduce synchronize at the speed of the slowest participant. Step time creeps up, throughput drops, and nobody gets paged because nothing “failed.” The job is just slower, and the GPU-hour bill is higher.

The dangerous version is the recurring one. A single Xid 74 under heavy load can be transient. Xid 74 repeating on the same link points at a physical problem: cable, connector, port, or the device at the remote end of the link. On NVSwitch systems (DGX/HGX), it can also be a symptom of a Fabric Manager or switch-side fault rather than a GPU problem at all.

This guide walks through isolating the physical link, distinguishing transient noise from hardware failure, and deciding when to reseat, reset, or RMA.

What this means

NVLink is the high-bandwidth GPU-to-GPU interconnect (up to 300 GB/s aggregate on V100, 600 GB/s on A100, 900 GB/s on H100). It has link-layer error detection and recovery built in: corrupted packets are detected via CRC and retransmitted (replay), and the link can retrain (recovery) when signal quality degrades. Xid 74 is the driver surfacing that this machinery fired - or, in the worse case, that the link failed outright.

Three properties of Xid 74 shape how you respond:

It is conditionally fatal. In the XID classification, Xid 74 is not an automatic “reboot the node” event like Xid 79 (GPU fallen off the bus). The standard action is a GPU reset on multi-GPU systems, and only after you have isolated which link and which side of the link is at fault.

It can be a downstream symptom. If the GPU at the remote end of an NVLink connection fails or resets, the surviving GPU can log Xid 74 because its link partner disappeared. Before blaming the reporting GPU’s hardware, check whether the peer GPU logged its own, more primary fault (Xid 48, 79, 95) at the same time.

It has a twin that gets confused with it. Xid 68 is a video decoder (NVDEC) exception, not NVLink. Xid 45 is preemptive cleanup following another error - a consequence, not an independent fault. If you see Xid 45 alongside Xid 74, the 45 is the echo; the 74 (or whatever preceded it) is the event. Misreading these codes leads to the wrong hardware being replaced.

Common causes

Quick checks

All of these are read-only and safe to run during an incident.

# 1. Find the Xid 74 events and their timestamps
dmesg -T | grep -i "NVRM: Xid"
journalctl -k | grep -i "NVRM: Xid"

# 2. Per-link NVLink status (up/down, per link index)
nvidia-smi nvlink -s

# 3. Per-link error counters: replay, recovery, CRC
nvidia-smi nvlink -e

# 4. Per-link throughput (cumulative KiB payload)
nvidia-smi nvlink -gt d -i 0

# 5. Topology: which links should exist between which GPUs
nvidia-smi topo --matrix

# 6. On NVSwitch systems: Fabric Manager health
systemctl status nvidia-fabricmanager
journalctl -u nvidia-fabricmanager --since "1 hour ago"

Two notes on the counters. First, nvidia-smi nvlink -e counters are volatile - they reset on driver reload, so capture them before any reset or reboot and record the values in your incident notes. Second, these commands are not available through --query-gpu, so your monitoring needs to collect them explicitly.

How to diagnose it

The goal is to answer three questions: which physical link, which side of the link, and is it recurring.

flowchart TD
  A[Xid 74 in dmesg] --> B[Check nvlink -e per-link counters]
  B --> C{One link with growing errors?}
  C -- Yes --> D[Check peer GPU for own fatal Xid]
  C -- No / multiple links --> E{NVSwitch system?}
  E -- Yes --> F[Check Fabric Manager and switch health]
  E -- No --> G[Check for recent driver reload or reset events]
  D -- Peer has Xid 48/79/95 --> H[Fault is remote GPU; triage that Xid]
  D -- Peer clean --> I[Suspect cable/connector/port on that link]
  I --> J{Recurs after reseat and reset?}
  J -- Yes --> K[RMA or vendor hardware ticket]
  J -- No --> L[Monitor counters; treat as transient]
  F --> M[Restart FM; if Xid 74 recurs, escalate to vendor]

1. Establish the timeline. Pull dmesg -T for all Xid events on the node, not just 74. If GPU 2 logged Xid 79 at 03:14:02 and GPU 0 logged Xid 74 at 03:14:03, the 74 is fallout from the peer dying. Triage the 79, not the 74.

2. Isolate the link. Run nvidia-smi nvlink -e on every GPU and compare counters across links. A hardware problem almost always shows up as one link with replay/recovery/CRC counts materially higher than its siblings. Uniformly low counts everywhere point away from a single physical link.

3. Check link state against expected topology. nvidia-smi nvlink -s shows which links are up; nvidia-smi topo --matrix shows what the topology should look like. A link that should exist but is down, on a node running multi-GPU training, is a page-level event: NCCL may have silently fallen back to PCIe for that path, which is a massive bandwidth loss.

4. Correlate with workload impact. Check training step time or collective latency around the event. If step time jumped at the same timestamp and stayed elevated, the link is degraded in a way that matters right now. If nothing changed and counters are not growing, you are looking at a transient.

5. On NVSwitch systems, check the fabric layer. On DGX/HGX, Fabric Manager manages NVSwitch configuration; if it is down or errored, NVLink connectivity through the switch is compromised and multiple GPUs can log Xid 74 simultaneously. Check systemctl status nvidia-fabricmanager and its journal before concluding any single GPU is at fault. Also confirm the Fabric Manager and driver versions are a matched pair - mismatches are a known cause of fabric failures.

6. Determine recurrence. One event with stable counters is watch-and-wait. Repeated events on the same link index, especially after a reseat, is hardware: cable, connector, port, or the link partner. That is a vendor support ticket or RMA conversation, and the per-link counter history is exactly the evidence you want attached to it.

Metrics and signals to monitor

On event-versus-state semantics: Xid 74 is a log event, so alert on new log entries, not on the historical presence of the string in an old log file. The error counters are cumulative and volatile, so alert on rate of change between samples, and expect them to zero out after a driver reload - do not let that reset look like a cure.

Fixes

Transient single event, no counter growth

Do nothing invasive. Record the event, keep collecting per-link counters, and compare again in 24-72 hours. A single recovery event under peak collective load can be a marginal signal integrity blip that never repeats. Resetting a healthy GPU costs you the workload running on it and gains nothing.

One link with growing error counters, peer GPU clean

Treat it as a physical layer problem on that link: cable, connector, or port. The sequence:

1. Drain the node or migrate the workload. Do not reseat hardware with jobs running.
2. Physically inspect and reseat the NVLink connection for that link index (bridge, cable, or backplane connector, depending on platform).
3. After reseat, reset the affected GPU(s). Note the reset scope: NVLink-connected GPUs have coupling requirements on reset, and on some platforms the whole connected set must be reset together. See the reset guide linked below for the safe procedure.
4. Watch counters under load. If the same link starts accumulating errors again, stop reseating and open a hardware ticket with your counter history attached.

Xid 74 caused by a peer GPU fault

The NVLink error is the symptom; the peer’s fatal Xid is the disease. Triage the peer per its own Xid (48 for double-bit ECC, 79 for fallen off the bus, 95 for uncontained ECC). Do not replace the GPU that only reported Xid 74 - its link hardware is probably fine.

Fabric Manager or NVSwitch involvement

Restart Fabric Manager (systemctl restart nvidia-fabricmanager) and verify it comes up clean in the journal. Confirm the Fabric Manager version matches the driver version. If Xid 74 events recur across multiple GPUs after a clean FM restart, escalate to the platform vendor - at that point you are dealing with switch ports or backplane, which is not field-serviceable by reseating GPU-side cables.

Virtualized environments

If Xid 74 appears at VM power-on, check the passthrough assignment: all NVLink-connected GPUs must be assigned to the same VM. Passing through only a subset of an NVLink-connected set produces exactly this error.

Prevention

• Baseline per-link counters. Collect nvidia-smi nvlink -e on a schedule and store it. Without a baseline, you cannot tell “growing for weeks” from “started today.”
• Alert on counter deltas, not raw values. Counters are cumulative and reset on driver reload. Rate-based alerting survives both properties.
• Correlate Xid 74 with peer Xids automatically. Your log pipeline should group Xid events by node and time window so the “74 is fallout from a 79” case is obvious in the alert, not discovered an hour into triage.
• Track step time per node. The slow-straggler version of Xid 74 is invisible to hardware health checks. A per-node step-time regression alert catches degraded links before anyone reads dmesg.
• Keep Fabric Manager and driver versions matched on NVSwitch systems, and monitor the FM service as infrastructure, not an afterthought.
• Record counter snapshots in every NVLink incident ticket. The recurrence pattern per link is the evidence that gets hardware replaced under warranty instead of bounced back as “no trouble found.”

How Netdata helps

Netdata’s NVIDIA GPU monitoring surfaces the signals that matter for this exact failure mode, at the per-second resolution that short-lived link events require:

• Xid event awareness via system logs, so a new Xid 74 shows up as an event you can correlate against GPU metrics at the same timestamp, not a string you grep for after a user complains.
• Per-GPU utilization and throughput charts that make the straggler pattern visible: one GPU diverging from its peers in a data-parallel job is the signature of a degraded NVLink path.
• PCIe throughput per GPU, which confirms the silent-fallback case: if collective traffic moved from NVLink to PCIe, PCIe TX/RX on the affected GPUs climbs while NVLink throughput drops.
• Cross-GPU correlation on one node, letting you overlay the reporting GPU and its NVLink peer to see whether the peer’s thermal, ECC, or utilization behavior explains the link error.
• Long retention for counter trends, so “is this link’s replay count growing week over week” is a query, not a spreadsheet.

Related guides

• NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver
• How an NVIDIA GPU actually works in production: a mental model for operators
• NVIDIA GPU monitoring checklist: the signals every production GPU fleet needs
• NVIDIA GPU monitoring maturity model: from survival to expert
• nvidia-smi hangs or is unresponsive: a wedged GPU or a stuck driver
• NVIDIA persistence mode: why the GPU keeps re-initializing and P-state flaps
• Resetting a wedged NVIDIA GPU: nvidia-smi –gpu-reset and when only a reboot works
• NVIDIA Xid 13: Graphics Engine Exception
• NVIDIA Xid 31: GPU memory page fault (invalid address)
• NVIDIA Xid 43: GPU stopped processing (the GPU hang)
• NVIDIA Xid 48: Double Bit ECC Error
• NVIDIA Xid 79: GPU has fallen off the bus