$ guides / nvidia-gpu / nvidia-gpu-nvlink-errors ▌

Operations Guides

NVIDIA GPU NVLink errors: CRC and replay errors on GPU-to-GPU links

Your training job is running. No crashes, no CUDA errors, no OOM. But step time is up roughly 20%, and the slowdown shows up exactly during the collective phases: AllReduce, gradient sync. The team has already blamed the data loader, shuffled hyperparameters, and re-run the job twice. The actual problem is one degraded NVLink cable retransmitting corrupted flits, and nothing in the default monitoring stack is looking at it.

NVLink CRC and replay errors are the classic silent degradation on multi-GPU NVIDIA systems. The link layer detects corruption and retransmits, so data still arrives intact. What you lose is bandwidth. Because nothing fails, the symptom is a slow job, not an error message, and misdiagnosis is common. This guide covers how to read the per-link error counters, tell one bad link from a systemic problem, correlate with Xid 74, and isolate GPU versus NVSwitch faults on HGX/DGX systems.

What this means

NVLink provides GPU-to-GPU bandwidth far above PCIe: roughly 300 GB/s on V100, 600 GB/s on A100, 900 GB/s on H100 class systems. Distributed training depends on that bandwidth during collective operations, where every GPU exchanges gradients with its peers. The communication is synchronous: the collective completes when the slowest participant finishes.

The NVLink link layer protects transfers with CRC checks on flits and data. When a CRC check fails, the link layer replays (retransmits) the affected data. When the link itself loses sync, it goes through a recovery event, briefly dropping and retraining. All three mechanisms preserve correctness but consume bandwidth and time. The counters the driver exposes are:

CRC errors (flit and data): corrupted transmissions detected on the link.
Replay errors: retransmissions triggered by those CRC failures.
Recovery events: the link went down momentarily and retrained. This is a link flap, not a counter tick.

A link with a modest but steady replay rate does not crash anything. It just delivers less effective bandwidth, so the GPU at the end of it becomes the straggler, and every other GPU in the job waits on it at each collective. On NVSwitch systems, the same mechanism applies between each GPU and the switch fabric.

flowchart TD
  A[Physical link degradation: cable, bridge, connector, transceiver] --> B[CRC errors on flits or data]
  B --> C[Replay: link layer retransmits]
  C --> D[Effective bandwidth drops on one link]
  D --> E[One GPU becomes the straggler]
  E --> F[Collectives slow: AllReduce, gradient sync]
  F --> G[Training step time up, no crash, no error]
  A --> H[Link loses sync]
  H --> I[Recovery event: brief link down and retrain]
  I --> J[Xid 74 in kernel log]

Common causes

Cause	What it looks like	First thing to check
Bad cable, loose NVLink bridge, or failing transceiver	One link’s counters far above the others on the same GPU	Compare per-link counters with `nvidia-smi nvlink -e`; look for the outlier link
Marginal link degrading over time	Replay/CRC counters climbing week over week on the same link	Persist counters across scrapes and trend the rate, not the absolute value
Physical disturbance (chassis vibration, reseating)	Sudden burst of recovery events and Xid 74 after maintenance or a move	Correlate Xid 74 timestamps with maintenance windows
NVSwitch or switch-side cabling fault (HGX/DGX)	Multiple GPUs on the same switch show errors on the corresponding path	Compare counters across GPUs that share the switch; switch-side faults hit many GPUs at once
NCCL silently fell back to PCIe	Training works but is much slower; NVLink throughput near zero while PCIe throughput is high	`nvidia-smi nvlink -s` for link state and `nvidia-smi topo --matrix` for topology

Quick checks

All of these are read-only and safe to run on a production node.

# Per-link NVLink error counters (CRC, replay, recovery) for GPU 0
nvidia-smi nvlink -e -i 0

# Link status: which links exist and whether they are up
nvidia-smi nvlink -s -i 0

# Cumulative NVLink data throughput (KiB) per link, GPU 0
nvidia-smi nvlink -gt d -i 0

# Topology: how the GPUs are actually connected
nvidia-smi topo --matrix

# Xid 74 and any other NVRM errors in the kernel log
dmesg -T | grep -i "NVRM: Xid"

Two things to know before reading the output:

The counters are per-link and volatile. They reset on driver reload and GPU reset. A single snapshot is nearly meaningless without history. Capture and persist them on every scrape, and alert on the rate of increase.
The comparison that matters is link versus link. A healthy system’s links sit at or near zero. One link with counters far above its siblings on the same GPU is a hardware problem on that specific path.

On Blackwell-generation GPUs, NVIDIA marks the older CRC, replay, and recovery NVML fields unsupported and points to the NVML_FI_DEV_NVLINK_COUNT_LINK_RECOVERY_* counters (successful, failed, and all recovery events). Do not assume nvidia-smi nvlink -e has the same output across generations; capture what your installed driver returns and use recovery-event counters where supported.

How to diagnose it

Snapshot the counters on every GPU in the node. Run nvidia-smi nvlink -e -i N for each GPU and save the output with a timestamp. You need a baseline before anyone reboots or resets anything, because a reset erases the evidence.
Find the outlier link. For each GPU, compare counters across its links. Ignore absolute values for the moment; you are looking for asymmetry. One link at thousands of replays while the rest sit near zero points at that link’s physical path.
Take a second snapshot a few minutes later and compute the rate. Any nonzero CRC or replay rate over a short window on a link that should be idle or lightly loaded is concerning. Steady incrementing under normal training load confirms active retransmission, not historical noise.
Check link state and topology. nvidia-smi nvlink -s shows links up or down. nvidia-smi topo --matrix shows which GPU pairs actually have NVLink paths. If a link that should exist is down, or NCCL-relevant pairs are traversing PCIe (PIX/PXB/NODE instead of NV# in the matrix), you have found why collectives are slow.
Check for Xid 74. dmesg -T | grep -i "NVRM: Xid" and look for Xid 74, the NVLink error event. Recovery events (the link briefly dropping and retraining) correlate with Xid 74 and should be treated as pageable on a node running active multi-GPU work, because they interrupt collectives in flight. Do not confuse Xid 74 with Xid 68, which is a video decoder exception and unrelated.
On NVSwitch systems, isolate the side of the fault. Compare the error counters across all GPUs connected to the same switch. If one GPU shows errors on its switch-facing links while its peers are clean, the problem is on that GPU’s path to the switch. If many GPUs show errors simultaneously, suspect the switch itself or its cabling. Fabric Manager health matters here too; on HGX/DGX systems, check systemctl status nvidia-fabricmanager and its journal.
Corroborate with workload impact. Confirm the symptom matches: elevated collective or gradient-sync time, elevated training step time, per-GPU utilization divergence with one GPU consistently behind. NVLink throughput near zero with high PCIe throughput during collectives indicates fallback, which is itself worth a ticket even with zero CRC errors.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Per-link CRC error rate	Direct measure of link corruption	Any sustained nonzero rate; one link far above siblings
Per-link replay rate	Retransmissions consuming effective bandwidth	Steady increments under normal load
Recovery events / link state transitions	Link flaps interrupt collectives in flight	Any unexpected Up-to-Down transition during an active job (PAGE)
Xid 74 in kernel log	Driver-level NVLink fault event	Any occurrence; recurring events on one GPU or one link
NVLink throughput per link	Confirms the link is carrying traffic and how much	Near-zero during collectives on a topology that should use NVLink
NCCL collective latency	Where the bandwidth loss becomes visible	Greater than 2x baseline for the same workload and topology
Per-GPU utilization divergence	Straggler detection	One GPU consistently behind its peers at each sync point
PCIe throughput during collectives	Detects silent PCIe fallback	High PCIe traffic where NVLink should carry the load

Two operational rules for these signals. First, alert on rates and deltas, not absolute counters: volatile counters reset on driver reload, so a level-triggered alert on a raw counter will both miss post-reset degradation and re-fire on historical state. Second, gate link-down paging on an active multi-GPU workload and confirmed impact (NCCL regression or PCIe fallback). A down link on an idle node is a ticket, not a 3 a.m. page.

Fixes

Physical path: cable, bridge, connector

The dominant cause of one outlier link is physical. Schedule a maintenance window, reseat the NVLink bridge or cable on the affected path, and re-check the counters under load. If the error rate follows the cable after a swap, replace the cable. If it stays with the port, the GPU or board connector is the suspect and the GPU is a replacement candidate. NVIDIA documents that Xid 74 requires a GPU reset or node reboot to clear the error; killing the process does not clear it, so plan the drain accordingly.

NCCL fallback to PCIe

If topology shows collectives traversing PCIe when NVLink paths exist, treat it as a fault even with clean error counters. Training will run correctly but much slower, and nothing in the application log will say why. Restore the NVLink path (fix the link, restart Fabric Manager on NVSwitch systems) and verify with throughput counters during a test collective.

Fabric-side faults on NVSwitch systems

If the fault pattern points switch-side (multiple GPUs, simultaneous errors), escalate to platform/vendor with the cross-GPU counter comparison you captured. Check Fabric Manager logs first; a Fabric Manager failure takes down NVLink connectivity through the switch while leaving each GPU individually healthy.

Straggler containment

If you cannot fix the link immediately and the job matters more than the node, draining the degraded GPU and continuing at reduced capacity is often cheaper than letting it gate every collective in the job. One slow participant sets the pace for everyone.

Prevention

Persist the counters. Scrape per-link CRC/replay/recovery counters at sub-minute resolution and store them externally. Volatile driver counters are the only source, and they vanish on reset.
Baseline per link, per node. Rate-of-change and link-versus-link asymmetry detection beats any absolute threshold.
Alert on recovery events as pageable when gated on active multi-GPU workloads; treat CRC/replay rate growth as a high-priority ticket with a hardware investigation attached.
Monitor Xid 74 explicitly in log pipelines, with correct classification (NVLink, not the frequently misattributed Xid 68).
Verify topology after maintenance. Any reseat, move, or driver reload should end with nvidia-smi topo --matrix and a counter baseline, so the next incident has something to compare against.
Correlate with collective latency. NVLink error monitoring without NCCL or step-time context will find hardware problems but miss whether they matter. Monitoring both closes the loop.

How Netdata helps

Netdata’s NVIDIA GPU collector tracks the interconnect signals alongside thermals, power, clocks, and utilization at per-second resolution, so a replay-driven bandwidth loss shows up next to the throttle and utilization context that rules out competing explanations.
Per-GPU time series make the straggler pattern visible directly: one GPU diverging from its peers in utilization, clocks, or throughput at each synchronization point.
Persisting NVLink counter scrapes externally solves the volatility problem: you keep rate history across driver reloads and GPU resets instead of losing the baseline exactly when you need it.
Correlating kernel-log Xid events with GPU metric timelines ties Xid 74 to the moment step time regressed, which shortens the “is this hardware or hyperparameters” argument to minutes.
Cross-signal correlation catches the silent variant too: NVLink throughput near zero with PCIe throughput elevated during collectives is a fallback signature, not an error counter, and it only shows up when both are on the same dashboard.