NVIDIA GPU NVLink down or degraded: a link that dropped in multi-GPU training

A link that drops mid-training is one of the nastiest GPU failures to catch, because the job usually does not crash. It hangs at a synchronization barrier, or it keeps running at a fraction of its normal step rate. Bandwidth between GPUs collapses from hundreds of GB/s (600 GB/s on A100, 900 GB/s on H100) to PCIe speeds, and every AllReduce now waits on the slowest path in the collective.

The worse variant is silent: the link never shows “down” in an obvious place because NCCL quietly fell back to PCIe or socket transport at job startup. Training initializes fine, losses decrease, and nobody notices until someone asks why the cluster is suddenly 3x slower. NVLink utilization sitting at 0% during multi-GPU training is the tell.

This guide covers how to confirm whether the link actually dropped, whether NCCL is even using NVLink, and what to do about it. For the broader interconnect picture, see how an NVIDIA GPU actually works in production.

What this means

NVLink is the high-bandwidth GPU-to-GPU interconnect. When a link drops or degrades, one of three things happens:

1. The collective path reroutes. NCCL picks a slower transport (PCIe P2P, shared memory, or sockets). The job runs but collectives are slow, and one degraded participant becomes a straggler that gates every synchronous step.
2. The job hangs or dies on NCCL timeout. If the link drops mid-collective, ranks wait on a peer that cannot respond fast enough. Depending on configuration this surfaces as an NCCL timeout abort, or as a hang that looks like a stuck job.
3. On NVSwitch systems (DGX/HGX), the whole fabric can be affected. Fabric Manager (nvidia-fabricmanager) manages NVSwitch configuration. If it fails, GPUs are individually healthy but cannot communicate over the fabric.

The critical detail: NCCL may fall back from NVLink to PCIe without logging an error. Training runs, dashboards look plausible, and the only symptoms are elevated step time and zero NVLink traffic.

flowchart TD
  A[Step time up or job hung] --> B{nvidia-smi nvlink -s}
  B -->|link down or inactive| C[XID 74 in dmesg?]
  B -->|all links up| D{NVLink traffic during job?}
  D -->|0 percent utilization| E[NCCL fell back to PCIe]
  D -->|traffic flowing| F[Check other straggler causes]
  E --> G[NCCL_DEBUG=INFO: via NET/Socket or P2P?]
  C -->|yes| H[Hardware: link, board, NVSwitch]
  C -->|no| I{NVSwitch system?}
  I -->|yes| J[Check nvidia-fabricmanager]
  I -->|no| K[Check NCCL env and container config]
  G --> K

Common causes

Quick checks

All read-only and safe to run on a live node.

# 1. Per-link NVLink status: every expected link should be active/up
nvidia-smi nvlink -s

# 2. NVLink error counters per link (CRC, replay, recovery)
nvidia-smi nvlink -e

# 3. Cumulative throughput counters (data payload, KiB) per link
nvidia-smi nvlink -gt d -i 0

# 4. Topology: which GPU pairs actually connect via NVLink vs PCIe
nvidia-smi topo --matrix

# 5. XID events: XID 74 is the NVLink error code
dmesg -T | grep -i "NVRM: Xid"

# 6. On NVSwitch systems (DGX/HGX): Fabric Manager state
systemctl status nvidia-fabricmanager
journalctl -u nvidia-fabricmanager --since "1 hour ago"

# 7. Live PCIe throughput while the job runs
nvidia-smi dmon -s t -d 1

Notes on interpretation:

• nvidia-smi nvlink -gt throughput counters are cumulative; sample twice and diff to see whether a link is carrying traffic right now.
• nvidia-smi dmon -s t shows PCIe throughput. NVLink traffic does not appear in PCIe counters, so heavy PCIe traffic between GPUs during collectives while NVLink sits idle is itself a fallback indicator.
• In nvidia-smi topo --matrix, entries marked NV# confirm a hardware NVLink path exists; PIX/PXB/PHB/SYS entries mean the pair communicates over PCIe. If a pair that should be NV-connected shows PCIe paths in use, you have your answer.
• Error counters from nvidia-smi nvlink -e are volatile and reset on driver reload. Compare across links: one link with rising errors while its peers stay clean points at that link’s hardware.

How to diagnose it

1. Confirm the symptom is interconnect, not compute. Compare per-GPU metrics across the node: one GPU with a different utilization pattern, higher temperature, or active throttle reasons points at a straggler. If all GPUs look healthy individually but step time is elevated or the job is stuck at a barrier, suspect the interconnect. See the straggler discussion in the production mental model.

2. Check link state. Run nvidia-smi nvlink -s on every GPU (-i N to isolate). Any expected link showing down or inactive is a hardware-side fault. Cross-reference with dmesg for XID 74.

3. Check the topology against expectations. nvidia-smi topo --matrix tells you which pairs can use NVLink at all. On partial-mesh systems, some pairs legitimately use PCIe; do not chase a “fault” that is the designed topology. If you are unsure what the chassis should provide, check the vendor’s topology documentation for that model.

4. Verify what NCCL actually selected. This is the decisive step for the silent-fallback case. Reproduce with debug logging:

   # Rerun the job (or a minimal all_reduce test) with NCCL debug output
   export NCCL_DEBUG=INFO
   export NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH
   

   In the logs, look at the transport chosen per channel. NVLink-backed paths show as P2P transports; via NET/Socket between GPUs on the same node is the red flag that NCCL gave up on direct paths and fell back to TCP. NVLink utilization at 0% in multi-GPU training usually means NCCL fell back to PCIe without logging an error; debug output plus the topology matrix is how you prove it.

5. Audit the job environment. Look for NCCL-related environment variables set by the launcher, container image, or scheduler that restrict P2P or force a network transport. A leftover debugging override from a previous incident is a common cause of permanent fallback. Remove overrides, let NCCL auto-detect, and re-verify with NCCL_DEBUG=INFO.

6. On NVSwitch systems, check Fabric Manager. If GPUs are healthy but NCCL cannot initialize or collectives hang across the fabric, check systemctl status nvidia-fabricmanager and its journal. Restarting nvidia-fabricmanager is disruptive to running jobs that use the fabric; coordinate with the workload owner first. If it fails to come back, check driver and Fabric Manager version compatibility.

7. Correlate with the timeline. Did the link drop mid-run (XID 74 at a specific timestamp, step time cliff) or was it never up for this job (slow from step one, NVLink at 0% throughout)? Mid-run drop is hardware. Slow-from-start is configuration or fallback.

Metrics and signals to monitor

Alerting guidance: page on an expected NVLink transitioning up to down for a sustained period, on a topology where that link should exist, while an active multi-GPU job is using it, with NCCL or step-time regression confirmed. If the link is down but no multi-GPU workload is affected, ticket. On NVSwitch systems, page on Fabric Manager being down long enough to matter with jobs failing to initialize NCCL; add an uptime gate to exclude cold start and planned restarts.

Fixes

Hardware link failure (XID 74, link down)

XID 74 is conditionally fatal: GPU reset on multi-GPU systems. In practice, on a shared training node the safer sequence is: drain or migrate the workload, then reset the affected GPU or reboot the node, then verify the link returns with nvidia-smi nvlink -s. A link that drops once will often drop again. Track recurrence per GPU and per link; repeated XID 74 on the same link is a hardware replacement conversation (board, connectors, or NVSwitch), not a monitoring problem.

If the job cannot be interrupted immediately and the topology allows it, continuing at degraded PCIe bandwidth is possible, but expect a large step-time penalty and treat it as a bridge to a maintenance window, not a resolution.

Silent NCCL PCIe fallback

Fix the cause, not the symptom:

• Remove environment overrides that restrict P2P or force network transports, and confirm with NCCL_DEBUG=INFO that channels bind to NVLink transports.
• If the fallback happens only inside containers, audit the container GPU and IPC configuration; anything that breaks GPU-to-GPU P2P or IPC handle exchange between ranks forces NCCL onto slower paths.
• After any change, re-run a minimal collective test with debug logging before releasing the node back to the scheduler. Do not trust “the job starts” as proof.

Fabric Manager failure

1. systemctl status nvidia-fabricmanager and journalctl -u nvidia-fabricmanager for the error.
2. Restart the service, knowing this is disruptive to jobs using the fabric.
3. If it fails to start or crashes again, check driver/Fabric Manager version compatibility. Mismatched versions are a common post-upgrade failure.
4. Verify recovery with nvidia-smi topo --matrix and a test collective.

Degraded link with rising error counters

A link that is up but accumulating CRC errors is on the way out. It may also be throttling effective bandwidth, producing an intermittent straggler that is hard to pin down. Track the error rate trend, correlate with step-time variance, and schedule hardware inspection. Do not wait for the full drop.

Prevention

• Baseline step time and collective latency per workload. Absolute thresholds are meaningless; the detection is “2x baseline” or “no progress for N minutes outside checkpoint windows.” Without a baseline you cannot see the 3x regression that silent fallback causes.
• Watch NVLink utilization during training, not just link state. Link state up plus zero traffic during collectives is the fallback signature. This catches the case where nothing ever logged an error.
• Delta-based XID monitoring. XIDs are log events; alert on new occurrences (especially 74 on multi-GPU nodes), not on historical presence. Track which events you have already paged on.
• Compare GPUs against each other. The straggler pattern only shows up in cross-GPU comparison: one GPU waiting, one link idle, one pair on PCIe. Per-GPU dashboards in isolation miss it.
• Gate alerts on job activity. Link flaps during boot, fabric restarts, and maintenance are expected. Page only when an active multi-GPU job is degraded.
• Sample fast enough. Interconnect anomalies and XID events need sampling at 10 seconds or faster; minute-resolution monitoring misses the event that explains the hang.

How Netdata helps

• Per-second GPU metrics make the step-time cliff and the straggler pattern visible in real time, instead of after the job has been slow for an hour.
• Cross-GPU correlation on one dashboard: utilization, temperature, throttle reasons, and PCIe throughput side by side, so the “one GPU is different” pattern stands out immediately.
• NVLink and PCIe throughput signals collected together let you spot the fallback signature: PCIe busy during collectives while NVLink sits at zero.
• XID events from kernel logs correlated against GPU metric timelines, so you can align “XID 74 at 03:12” with the exact moment step time jumped.
• Fabric Manager and driver-health signals alongside GPU metrics on NVSwitch nodes, covering the case where every GPU looks fine but the fabric is down.
• Anomaly detection on per-workload baselines flags the 2-3x step-time regression that silent PCIe fallback produces, without hand-tuned thresholds.

Related guides

• NVIDIA BAR1 memory exhaustion: mapping failures with free framebuffer
• NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver
• CUDA out of memory with free memory available: GPU memory fragmentation
• CUDA out of memory: diagnosing NVIDIA GPU framebuffer exhaustion
• NVIDIA GPU ECC disabled: the silent data-corruption risk
• NVIDIA GPU ECC errors: corrected, uncorrected, volatile, and aggregate
• NVIDIA GPU fan at 0%: fan failure on air-cooled cards
• NVIDIA GPU HBM (memory) temperature: the thermal limit most teams miss
• NVIDIA GPU HBM progressive failure: from single-bit errors to a dead GPU
• How an NVIDIA GPU actually works in production: a mental model for operators
• NVIDIA GPU HW Power Brake Slowdown: the chassis is cutting GPU power
• NVIDIA GPU memory leak: framebuffer usage climbing without a plateau