The job runs. Health checks pass. nvidia-smi shows the GPU, temperature is fine, ECC is clean, and utilization looks normal. Yet training throughput is half of what it should be, or one GPU in an eight-GPU node drags every collective operation. A common root cause is a PCIe link that negotiated below its capability: a Gen4 x16 card silently running at Gen3, or worse, at x8 or x4.

Nothing errors. The link trained successfully, just at a lower speed or width. The driver is happy, the kernel is happy, and every green dashboard panel is technically true. The only symptom is bandwidth you no longer have: Gen4 to Gen3 costs roughly 50% of link bandwidth, x16 to x8 costs 50%, and x16 to x4 costs 75%.

Width reduction matters more than generation reduction. A generation drop at idle is normal power management and should recover under load. A width drop means physical lanes failed to train, which points at connectors, risers, slots, retimers, or BIOS lane allocation, and it tends to get worse over time, not better.

What this means

PCIe link parameters are negotiated (“trained”) between the GPU and the root complex or switch at link-up time. Both sides advertise a maximum generation (Gen3/4/5) and a maximum lane width (x16 for a full-size slot). The negotiated result can be lower on either axis, and the link runs at whatever both sides agreed on.

Two different things can push negotiated values below capability:

  1. Dynamic, by design. NVIDIA drivers downshift link generation at idle to save power. A Gen5 GPU reporting Gen1 or Gen2 while idle is normal. ASPM (Active State Power Management) does the same class of thing. The link must climb back to full speed under load. If it does not, that is the fault.
  2. Static, by failure. The link trained low at boot and never retrains upward. Causes are physical or configurational: a loose or dirty riser or slot contact, a damaged slot or bent connector, a failed PCIe retimer, a BIOS lane-sharing or bifurcation setting that only wired x8 to the slot, or thermal-related signal integrity problems. Width training low is almost always in this category, because lane count does not dynamically scale the way generation does.

In multi-GPU training, one degraded link creates a straggler: every collective waits on the GPU with the slowest host and peer path, so a single x8 GPU can tax the whole node. See how an NVIDIA GPU actually works in production.

Common causes

CauseWhat it looks likeFirst thing to check
Idle downshift (ASPM, driver power management)Gen low at idle, recovers to max under load; width stays at maxPut the GPU under load and re-query link gen
Loose, dirty, or oxidized slot/riser contactsWidth below max since boot; often after a move, reseat, or maintenancePower down, reseat the card, inspect and clean contacts
Riser cable signal integrityWidth or gen low only through the riser; worse on Gen4/Gen5 linksTest the card directly in the slot
Damaged slot or bent connectorWidth stuck low, follows the slot not the cardMove the GPU to a known-good slot and compare
BIOS lane sharing or bifurcationWidth halved since build; often an M.2 slot shares lanes with the x16 slotMotherboard lane map; BIOS bifurcation settings
Failed or marginal PCIe retimerLink trains low or flaps; replay errors climbReplay counters in nvidia-smi -q -d PCIE
Thermal signal-integrity degradationLink degrades or errors rise as temperature climbsCorrelate link state and replay errors with GPU/board temp
Driver/ASPM bugGen falls back under load on specific driver/hardware combosDriver release notes; a confirmed bug (open-gpu-kernel-modules #1010) causes Gen1 fallback under load on some 500-series GPUs
Card physically wired for fewer lanesWidth max itself reports x8If pcie.link.width.max is x8, the card or slot wiring is x8 by design, not degraded

Quick checks

All of these are read-only and safe to run in production.

# 1. Current vs maximum link generation and width, per GPU
nvidia-smi --query-gpu=index,name,pcie.link.gen.gpucurrent,pcie.link.gen.gpumax,pcie.link.gen.hostmax,pcie.link.width.current,pcie.link.width.max --format=csv

Note the field names. pcie.link.gen.current is deprecated; use pcie.link.gen.gpucurrent. Also capture hostmax: if the host’s maximum is lower than the GPU’s, the limit is the slot, chipset, or CPU, not the card.

# 2. The same story from the OS side: LnkCap (capability) vs LnkSta (status)
lspci -vv -s $(lspci | grep -i nvidia | head -1 | awk '{print $1}') | grep -i "lnk"

LnkSta below LnkCap on width or speed is the kernel’s view of the downgrade. Some kernels tag the line with “(downgraded)”.

# 3. Re-query under load, the critical step
# Start a real workload or a bandwidth-heavy job, then re-run check 1.
# Generation should climb to gpumax. Width should already be at max and stay there.

# 4. PCIe replay errors: retransmissions indicate marginal signal integrity
nvidia-smi -q -d PCIE | grep -i replay

The counter is cumulative since boot. A nonzero static value is history; a value that increases between checks is an active problem.

# 5. Actual PCIe throughput in both directions
nvidia-smi dmon -s t -d 1
# rxpci / txpci in MB/s; compare against the practical max for the negotiated link

For reference, Gen4 x16 delivers roughly 26-28 GB/s per direction in practice. If the link trained at x8, expect roughly half.

# 6. Check the kernel log for PCIe-related GPU events
dmesg -T | grep -i "NVRM: Xid"

XID 79 (GPU has fallen off the bus) is a catastrophic PCIe failure and a different incident class, but sustained link-layer trouble can precede it.

How to diagnose it

flowchart TD
  A[current gen or width below max] --> B{Under active load?}
  B -->|No| C[Put GPU under load and re-query]
  C --> D{Recovered to max?}
  D -->|Yes| E[Normal idle downshift - no action]
  D -->|No| F[Real downgrade]
  B -->|Yes| F
  F -->|Width low| G[Physical or BIOS lane issue]
  F -->|Gen low only| H[Signal integrity, retimer, or driver/ASPM bug]
  G --> I[Reseat card, clean contacts, try another slot]
  G --> J[Check BIOS lane sharing and bifurcation]
  H --> K[Check replay counters and riser]
  H --> L[Check driver version against known ASPM bugs]

Work through it in this order:

  1. Establish the baseline. Run check 1 on every GPU in the node. Note which axis is degraded (gen, width, or both) and whether it is one GPU or several. One GPU degraded points at that card’s slot, riser, or connector. All GPUs degraded points at BIOS configuration, the host’s PCIe capability (hostmax), or a shared switch/retimer.
  2. Rule out idle downshift. Re-query during a real workload. If generation climbs to maximum under load and width was always at max, there is no fault. This is the most common false alarm, and the reason PCIe link alerts must be gated on workload activity.
  3. Check whether width.max itself is x8. If pcie.link.width.max reports x8, the GPU is wired for x8: either the card design is x8 (some SKUs are), or the slot only has x8 routed. Compare against the hardware spec before treating it as degradation.
  4. Cross-check with lspci. Confirm LnkSta against LnkCap. If the OS agrees with nvidia-smi, the downgrade is real and at the link layer, not a reporting artifact.
  5. Look at replay errors. Take two readings of the replay counter a few minutes apart under load. Growth means marginal signal integrity: riser, contacts, retimer, or thermals. A growing replay count with a degraded link is strong evidence for a hardware path problem.
  6. Bisect physically. Move the GPU to a known-good slot, or put a known-good GPU in the suspect slot. The fault follows the card: card or connector. The fault follows the slot: slot, riser, retimer, or BIOS lane wiring.
  7. Check BIOS lane allocation. Many boards share lanes between the primary x16 slot and M.2 slots or between slots. Populating a specific M.2 slot can force the GPU slot to x8. Verify against the motherboard lane map and the bifurcation setting.
  8. Eliminate the riser. If the GPU is on a riser cable, test it seated directly in the slot. Risers are a leading cause of training failures, and newer generations are less tolerant of marginal risers than older ones.
  9. Check the driver. If generation falls back under load with no physical explanation, check the driver branch against known issues. A confirmed bug (open-gpu-kernel-modules #1010) has ASPM causing Gen1 fallback under load on some 500-series GPUs.

One caution on false positives: some platforms misreport. There is a documented case where a GPU’s firmware advertised x16 capability while the card was physically x8, causing the platform to log a spurious “link width degraded” event. If width.max disagrees with the card’s published spec, trust the spec and the vendor’s advisories.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
pcie.link.gen.gpucurrent vs gpumaxDetects generation downgradeBelow max during active workload
pcie.link.width.current vs width.maxDetects lane loss, the more serious axisBelow max at any time after training
PCIe replay errorsSignal integrity leading indicatorRate of increase under load
PCIe throughput (rxpci/txpci)Confirms bandwidth actually lostCeiling well below the negotiated link’s practical max
Per-GPU throughput symmetry in multi-GPU jobsOne degraded GPU becomes a stragglerOne GPU consistently slower than its peers
XID events in dmesgLink-layer faults escalating to device lossXID 79, or repeated PCIe-related XIDs

Two rules keep this from paging you at 3 a.m. for nothing. First, gate link-state alerts on workload activity, because idle downshift is by design. Second, treat a link below baseline as a ticket unless corroborated: page only when the downgrade is sustained under load and there is measured throughput or SLO impact, or replay errors are climbing. Even a drop to x1 or Gen1 under load can have benign causes (ASPM quirks, driver bugs, maintenance renegotiation), so corroborate before waking someone up.

Fixes

Physical reseat and contact cleaning

Power the node down, remove the GPU, inspect the edge connector and slot for debris, oxidation, or bent pins, clean contacts appropriately, and reseat firmly. This resolves a large share of width-training faults, especially after maintenance, shipping, or chassis work. This is disruptive: it requires downtime for the node or at least the GPU.

Remove or replace the riser

If the GPU trains at full width and gen in the slot but not through the riser, the riser is the fault. Replace it with one rated for the generation you are running. Gen5 links are substantially less forgiving of riser quality than Gen3 or Gen4. If the platform allows it, forcing one generation lower in BIOS is a valid stability workaround at the cost of bandwidth.

BIOS lane configuration

If an M.2 device or a second slot is stealing lanes, move the device to a slot on a different lane group or accept x8 knowingly. If bifurcation is configured for a different device population than what is installed, correct it. Check for board vendor BIOS updates as well; lane-init bugs at power-on have shipped and been fixed in firmware on some platforms.

Retimer or slot hardware

If a known-good card trains low in one slot and the suspect card trains fine elsewhere, the slot, retimer, or board routing is bad. That is a hardware replacement or vendor RMA conversation, not a software fix. Bring the replay counter trend and the lspci LnkCap/LnkSta output; it shortens the RMA loop considerably.

Driver and ASPM issues

If the evidence points at software (generation fallback under load, clean replay counters, fault appearing after a driver update), test with ASPM policy changed in BIOS or the driver upgraded or rolled back per the known-issue guidance for your branch. Validate the fix under real load, not at idle.

Living with x8

Not every workload cares. If the model and working set fit in GPU memory and host-to-device traffic is light, an x8 link may cost little. Measure the impact with PCIe throughput during the actual workload before scheduling downtime. If the workload is PCIe-bound (heavy data loading, frequent checkpointing, peer-to-peer traffic without NVLink), fix the link.

Prevention

  • Baseline link state at provisioning. Record gen and width current/max for every GPU at node acceptance, under load. A downgrade is only detectable if you know what “trained correctly” looked like.
  • Alert on the delta, not the raw value. Compare current to max during active workloads only. Never alert on idle readings.
  • Track replay errors as a rate. Marginal signal integrity shows up as a climbing replay rate weeks before a link drops or a GPU falls off the bus.
  • Re-verify after any physical work. Reseats, riser changes, chassis moves, and BIOS updates can all change link training. Make link-state verification a step in the maintenance checklist.
  • Monitor per-GPU symmetry in multi-GPU jobs. Cross-GPU throughput comparison catches a straggler even when each individual GPU looks healthy in isolation.
  • Sample fast enough. PCIe link transitions and replay bursts are short-lived. Minute-resolution polling will miss them; 10-second or faster sampling is the floor for GPU signals.

How Netdata helps

  • Netdata collects per-GPU PCIe link generation and width alongside utilization, power, and temperature, so a link downgrade is visible in the same view as the workload that should have raised it.
  • Correlating link width with PCIe throughput confirms whether the downgrade is costing real bandwidth or is irrelevant to the current workload.
  • Per-second sampling catches link retraining events and replay-error bursts that minute-resolution monitoring misses.
  • Cross-GPU comparison on a node makes the straggler pattern obvious: one GPU with a degraded link and lower effective throughput while its peers wait in collectives.
  • Tracking link state over time separates “always x8 since build” (BIOS lane allocation) from “dropped to x8 last Tuesday” (hardware fault in progress).