$ guides / nvidia-gpu / nvidia-gpu-pcie-straggler-multi-gpu ▌

Operations Guides

NVIDIA GPU PCIe straggler: one slow link throttling multi-GPU training

Your distributed training job is running. No errors, no crashes, no XIDs. But step time jumped 30-60% and never came back, and scaling from 4 GPUs to 8 GPUs bought almost nothing. Per-GPU utilization shows one GPU a few points below the rest, and its PCIe throughput is consistently below its peers.

That is the PCIe straggler pattern. One GPU’s link negotiated at a lower generation or narrower width than it should have: Gen3 instead of Gen4, x8 instead of x16, or worse. That GPU now feeds data and exchanges gradients at half (or less) of its peers’ bandwidth. In synchronous data-parallel training, every collective waits for the slowest participant, so the whole job runs at the speed of the worst link.

The dangerous part is what does not happen: nothing fails. The GPU is reachable, ECC is clean, temperatures are fine, and nvidia-smi shows no alarm. The only symptom is throughput. This pattern survives for weeks in fleets that monitor individual GPUs but never compare them against each other.

What this means

PCIe Gen3/4/5 at x16 gives roughly 16/32/64 GB/s per direction of theoretical bandwidth. One generation step down or one width step down (x16 to x8) each cost about half; both together is a quarter of expected bandwidth.

In data-parallel training, every step ends with a gradient synchronization collective (AllReduce or equivalent) that cannot complete until every rank contributes. If one rank’s data path is degraded, every other GPU idles during that window. You see: one GPU with slightly lower compute utilization (waiting for data), slightly lower PCIe throughput, and elevated step time across the whole job.

flowchart TD
  A[PCIe link downgraded on one GPU] --> B[Halved or worse bandwidth on that GPU]
  B --> C[Gradient sync waits for slow rank]
  C --> D[Other GPUs idle during collectives]
  D --> E[Whole job runs at straggler speed]
  A --> F[No errors, no XIDs, no alerts]
  F --> G[Pattern survives unless GPUs are compared]

The timing of the downgrade tells you where to look. Degraded since boot: suspect BIOS lane allocation or physical seating. Degraded mid-operation: suspect hardware, a loose riser, thermal-induced signal integrity, or a failing retimer.

Common causes

Cause	What it looks like	First thing to check
Loose PCIe riser or damaged slot	Width dropped (x16 to x8 or lower), possibly mid-operation	`lspci -vv` LnkSta vs LnkCap on the affected GPU
Bent connector or poor seating	Downgrade present since boot, persists across reboots	Physical inspection; reseat the card
BIOS lane allocation misconfiguration	Width degraded since boot, often after a BIOS update or new card added	BIOS PCIe lane settings against the board manual
Thermal-induced signal integrity	Degrades under sustained load, recovers when cool	Correlate downgrade timing with chassis temperatures
Failed or failing PCIe retimer	Gen downgrade with rising PCIe replay errors	`nvidia-smi -q -d PCIE` replay counter delta between samples
Driver/ASPM power-management bug	Gen1 fallback under load on affected driver branches	Driver version; reported in open-gpu-kernel-modules issue #1010
NUMA affinity mistake	Bandwidth fine per-link but transfers cross the CPU socket interconnect	`nvidia-smi topo --matrix` vs process CPU affinity

Quick checks

All read-only and safe during a live job.

# 1. Compare negotiated link state across all GPUs (must be sampled under load)
nvidia-smi --query-gpu=index,pcie.link.gen.gpucurrent,pcie.link.gen.gpumax,pcie.link.gen.hostmax,pcie.link.width.current,pcie.link.width.max --format=csv

# 2. Confirm the GPU is actually under load right now
# (idle downgrade is by design and NOT a fault)
nvidia-smi --query-gpu=index,utilization.gpu,power.draw --format=csv

# 3. Live PCIe throughput per GPU (rxpci/txpci in MB/s)
nvidia-smi dmon -s t -d 1

# 4. Kernel-side view: capability vs negotiated state
# (run as root; non-root lspci may not show full capability data)
sudo lspci -vv -s <bus_id_of_suspect_gpu> | grep -i "lnk"

# 5. PCIe replay errors (cumulative since boot; sample twice, watch the delta)
nvidia-smi -i <N> -q -d PCIE | grep -i replay

# 6. Topology: is traffic crossing NUMA or socket boundaries?
nvidia-smi topo --matrix

# 7. Any XIDs at all? (straggler pattern usually has none; if you see them, reassess)
dmesg -T | grep -i "NVRM: Xid"

Two interpretation rules before you conclude anything:

Idle downgrade is normal. NVIDIA drivers downclock the PCIe link at idle (a Gen5 GPU showing Gen2 or lower at idle is expected). Only compare pcie.link.gen.gpucurrent against gpumax while the workload is actively running. Alerting on idle link state is a classic false positive.
pcie.link.gen.current is deprecated. Use pcie.link.gen.gpucurrent. If your monitoring silently returns nothing, check field names against nvidia-smi --help-query-gpu on your driver branch.

How to diagnose it

Confirm the asymmetry. Run check 1 under active load. The straggler GPU shows gpucurrent < gpumax or width.current < width.max while peers are at full spec. If every GPU is degraded identically, this is not a straggler; it is a platform configuration issue (chipset link limits, global BIOS setting).
Rule out the idle artifact. Verify utilization.gpu and power.draw on the suspect GPU are elevated at the moment you sampled link state. If utilization was near zero, resample during a compute phase.
Quantify the throughput gap. Watch nvidia-smi dmon -s t for a minute. The degraded GPU’s rxpci/txpci should ceiling well below its peers. A healthy Gen4 x16 link sustains roughly 26-28 GB/s per direction in practice; a Gen3 x8 link ceilings far below that. Then confirm the gap matters for your workload: if the model fits in GPU memory and gradients flow over NVLink, a PCIe downgrade may be nearly invisible.
Correlate with the collective. What separates “degraded link” from “degraded link that is hurting the job” is elevated collective latency plus idle time on the healthy GPUs during sync phases. If you have step-time instrumentation, check whether the step time increase coincides with the downgrade date.
Check replay errors. Take two nvidia-smi -q -d PCIE samples a few minutes apart on the suspect GPU. A rising replay count points at physical signal integrity (riser, slot, retimer, dust), not configuration.
Establish the timeline. Did the link train low at boot, or drop during operation? Since-boot points to BIOS lane allocation or seating. Mid-operation points to hardware degradation or a driver power-management path. The reported open-gpu-kernel-modules #1010 case, where ASPM-related behavior forces a Gen1 fallback under load on some 500-series GPUs, is an example of the latter: identical hardware works on a different driver branch.
Check NUMA placement. If the link negotiates correctly but effective bandwidth is still poor, run nvidia-smi topo --matrix and check whether the training process’s CPUs and memory sit on the same NUMA node as the GPU. Transfers crossing the CPU socket interconnect pay a latency and bandwidth penalty that looks like a link problem but is a placement problem.
Isolate. If hardware is suspected and you can afford it, drain the GPU from the scheduler and rerun on the remaining GPUs. If step time recovers to baseline, you have confirmed the straggler and contained it in one move.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`pcie.link.gen.gpucurrent` vs `gpumax` (under load)	Direct measure of generation downgrade	Current below max during active workloads
`pcie.link.width.current` vs `width.max`	Direct measure of width downgrade	x8 or less where x16 is expected, under load
PCIe throughput (dmon rxpci/txpci)	Shows whether the downgrade caps data flow	One GPU ceiling well below peers
PCIe replay error rate	Signal-integrity leading indicator	Count increasing between samples
Per-GPU `utilization.gpu` spread	Straggler waits for data; peers wait for straggler	One GPU consistently a few points below peers during compute phases
Training step time	The user-facing impact	Step time regression correlating with link state change
Collective/sync latency	Confirms the downgrade is gating the job	Latency above baseline with no other fault signal

Gate every link-state alert on workload activity. Without that gate, idle power management will page you for by-design behavior.

Fixes

Physical: riser, slot, seating

If the downgrade appeared mid-operation or replay errors are climbing, treat it as hardware. Power down the node, reseat the GPU and any riser card, inspect the connector for bent pins or contamination, and check adjacent cabling for airflow obstruction. This is disruptive: drain the node first. If the link recovers to full spec after reseating, keep the GPU under observation; a slot that degraded once tends to degrade again.

BIOS: lane allocation

If the link has been wrong since boot, check BIOS PCIe settings. Lane allocation shifts when cards are added or defaults change after an update. Verify the slot is configured for full width and expected generation rather than “Auto” negotiating down. A BIOS change requires a reboot; schedule it against the workload calendar.

Driver: known downgrade bugs

If the timeline matches a recent driver update and the downgrade persists under load with clean replay counters, check whether your driver branch is affected by a known link-training bug (the #1010 ASPM/Gen1 fallback case is the documented example). Validating against a different driver branch on identical hardware is the fastest confirmation. A driver rollback requires a driver reload; coordinate with the workload team.

NUMA placement

If the topology shows the GPU attached to a different NUMA node than the CPUs running the data pipeline, fix affinity: pin the process to CPUs and memory local to the GPU. This is a launcher or scheduler configuration change and needs no hardware intervention.

Workaround: run degraded or run without it

If you cannot fix the link immediately, you have two honest options. Keep the GPU and accept the reduced step time, which may still be a net win if the workload is not PCIe-sensitive. Or exclude the GPU and run at reduced capacity at full per-GPU speed. For bandwidth-sensitive data-parallel jobs, fewer healthy GPUs frequently beat more GPUs with a straggler. Measure both before committing.

Prevention

Baseline link state per node. Record expected gen/width per GPU at provisioning. Alert on deviation under load, not on absolute values.
Compare across GPUs, not just within them. The straggler pattern is invisible to per-GPU threshold alerting. Node- or fleet-level comparison of link state, throughput, and utilization spread is the detection mechanism.
Track replay error rate as a trend. A slowly rising replay counter is your early warning for riser and retimer degradation, weeks before the link drops.
Gate PCIe alerts on activity. Pair a utilization or power-draw condition with link-state alerts so idle power management never pages.
Re-verify after any chassis, BIOS, or driver change. Link state is negotiated state. Anything touching the physical or firmware layer can renegotiate it downward.

How Netdata helps

Per-second PCIe throughput per GPU makes the asymmetric ceiling visible immediately: one GPU capping below its peers during the same training step is the straggler fingerprint.
Link gen and width collected alongside utilization lets you gate link-state alerts on real workload activity, eliminating the idle-downgrade false positive that makes teams disable these alerts entirely.
Cross-GPU correlation on one node shows utilization spread, power draw, and throughput side by side, the exact comparison that separates a straggler from a uniformly configured platform.
PCIe replay error trending surfaces signal-integrity degradation as a slope, not a surprise, giving you a maintenance window instead of an incident.
ML-based anomaly detection on step-time-adjacent signals (power, utilization, throughput) catches the “everything looks fine but the job got slower” class of regression that threshold alerts miss because nothing crosses a threshold.