$ guides / nvidia-gpu / nvidia-gpu-pcie-replay-errors ▌

Operations Guides

NVIDIA GPU PCIe replay counter rising: physical-layer signal integrity

The PCIe replay counter on one of your GPUs is climbing. Not a burst at boot, but a sustained rate of hundreds or thousands per second. Nothing has crashed yet: training still runs, nvidia-smi still answers, and there are no XID errors in dmesg. That is exactly why this signal gets ignored, and exactly why it should not be.

A PCIe replay is a link-layer retransmission. The GPU (or the root complex, switch, or retimer on the path) received a packet that failed its CRC check and asked for it again. Occasional replays are normal PCIe behavior. A sustained high rate means bits are being corrupted on the wire often enough that retransmission is routine, not exceptional. That is a physical-layer signal integrity problem: a cable, connector, slot, riser, or an overheating retimer or PCIe switch somewhere between the CPU and the GPU.

The operational risk is twofold. First, every replayed packet eats effective bandwidth even though the link still reports full width and speed. Second, a deteriorating link does not stay at “high replay rate” forever. After repeated failed transmissions the link retrains, and retraining at degraded signal quality is how you end up at x8 instead of x16, or Gen3 instead of Gen4, with half your host-to-device bandwidth gone. An accelerating replay rate is the early warning before that downgrade, and watching it today is how you avoid an XID 79 (“GPU has fallen off the bus”) next month.

What this means

PCIe handles transient errors by design. The data link layer ACKs good packets and NAKs bad ones, and the transmitter replays anything that was NAKed. The GPU exposes a monotonic counter of these replays. Because the counter is cumulative since boot or driver load, the absolute number is meaningless on its own. What matters is the rate: replays per second, sampled over minutes and hours.

The progression you are watching for:

flowchart LR
  A["Signal integrity degrades
cable, connector, slot, retimer"] --> B["Replay rate rises
hundreds to thousands per sec"]
  B --> C["Effective bandwidth drops
retransmissions eat throughput"]
  B --> D["Repeated failures trigger
link retraining"]
  D --> E["Link downgrades
gen or width reduced"]
  E --> F["Sustained degradation risks
XID 79: GPU off the bus"]

The replay counter sits at step B. Everything downstream of it is worse and harder to fix mid-incident. Catching the problem here means a planned reseat or cable swap during a maintenance window instead of an unplanned node outage.

One important distinction: the replay counter and the link speed/width fields measure different things. A GPU can sit at full Gen4 x16 with a replay rate in the thousands per second. The link negotiated fine; it just cannot hold a clean signal. Do not let healthy-looking pcie.link fields talk you out of a replay problem.

Common causes

Cause	What it looks like	First thing to check
Contaminated or loose slot/connector	Replay rate climbing over weeks; one GPU affected	Physical inspection; reseat the card
Cable or riser degradation (riser-based chassis)	High replay rate on GPUs behind a specific riser; often temperature-sensitive	Swap the riser/cable with a known-good slot and see if the problem follows
Overheating retimer or PCIe switch	Replay rate correlates with chassis temperature or workload intensity	Compare replay rate against temperature trends; check switch/retimer temps via platform sensors
Motherboard or slot damage	GPU works elsewhere; slot misbehaves with any card	Move the GPU to a different slot
Thermal-induced marginality	Replay rate rises under sustained load, falls at idle	Replay rate vs. workload timeline; fan and intake temps
ASPM or power-management transitions	Replay bursts correlated with idle/load transitions rather than steady state	Kernel command line and AER logs

Dust in the PCIe slot, cable and riser issues, and board-level slot problems account for most cases in practice. In multi-GPU servers with PCIe switches, remember the GPU-to-switch link and the switch-to-CPU link are separate segments. A replay counter on the GPU tells you the GPU’s own link segment is retransmitting; the upstream segment needs AER logs to characterize.

Quick checks

All of these are read-only and safe to run on a production node.

# 1. Current replay counter (cumulative since boot/driver load)
nvidia-smi -i 0 -q -d PCIE | grep -i replay

# 2. Watch the rate of change, not the absolute value
watch -n 5 'nvidia-smi -i 0 -q -d PCIE | grep -i replay'

# 3. Current vs max link gen and width (check under load, not at idle)
nvidia-smi --query-gpu=pcie.link.gen.gpucurrent,pcie.link.gen.gpumax,pcie.link.width.current,pcie.link.width.max --format=csv

# 4. Kernel view: PCIe Advanced Error Reporting for this device
dmesg -T | grep -i "aer\|pcie" | grep -i "error\|corrected"

# 5. Link capability vs status from the kernel's perspective
lspci -vv -s $(lspci | grep -i nvidia | head -1 | awk '{print $1}') | grep -i "lnk"

# 6. Any XID events, especially 79 (GPU off the bus)
dmesg -T | grep -i "NVRM: Xid"

# 7. If DCGM is deployed, stream the replay counter (field 411) as a rate
dcgmi dmon -e 411 -c 10 -d 1000

Notes on interpreting what you see:

The replay counter is monotonic. Two samples five seconds apart give you a per-second rate. A handful of replays over days is noise. Sustained hundreds to thousands per second is a deteriorating physical connection.
Idle link downgrade is normal. NVIDIA drivers downclock the PCIe link at idle by design, and a Gen5 GPU sitting at Gen2 while idle is not a fault. Compare gpucurrent to gpumax only while the GPU is doing real work.
lspci shows LnkCap (what the device can do) and LnkSta (what it actually trained to). A gap between them under load confirms a downgrade has already happened.
AER correctable errors in dmesg are the kernel’s view of the same physical-layer problem. The error type in the AER message (receiver error, bad TLP, bad DLLP) tells you which direction and which layer is corrupting, which helps localize the segment.

How to diagnose it

Establish the rate, not the count. Sample the replay counter every 5 to 10 seconds for several minutes. Compute replays/second. A one-time burst during boot or driver load is normal; a sustained nonzero rate under load is the finding.
Confirm the counter is still moving under steady workload. Replay rate that scales with PCIe traffic (data loading, checkpoint writes, host-to-device copies) is consistent with a marginal link. Replay bursts only at idle-to-load transitions point at power management instead of pure signal integrity.
Cross-reference AER. dmesg -T | grep -i aer on the host shows which device and which error type the kernel sees. Correctable errors logged against the GPU’s bus address at the same time the replay counter jumps confirm the physical layer is the problem, not the driver.
Check link gen and width under load. If gpucurrent or width is already below max during active work, the link has already retrained at least once. LnkSta vs LnkCap in lspci -vv confirms from the host side.
Correlate with temperature. Overlay the replay rate against GPU die temperature and, if your platform exposes them, chassis intake and switch/retimer temperatures. A replay rate that tracks temperature points at a marginal connector or an overheating retimer. One that is flat regardless of temperature points at physical damage or contamination.
Compare across GPUs. On a multi-GPU node, is it one GPU or several behind the same switch or riser? One GPU: suspect the card, its connector, or its slot. Several GPUs sharing a riser, cable bundle, or switch: suspect the shared component.
Check history. Pull the counter trend over days or weeks. An accelerating rate is the decisive signal: physical connections that are failing get worse, not better. A flat, low, stable rate on an otherwise healthy GPU can be documented and watched rather than acted on.
Physical intervention last, and planned. Reseat the card, clean the slot, swap the riser or cable, or move the GPU to a known-good slot during a maintenance window. If the problem follows the GPU, the card is the suspect. If it stays with the slot, the board or riser is. None of this is safe to improvise on a live production node; drain the node first.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
PCIe replay rate (replays/sec, derived from the monotonic counter; DCGM field 411)	The primary signal. Rate of link-layer retransmissions	Sustained hundreds to thousands/sec; any accelerating trend
`pcie.link.gen.gpucurrent` vs `gpumax` under load	Detects a gen downgrade after retraining	Current below max during active workloads
`pcie.link.width.current` vs `max` under load	Detects a width downgrade	Width below max during active workloads
PCIe throughput (rxpci/txpci via `nvidia-smi dmon -s t`)	Replays eat effective bandwidth before any downgrade	Throughput ceiling below what the workload used to achieve
AER correctable error count in dmesg	Kernel-side confirmation and error-type detail	Growing count against the GPU’s bus address
XID events (especially 79)	The failure you are trying to preempt	Any XID 79; XIDs on the same GPU that shows replay growth
GPU and chassis temperatures	Thermal marginality mechanism	Replay rate tracking temperature under load

Sampling matters here. Replay bursts, AER bursts, and retrain events are short-lived. Minute-resolution collection will smooth a real problem into invisibility. Use 10-second-or-faster sampling for the replay rate and link state.

If you run DCGM, note that the background health monitor has its own replay-rate threshold and that field naming and counter semantics for PCIe replays have changed across DCGM releases. On older DCGM versions, policy violations could fire from the cumulative counter rather than new activity, producing false positives. Verify your DCGM version’s behavior before trusting its PCIe alerts, and prefer computing your own rate from field 411 deltas.

Fixes

Fixes map to physical causes, and almost all of them require hands on the machine. There is no software fix for a dirty connector.

Reseat and clean the card and slot

Power down, remove the GPU, clean the slot and card edge connector, and reseat firmly. Dust in the PCIe slot is a documented cause and the cheapest fix available. Verify the replay rate returns to baseline over the following days under comparable load. If the rate resumes climbing, the contamination was not the whole story.

Replace the riser or cable

In riser-based and cabled chassis (common in dense multi-GPU servers), the riser PCB and its cabling are wear items and a frequent replay source. Swap with a known-good riser. If the elevated replay rate follows the riser to the new GPU position, retire the riser. If it stays with the GPU, suspect the card.

Move the GPU to a different slot

This is the isolation test. Problem follows the card: card-side connector or the GPU itself. Problem stays with the slot: motherboard slot, solder joints, or the upstream switch port. A slot that produces replays with any card should be flagged and the board escalated to the hardware vendor.

Address thermal marginality

If the replay rate tracks temperature, fix the cooling before replacing hardware: verify chassis fan health, clear airflow obstructions, and check intake temperatures. Retimers and PCIe switches have their own thermal limits and can degrade signal quality when hot even while the GPU die temperature looks fine.

Do not mask the symptom

Forcing a lower PCIe generation in firmware or disabling link power management can reduce replay counts on a marginal link, but it does so by lowering signaling rates or removing transitions, not by fixing the physics. Treat these as diagnostic steps or temporary risk reduction with a documented expiry, not as fixes. A link that only works at reduced speed is telling you something; listen to it.

What not to do

Do not reload the driver or reboot as a first response. The replay counter resets, your trend data resets, and the physical problem is untouched. Reboot only when you have exhausted read-only diagnosis or when the link has already failed. And do not ignore a high but stable replay rate because “it has always been like that”: establish the fleet baseline so you know what “normal” actually is for your hardware.

Prevention

Baseline every node. Record the steady-state replay rate, link gen, and width per GPU after commissioning. Deviations from a known baseline are far easier to act on than absolute thresholds.
Alert on rate and acceleration. Page-worthy is rare here, but a ticket on sustained replay rate above baseline, and an escalated ticket on a rate that is accelerating week-over-week, catches physical degradation while it is still a maintenance task.
Gate link-state alerts on workload activity. Idle PCIe downgrade is by design. Only compare current vs. max gen and width while the GPU is loaded, or you will train the team to ignore the alert.
Correlate with AER and temperature in one view. Replay rate plus AER error type plus chassis temperature is what turns “counter went up” into “riser 2 is heat-soaked.”
Sample fast. Sub-10-second collection on replay rate and link state. Short retrain events and replay bursts vanish at minute resolution.
Track per-node repair history. A slot that has produced replay growth twice is a slot with a hardware problem, not a coincidence.

How Netdata helps

Netdata’s NVIDIA GPU monitoring collects per-GPU PCIe link state and throughput at per-second resolution, so retrain events and replay-driven bandwidth loss are visible instead of averaged away.
The replay counter is monotonic; Netdata’s per-second collection makes deriving a replays-per-second rate straightforward, which is the number that actually matters.
Correlating replay rate with PCIe throughput on the same GPU shows bandwidth being consumed by retransmission before any gen or width downgrade appears.
Per-GPU temperature alongside link state exposes the thermal-marginality pattern: replay rate rising and falling with heat rather than with traffic.
Comparing link gen and width across all GPUs on a node makes the asymmetric case obvious: the one GPU training below max under load stands out against its peers.
Long retention on these signals gives you the acceleration trend, which is the real early warning, and the before/after evidence for any reseat, cable swap, or RMA decision.