NVMe PCIe link degraded: current link speed and width below maximum

An NVMe drive negotiates its PCIe link at boot, at hot-plug, and after some error events. When negotiation lands below the device’s capability, nothing breaks. The drive keeps serving I/O, SMART stays clean, and the filesystem sees no errors. The only symptom is a lower bandwidth ceiling: a Gen4 x4 drive running at Gen3 x2 delivers roughly one quarter of its rated throughput, and every application on top of it just runs slower.

That is why this fault is easy to miss. Teams burn time on application-level explanations for a throughput plateau while the drive has been degraded for weeks. The check itself is four sysfs reads.

This article covers how to confirm the downgrade, how to tell a hardware problem from a platform limit, and how to fix it without guesswork.

What this means

Every PCIe endpoint advertises a maximum link capability: a generation (Gen3, Gen4, Gen5) and a lane width (x1, x2, x4 for typical NVMe). During link training, the endpoint and the upstream port agree on the best speed and width both sides can sustain. If signal integrity is marginal, training backs off: fewer lanes, a lower generation, or both. The link still works. It just works slower.

Linux exposes both sides of this negotiation per device:

• max_link_speed and max_link_width: what the endpoint can do
• current_link_speed and current_link_width: what the link actually trained to

When current is below max, the link has degraded. There is no NVMe-level error for this because it is not an NVMe event. It is a PCIe transport event, one layer below the protocol your monitoring usually watches. I/O completes successfully, latency at low queue depth can look normal, and the observable change is that throughput saturates at a lower ceiling under load.

flowchart TD
  A[Throughput capped below spec] --> B{current_link < max_link?}
  B -- No --> C[Look elsewhere: thermal, GC, queue depth]
  B -- Yes --> D{AER correctable errors accumulating?}
  D -- Yes --> E[Physical layer: reseat, inspect connector, check retimer]
  D -- No --> F{Deliberate platform limit?}
  F -- Bifurcation x16 to 4x x4 --> G[Expected, not a fault]
  F -- BIOS gen limit or root port cap --> H[Fix BIOS setting or accept platform max]
  F -- Neither --> I[Retrain: cold boot or rescan, then re-check]
  I -- Still degraded --> E
  I -- Recovered --> J[Monitor AER and link state going forward]

Common causes

Two of these are not faults. Bifurcation is intentional: an x16 slot split into four x4 links gives each drive its maximum of x4, and current equals max per device. A BIOS or root-port generation cap means the endpoint shows its own higher capability as max while the platform holds the link lower. The Dell-style case, where an endpoint advertises Gen4 but the root port only supports Gen3, falls here. Distinguish “the link is below the drive’s capability” from “the link is below the platform’s capability” before touching hardware.

Quick checks

All read-only. Run as root where noted. Cold boots, reseats, and BIOS changes come later and are disruptive; schedule them.

# 1. Read current and max link parameters for the device
cat /sys/class/nvme/nvme0/device/current_link_speed
cat /sys/class/nvme/nvme0/device/max_link_speed
cat /sys/class/nvme/nvme0/device/current_link_width
cat /sys/class/nvme/nvme0/device/max_link_width

# 2. Cross-check with lspci: LnkCap is capability, LnkSta is current state
dev_bdf=$(basename "$(readlink -f /sys/class/nvme/nvme0/device)")
lspci -vvs "$dev_bdf" | grep -Ei 'LnkCap|LnkSta'

# 3. Check AER correctable and uncorrectable counters (root required)
cat /sys/class/nvme/nvme0/device/aer_dev_correctable
cat /sys/class/nvme/nvme0/device/aer_dev_fatal
cat /sys/class/nvme/nvme0/device/aer_dev_nonfatal

# 4. Look for link retrain and AER messages in the kernel log
dmesg | grep -Ei 'aer|link.*train|pcie.*error'

# 5. Check controller state is otherwise healthy
cat /sys/class/nvme/nvme0/state

# 6. Confirm the drive reports clean at the NVMe layer
nvme smart-log /dev/nvme0 | grep -Ei 'critical_warning|media_errors'

If the aer_dev_* files do not exist, AER may be disabled or not exposed for that device; absence is not proof of a healthy link.

A note on lspci: when LnkSta is below LnkCap, it appends “(downgraded)” for endpoint devices. That label compares the endpoint’s current state to its own capability only. It does not account for upstream limits, so “(downgraded)” can appear on a link that is running at the maximum the platform supports. Treat the label as a prompt to investigate, not a verdict.

How to diagnose it

1. Confirm the mismatch. Read the four sysfs attributes. If current_link_speed is below max_link_speed, or current_link_width is below max_link_width, the link is degraded. If they match, your throughput problem is elsewhere: thermal throttling, GC pressure, queue depth, or a workload change.

2. Quantify the impact. Per-lane bandwidth roughly doubles per PCIe generation. Dropping one generation on the same width halves peak bandwidth; halving the width halves it again. Gen4 x4 to Gen3 x2 is approximately a 4x reduction. Compare this against the throughput ceiling you actually observe under load to confirm the link, not something downstream, is the binding constraint.

3. Check AER counters. Read aer_dev_correctable (per-type fields plus TOTAL_ERR_COR), aer_dev_fatal, and aer_dev_nonfatal. On a healthy link these counters are zero. Accumulating correctable errors (RxErr, BadTLP, BadDLLP class events) mean the physical layer is marginal and retransmitting silently. Correctable errors that accumulate before or alongside a downgrade point hard at the physical path: connector, cable, riser, retimer. Correctable errors can also accumulate without a downgrade, so a zero counter does not fully exonerate the physical layer, but a rising counter is strong corroboration.

4. Rule out platform limits before blaming hardware. Check whether the slot is bifurcated, whether BIOS pins the slot to a lower generation, and whether the upstream root port or bridge tops out below the drive’s capability. If the root port caps at Gen3, a Gen4 drive will always show current below max, and that is by design.

5. Check for a kernel link-training regression. A regression was reported in the v6.5 to v6.13 window where a failed retrain could leave devices stuck at 2.5 GT/s; the draft notes partial mitigation in v6.12 and a full fix in v6.13, with Ubuntu 24.04’s GA kernel (6.8.x) affected and the 6.14 HWE kernel carrying the fix. Treat the exact range as something to validate against your kernel and distro changelog before acting on it.

6. Force a retrain and re-check. A cold boot (full power cycle, not a warm reboot) forces link training from scratch. This is disruptive; fail over workloads and confirm redundancy first. If the link comes back at full speed and stays there, the downgrade was a transient training failure. If it degrades again under load or after thermal events, the physical path is marginal. Some controllers drop speed after sustained writes and recover after a cold boot, which indicates a thermal or signal-integrity problem rather than permanent damage.

7. Isolate slot versus drive. If physically accessible, move the drive to a known-good slot during a maintenance window. Degradation follows the drive: suspect the drive’s connector or controller. Degradation stays with the slot: suspect the slot, riser, backplane, or retimer.

Metrics and signals to monitor

Severity guidance from the playbook: a mismatch between current and max link parameters is TICKET-level. Escalate when the reduced bandwidth is insufficient for the workload, or when AER uncorrectable errors appear.

One extreme worth calling out: a drive that should run Gen3 or Gen4 x4 and negotiates to 2.5 GT/s x1 (roughly 250 MB/s) is often a failing controller hitting training errors and backing off to the minimum survivable link. Treat that as a drive-failure signal, not a cabling nuisance, and correlate with the error log and SMART data.

Fixes

Reseat and force renegotiation

Power down, remove and reseat the drive (M.2 screw and connector, U.2/U.3 carrier, EDSFF sled), then cold boot. Insufficient insertion force and contaminated contacts are common, especially after chassis work. Tradeoff: requires downtime on that node; do it only after failover or in a maintenance window.

Fix BIOS/UEFI slot configuration

If a per-slot generation setting is forcing the link low, set it to the generation both the slot and drive support, or to Auto if Auto trains correctly. If bifurcation is the cause, decide whether the lane split is still what you want; it is a design choice, not a defect. BIOS changes require a reboot and should be treated like firmware work, not a quick toggle.

Replace the physical path

If AER counters keep rising after reseating, swap components in order of suspicion: cable or riser, then slot (move the drive), then backplane or retimer assembly. Retimers on server NVMe backplanes are a known failure point and are invisible to everything except link behavior and AER counters.

Update the kernel if you are in the regression window

If your kernel is in the reported v6.5 to v6.13 range and retrains leave devices stuck low, move to a kernel carrying the fix (v6.13 or later upstream, or a distro kernel with the backport). On Ubuntu 24.04, the 6.14 HWE kernel is described as resolving it; confirm against your release notes before planning the upgrade.

Accept and document platform limits

When the root port or platform genuinely caps the link, no fix exists short of hardware change. Record the expected negotiated values for that slot so future comparisons have a correct baseline.

Do not mask the problem

Increasing queue depth or switching I/O schedulers will not recover lost link bandwidth. ASPM (Active State Power Management) transitions can cause transient retrains and spurious correctable errors on some platforms; disabling ASPM is common for production NVMe, but treat that as a stability measure, not a fix for a marginal connector.

Prevention

• Baseline at provisioning. Record current_link_speed, current_link_width, and the max values for every drive at deployment. Degradation is only detectable against a known-good baseline.
• Check link state after any chassis work. Reseating, cable routing, riser swaps, and backplane maintenance are the top triggers. Add a link-state check to the post-maintenance checklist.
• Alert on the comparison, not the values. The check is current < max for either speed or width. A one-line script in cron or a collector plugin is enough; the data is four file reads.
• Track AER counters continuously. Correctable errors accumulating over days are the earliest warning of a link that will eventually retrain down. In healthy systems these counters stay at zero.
• Manage thermals. Sustained high temperatures degrade connector and trace signal integrity over time. Keep composite temperature comfortably below WCTEMP under sustained load.
• Beware hot-plug churn. Repeated hot-plug events increase the chance of a bad retrain, especially on kernels in the suspected regression window.

How Netdata helps

Netdata does not collect the PCIe link speed and width attributes or the AER sysfs counters by default, so the link check itself stays a manual or scripted step. Where Netdata shortens this investigation is the surrounding context:

• Throughput and IOPS per device: the visible symptom of a degraded link is a lowered bandwidth ceiling, which shows up directly in block device throughput charts under load.
• Composite temperature and thermal management transitions: helps confirm or rule out thermal contribution to signal degradation and retrains.
• Critical warning bits and media error rates: a fast way to confirm the NVMe layer is clean, which is what makes a link downgrade so deceptive.
• Unsafe shutdown and power cycle counters: hot-plug and power events are when retrains happen; correlating link degradation timing with these counters narrows the cause.
• Latency trends: latency that stays flat at low load but degrades sharply as throughput approaches the new, lower ceiling is the classic signature of a transport cap rather than a controller problem.

The practical workflow: notice the throughput ceiling in Netdata, confirm the NVMe layer is clean from SMART charts, then run the four sysfs reads to confirm the link downgrade and start the physical-layer investigation.

Related guides

• How NVMe actually works in production: a mental model for operators
• NVMe critical_warning is nonzero: decoding the SMART critical warning bitmask
• NVMe controller reset loop: repeated resets from a firmware hang
• nvme nvme0: I/O timeout, Resetting controller: what an NVMe controller reset means
• NVMe device disappeared: nvme0: Removing and a drive that fell off the PCIe bus
• blk_update_request: I/O error, dev nvme0n1: reading NVMe I/O errors in the kernel log
• NVMe error log entries growing: num_err_log_entries beyond media errors