NVMe PCIe AER errors: correctable and uncorrectable transport-layer faults

You opened the kernel log because something felt off, and found lines like this repeating:

pcieport 0000:00:1b.0: AER: Corrected error received: 0000:01:00.0
nvme 0000:01:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)

That is PCI Express Advanced Error Reporting (AER), the transport layer underneath NVMe telling you the link between the root port and the drive is producing errors. These errors happen below the NVMe protocol: the drive’s SMART data can look perfectly healthy while the PCIe link is retransmitting constantly, quietly adding latency to every I/O.

The first decision is severity. AER splits errors into two classes. Correctable errors (RxErr, BadTLP, BadDLLP, and friends) are fixed by hardware retransmission: no data is lost, but each retry costs time, and a high rate means the physical link is marginal. Uncorrectable errors, fatal or non-fatal, mean the link layer could not recover: these precede link retraining, controller resets, or the device falling off the bus entirely.

This article covers how to read the kernel messages, pull the persistent counters from sysfs, and decide whether you are looking at a loose connector or an impending device loss. For the broader NVMe signal model, see how NVMe actually works in production.

What this means

PCIe is a packet-based serial transport. Data moves in TLPs (transaction layer packets) protected by CRCs and a link-layer replay mechanism. When a packet arrives corrupted, the receiver NAKs it and the sender retransmits. That retry is a correctable error: functionally invisible, temporally real. When retries are exhausted or the error class is not recoverable (a malformed TLP that breaks protocol, or a link training failure), the error is uncorrectable.

AER is the standardized mechanism by which PCIe devices and root ports report these events. On Linux, the pcieport AER driver logs them, and the kernel exposes per-device counters in sysfs. NVMe sits on top of all of this and knows nothing about it. The controller sees slightly slower completions; your application sees slightly higher latency; the filesystem sees nothing at all. That is why transport errors are the blind spot of NVMe monitoring: every standard tool watches the drive, nobody watches the bus.

The practical escalation model:

flowchart TD
  A[AER messages in kernel log] --> B{Severity?}
  B -->|Corrected| C[Check TOTAL_ERR_COR rate]
  B -->|Uncorrectable non-fatal| D[Check TOTAL_ERR_NONFATAL + dmesg context]
  B -->|Uncorrectable fatal| E[Device likely frozen or removed]
  C -->|Low or zero rate| F[Baseline noise - document and watch]
  C -->|Sustained rate| G[Physical layer fault: connector, cable, slot, retimer]
  D --> H[Driver-level recovery - watch for recurrence]
  E --> I[Check /sys/class/nvme/nvmeX/state and device presence]
  G --> J[Reseat drive, inspect slot, monitor counters after]

Common causes

A loose M.2 connector deserves special mention because it is the single most common root cause: vibration or thermal cycling breaks contact on a lane, and the link starts generating thousands of correctable errors while the drive reports perfectly healthy SMART data. Latency gets measurably worse and nothing else alerts.

Quick checks

All of these are read-only and safe to run during an incident.

# 1. Find the AER messages and their severity
dmesg -T | grep -i "aer\|pcie bus error" | tail -50

# 2. Read the persistent correctable-error counter for the device
cat /sys/class/nvme/nvme0/device/aer_dev_correctable

# 3. Read the uncorrectable counters (root-only)
sudo cat /sys/class/nvme/nvme0/device/aer_dev_fatal
sudo cat /sys/class/nvme/nvme0/device/aer_dev_nonfatal

# 4. Check whether the link has trained down
cat /sys/class/nvme/nvme0/device/current_link_speed
cat /sys/class/nvme/nvme0/device/max_link_speed
cat /sys/class/nvme/nvme0/device/current_link_width
cat /sys/class/nvme/nvme0/device/max_link_width

# 5. Check controller state (fatal errors can freeze or remove the device)
cat /sys/class/nvme/nvme0/state

# 6. Rule out drive-level health problems
nvme smart-log /dev/nvme0 | grep -E "critical_warning|media_errors|num_err_log_entries"

The three aer_dev_* files each list per-error-type counts followed by a total: TOTAL_ERR_COR, TOTAL_ERR_FATAL, and TOTAL_ERR_NONFATAL respectively. All three files are world-readable (the kernel exposes each as mode 0444), so the sudo in the examples below is not strictly required. All three appear on AER-capable devices, the NVMe endpoint included, whenever the kernel initialized AER for that device. If the files are missing entirely, check dmesg | grep -i aer at boot to see whether AER initialized; some platforms run in firmware-first mode where the BIOS handles errors and Linux never sees them.

Two things about these counters that trip people up:

• transport_err_cnt does not exist in sysfs. Operators regularly go looking for it. The AER counters are the only transport error signal the kernel exposes.
• The counters are cumulative and read-only. They are in-memory statistical counters with no reset mechanism in any kernel version, so they cannot be cleared short of a reboot; track deltas over time, not absolute values.

How to diagnose it

1. Classify the severity from the log. Look at the most recent AER lines in dmesg -T. severity=Corrected means retransmission is handling it. severity=Uncorrectable (Non-Fatal) means the error reached the driver layer but recovery was possible. severity=Uncorrectable (Fatal) means the hierarchy was reset; check immediately whether the device is still present (ls /sys/class/nvme/ and cat /sys/class/nvme/nvme0/state).

2. Quantify the correctable rate. Read aer_dev_correctable, note TOTAL_ERR_COR and the per-type breakdown (RxErr, BadTLP, BadDLLP are the usual suspects). Wait a fixed interval under typical load and read it again. A rate of zero to a handful per hour on a platform that has always done that is baseline noise. A sustained rate, or thousands accumulating, is a physical-layer problem.

3. Check for a trained-down link. Compare current vs. max link speed and width. A Gen4 x4 drive running at Gen3 x2 delivers a quarter of the bandwidth with no errors at the NVMe layer. AER errors plus a degraded link strongly suggests signal integrity problems rather than a drive problem.

4. Isolate drive vs. slot vs. platform. If you can schedule a maintenance window: reseat the drive first (this fixes the majority of correctable storms on M.2), then move it to a different slot. If errors follow the drive, the drive or its connector is at fault. If errors stay with the slot, suspect the riser, backplane, retimer, or root port.

5. Correlate with the NVMe layer. Pull nvme smart-log and check critical_warning, media_errors, and num_err_log_entries. Transport errors with a clean SMART log point at the physical link. Transport errors plus rising media errors or a nonzero critical warning means the drive itself is sick and the AER errors may be a symptom of failing device electronics.

6. Watch for the downstream pattern. Uncorrectable errors are the early edge of a worse failure: they precede link retraining, controller resets (“Resetting controller” in dmesg), and in the fatal case, device removal. If non-fatal uncorrectable errors are recurring, treat it as a pre-incident for the patterns in the reset-loop and device-disappeared guides linked below.

Metrics and signals to monitor

Fixes

Reseat and physically remediate

For a correctable-error storm, the highest-probability fix is physical: power down, remove the drive, inspect the connector for debris or damage, reseat with proper retention (M.2 screw fully seated, U.2/U.3 carrier fully latched), and reboot. Watch TOTAL_ERR_COR after. If the rate drops to zero, you are done. Do not attempt to reseat a live drive unless your chassis and configuration explicitly support NVMe hot-plug.

Isolate and replace the failing component

If reseating does not help, move the drive to a known-good slot. Errors that follow the drive mean drive-side electronics or connector damage: replace the drive. Errors that stay with the slot mean riser, cable, backplane, or root port: replace that component or re-route to a different slot. On systems with retimers in the backplane, errors affecting multiple drives behind the same backplane point there rather than at any single drive.

Address ASPM-related noise

If correctable errors correlate with idle-to-load transitions and the rate is otherwise low, test with PCIe active state power management disabled (a kernel boot parameter or BIOS setting, depending on platform) and observe whether the rate drops. This trades power efficiency for link stability; on latency-sensitive production NVMe that trade is usually correct, but validate on your hardware rather than applying it blindly.

Do not paper over uncorrectable errors

Boot parameters exist that suppress AER logging entirely (pci=noaer). That hides the messages; it does not fix the link. If you use it to stop a log flood from a known-noisy platform, keep the sysfs counters under monitoring so the signal is not lost. Fatal uncorrectable errors are never a logging problem: the device was reset or isolated, and the next occurrence may take it off the bus. Treat those as hardware incidents.

Prevention

• Baseline per platform. Some hardware generates a low background rate of correctable errors that the vendor considers normal. Record the baseline per machine model so you can tell noise from regression.
• Reseat after any chassis service. The highest-correlation event with new AER storms is someone opening the chassis. Add “check AER counters” to the post-maintenance checklist.
• Alert on rate, not on messages. AER messages in dmesg are ratelimited and easy to miss in log volume; the sysfs counters are not. Alert on any sustained increase in TOTAL_ERR_COR and any increment of the fatal or non-fatal totals.
• Track link speed/width continuously. A trained-down link is the silent companion of signal integrity problems. Alert on current != max.
• Correlate before replacing drives. AER errors with clean SMART data is a platform problem, not a drive problem. Swapping the drive without checking the slot wastes hardware and leaves the fault in place.

How Netdata helps

Netdata’s NVMe collector covers the drive side of this picture: SMART health, media errors, critical warning bits, temperature, and endurance. That is exactly the correlation you need when AER errors appear, because the first question is always “is the bus noisy or is the drive dying.” Specifically:

• nvme.device_media_errors_rate and nvme.device_error_log_entries_rate tell you whether the drive is reporting its own errors alongside the transport noise.
• nvme.device_critical_warnings_state shows whether the controller itself considers its reliability degraded.
• Block-layer and disk latency metrics let you quantify what the retransmissions are actually costing in I/O time, which is otherwise invisible.
• Temperature and throughput charts help confirm or rule out the adjacent failure patterns (thermal throttling, link degradation) that share symptoms with AER storms.

One honest limitation: Netdata’s NVMe collector does not read the AER sysfs counters, and the fatal/nonfatal files require root plus host sysfs access, so they are not visible in containers either. If you want AER rates in your dashboards alongside the NVMe signals, collect the TOTAL_ERR_* values with a small custom collector or node-exporter-style textfile job and graph the deltas.

Related guides

• How NVMe actually works in production: a mental model for operators
• NVMe controller reset loop: repeated resets from a firmware hang
• nvme nvme0: I/O timeout, Resetting controller: what an NVMe controller reset means
• NVMe device disappeared: nvme0: Removing and a drive that fell off the PCIe bus
• NVMe controller state not live: reading resetting, deleting, and dead from sysfs
• blk_update_request: I/O error, dev nvme0n1: reading NVMe I/O errors in the kernel log
• NVMe critical_warning is nonzero: decoding the SMART critical warning bitmask
• NVMe media_errors increasing: uncorrectable data-integrity errors on NAND
• NVMe error log entries growing: num_err_log_entries beyond media errors
• NVMe available spare below threshold: critical warning bit 0 and end-of-life wear
• NVMe available spare declining: watching the wear trajectory before the threshold
• NVMe endurance runway: projecting time-to-replacement from wear signals