NVMe ASPM latency spikes: PCIe power states adding first-request latency

You are chasing tail latency on an NVMe-backed service. The p99 graph shows sporadic spikes: tens of microseconds added to reads that should complete in well under 100us. The spikes do not correlate with load, temperature, media errors, queue depth, or garbage collection. SMART is clean. The block layer looks clean. The one pattern you can find: the slow I/Os tend to be the first request after the device sat idle.

That pattern is the signature of PCIe Active State Power Management (ASPM). When the NVMe link has no traffic, ASPM lets it drop into a low-power state. The next I/O has to wake the link before it can be serviced, and the wake-up time lands directly on that first request’s latency. There are no errors, no log lines, nothing in SMART: just tail latency that looks random until you line it up against idle gaps.

This article covers how to recognize ASPM-induced latency, how to confirm it on a live system, how it differs from NVMe APST (a separate power mechanism with similar symptoms), and how to turn it off without breaking power management for the rest of the machine.

What this means

ASPM lets an idle PCIe link save power by transitioning out of the fully-active L0 state. The states that matter for NVMe latency:

• L0s: shallow idle, fast entry and exit.
• L1: deeper idle, the link is effectively off until something needs it.
• L1.1 / L1.2: L1 substates with progressively lower power and longer exit latency. L1.2 powers down high-speed circuits; exit involves the device’s advertised T_CommonMode and T_PowerOn delays, which can run to tens of microseconds on some endpoints.

When the host submits a command after an idle period, the link must return to L0 before the transaction moves. That exit latency is added to the first request and nothing else. Under steady load the link never idles, so the symptom disappears under load and reappears exactly when traffic is bursty or low-rate. That is why it reads as random tail latency rather than a consistent slowdown.

A second, independent mechanism produces almost identical symptoms: NVMe APST (Autonomous Power State Transition), where the NVMe controller itself drops into a low-power operational state. Disabling one does not disable the other, and confusing the two is the most common way operators “fix” the problem and change nothing. More on that below.

A smaller contributor in the same family: deep CPU C-states add roughly 100-500ns when the core handling the completion has to wake up. That is an order of magnitude below ASPM L1.2 exit, but on very latency-sensitive paths it stacks.

flowchart TD
  A[Workload goes idle] --> B[PCIe link enters L1/L1.2 via ASPM]
  B --> C[Next I/O submitted]
  C --> D[Link exit latency added to first request]
  D --> E[Tail latency spike with no errors]
  A2[Workload goes idle] --> B2[NVMe controller enters low power state via APST]
  B2 --> C
  A3[Core goes idle] --> B3[Deep CPU C-state]
  B3 --> C

Common causes

Quick checks

All of these are read-only.

# Check whether ASPM is enabled on the NVMe device's link
# Look for "LnkCtl: ASPM L1 Enabled" (or L0s) in the capabilities section
lspci -vv -s $(basename $(readlink -f /sys/class/nvme/nvme0/device))

# Check the current ASPM policy
cat /sys/module/pcie_aspm/parameters/policy

# Check the NVMe controller's autonomous power state setting
nvme get-feature /dev/nvme0 -f 0x0c

# Check the APST latency tolerance the kernel applies (0 disables APST)
cat /sys/module/nvme_core/parameters/default_ps_max_latency_us

# Look for the kernel's power-management hint in the logs
journalctl -k --no-pager | grep -i "power saving mode\|default_ps_max_latency"

For confirmation, correlate block-layer latency against idle time. /sys/block/nvme0n1/stat only gives you averages, which hide this entirely: a ten-microsecond spike on one request per minute does not move an average. You need per-I/O visibility, for example biolatency or a bpftrace one-liner on block_rq_complete, sampled during a low-traffic window. If the outlier I/Os line up with idle gaps of a few hundred milliseconds or more, power-state exit is the likely cause.

How to diagnose it

1. Establish that the symptom is first-request-after-idle latency. Capture per-I/O latency during a quiet period (eBPF/biolatency, or application-side latency logging with timestamps). Plot latency against time-since-previous-I/O. ASPM shows a clear step: requests after a gap are slow, requests in a stream are fast. GC stalls and thermal throttling do not produce this idle-gap correlation; they correlate with write pressure and temperature instead.

2. Check the link state configuration. Run the lspci -vv check above. If LnkCtl shows ASPM Disabled, ASPM is not your cause; look at APST or elsewhere. Note that LnkCap shows what the link supports and LnkCtl shows what is currently enabled. You want the latter.

3. Check APST independently. nvme get-feature /dev/nvme0 -f 0x0c shows whether the controller is allowed to autonomously transition power states. A controller can sit in a deep non-operational power state even with ASPM fully off, because APST is controller-internal and does not depend on the PCIe link state.

4. Rule out the lookalikes. Idle-correlated first-request latency has a few neighbors worth excluding before you change anything: post-idle GC backlog (the first write after a long idle triggers deferred garbage collection), thermal behavior after idle (rare), and C-state effects (too small to explain multi-microsecond spikes). The distinguishing test: disable ASPM on one node, keep it on another, and compare p99 after-idle latency.

5. If you see actual failures, not just latency, check the kernel log for the driver’s own hint. On kernels since roughly 5.18, when an NVMe controller stops responding (CSTS reads as 0xffffffff) the driver prints “Does your device have a faulty power saving mode enabled?” and suggests nvme_core.default_ps_max_latency_us=0 pcie_aspm=off. Treat that message as a heuristic, not a diagnosis; the same symptom comes from firmware bugs, power delivery problems, and PCIe signal integrity issues.

Metrics and signals to monitor

Fixes

Disable ASPM for latency-sensitive devices

The cleanest runtime control is the kernel policy:

# Set ASPM policy to performance (disables ASPM link power management)
# Note: this is system-wide, affecting every PCIe device, not just NVMe
echo performance > /sys/module/pcie_aspm/parameters/policy

Persist it across reboots with the kernel parameter pcie_aspm.policy=performance. Two caveats that bite people:

• Do not use pcie_aspm=force. It enables ASPM even on devices that do not claim support and can lock up systems. It is the opposite of what you want here.
• Changing the policy can fail with “Operation not permitted” if the firmware owns ASPM configuration and has it disabled or locked. In that case the BIOS/UEFI setting is the only control.

The pcie_aspm=off trap

pcie_aspm=off does not reliably mean “disable ASPM.” The documented semantics are that it tells the kernel to leave ASPM configuration untouched, which means firmware-enabled ASPM stays on. If you added pcie_aspm=off to your cmdline years ago and assumed the link was in L0 permanently, verify with lspci -vv rather than trusting the parameter. To actually force the link active, use pcie_aspm.policy=performance and, where available, disable ASPM in the platform firmware.

Disable NVMe APST separately

If the controller itself is the latency source, cap its autonomous power state latency at the module level:

# Runtime: allow only power states with 0us exit latency (effectively disables APST)
echo 0 > /sys/module/nvme_core/parameters/default_ps_max_latency_us

Persist as nvme_core.default_ps_max_latency_us=0 on the kernel cmdline. This is also the first half of the workaround the kernel itself suggests when it suspects power-state-related controller hangs, and it has resolved recurring I/O errors on drives with buggy power-state implementations. Community reports are heavily weighted toward consumer hardware; enterprise drives with mature firmware are less affected.

Tradeoffs

Disabling ASPM and APST raises idle power draw. On a single drive in a server this is typically a watt or two per device; on a dense NVMe box or a laptop it is measurable. That is the actual trade: idle power versus first-request tail latency. For latency-sensitive databases and queues, tail latency wins. For capacity tier storage and batch workers, leave power management on. A per-device approach (performance policy on latency-critical nodes only) is usually the right fleet posture.

Prevention

• Baseline the setting at provisioning. Record ASPM policy and APST configuration alongside firmware version and PLP capability in your drive inventory. This class of latency is invisible until someone graphs tail latency against idle gaps, which usually happens during an incident.
• Set the policy by role, not by default. Latency-sensitive hosts get pcie_aspm.policy=performance and nvme_core.default_ps_max_latency_us=0 baked into their image; everything else keeps platform defaults.
• Graph tail latency, not averages. Block-layer stats provide only averages, and averages cannot see this failure. Per-I/O latency histograms belong in the NVMe monitoring baseline on any host where tail latency matters.
• Re-verify after kernel or firmware changes. ASPM behavior has shifted across kernel releases (including D3cold handling changes and the pcie_aspm parameter semantics clarification), and firmware updates can re-enable link power states. Include the lspci LnkCtl check in post-upgrade validation.

How Netdata helps

• Netdata’s NVMe collector surfaces the health and transport signals that sit next to this symptom: AER-adjacent PCIe errors, controller resets, and SMART health, so you can quickly exclude the loud failure modes (throttling, media errors, resets) and isolate the quiet one.
• Per-second block-level I/O latency and throughput charts show the bursty-traffic pattern ASPM punishes: latency outliers appearing exactly when IOPS dips.
• Correlating NVMe latency against CPU idle behavior on the same dashboard helps separate link power-state exit from C-state wake-up effects.
• Alerting on controller state transitions and kernel-log reset patterns catches the escalation case, where a buggy power-state implementation progresses from latency spikes to controller hangs.

Related guides

• NVMe high I/O latency: reading block-layer latency and the outliers that matter
• How NVMe actually works in production: a mental model for operators
• NVMe controller reset loop: repeated resets from a firmware hang
• nvme nvme0: I/O timeout, Resetting controller: what an NVMe controller reset means
• NVMe controller state not live: reading resetting, deleting, and dead from sysfs
• blk_update_request: I/O error, dev nvme0n1: reading NVMe I/O errors in the kernel log
• NVMe device disappeared: nvme0: Removing and a drive that fell off the PCIe bus
• NVMe critical_warning is nonzero: decoding the SMART critical warning bitmask
• NVMe available spare below threshold: critical warning bit 0 and end-of-life wear
• NVMe available spare declining: watching the wear trajectory before the threshold
• NVMe endurance runway: projecting time-to-replacement from wear signals
• NVMe error log entries growing: num_err_log_entries beyond media errors