$ guides / nvme / nvme-high-io-latency ▌

Operations Guides

NVMe high I/O latency: reading block-layer latency and the outliers that matter

Latency complaints about NVMe arrive as “the database is slow” or “queries time out randomly.” By the time the report reaches you, the mean latency from iostat often looks fine, because NVMe mean latency is almost always fine. The damage comes from the tail: the occasional 10 ms or 1 s outlier that breaks a request deadline and cascades into application timeouts.

The block layer gives you averages, and averages hide exactly the outliers that matter. This guide covers how to read the block-layer counters, how to get the actual latency distribution, and how to map a latency spike to its cause: thermal throttling, garbage collection pressure, PCIe transport degradation, or a controller on its way to a reset.

One thing up front: if the machine booted in the last few minutes, wait. Post-boot elevated latency for 1-5 minutes is normal while the controller’s flash translation layer loads mapping tables into DRAM. Do not alarm on that window.

What this means

“High NVMe latency” is a symptom with at least five distinct root causes, and they need different responses. The block layer measures the time from I/O submission to completion, which includes block-layer queuing, NVMe driver processing, controller firmware processing, and NAND access time. A spike can originate at any of those stages:

Host side: queuing at high queue depth, a misconfigured I/O scheduler, PCIe link power management (ASPM) transitions.
Transport: PCIe correctable errors silently retransmitting, or a link retrained to a lower speed or width.
Controller: thermal throttling, garbage collection competing with host I/O, firmware hangs that precede a controller reset.
Media: read retries on degrading NAND, showing up as sporadic read latency on cold data.

Under light load, a healthy drive delivers roughly 70-150 us for 4K random reads and 10-30 us for writes at the device level. Mean latency within 2x of your baseline is normal. Mean at 2-5x baseline, or p99 above 1 ms for reads, deserves investigation. Any single I/O taking more than 1 second indicates a controller stall, a garbage collection storm, or an imminent controller reset. Treat that as page-worthy.

Common causes

Cause	What it looks like	First thing to check
Thermal throttling	Gradual latency increase that tracks temperature; recovers when load drops	Composite temperature vs throttling history in SMART
GC stall / write cliff	Sudden 50-90% throughput drop and latency spike with normal temperature	Drive fill level, TRIM/discard configuration
PCIe link degradation	Uniformly higher latency, capped bandwidth, no NVMe errors	current vs max link speed/width, AER counters
SLC cache exhaustion	Step-function write latency jump during sustained sequential writes	Does latency recover after idle time
Controller hang (pre-reset)	Latency climbing toward the I/O timeout, then a reset in the kernel log	dmesg for timeout and reset messages
APST power-state transitions	Periodic latency spikes at low load, otherwise healthy drive	Whether autonomous power state transitions are enabled
Post-boot FTL load	High latency for 1-5 minutes after boot, then normal	Uptime; expected, no action
Queue depth effects	Latency 3-5x higher at QD32 than QD1 on a healthy drive	Compare latency at matched queue depths

Quick checks

All read-only and safe to run during an incident.

# 1. Snapshot the block-layer counters (cumulative since boot)
cat /sys/block/nvme0n1/stat

# 2. Instantaneous queue depth (field 9)
awk '{print $9}' /sys/block/nvme0n1/stat

# 3. Average read and write latency over 5 seconds
S1=$(cat /sys/block/nvme0n1/stat); sleep 5; S2=$(cat /sys/block/nvme0n1/stat)
echo "$S1" | awk '{print "r_ios="$1" r_ms="$4" w_ios="$5" w_ms="$8}'
echo "$S2" | awk '{print "r_ios="$1" r_ms="$4" w_ios="$5" w_ms="$8}'
# avg_read_ms = (delta field4) / (delta field1); avg_write_ms = (delta field8) / (delta field5)

# 4. Latency distribution, not averages (requires bcc-tools or bpftrace)
biolatency -D 10 1   # per-disk histograms for all disks; read the nvme0n1 section

# 5. Controller temperature and thermal history
nvme smart-log /dev/nvme0 | grep -i "temperature\|thm_temp"

# 6. Kernel log for timeouts and resets
dmesg | grep -i "nvme" | grep -i "timeout\|reset"

# 7. PCIe link state: current vs maximum
cat /sys/class/nvme/nvme0/device/current_link_speed
cat /sys/class/nvme/nvme0/device/max_link_speed
cat /sys/class/nvme/nvme0/device/current_link_width
cat /sys/class/nvme/nvme0/device/max_link_width

# 8. PCIe correctable errors (silent retransmissions)
cat /sys/class/nvme/nvme0/device/aer_dev_correctable  # absent if AER unsupported

# 9. I/O scheduler (should be "none" for NVMe)
cat /sys/block/nvme0n1/queue/scheduler

# 10. Drive fill level and TRIM state
df -h /your/mountpoint
lsblk -D | grep nvme

How to diagnose it

Work from the symptom shape to the cause. The shape of the latency curve is the strongest discriminator you have.

Rule out the post-boot window. Check uptime. If the host booted less than 5 minutes ago, the FTL is still loading mapping tables into DRAM and elevated latency is expected. Re-check after the window.
Establish whether the tail or the mean is the problem. Compute average latency from /sys/block/nvme0n1/stat deltas over several intervals, then run biolatency -D 10 1 and compare. If the average is normal but the histogram shows a second mode or outliers above 10 ms, you have a tail-latency problem, and the average will never show it. This is the single most common instrumentation mistake: /sys/block gives averages, and the p99/p999 tail is what actually breaks SLAs.
Check for any I/O over 1 second. In the biolatency histogram, anything in the >1 s bucket means a controller stall, a GC storm, or a pending controller reset. Check dmesg immediately for timeout messages; the default I/O timeout is typically 30 seconds (/sys/module/nvme_core/parameters/io_timeout), and a hang that long ends in a reset. See nvme nvme0: I/O timeout, Resetting controller.
Correlate with temperature. Pull composite temperature and the thermal management transition counters (thm_temp1_trans_count, thm_temp2_trans_count). If latency climbs as temperature climbs and recovers when load is removed, the controller is throttling. No media errors, no error log entries.
Correlate with write pressure and fill level. If latency spikes are periodic, temperature is normal, and write throughput drops 50-90% during the spikes, the FTL’s garbage collection is competing with host I/O. Check drive fill level (the pattern turns severe above roughly 80-90% full) and confirm TRIM is actually running. Without TRIM, the FTL treats the whole device as live data and GC runs at maximum pressure continuously.
Check the PCIe transport. Compare current vs max link speed and width. A Gen4 x4 device retrained to Gen3 x2 delivers one-quarter the bandwidth with zero NVMe-level errors, and every retransmission from a marginal link shows up as latency. Non-zero and growing aer_dev_correctable counters confirm physical layer degradation. Latency from this cause is uniform, not spiky.
Match queue depth before concluding anything. Latency naturally rises with queue depth: a jump from QD1 to QD32 raises average latency 3-5x on a perfectly healthy drive. Sample field 9 of /sys/block/nvme0n1/stat frequently and compare latency only at similar queue depths.
Consider power management. If spikes happen at low load and everything else is clean, autonomous power state transitions (APST) or PCIe ASPM are prime suspects. The controller or the link drops into a low-power state and the wake-up latency lands on the next I/O. See the power-state fix below.

flowchart TD
  A[Latency outlier observed] --> B{Host booted < 5 min ago?}
  B -- yes --> C[Normal: FTL table load, re-check later]
  B -- no --> D{Tail or mean? Run biolatency}
  D -- "any I/O > 1s" --> E[Check dmesg for timeouts - controller stall or imminent reset]
  D -- "p99 elevated, spiky" --> F{Temperature rising with latency?}
  F -- yes --> G[Thermal throttling]
  F -- no --> H{Write-heavy, drive >80% full?}
  H -- yes --> I[GC stall / write cliff]
  H -- no --> J{Link speed/width degraded or AER errors?}
  J -- yes --> K[PCIe transport degradation]
  J -- no --> L[APST/ASPM power-state transitions]

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Block stat deltas (ms_reading, ms_writing / ops)	Average read/write latency per interval	Mean 2-5x baseline sustained
Latency histogram (biolatency/BPF)	The only way to see p99/p999 and multi-modal distributions	Any I/O over 1 s; p99 > 1 ms reads
Composite temperature + TMT transitions	Confirms or rules out thermal throttling	Latency and temperature rising together
ios_in_progress (stat field 9)	Queue depth context for every latency reading	QD rising while latency rises (saturation)
PCIe link current vs max speed/width	Silent bandwidth cap with no errors	current < max on either axis
AER correctable error counters	Retransmissions that look like latency	Any sustained non-zero rate
Kernel log timeout/reset messages	Controller hangs surface here before anywhere else	Any timeout; resets more than once per week
Drive fill level + discard enabled	GC pressure is a function of how full the FTL thinks the drive is	Above 80-90% full, or no TRIM configured

Fixes

Thermal throttling

Reduce write load immediately to let the controller cool, then fix the thermal path: verify airflow, fan function, and heatsink contact. M.2 drives without heatsinks throttle under sustained load routinely; this is a hardware/environment fix, not a tuning fix. Track the thermal management transition counters afterward to confirm the drive is no longer entering throttled states.

GC pressure and the write cliff

Stop or throttle the write workload if latency is SLA-breaking. Run fstrim on the mounted filesystem for immediate relief, and make sure either continuous discard or a scheduled fstrim is configured permanently. Keep more than 15-20% free space so the FTL has headroom for garbage collection. On a nearly full consumer drive under an enterprise workload, no configuration change fixes this; the fix is a correctly classed drive or more overprovisioning.

APST or ASPM power-state spikes

Disable autonomous power state transitions on the controller:

# Disable APST on the controller (feature 0x0c)
nvme set-feature /dev/nvme0 -f 0x0c --value=0

This changes device behavior and does not persist across reboots or controller resets; reapply it at boot via a udev rule or systemd unit. It trades idle power savings for consistent latency, which is usually the right trade on latency-sensitive database hosts. Disabling ASPM at the platform level is a separate, boot-time or firmware-level change; test before applying broadly.

PCIe link degradation

Reseat the drive and inspect connectors, cables, and retimers. Power cycle to force link renegotiation and re-check current vs max link parameters. If a different slot resolves it, the slot or riser was the problem, not the drive.

Controller hang pattern

If latency climbs toward the timeout and resets repeat, check the firmware version (nvme id-ctrl /dev/nvme0 | grep fr) against known issues and plan a firmware update. A single reset may be transient; repeated resets in a loop indicate a firmware or hardware fault. See NVMe controller reset loop.

Prevention

Baseline latency at your real queue depths. Absolute thresholds mislead; a 24-hour baseline at production QD is the reference that makes deviations meaningful.
Alert on the tail, not the mean. If you only collect /sys/block averages, add BPF-based histogram collection and alert on p99, not average.
Alert on any I/O over 1 second. That threshold catches controller stalls and GC storms regardless of what the average says.
Track temperature and TMT transition counters continuously. Thermal throttling leaves no errors; the transition counters are the only record that it happened.
Verify TRIM and fill-level policy at provisioning. Chronic write-latency spikes from GC pressure are usually a missing fstrim schedule discovered months late.
Record link speed/width at deployment and alert on any downgrade. Silent retraining persists indefinitely because nothing errors.
Suppress alerts for the first 5 minutes after boot so the FTL load window does not page anyone.

How Netdata helps

Netdata collects NVMe SMART signals continuously, so a latency spike can be correlated against composite temperature, thermal management transition counts, and warning/critical temperature time in the same dashboard, confirming or ruling out throttling in one view.
Critical warning bits are exposed per-bit, so bit 1 (temperature) during a latency event is distinguishable from bit 0 (spare) or bit 2 (reliability degraded), each of which has a different latency signature.
Media error and error log entry rates surface read-retry latency on degrading NAND, the case where read latency is sporadically high on cold data with no thermal cause.
Unsafe shutdowns, power cycles, and endurance signals (available spare, percentage used) give the wear context that explains why GC pressure and retry latency get worse over time on an aging drive.
Block-layer averages from /sys/block are useful for trend, but the tail still needs BPF tooling; use Netdata for correlation across thermal, wear, and error signals, and biolatency-style histograms for the distribution itself.