blk_update_request: I/O error, dev nvme0n1: reading NVMe I/O errors in the kernel log

dmesg shows lines like this:

blk_update_request: I/O error, dev nvme0n1, sector 12345678

Maybe one line. Maybe thousands. Maybe interleaved with nvme nvme0: I/O 4096 QID 1 timeout, reset controller or nvme nvme0: Removing. The operational question is always the same: is this a single bad block the drive surfaced on a read, or is the device, the controller, or the PCIe link underneath it failing?

This guide covers what the kernel is actually telling you, how to classify the failure from the surrounding log lines, and which SMART and sysfs signals confirm or rule out each cause.

What this means

blk_update_request() is the block layer function that processes completed I/O. When a request finishes with a non-zero error status, the kernel logs blk_update_request: I/O error, dev <device>, sector <N>. For an NVMe device, that status came from the NVMe controller: a command completed with a failure status, and the NVMe driver translated it to EIO.

The critical point: this message is the block layer reporting a completed-but-failed command. It is a symptom, not a root cause. Three very different failures produce the same line:

1. Media fault. The controller’s internal ECC could not recover data from a NAND page. The read of that sector fails, everything else works. This is the classic latent bad block surfacing on a read of cold data.
2. Controller fault. The controller hung or crashed, the driver timed out and reset it, and in-flight commands completed with errors during the recovery window. You will see nvme nvme0: I/O 4096 QID 1 timeout, reset controller nearby.
3. Transport fault or device loss. The PCIe link dropped or the device was removed (surprise removal, fatal PCIe error, power loss to the slot). CSTS reads as all-ones, the driver declares the controller down, and every in-flight I/O fails with EIO. You will see controller is down and often Removing in the log.

Newer kernels add the operation code to the message (op 0x0:(READ), op 0x1:(WRITE), and so on) along with flags and priority class. Older kernels print just the device and sector. The diagnostic logic is the same either way.

The triage decision tree:

flowchart TD
    A["blk_update_request: I/O error, dev nvme0n1"] --> B{"Same sector repeating
or scattered sectors?"}
    B -->|"Same sector, isolated"| C["Likely latent bad block
Check SMART media_errors"]
    B -->|"Scattered sectors"| D{"Reset or down messages
in dmesg?"}
    D -->|"timeout, reset controller"| E["Controller fault
firmware hang, thermal, PCIe"]
    D -->|"controller is down / Removing"| F["Transport or device loss
PCIe, power, surprise removal"]
    D -->|"No reset lines"| G["Check media_errors and
error log entries"]
    C --> H["media_errors incrementing?
TICKET and monitor"]
    E --> I["Check AER counters, temperature,
firmware version"]
    F --> J["PAGE: check controller state,
filesystem damage, physical seating"]

Common causes

A note on the last row: some drives with buggy firmware reject WRITE_ZEROES and the block layer logs operation not supported error rather than I/O error. These are noise if the kernel has a quirk for the drive, but worth checking if the line is new after a kernel upgrade.

Quick checks

All read-only, all safe to run during an incident.

# 1. Get the full context around the errors
dmesg -T | grep -i "nvme0\|blk_update_request" | tail -50

# 2. Controller state: live, resetting, dead, or deleting
cat /sys/class/nvme/nvme0/state

# 3. SMART health: media errors, error log entries, critical warnings
nvme smart-log /dev/nvme0 | grep -i "critical_warning\|media_errors\|num_err_log_entries\|available_spare\|percentage_used\|unsafe_shutdowns"

# 4. Actual error log entries with status codes and LBAs
nvme error-log /dev/nvme0 --log-entries=16

# 5. Controller resets and removal events
dmesg | grep -i "nvme.*reset\|nvme.*timeout\|removing\|controller is down"

# 6. PCIe AER counters: transport-layer health
cat /sys/class/nvme/nvme0/device/aer_dev_correctable
cat /sys/class/nvme/nvme0/device/aer_dev_fatal
cat /sys/class/nvme/nvme0/device/aer_dev_nonfatal

# 7. Is the block device still present and consistent
lsblk | grep nvme
ls /sys/class/nvme/

The AER paths require root and may not exist if AER is disabled. lsblk can show stale entries briefly after a device vanishes; /sys/class/nvme/ is ground truth for what the kernel actually has.

How to diagnose it

Work through these in order. Each step eliminates a branch of the tree above.

1. Establish the pattern: isolated or storm. Look at timestamps and sector numbers in dmesg. One line, or a handful of lines on the same sector over hours, points at a media fault on cold data. Hundreds of lines with random sectors in a burst points at a controller or transport event.

2. Look for the lines that precede the errors. The ordering matters more than the error line itself:

• nvme nvme0: I/O <N> QID <N> timeout, reset controller, then a burst of EIO: controller hang. In-flight commands failed during the 5-30 second reset window.
• controller is down; will reset: CSTS=0xffffffff: the controller vanished from the driver’s perspective. This is PCIe link loss, device power loss, or hardware death, not a media problem.
• nvme nvme0: Removing or removal after a failed reset with a negative status: the kernel has given up on the device. Everything after that is EIO to a device that no longer exists.
• No NVMe lines at all, just the block layer error: the command completed with a failure status but the controller stayed healthy. That is the media-error signature.

3. Check controller state right now. cat /sys/class/nvme/nvme0/state. live means the device recovered or never went down. resetting for more than a few seconds means a stuck recovery. dead or deleting means the kernel has detached or is detaching it. Brief resetting during a sanitize or self-test is expected; check nvme sanitize-log /dev/nvme0 if the timing fits.

4. Interrogate SMART for the media story. If media_errors incremented by exactly the number of blk_update_request lines, and critical_warning is zero, you are likely looking at isolated cell failures the controller has already remapped or flagged. If media_errors is climbing continuously, available_spare is declining, and critical_warning bit 2 (reliability degraded) is set, the NAND is actively failing and the drive needs replacement, not investigation. If num_err_log_entries is climbing but media_errors is flat, the errors are not media errors: think firmware, driver, or command-level issues.

5. Pull the error log. nvme error-log /dev/nvme0 gives you status codes, the failing command, the namespace, and the LBA for each entry. This is how you confirm whether the failing LBAs match the sectors in dmesg. Do this early: the log is a circular buffer and high error rates overwrite the entries that explain the root cause.

6. Rule the PCIe layer in or out. Non-zero aer_dev_fatal or aer_dev_nonfatal counters, or a burst of correctable errors before the event, mean the transport was degraded. Combine that with controller is down and you have a link, connector, riser, or power problem. Check current_link_speed and current_link_width against the max values while you are there.

7. Assess blast radius. If the device went away or threw write errors, assume any mounted filesystem took damage. Check for EXT4-fs error, XFS, or Buffer I/O error lines after the NVMe errors in dmesg, and plan a filesystem check before trusting the data.

Metrics and signals to monitor

Fixes

Grouped by what you found. None of these start with “reboot”.

Isolated bad block, drive otherwise healthy. A single new media error during a boot-time scan or backup that touches cold data can be a latent defect, not an emergency. Verify backups or replicas for the affected data, rewrite the affected file (or the block, if you can map sector to file) to force the controller to reallocate it, and put a trend watch on media_errors. Same-shift ticket, not a page.

Controller hang with successful reset. Check the firmware version (nvme id-ctrl /dev/nvme0 | grep -i "fr ") against known issues for that model and look for a vendor firmware update. Check whether the reset correlated with temperature (thermal shutdown) or with PCIe AER errors (link instability). A single reset can be a transient firmware glitch; repeated resets in a loop with no environmental correlation is a firmware hang pattern, and the drive or its firmware must change. If autonomous power state transitions are suspected as the trigger, disabling APST is a known mitigation on some consumer drives. On Linux that means either the boot parameter nvme_core.default_ps_max_latency_us=0 (the nvme-core module parameter that turns autonomous power state transitions off) or, at runtime, nvme set-feature /dev/nvme0 -f 0x0c --value=0 (feature 0x0c is the Autonomous Power State Transition feature).

Device down or removed. This is a page. If it is a hot-swap bay and someone pulled the drive, correlate with the maintenance window before treating it as a failure. Otherwise: reseat the drive if physically accessible, inspect the connector and slot, check AER counters for fatal errors, and check whether a power-cycle brings it back. Some BIOS configurations require a reboot to re-detect a device even after physical remediation. Plan a filesystem check for anything that was mounted on it.

Active NAND degradation. Critical warning bit 2 plus rising media_errors plus declining spare is the drive telling you it is failing. Verify redundancy (RAID, replicas, backups) first, force rewrites of at-risk cold data if your stack supports it, and replace the drive. Do not wait for it to go read-only.

Firmware quirk noise (WRITE_ZEROES). If the only symptom is operation not supported error on op 0x9:(WRITE_ZEROES) and SMART is clean, confirm the kernel carries a quirk for your drive model. Updating firmware or the kernel usually silences it. It is cosmetic, but verify before you dismiss it.

Prevention

• Alert on the causes, not just the symptom. Kernel log matching on blk_update_request alone gives you a noisy alert with no context. Pair it with alerts on controller state changes, reset lines, and media_errors rate so the page arrives with the classification already attached.
• Watch the rate, not the counter. media_errors, num_err_log_entries, and unsafe_shutdowns are lifetime counters. A static value of 3 from two years ago is history; a value that moved today is signal.
• Track available_spare trajectory. Spare consumption accelerates non-linearly. The rate of change is one of the strongest leading indicators of a drive that will start throwing read errors months from now.
• Baseline PCIe health. AER counters and link speed/width are transport signals that SMART never shows. A degraded link produces exactly the EIO storms this article is about, and it produces them with a perfectly healthy drive behind it.
• Track firmware versions fleet-wide. Firmware hangs and command-handling quirks are version-specific. Knowing which drives run which firmware turns a fleet-wide mystery into a targeted update.

How Netdata helps

• Netdata’s NVMe collector surfaces media_errors as an incremental rate (nvme.device_media_errors_rate), so a new bad block shows up as a spike the moment it happens, not at the next manual nvme smart-log.
• The nvme.device_critical_warnings_state chart breaks the critical_warning bitmask into per-bit dimensions, so you can distinguish bit 2 (reliability degraded, investigate) from bit 3 (read-only, outage) instead of alerting on any nonzero value.
• nvme.device_error_log_entries_rate tracks error log growth separately from media errors, which is the correlation that separates NAND failure from firmware or command-level problems.
• Trending nvme.device_available_spare_perc and nvme.device_estimated_endurance_perc over weeks shows the wear trajectory that predicts future read errors, and Netdata’s ML anomaly detection flags accelerations in that trend before thresholds trip.
• Correlating these per-second device metrics with system-level I/O latency and throughput on the same dashboard is what turns “blk_update_request in dmesg” into “controller reset at 03:14 preceded by rising latency” in one view.

Related guides

• nvme nvme0: I/O timeout, Resetting controller: what an NVMe controller reset means
• How NVMe actually works in production: a mental model for operators
• NVMe monitoring checklist: the signals every production SSD needs
• NVMe monitoring maturity model: from survival to expert