$ guides / nvme / nvme-controller-reset-timeout ▌

Operations Guides

nvme nvme0: I/O timeout, Resetting controller: what an NVMe controller reset means

You are here because dmesg or journalctl -k shows a sequence like this:

nvme nvme0: I/O 24 QID 3 timeout
nvme nvme0: Abort status: 0x0
nvme nvme0: Resetting controller

The NVMe driver submitted a command, the controller did not complete it within the I/O timeout (30 seconds by default), the driver tried to abort the command, the abort did not resolve things, and the driver forced a full controller reset. During the reset-recovery cycle, which typically takes 5 to 30 seconds, all I/O to that device stalls. In-flight commands are replayed after the reset completes.

This is the NVMe equivalent of a SCSI bus reset: a brute-force recovery mechanism. Data is typically preserved because commands replay, but the multi-second stall cascades upward. Applications with timeouts shorter than 30 seconds (most databases, most request handlers) fail before the reset even finishes. One reset shows up downstream as database connection timeouts, failed requests, and cluster rebalancing events. The application errors are the symptom; the reset is the cause.

The most important operational fact: this event does not appear in SMART data. A controller reset is a transport and firmware-level event, not a media error. nvme smart-log will often show zero media errors, no new error log entries, and a healthy drive while resets are actively happening. If you only monitor SMART, you will miss this entirely. The reliable sources are the kernel log and controller state in sysfs.

What this means

The kernel NVMe driver tracks every outstanding command. When a command exceeds io_timeout (check /sys/module/nvme_core/parameters/io_timeout, 30 seconds by default), the driver’s timeout handler fires. It first attempts to abort the specific command. If the abort succeeds, life goes on. If the abort itself times out or fails, the driver schedules a controller reset.

A reset is a full reinitialization: the driver tears down the queue pairs, resets the controller hardware, recreates the admin and I/O queues, and replays every in-flight command. From the block layer’s perspective, requests submitted before the reset eventually complete. From the application’s perspective, I/O simply stopped for 5 to 30 seconds.

flowchart TD
  A[Command exceeds io_timeout] --> B[Driver aborts command]
  B -->|abort succeeds| C[I/O resumes normally]
  B -->|abort fails or times out| D[Controller reset scheduled]
  D --> E[All I/O stalled 5-30s]
  E -->|reset succeeds| F[In-flight commands replayed]
  E -->|reset fails| G[Controller state: dead, device removed]
  F --> H{Does it recur?}
  H -->|repeated resets| I[Dying controller, firmware bug, or PCIe fault]
  H -->|isolated| J[Investigate anyway: firmware, APST, thermals, AER]

You can watch the controller move through this in real time via sysfs:

# Controller state: live, resetting, connecting, deleting, dead, new
cat /sys/class/nvme/nvme0/state

A controller stuck in resetting for more than a few seconds, or cycling between resetting and live, is the reset loop pattern: the controller crashes shortly after coming online, the driver resets it again, and the cycle repeats. That pattern is usually unrecoverable without a firmware update or hardware replacement.

Common causes

Cause	What it looks like	First thing to check
Controller firmware hang or bug	Resets with no temperature spike, no AER errors, no media errors; often reproducible under a specific workload	`nvme id-ctrl /dev/nvme0 \| grep -i fr` for firmware version; search for known issues
APST / aggressive power management	Timeouts after idle periods, on boot, or under light load; consumer drives especially	Kernel log for timeouts correlating with idle-to-active transitions
PCIe link instability	AER correctable errors accumulating, link retrains in dmesg, possibly link speed/width downgrade	`/sys/class/nvme/nvme0/device/aer_dev_correctable` and link speed vs max
Thermal emergency	Temperature climbing before the reset, warning_temp_time increasing, throttling transitions	`nvme smart-log` temperature and thermal management counters
Power delivery instability	Resets plus unsafe shutdown count increasing, or power cycles without reboots	`nvme smart-log` unsafe_shutdowns and power_cycles rates
Dying controller / reset loop	Repeated resets (2 or more per hour), device disappears and reappears, failed recovery	`dmesg` reset frequency; `/sys/class/nvme/nvme0/state` stuck non-live

Two benign cases to rule out first: a VM or guest reboot produces expected controller reset messages on the host’s virtual NVMe device, and host-initiated resets (driver reload, manual trigger via /sys/class/nvme/nvme0/reset_controller) are expected. Some virtualization layers also emulate controller resets during live migration or backend maintenance. Correlate with maintenance windows and VM lifecycle events before treating a reset as a hardware fault.

Quick checks

All read-only and safe to run during an incident.

# 1. Count resets and see the full sequence
journalctl -k --no-pager | grep -c "nvme0: Resetting controller"
dmesg -T | grep -i "nvme.*reset\|nvme.*timeout" | tail -50

# 2. Current controller state (ground truth for availability)
cat /sys/class/nvme/nvme0/state

# 3. Confirm the timeout value the driver is using
cat /sys/module/nvme_core/parameters/io_timeout

# 4. PCIe AER counters (should be zero on a healthy link)
cat /sys/class/nvme/nvme0/device/aer_dev_correctable
cat /sys/class/nvme/nvme0/device/aer_dev_nonfatal
cat /sys/class/nvme/nvme0/device/aer_dev_fatal

# 5. Link degradation check: current vs max
cat /sys/class/nvme/nvme0/device/current_link_speed
cat /sys/class/nvme/nvme0/device/max_link_speed
cat /sys/class/nvme/nvme0/device/current_link_width
cat /sys/class/nvme/nvme0/device/max_link_width

# 6. SMART cross-check: expect this to look clean during a timeout incident
nvme smart-log /dev/nvme0 | grep -i "critical_warning\|media_errors\|num_err_log_entries\|temperature\|unsafe_shutdowns\|power_cycles"

# 7. Firmware version and firmware log
nvme id-ctrl /dev/nvme0 | grep -i "^fr "
nvme fw-log /dev/nvme0

# 8. Detailed error entries (circular buffer, check early)
nvme error-log /dev/nvme0

The AER counters live under the PCI device path (/sys/class/nvme/nvme0/device/ resolves to the PCI device), so read permissions depend on system policy, and in containers sysfs may not be mounted.

How to diagnose it

The goal is to attribute the reset to one of the causes above. Work the correlation window: the minutes before the first timeout message.

Establish the timeline. Pull the full kernel log around the event with journalctl -k --since "1 hour ago". Find the first I/O ... timeout line, not just the reset. Note whether anything preceded it: AER messages, link retrain notices, thermal messages, or a period of silence after idle.
Classify the pattern. One reset ever: investigate but do not treat as an emergency. Two or more resets in an hour, or a reset with failed recovery (controller goes to dead or the device disappears): this is the reset loop, treat as page-worthy and plan hardware action.
Rule out the transport layer. Check AER counters. Any uncorrectable fatal or non-fatal error is a severe hardware issue (cable, connector, slot, retimer). A sustained correctable error rate points to marginal signal integrity. Also compare current vs max link speed and width; a downgraded link means the physical layer has already been renegotiating.
Rule out thermals. Pull composite temperature history for the window before the reset. Check warning_temp_time and critical_comp_time in the SMART log: these are cumulative minutes above the warning and critical thresholds, so they reveal thermal stress even if you missed the live event. Rising thermal management transition counts (thm_temp1_trans_count, thm_temp2_trans_count) confirm throttling was active.
Rule out power. Compare unsafe_shutdowns and power_cycles rates against actual reboot and maintenance records. Power cycles without corresponding system reboots suggest slot or PDU power instability.
Suspect firmware and power management last. If temperature is normal, AER is clean, power is stable, and SMART is clean, the most likely remaining causes are a controller firmware hang (often triggered by a specific command pattern or queue depth) or autonomous power state transitions (APST) on consumer-class drives, where the drive enters a deep idle state and fails to wake within the timeout.
Preserve the error log now. The NVMe error information log is a circular buffer. Under a high error rate, root-cause entries get overwritten. Run nvme error-log /dev/nvme0 and save the output before continued operation rotates it.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Kernel log reset/timeout count	The reset itself exists nowhere else; SMART will not show it	Any reset; 2+ in 1 hour is a dying-controller pattern
`/sys/class/nvme/nvme0/state`	Direct availability signal	Non-`live` sustained over 30 seconds
PCIe AER correctable error rate	Link integrity degrades before resets start	Sustained non-zero rate
PCIe AER uncorrectable (fatal/non-fatal)	Severe transport fault, often precedes device loss	Any occurrence
Composite temperature + warning/critical temp time	Thermal events cause controller hangs and throttling	Temperature near WCTEMP; warning_temp_time rate > 0
Block layer I/O latency	Latency climbs toward the 30s timeout before a reset fires	Sustained multi-second latency; any I/O over 1s
unsafe_shutdowns / power_cycles rate	Power instability triggers controller faults	Increments without matching reboots
num_err_log_entries rate	Firmware and internal errors appear here, not in media_errors	Rising rate with flat media_errors

Fixes

Firmware hang or known firmware bug

Check the firmware revision (nvme id-ctrl | grep -i fr) against the vendor’s release notes and advisories. Firmware bugs that trigger hangs under specific workloads are more common than vendors admit, and fixes ship as firmware updates. Apply updates during a maintenance window; firmware activation can itself involve a controller reset, so quiesce I/O first. If the trigger is reproducible, changing the I/O pattern (queue depth, scheduler, I/O size) sometimes avoids the buggy path until you can update.

APST / power management timeouts

Common on consumer NVMe in server or workstation roles. The drive enters an autonomous low-power state and does not wake within the I/O timeout. The widely used workaround is disabling deep power state transitions, either via the kernel command line (nvme_core.default_ps_max_latency_us=0, as documented in the Arch Wiki NVMe troubleshooting page) or per-device:

# Disable autonomous power state transitions (feature 0x0c)
nvme set-feature /dev/nvme0 -f 0x0c -v 0

This does not persist across resets or reboots on all controllers, so the kernel parameter is the durable form. Tradeoff: higher idle power draw and slightly higher idle temperatures. PCIe ASPM can also cause latency spikes and transient link issues, and some operators disable it for the same reason. Validate on one node before rolling out.

PCIe link instability

Reseat the drive (M.2 connectors especially; vibration and thermal cycling work them loose). Inspect or replace cables and risers on U.2/U.3 paths. If the link has negotiated below max speed or width, a full power cycle (not a warm reboot) forces renegotiation. If AER errors persist after reseating, try a different slot to isolate drive vs slot vs backplane.

Thermal

Restore cooling headroom: verify heatsink contact, chassis airflow, and fan function. Reduce write load while the drive is hot. A drive that has accumulated critical_comp_time has operated in a zone that accelerates NAND wear, so increase monitoring frequency on its endurance signals afterward.

Reset loop / failed recovery

If resets repeat and recovery fails, plan replacement. Verify your redundancy (RAID, replication) is healthy first, then fail the device out cleanly rather than letting the loop continue to stall I/O for the whole box. Check for a firmware update as a last resort before RMA.

What not to do

Do not raise nvme_core.io_timeout or set it to a huge value to “fix” the resets. That only makes applications hang longer before failing and hides the underlying hardware or firmware problem. Do not treat a single successful recovery as resolution; resets recur.

Prevention

Alert on the kernel log pattern. nvme.*timeout and Resetting controller must be a structured log alert, not something you find in a postmortem. Single reset: ticket. Two or more in an hour, or failed recovery: page.
Baseline AER counters and link state at provisioning. A Gen4 x4 drive running at Gen3 x2 delivers a quarter of its bandwidth with zero errors. Catch degradation at deployment, not during an incident.
Track firmware versions fleet-wide. A known-bad firmware revision is only actionable if you know which drives run it.
Baseline PLP and drive class. Consumer drives without power-loss protection and with aggressive APST behave differently under power events and idle transitions. Monitor them accordingly.
Monitor temperature continuously, not just SMART thresholds. The cumulative warning and critical temperature time counters reveal thermal events you missed live.
Distinguish resets from unsafe shutdowns in your runbooks. Resets are software recovery; unsafe shutdowns are power loss. Different causes, different responses.

How Netdata helps

Netdata does not see the reset itself (nothing in SMART reports it), but it gives you the attribution context around the event:

nvme.device_composite_temperature plus nvme.device_warning_composite_temperature_time and nvme.device_critical_composite_temperature_time let you confirm or rule out a thermal lead-up to the reset, including thermal stress that happened while nobody was watching.
nvme.device_error_log_entries_rate rising while nvme.device_media_errors_rate stays flat is the fingerprint of firmware or internal controller faults rather than NAND problems.
nvme.device_critical_warnings_state per-bit dimensions tell you whether the drive itself assessed a reliability or thermal problem in the same window.
nvme.device_unsafe_shutdowns_count and power cycle trends separate power delivery problems from driver-level resets.
Per-second block device I/O latency around the event shows the stall building toward the 30 second timeout, which helps you distinguish a slow-burn degradation from a sudden firmware hang.

The practical workflow: alert on the kernel log pattern, then open the Netdata dashboard for that node at the reset timestamp and correlate temperature, error log rate, and block latency in the minutes before the event.