NVMe controller reset loop: repeated resets from a firmware hang

Your kernel log shows the same cycle over and over: an I/O command stalls until the 30-second io_timeout fires, the NVMe driver resets the controller, I/O briefly recovers, then the whole thing hangs again. Each cycle costs your applications 5 to 30 seconds of stalled I/O. Databases time out, requests fail, clusters rebalance, and minutes later it happens again.

This is the controller firmware hang loop. The firmware hits a bug triggered by a specific command sequence, queue depth, I/O size, or power state transition. The kernel detects the timeout, resets the controller, and replays the in-flight commands. The same trigger recurs, so the controller hangs again. The loop can continue indefinitely.

The defining characteristic is what is absent: temperature is normal, PCIe AER counters are clean, and SMART shows no media errors. Nothing in the environment explains the resets. That absence is the fingerprint of a firmware defect rather than a hardware or thermal problem.

What this means

A single controller reset is the kernel’s brute-force recovery: a command timed out, the driver gave up waiting, and it reset the controller to replay in-flight I/O. One reset can be a transient glitch. Repeated resets with the same shape (latency spike to the timeout value, reset, brief recovery, another hang) mean the controller keeps walking into the same faulty internal state.

The trigger is usually one of:

• A specific command sequence or opcode pattern the firmware mishandles.
• A queue depth or I/O size that pushes the firmware into an unhandled path.
• An APST (Autonomous Power State Transition) event, where the controller enters a low-power state and fails to wake cleanly.

Because the trigger is workload-dependent, the loop often correlates with a particular application, a time of day, or a recent change in I/O pattern. Because it is firmware-internal, no host-side counter will show you the cause. SMART stays clean. The drive reports healthy right up until it stops answering.

flowchart LR
  A[Normal I/O] --> B[Trigger: command seq, queue depth, I/O size, or APST]
  B --> C[Firmware hangs, controller stops responding]
  C --> D[Latency spikes to 30s io_timeout]
  D --> E[Kernel resets controller, replays I/O]
  E --> F[Brief recovery]
  F --> B
  E -.->|recovery fails| G[Device dead or removed, PAGE]

Common causes

The lookalikes matter because their fixes differ completely. A PCIe fault is a reseat or retimer problem. A firmware hang is a firmware update or an RMA. Do not skip the rule-out steps.

Quick checks

All read-only and safe to run during an incident.

# Confirm the reset loop pattern in the kernel log
journalctl -k --no-pager | grep -i "nvme" | grep -i "timeout\|reset\|aborting"

# Count resets per device over the current boot
journalctl -k --no-pager | grep "nvme0" | grep -c "reset controller"

# Check current controller state (live, resetting, dead, deleting)
cat /sys/class/nvme/nvme0/state

# Capture the firmware revision before anything else
nvme id-ctrl /dev/nvme0 | grep -E "^(fr|mn|sn)"

# Check firmware slot history for recent updates
nvme fw-log /dev/nvme0

# Rule out media degradation: these should be flat and near zero
nvme smart-log /dev/nvme0 | grep -E "media_errors|num_err_log_entries|critical_warning|temperature"

# Rule out PCIe transport faults: all counters should be zero
cat /sys/class/nvme/nvme0/device/aer_dev_correctable
cat /sys/class/nvme/nvme0/device/aer_dev_fatal
cat /sys/class/nvme/nvme0/device/aer_dev_nonfatal

# Check the error log for what the controller recorded before each hang
nvme error-log /dev/nvme0

Note the ordering. Capture the firmware version first, before any reset, workaround, or swap. Once you replace the drive, the evidence of which firmware was running is gone, and you need it for the vendor case and for fleet-wide exposure assessment.

How to diagnose it

1. Confirm the loop shape. In journalctl -k, you should see I/O <N> QID <N> timeout, reset controller messages, repeating with a period of minutes. Between resets, I/O completes normally. If resets appear without preceding I/O timeouts, or the device disappears entirely instead of resetting, you are looking at a different failure (see the lookalikes above).

2. Rule out thermals. Check composite temperature against the drive’s WCTEMP. A thermal reset comes with temperature near or above the warning threshold and TMT transition counters climbing. In a firmware hang loop, temperature is normal and stays normal through the resets.

3. Rule out the PCIe transport. AER correctable and uncorrectable counters must be zero or flat. Any uncorrectable error, or a rising correctable rate, points at the physical layer: connector, riser, retimer, slot. Also confirm current_link_speed and current_link_width still match max. A degrading link produces retransmissions and latency, not a clean timeout-then-reset cycle.

4. Rule out media failure. media_errors flat, critical_warning zero, available spare stable. Media failure produces errors and read retries, not a silent controller that stops answering commands.

5. Correlate the trigger. Look at what was running when each hang started. The triggers are specific: a command sequence, a queue depth, an I/O size, or an APST transition. If resets cluster around idle periods or wake-from-idle, suspect APST. If they cluster under one application’s load, suspect a workload-pattern trigger.

6. Check the firmware version against known issues. Firmware hang loops are typically version-specific. Search the vendor’s release notes and advisories for your exact fr revision. Also check nvme fw-log for a recent update: a loop that started right after a firmware commit points at the new firmware, not the workload.

7. Assess recovery quality. After each reset, does the controller return to live and stay there until the next trigger? If recovery starts failing (state stuck in resetting beyond about 30 seconds, or transitioning to dead), escalate immediately. The threshold: PAGE at two or more resets in an hour with failed recovery. At that point you are one bad cycle from losing the device mid-write.

Metrics and signals to monitor

Fixes

Disable APST (workaround, not a fix)

If resets correlate with idle transitions, disable autonomous power state transitions:

# Disable APST on the controller
nvme set-feature /dev/nvme0 -f 0x0c -v 0

This is disruptive-adjacent rather than destructive, but test it on one host first. If the loop stops, APST was the trigger. The underlying firmware bug is still there; you have only removed the stimulus. Treat this as a bridge to a firmware update, and if you keep it, make it persistent across reboots via the kernel parameter nvme_core.default_ps_max_latency_us=0 (kernel cmdline or /etc/modprobe.d/); a runtime set-feature alone is re-programmed by the kernel at controller initialization.

Modify the I/O pattern (workaround)

If the trigger is a specific queue depth or I/O size, changing the workload shape can stop the loop: different I/O scheduler settings, reduced queue depth, or batching changes at the application layer. This is only practical if you have identified the trigger, and it carries the same caveat as APST: you are avoiding the bug, not fixing it.

Firmware update (the actual fix)

If a newer firmware revision addresses the hang, schedule a maintenance window and update. Firmware updates on NVMe are controller-level operations: brief I/O interruption is expected, and a controller reset or activation step is part of the process. Verify the new revision afterward with nvme id-ctrl and record the change in nvme fw-log. Keep the old version string in your records; if the new firmware introduces its own regression, you need the before-and-after.

Hardware replacement

If no fixed firmware exists, or the vendor confirms the revision you are on cannot be patched, replace the drive. Before swapping, capture the firmware version, serial number (nvme id-ctrl | grep sn), SMART log, and error log from the old device. You need these for the RMA and for confirming whether the replacement ships with the same affected firmware.

What not to do

Do not keep rebooting or reloading the driver as remediation. A host-initiated reset clears the current hang but the trigger recurs, and every cycle is another 5 to 30 second I/O stall against your applications. Do not raise io_timeout to mask the symptom either: it stretches each stall window without addressing why the controller stopped answering.

Prevention

• Track firmware versions fleet-wide. Once one drive hits a firmware hang loop, every drive on the same revision is exposed. Firmware version tracking is operational hygiene, and this failure mode is exactly why. Unexpected firmware changes should also alert.
• Alert on resets, not just device loss. A single reset is a TICKET. Two or more in an hour, or any reset with failed recovery, is a PAGE. Resets are only visible in kernel logs, not SMART, so you need log-pattern monitoring, not just device polling.
• Baseline controller state. Alert on /sys/class/nvme/nvmeX/state leaving live for more than 30 seconds. This catches stuck resets and failed recoveries faster than waiting for filesystem errors.
• Watch pre-reset latency. A latency trend climbing toward the timeout value, on a device that normally answers in microseconds, is your earliest warning that the next hang is coming.
• Stage firmware updates. Roll new firmware to a small cohort first and watch reset counts and error log rates before fleet-wide deployment. A firmware hang loop that starts right after an update is a rollout you want to halt, not propagate.
• Record workload context at each incident. The trigger is workload-specific. Noting which application, I/O size, or queue depth was active when each hang occurred is what lets you, or the vendor, reproduce it.

How Netdata helps

• Netdata’s NVMe collector tracks media errors rate, error log entries rate, and critical warning bits continuously, which gives you the rule-out evidence in one view: a clean SMART picture alongside active resets is the firmware hang fingerprint.
• Composite temperature and thermal management transition charts let you confirm or exclude a thermal cause in seconds instead of correlating logs by hand.
• Per-device I/O latency and throughput charts show the pre-reset spike and post-reset recovery pattern, so you can see the loop’s rhythm and measure how long each stall lasts.
• Unsafe shutdown and power cycle counters stay flat through a reset loop, which helps distinguish software recovery (controller reset) from power events.
• Alerting on state and error-rate signals means the second reset of the hour pages you before the third one takes the device offline mid-write.

Related guides

• nvme nvme0: I/O timeout, Resetting controller: what an NVMe controller reset means
• How NVMe actually works in production: a mental model for operators
• NVMe monitoring checklist: the signals every production SSD needs
• NVMe monitoring maturity model: from survival to expert