$ guides / nvme / nvme-controller-state-not-live ▌

Operations Guides

NVMe controller state not live: reading resetting, deleting, and dead from sysfs

The fastest way to answer “is this NVMe device actually usable right now” is a single file: /sys/class/nvme/nvmeX/state. It is the kernel NVMe driver’s own view of the controller, exposed as one word: live, resetting, connecting, deleting, dead, or new. No SMART parsing, no log pages, no vendor tooling. If the state is not live, I/O to that device is stalled, failing, or about to.

This signal is easy to miss because it lives in sysfs, not in the SMART log. nvme smart-log will report a healthy drive while the controller sits in resetting and every I/O queues behind a timeout. It is also not exposed by Netdata’s NVMe collector, so many monitoring setups have no visibility into it at all.

What the state file is and why it matters

The kernel NVMe driver tracks every controller through a lifecycle state machine: created, connected, live, reset, teardown, and dead. The state attribute under /sys/class/nvme/nvmeX/ exposes the current position in that machine as a string:

# Read controller state for nvme0
cat /sys/class/nvme/nvme0/state
live

Three properties make this the most direct availability signal for local NVMe:

It reflects the driver, not the drive’s self-report. SMART data is the controller telling you how it feels. The state file is the host telling you whether it can currently submit commands. During a firmware hang, SMART can look fine because the controller is too hung to update it, while the state file shows resetting.
It changes before I/O errors surface. deleting and sustained resetting typically precede the block device disappearing and the flood of blk_update_request: I/O error messages.
It is cheap. Reading a sysfs attribute costs nothing and can be polled at any interval.

The kernel defines more states internally than sysfs exposes. Notably, the internal NVME_CTRL_DELETING_NOIO state (teardown with I/O already blocked) maps to the same deleting string as a normal deletion, so you cannot distinguish the two from sysfs alone. Internally, both deleting and dead are treated as terminal states: once a controller is dead, the kernel makes no further reset attempts.

The states, one by one

State	Meaning	Operational impact
`live`	Controller initialized and processing commands	Normal. The only acceptable steady state in production.
`new`	Controller object created, not yet initialized	Brief during boot or hot-add.
`connecting`	Controller association being established	Normal during init; mostly relevant to fabrics transports, rare on local PCIe.
`resetting`	Driver is resetting the controller and will replay in-flight I/O	All I/O stalled for the duration. Transient is normal; sustained is an incident.
`deleting`	Kernel is detaching the controller	Often the last thing you see before the device vanishes entirely.
`dead`	Kernel has given up on the controller	Outage. No further automatic recovery; I/O fails.

live

Steady state. Queues are up, commands complete. Anything else on a production device deserves at least a second look.

new and connecting

You see these during boot, after a PCIe hot-add, or when a controller is first initialized. On local PCIe devices both should resolve to live within seconds. A controller parked in connecting on a local device is unusual; check dmesg.

resetting

This state generates the most confusion, because it has a legitimate and a pathological form.

Legitimate: the driver resets the controller as part of normal recovery and certain administrative flows. Brief resetting during a firmware-activated sanitize or device self-test is expected, and a VM or guest reboot can produce reset messages on the host. These transitions resolve within seconds.

Pathological: a firmware hang, a PCIe link fault, or a power state bug causes the kernel’s I/O timeout to fire, the driver resets, and the controller either takes a long time to come back or never does. While the state is resetting, every I/O to the device is stalled. Operational line: any non-live state sustained for more than 30 seconds is abnormal in production, and resetting held past 30 seconds is a stuck reset.

deleting

The kernel is tearing the controller down. This appears during intentional removal (hot-unplug, driver unbind, nvme disconnect on fabrics) but also when a fatal PCIe error forces device isolation. On an unmanaged production device, deleting with no corresponding maintenance action is effectively a pre-mortem: expect /dev/nvmeXnY to disappear and mounted filesystems to start failing.

dead

Terminal. The kernel has exhausted its recovery attempts and will not reset the controller again. I/O fails, and the device will not return on its own. Recovery requires re-initializing the hardware, which in practice means a PCIe rescan or a reboot, and possibly physical intervention. Treat dead as an immediate outage.

The state lifecycle

stateDiagram-v2
    [*] --> new: device detected
    new --> connecting: init started
    connecting --> live: queues up
    live --> resetting: timeout or admin reset
    resetting --> live: recovery, I/O replayed
    resetting --> dead: recovery failed
    live --> deleting: removal or fatal PCIe error
    dead --> [*]: rescan or reboot required
    deleting --> [*]: device detached

The two edges that matter operationally: resetting to live is the recovery path you want, and resetting to dead is the one where the device is gone until someone touches the machine.

Distinguishing a transient reset from a stuck one

The state word alone does not tell you severity. Duration and context do. Work through this sequence:

Sample the state more than once. A single read of resetting tells you nothing. Poll a few times over 30 seconds. If it returns to live, you observed a transient reset; note it and correlate later.
Check for a sanitize or self-test in progress. Brief resetting during firmware-activated sanitize is expected behavior:
```
# Check whether a sanitize operation is in progress
nvme sanitize-log /dev/nvme0
```
If a sanitize is running, the reset is a side effect, not a fault. Do not remediate.
Check the kernel log for the trigger. The reset reason is almost always in dmesg:
```
# Look for the timeout/reset pair and any PCIe errors around it
dmesg | grep -i "nvme.*reset\|nvme.*timeout"
dmesg | grep -i "aer\|pcie.*error"
```
I/O timeout followed by Resetting controller points at a hung controller. Uncorrectable AER errors point at the physical link.

Check PCIe transport health. A controller that keeps falling into reset often has a sick link underneath it:

# Compare negotiated link vs capability, and check AER counters
cat /sys/class/nvme/nvme0/device/current_link_speed
cat /sys/class/nvme/nvme0/device/max_link_speed
cat /sys/class/nvme/nvme0/device/aer_dev_correctable
cat /sys/class/nvme/nvme0/device/aer_dev_fatal

If it reaches dead, stop expecting self-recovery. The kernel is done. Verify your redundancy holds, and pull the SMART log (nvme smart-log, nvme error-log) before the device potentially disappears from the bus entirely. Recovery means a PCIe rescan or reboot; both are disruptive to anything sharing that bus segment, so plan them rather than improvising mid-incident.

One known pattern behind recurring resets: some SSD firmware has broken autonomous power state transitions (APST), and the controller fails to wake from a deep power state. The kernel log in that case commonly shows controller is down; will reset: CSTS=0xffffffff. A widely used mitigation is disabling the deepest power states with the nvme_core.default_ps_max_latency_us=0 kernel parameter. Test before applying fleet-wide; it increases idle power draw.

Reading the state from containers and automation

The state file is only readable where sysfs is populated. Two practical consequences:

Containers. Passing /dev/nvme0 into a container is not enough. nvme list and friends discover devices through sysfs, so without the NVMe sysfs subtrees they return empty, and the state file will not exist either. To read controller state inside a container, bind-mount at minimum:

/sys/class/nvme
/sys/class/nvme-subsystem
/sys/class/nvme-generic

Alternatively run the container privileged, though bind-mounting the three subtrees is the smaller privilege. Device-handle commands like nvme id-ctrl /dev/nvme0n1 still work with just device passthrough, but state discovery does not.

Alerting logic. Suggested thresholds, stated plainly:

state != live sustained for more than 30 seconds: page. The 30-second grace excludes transient firmware-initiated resets during sanitize or self-test.
state = dead at any point: page immediately. It is an outage, not a warning.
state = deleting with no change window in progress: treat as imminent device loss.
Repeated transitions into resetting (two or more resets in an hour with failed or incomplete recovery): page, even if each individual reset recovers. That is the reset loop pattern, and it does not resolve on its own.

A simple collector is a cron job or a small script that reads the file and exports the value as a metric or log line. There is no excuse for this signal existing nowhere in your telemetry.

Signals to correlate

Controller state rarely fails alone. These are the surrounding signals that tell you why:

Signal	Why it matters	Warning sign
Kernel log reset/timeout messages	Root cause of each reset event	`I/O timeout` + `Resetting controller` pairs, especially repeating
PCIe AER counters (`aer_dev_correctable`, `aer_dev_fatal`, `aer_dev_nonfatal`)	Physical link health underneath the controller	Any uncorrectable error; sustained correctable rate
Link speed/width (`current_link_speed` vs `max_link_speed`)	Silent transport degradation preceding resets	Current below max
SMART `critical_warning` bits	The drive’s own hardware assessment	Bit 3 (read-only) or bit 2 (reliability degraded) asserted
`media_errors` rate	Active NAND degradation behind instability	Any increment during operation
`unsafe_shutdowns`	Power events that can hang a controller	Increment correlated with reset events
`nvme sanitize-log` status	Explains legitimate `resetting`	Sanitize in progress during the state transition
Device presence (`/sys/class/nvme/`, `lsblk`)	Ground truth after `deleting`	Controller directory or block device gone

How Netdata helps

Netdata’s NVMe collector does not expose the controller state file itself, so it will not alert on state != live directly. What it provides is the correlation context around a state event, which is what shortens the diagnosis:

Per-bit critical warning charts (nvme.device_critical_warnings_state) show whether the drive had already declared a hardware problem, such as read-only mode or degraded reliability, before it fell out of live.
Media error and error-log rates (nvme.device_media_errors_rate, nvme.device_error_log_entries_rate) tell you whether a hung controller sits on top of actively failing NAND.
Temperature, thermal throttle transitions, and cumulative time above thresholds let you confirm or rule out a thermal cause for repeated resets.
Unsafe shutdown counts separate power-loss events from genuine controller faults.
Per-second block device metrics show the exact I/O stall window a resetting episode caused, which is what you need to explain the application-level timeouts that followed.

Pair those with your own collection of the state file and kernel log monitoring for reset messages, and a controller event goes from “the database timed out for 20 seconds, no idea why” to a named failure with a known recovery posture.