NVMe Operations Guides on Netdata

blk_update_request: I/O error, dev nvme0n1: reading NVMe I/O errors in the kernel log

Mon, 01 Jan 0001 00:00:00 +0000

dmesg shows lines like this:

blk_update_request: I/O error, dev nvme0n1, sector 12345678

Maybe one line. Maybe thousands. Maybe interleaved with nvme nvme0: Resetting controller or nvme nvme0: Removing. The operational question is always the same: is this a single bad block the drive surfaced on a read, or is the device, the controller, or the PCIe link underneath it failing?

This guide covers what the kernel is actually telling you, how to classify the failure from the surrounding log lines, and which SMART and sysfs signals confirm or rule out each cause.

How NVMe actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

Most storage incidents on NVMe are misdiagnosed for the same reason: operators debug them as if NVMe were a faster SATA disk. It is not. NVMe is a host-to-controller communication protocol that exposes flash storage over PCIe, and the failure modes live in places generic disk monitoring never looks: the PCIe link, the controller firmware, and the flash translation layer sitting between your filesystem and the NAND.

The mental model that makes NVMe behavior predictable has three layers: the PCIe transport, the controller, and the flash media. Once you hold this model, most “mystery slowness” and “healthy SMART, dead drive” incidents stop being mysterious. Scope here is local PCIe-attached NVMe. NVMe-oF, ZNS, and SPDK change the monitoring model fundamentally and are not covered.

NVMe ASPM latency spikes: PCIe power states adding first-request latency

Mon, 01 Jan 0001 00:00:00 +0000

You are chasing tail latency on an NVMe-backed service. The p99 graph shows sporadic spikes: tens of microseconds added to reads that should complete in well under 100us. The spikes do not correlate with load, temperature, media errors, queue depth, or garbage collection. SMART is clean. The block layer looks clean. The one pattern you can find: the slow I/Os tend to be the first request after the device sat idle.

NVMe available spare below threshold: critical warning bit 0 and end-of-life wear

Mon, 01 Jan 0001 00:00:00 +0000

Your monitoring fired on an NVMe drive: critical warning bit 0 is set, or available_spare has dropped to (or below) the vendor threshold. The drive still works. Reads and writes complete, latency looks fine, nothing in the application layer is complaining. That is exactly what makes this signal easy to ignore and expensive to ignore.

Bit 0 means the controller’s pool of spare NAND blocks, the reserve it uses to transparently replace failed cells, has fallen below the safety margin the vendor baked into the firmware. The drive is telling you it is approaching end-of-life. It is not telling you it has failed.

NVMe available spare declining: watching the wear trajectory before the threshold

Mon, 01 Jan 0001 00:00:00 +0000

The drive reported available_spare at 100% for two years. This quarter it reads 94%, and last month it was 96%. Nothing has alerted: critical_warning is zero, I/O is clean, latency is normal. The question is whether you are watching normal aging or the early edge of a failure curve. The answer is almost never in the current value. It is in the rate of change.

Available spare is a gauge, not an event. By the time the drive asserts critical_warning bit 0 (available spare below threshold), you are inside the vendor’s safety margin and replacing the drive on the failure’s schedule instead of yours. An accelerating, non-linear spare decline is one of the strongest leading indicators of premature NVMe failure, and you can only see it if you track consumption months before the threshold crossing.

NVMe controller reset loop: repeated resets from a firmware hang

Mon, 01 Jan 0001 00:00:00 +0000

Your kernel log shows the same cycle over and over: an I/O command stalls until the 30-second io_timeout fires, the NVMe driver resets the controller, I/O briefly recovers, then the whole thing hangs again. Each cycle costs your applications 5 to 30 seconds of stalled I/O. Databases time out, requests fail, clusters rebalance, and minutes later it happens again.

This is the controller firmware hang loop. The firmware hits a bug triggered by a specific command sequence, queue depth, I/O size, or power state transition. The kernel detects the timeout, resets the controller, and replays the in-flight commands. The same trigger recurs, so the controller hangs again. The loop can continue indefinitely.

NVMe controller state not live: reading resetting, deleting, and dead from sysfs

Mon, 01 Jan 0001 00:00:00 +0000

The fastest way to answer “is this NVMe device actually usable right now” is a single file: /sys/class/nvme/nvmeX/state. It is the kernel NVMe driver’s own view of the controller, exposed as one word: live, resetting, connecting, deleting, dead, or new. No SMART parsing, no log pages, no vendor tooling. If the state is not live, I/O to that device is stalled, failing, or about to.

This signal is easy to miss because it lives in sysfs, not in the SMART log. nvme smart-log will report a healthy drive while the controller sits in resetting and every I/O queues behind a timeout. It is also not exposed by Netdata’s NVMe collector, so many monitoring setups have no visibility into it at all.

NVMe critical_warning is nonzero: decoding the SMART critical warning bitmask

Mon, 01 Jan 0001 00:00:00 +0000

Your monitoring fired because critical_warning in the NVMe SMART log is nonzero. Or you ran nvme smart-log /dev/nvme0 during an investigation and saw something like critical_warning : 0x04 where you expected 0x00. You need two things fast: which bit is set, and how bad it is.

The most common mistake here is treating critical_warning != 0 as one alert with one severity. It is a bitmask of six independent conditions with wildly different severity. Bit 3 means the drive has gone read-only and is refusing writes: a page-right-now outage. Bit 0 means spare capacity is below the vendor threshold: a procurement ticket, not a 3 a.m. incident. Bit 1 may be a transient thermal event during a backup run that clears itself.

NVMe device disappeared: nvme0: Removing and a drive that fell off the PCIe bus

Mon, 01 Jan 0001 00:00:00 +0000

You are looking at a host where an NVMe drive was present and is now gone. The kernel log shows some version of this sequence: a pcieport: AER: Uncorrectable (Fatal) line, possibly followed by nvme nvme0: controller is down; will reset: CSTS=0xffffffff, then nvme nvme0: Removing (or nvme nvme0: Removing after probe failure status: -19), and often nvme0n1: detected capacity change from X to 0. After that, /dev/nvme0n1 no longer exists and /sys/class/nvme/nvme0 is empty.

NVMe drive in read-only mode: critical warning bit 3 and rejected writes

Mon, 01 Jan 0001 00:00:00 +0000

Your monitoring fires a PAGE: the NVMe device has set critical_warning bit 3 (0x08), meaning the drive has autonomously placed its media in read-only mode. Every write command the host sends is now rejected by the drive’s firmware. Reads still work. Writes do not.

This is not transient, not load-dependent, and not cleared by a reboot or controller reset. The drive has decided, based on its own internal assessment, that it can no longer safely accept writes, typically because it has run out of spare NAND blocks to remap failing cells into. It has switched itself into data-preservation mode so you can get your data off before it dies completely.

NVMe endurance runway: projecting time-to-replacement from wear signals

Mon, 01 Jan 0001 00:00:00 +0000

NVMe wear-out is one of the few storage failures that usually announces itself months in advance. The announcement is quiet: a monotonically rising percentage_used, a slow drift in available_spare, maybe the first few media_errors. If you only look at current values, you find out when the drive crosses a threshold. If you trend the rates, you can procure before the drive becomes an incident.

This guide is a planning procedure for local PCIe-attached NVMe devices. It turns three wear signals into a replacement date, then cross-checks that date against spare-block consumption and early media failure. It does not apply to NVMe-oF, ZNS, SPDK, or virtualized cloud “NVMe” devices where SMART data is absent, synthetic, or owned by the provider.

NVMe error log entries growing: num_err_log_entries beyond media errors

Mon, 01 Jan 0001 00:00:00 +0000

Your NVMe alert fired because num_err_log_entries is climbing. You pull the SMART log, and media_errors is zero. The drive looks healthy, but something is writing error entries at a steady, sometimes alarming, rate.

num_err_log_entries is a superset counter: it counts every error the controller records, including admin command errors, I/O command errors, internal controller errors, and thermal events. media_errors counts only data integrity failures against the NAND. A rising num_err_log_entries with flat media_errors points away from failing flash and toward firmware, driver, or command-level issues. In a large share of cases, it points at your own monitoring stack.

NVMe firmware version tracking: catching the bugs vendors do not advertise

Mon, 01 Jan 0001 00:00:00 +0000

The NVMe controller is an embedded computer running proprietary firmware that owns your data path: the flash translation layer, wear leveling, garbage collection, error correction, and the queue interface to the host. When that firmware has a bug, the symptoms show up as reset loops, erratic latency, premature wear, or in the worst cases silent data loss. Controller firmware bugs are more common than vendors admit, and they are usually triggered by specific command sequences or power-state transitions, so two identical drives on different firmware revisions can behave like different hardware.

NVMe high I/O latency: reading block-layer latency and the outliers that matter

Mon, 01 Jan 0001 00:00:00 +0000

Latency complaints about NVMe arrive as “the database is slow” or “queries time out randomly.” By the time the report reaches you, the mean latency from iostat often looks fine, because NVMe mean latency is almost always fine. The damage comes from the tail: the occasional 10 ms or 1 s outlier that breaks a request deadline and cascades into application timeouts.

The block layer gives you averages, and averages hide exactly the outliers that matter. This guide covers how to read the block-layer counters, how to get the actual latency distribution, and how to map a latency spike to its cause: thermal throttling, garbage collection pressure, PCIe transport degradation, or a controller on its way to a reset.

NVMe media_errors increasing: uncorrectable data-integrity errors on NAND

Mon, 01 Jan 0001 00:00:00 +0000

Your monitoring shows the media_errors counter on an NVMe drive going up. In nvme smart-log output the field is labelled media_errors (or “Media and Data Integrity Errors” in some nvme-cli versions), and it is the one SMART field you should never explain away: each increment is a read or write where the controller could not maintain data integrity even after its internal ECC and retry mechanisms were exhausted.

The absolute value is almost meaningless on its own. A drive with 3 lifetime errors after four years of service can be perfectly healthy; a drive that went from 0 to 12 errors this week is failing. The rate of change, and what changes alongside it, is the diagnostic. A single new error during a backup run that scanned cold data can be one latent bad block finally being touched. A steady climb combined with critical_warning bit 2 (NVM subsystem reliability degraded) is the drive telling you it is dying.

NVMe monitoring checklist: the signals every production SSD needs

Mon, 01 Jan 0001 00:00:00 +0000

Most NVMe monitoring collects a handful of SMART counters and stops there. That works until a controller hangs without touching SMART, a PCIe link silently retrains to half speed, or a drive sets a critical warning bit that your single blanket alert treats as noise. NVMe failures rarely arrive as a clean “disk error.” They arrive as latency, throttling, stalled queues, or a device that vanishes from the bus.

Scope: local PCIe-attached NVMe devices (M.2, U.2, U.3, EDSFF). NVMe-oF, ZNS, and SPDK userspace configurations have different monitoring models and are out of scope.

NVMe monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Most teams discover their NVMe monitoring gaps during an incident: a drive throttling silently at 80 C while every dashboard shows green, or a Gen4 x4 device running at Gen3 x2 for weeks with zero errors. NVMe fails differently from SATA and SAS. It has a PCIe transport layer with its own error reporting, a controller running a flash translation layer that causes latency variance invisible to block-layer averages, and wear signals that only matter as rates of change. Generic disk monitoring misses all of this.

NVMe namespaces and the 512e vs 4K sector-size trap

Mon, 01 Jan 0001 00:00:00 +0000

A namespace formatted for 512-byte logical blocks (512e) while the workload, filesystem, or encryption layer assumes 4K blocks produces no errors, no SMART warnings, and no kernel log entries. It just makes the controller do more work per host write than it should. This is one of the genuinely silent NVMe failure modes: everything looks healthy while write amplification quietly burns endurance and adds latency.

The trap is a provisioning-time decision with runtime consequences. Many drives ship formatted 512e for compatibility. Someone creates a filesystem, a LUKS volume, or a ZFS pool on top, and the mismatch is locked in until the namespace is reformatted, which destroys the data on it.

NVMe NVM subsystem reliability degraded: critical warning bit 2

Mon, 01 Jan 0001 00:00:00 +0000

Your monitoring just reported critical_warning : 0x4 on an NVMe device, or smartctl printed “SMART overall-health self-assessment test result: FAILED! - NVM subsystem reliability has been degraded.” The drive is still serving I/O. Nothing in the kernel log looks catastrophic. Is the drive dying, or is this noise?

It depends on what else the SMART log says. Critical warning bit 2 is the vaguest bit in the critical warning byte. The specification language is broad: the controller has detected significant media-related errors, or some internal error, that degrades NVM subsystem reliability. What counts as “significant” and “degraded” is left to the vendor, and vendors interpret it very differently. Some drives set bit 2 only when the flash is genuinely failing. Others, notably several Samsung consumer models, set it preemptively the moment Percentage Used crosses 100 percent, which is a warranty-consumed signal, not a failure signal.

nvme nvme0: I/O timeout, Resetting controller: what an NVMe controller reset means

Mon, 01 Jan 0001 00:00:00 +0000

You are here because dmesg or journalctl -k shows a sequence like this:

nvme nvme0: I/O 24 QID 3 timeout
nvme nvme0: Abort status: 0x0
nvme nvme0: Resetting controller

The NVMe driver submitted a command, the controller did not complete it within the I/O timeout (30 seconds by default), the driver tried to abort the command, the abort did not resolve things, and the driver forced a full controller reset. During the reset-recovery cycle, which typically takes 5 to 30 seconds, all I/O to that device stalls. In-flight commands are replayed after the reset completes.

NVMe PCIe AER errors: correctable and uncorrectable transport-layer faults

Mon, 01 Jan 0001 00:00:00 +0000

You opened the kernel log because something felt off, and found lines like this repeating:

pcieport 0000:00:1b.0: AER: Corrected error received: 0000:01:00.0
nvme 0000:01:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)

That is PCI Express Advanced Error Reporting (AER), the transport layer underneath NVMe telling you the link between the root port and the drive is producing errors. These errors happen below the NVMe protocol: the drive’s SMART data can look perfectly healthy while the PCIe link is retransmitting constantly, quietly adding latency to every I/O.

NVMe PCIe link degraded: current link speed and width below maximum

Mon, 01 Jan 0001 00:00:00 +0000

An NVMe drive negotiates its PCIe link at boot, at hot-plug, and after some error events. When negotiation lands below the device’s capability, nothing breaks. The drive keeps serving I/O, SMART stays clean, and the filesystem sees no errors. The only symptom is a lower bandwidth ceiling: a Gen4 x4 drive running at Gen3 x2 delivers roughly one quarter of its rated throughput, and every application on top of it just runs slower.

NVMe percentage used at 100%: reading the endurance-consumed estimate

Mon, 01 Jan 0001 00:00:00 +0000

Your monitoring just told you an NVMe drive is at 100% “percentage used”, or 123%, or 167%. The number looks like a fuel gauge hitting empty, and the instinct is to treat it as an emergency. It is not one. percentage_used is a vendor estimate of rated endurance consumed, the NVMe specification explicitly allows it to exceed 100, and drives routinely operate past 100% for months or years. It is also monotonic: it never goes back down.

NVMe power cycles and power-on hours: fleet age and lifecycle tracking

Mon, 01 Jan 0001 00:00:00 +0000

power_cycles and power_on_hours become operational when you need to know how old a drive really is, whether it has been losing power independently of the OS, and which fleet cohort is approaching replacement together.

They are not failure alerts by themselves. Treat them as the lifecycle ledger for local PCIe-attached NVMe devices: power_cycles counts full device power-off and power-on events, not NVMe controller resets, while power_on_hours counts electrical on-time, not active I/O time. For controller-reported I/O occupancy, use controller_busy_time.

NVMe power-loss protection: knowing whether your drive has PLP at all

Mon, 01 Jan 0001 00:00:00 +0000

Power-loss protection (PLP) is the difference between an unsafe shutdown being a non-event and an unsafe shutdown being silent data corruption. Enterprise NVMe drives carry capacitors that hold the controller up long enough to flush the volatile write cache to NAND when power drops. Consumer drives do not. On a drive without PLP, every power loss, PSU failure, or hard reset risks losing writes that the kernel and the application already believe are durable.

NVMe queue depth saturation: command slots, io_timeout, and deep queues

Mon, 01 Jan 0001 00:00:00 +0000

You are looking at a host where NVMe latency has climbed, iostat shows the device busy, and nothing is erroring. No media errors, no kernel I/O error lines, SMART looks clean. The question is whether the drive is simply working as hard as it can, or whether something is wrong inside it. Queue depth is the signal that separates those two cases.

NVMe was designed for deep parallelism: each CPU core typically gets its own submission queue (SQ) and completion queue (CQ) pair, and a single queue can hold up to 64K command entries. Most enterprise drives reach peak throughput somewhere between QD 64 and QD 256 across all queues. Past that point, more outstanding commands buy nothing but latency: every extra command sits in a queue slot waiting for the controller to drain the ones ahead of it, and completion time grows linearly with how deep you stack.

NVMe sanitize, format, and secure erase: expected events versus red flags

Mon, 01 Jan 0001 00:00:00 +0000

Sanitize, Format NVM, and secure erase are the three ways an NVMe controller destroys data on its own media. They are admin commands executed inside controller firmware, and once started they are largely outside the host’s control. In a healthy fleet they appear during commissioning (occasionally) and during decommission or repurpose. Any other occurrence is a security or integrity event.

The operational problem is that these operations look alarming even when they are working correctly. Progress counters sit at 0% or 50% for hours. The controller briefly shows a resetting state. Estimated completion times report meaningless values. Operators page on all of this, or worse, they try to “fix” a running sanitize and discover it cannot be stopped.

NVMe silent data degradation: media errors, reliability bit, and failing cold reads

Mon, 01 Jan 0001 00:00:00 +0000

The drive is still there. It answers I/O, the filesystem is mounted, latency looks mostly normal, and nothing in dmesg is screaming. But media_errors has been climbing for days, critical_warning now shows bit 2 set, and a few reads per hour are inexplicably slow. A backup verification job just failed a checksum on a file nobody has written to in months.

This is NVMe silent data degradation: NAND cells are losing their ability to hold charge, reads need multiple internal retry passes, and some reads fail outright even after ECC. The device stays responsive the whole time. There is no crash to page you, only a slow accumulation of uncorrectable reads, usually on data that is written once and rarely touched.

NVMe temperature threshold exceeded: critical warning bit 1, WCTEMP, and CCTEMP

Mon, 01 Jan 0001 00:00:00 +0000

Your monitoring fired on an NVMe drive: critical_warning is non-zero, and after decoding the bitmask you find bit 1 set (value 0x02). The drive is reporting that its composite temperature has crossed a vendor-defined over-temperature threshold. The question that matters is whether this is a transient self-protecting throttle under heavy load, or a sustained thermal condition that is getting worse.

Bit 1 can fire during a perfectly healthy backup job and clear itself minutes later. It can also be the first signal of a cooling failure that ends in controller shutdown. The difference is not in the bit itself but in the surrounding counters: how long the condition has persisted, and whether time above the critical threshold is accumulating.

NVMe thermal management transitions: TMT1 and TMT2 as an early throttle signal

Mon, 01 Jan 0001 00:00:00 +0000

An NVMe drive that is thermally throttling rarely looks broken. There are no I/O errors, no kernel messages, no failed commands. Throughput drifts down, latency drifts up, and the application team files a ticket about “the database being slow.” By the time critical_warning bit 1 (temperature threshold exceeded) is set, the drive has already been protecting itself for a while.

The thermal management transition counters close that gap. They tell you the drive has entered a throttling state, how many times, and for how long, well before the warning or critical composite temperature thresholds are crossed. In the monitoring maturity model this is a Mature-level signal: it sits between “watch the composite temperature” and “the drive has declared a thermal problem.” See the NVMe mental model for where thermal throttling fits among the characteristic NVMe failure archetypes.

NVMe thermal throttling: the drive runs hot and performance quietly drops

Mon, 01 Jan 0001 00:00:00 +0000

The database is slow. CPU utilization is low, the query plan has not changed, and there are no I/O errors in dmesg, no media errors in SMART, no failed disk anywhere. Yet p99 query latency has doubled and IOPS are half of what they were last week.

This is the classic NVMe thermal throttling incident. The controller detected that it was running too hot and quietly reduced its own performance to protect itself. There is no error, no log entry, no kernel message. The drive simply gets slower until it reaches thermal equilibrium, and it gets faster again when the load drops. If you are only watching error counters and application logs, you will never see it.

NVMe throughput collapse: busy controller, low IOPS, and internal contention

Mon, 01 Jan 0001 00:00:00 +0000

The drive is flat out. Whatever metric you look at says the controller is working constantly. And yet the application is starving: IOPS are a fraction of what the drive is rated for, write throughput has fallen off a cliff, and latency is climbing. Nothing in the kernel log looks broken. No media errors. No resets. The drive is busy, and it is delivering almost nothing.

The standard instinct (“the disk is saturated”) is both right and wrong here. The controller is saturated. The host-visible I/O path is not. The drive is burning its internal capacity on work you cannot see: garbage collection, FTL remapping, wear leveling, or simply running its clocks slower because it is too hot.

NVMe unsafe shutdowns increasing: power-loss events and silent corruption risk

Mon, 01 Jan 0001 00:00:00 +0000

The unsafe_shutdowns field in the NVMe SMART log is a lifetime counter. It increments every time the drive loses power without first receiving a shutdown notification (CC.SHN) from the host. A clean reboot increments power_cycles but not unsafe_shutdowns. A power cut, kernel panic, or someone holding the power button increments both.

The counter never goes down, so the absolute number is history. What matters is the rate of change: every new increment means something cut power to the drive unexpectedly. On enterprise drives with power-loss protection (PLP) capacitors, in-flight writes in DRAM get flushed to NAND before the power dies, and the event is a footnote. On consumer drives without PLP, every increment is a data-loss roll of the dice: writes the host believes were completed may never have reached NAND.

NVMe volatile memory backup failed: critical warning bit 4 and a dead PLP capacitor

Mon, 01 Jan 0001 00:00:00 +0000

Your monitoring shows critical_warning nonzero on an enterprise NVMe drive. Decoding the bitmask, it is bit 4: volatile memory backup failed. The drive itself is behaving normally. IOPS are fine, latency is fine, media errors are zero. If you only looked at performance dashboards, nothing would look wrong.

That is exactly the problem. Bit 4 means the power-loss-protection (PLP) capacitor bank on the drive has failed or is degraded. The drive still accepts and acknowledges writes at full speed, but the moment this host loses power unexpectedly, any writes sitting in the controller’s volatile DRAM or FTL state are gone. You have lost the one feature that made every previous unsafe shutdown survivable.

NVMe warning and critical temperature time: reading the thermal-stress counters

Mon, 01 Jan 0001 00:00:00 +0000

Most NVMe thermal monitoring is live-state monitoring: you watch composite temperature and alert when it crosses a threshold. That works while you are looking. It tells you nothing about the 40 minutes last night when a backup job pushed an M.2 drive past its warning threshold and the controller quietly throttled your database.

Two fields in the NVMe SMART/Health Information log close that gap: warning_temp_time (Warning Composite Temperature Time) and critical_comp_time (Critical Composite Temperature Time). They are cumulative counters, in minutes, recording how long the drive has spent above its Warning Composite Temperature Threshold (WCTEMP) and Critical Composite Temperature Threshold (CCTEMP). They are the drive’s own thermal-stress history, kept whether or not anyone was watching.

NVMe write amplification: why data_units_written understates real NAND wear

Mon, 01 Jan 0001 00:00:00 +0000

The usual incident goes like this: percentage_used on a drive is climbing faster than planned, someone pulls data_units_written from the SMART log, divides by power-on hours, and concludes the workload is well within the drive’s DWPD rating. Six months later the drive sets critical warning bit 0 and procurement is scrambling. The math was not wrong. The input was.

data_units_written counts what the host sent to the controller. It does not count what the controller wrote to the NAND. Between those two numbers sits the flash translation layer: garbage collection relocating valid pages, wear leveling shuffling cold blocks, metadata updates, SLC cache destaging. Every one of those operations programs NAND without appearing in a host-visible counter.

NVMe write cliff: SLC cache exhaustion and garbage-collection stalls

Mon, 01 Jan 0001 00:00:00 +0000

Write throughput on an NVMe drive falls off a cliff: 50-90% below baseline, arriving as a step function rather than a gradual decline. Write latency jumps 3-10x at the same moment. The application layer starts reporting slow queries, stalled flush operations, or request timeouts, and the drive looks guilty.

The confusing part is what is absent. The drive is not hot. There are no media errors. The kernel log is quiet. SMART looks clean. Standard disk monitoring shows a busy device, which tells you nothing you did not already know.