$ guides / nvme ▌

NVME · OPERATIONS PLAYBOOK

NVMe fails quietly, then all at once: it throttles, it wears, and one day the controller stops answering

A flash SSD is a small computer that grades its own homework. It reports its health over SMART, throttles under heat with no error, and wears on a curve you only see if you trend it — until it crosses a cliff and goes read-only, drops into a reset loop, or vanishes from the PCIe bus. We trace how the drive behaves under load, which signals lead the failure, and what to do when the cliff arrives.

> Start with the monitoring checklist → # Jump to the full guide list

NVMe defaults look healthy right up until they don't. The drive self-reports, degrades on a monotonic curve, and hides its worst problems behind a green SMART status.

The drive works. Until an M.2 with no heatsink crosses WCTEMP under a backup job and throttles database queries while the CPU sits idle. Until available_spare drifts below the vendor threshold, the last bad block finds no replacement, and the controller sets critical_warning bit 3 and rejects every write. Until a firmware bug hangs the controller, the driver logs I/O timeout, Resetting controller, and I/O stalls for thirty seconds. Until a fatal PCIe error trips nvme0: Removing and the block device is simply gone. Until media_errors climbs on cold reads and data you thought was safe is quietly rotting.

These guides are written for engineers who already run NVMe storage, not for people learning what an SSD is. The goal is the mental model of how the controller, the flash translation layer, and the NAND actually behave under load; the failure patterns that keep recurring; the monitoring story that catches them before they page anyone; and the runbooks you wish someone had handed you before your last incident. Scope is local PCIe-attached NVMe — NVMe-oF, ZNS, and SPDK have different monitoring models and are out of scope.

How NVMe actually runs in production

NVMe is not a disk. It is a PCIe endpoint with an embedded controller running firmware, a flash translation layer mapping logical blocks onto wearing NAND, and a SMART log through which the drive reports its own health. Most production failures live between these layers — a thermal throttle, a link trained down, a garbage-collection stall, a controller reset — not in any one of them.

host + block layer

The OS side. Applications issue reads and writes through the block layer to <code>/dev/nvmeXnY</code>; the I/O scheduler, queue depth, and the <code>io_timeout</code> (default 30s) live here. When the device stalls, this is where <code>blk_update_request: I/O error</code> and command timeouts surface.

HOST

PCIe transport

The drive is a PCIe endpoint that negotiates a link generation (Gen3/4/5) and width (x1-x4). The link can quietly train down — a Gen4 x4 device at Gen3 x2 delivers a quarter of the bandwidth with no error. PCIe AER reports correctable and uncorrectable faults before NVMe even sees them.

LINK

NVMe controller

An embedded computer with firmware and DRAM (or borrowed host memory via HMB). It exposes submission/completion queue pairs — usually one per CPU core — and executes commands via DMA. When it hangs, the driver forces a controller reset; when it dies, the device disappears.

CTRL

flash translation layer

The FTL maps logical block addresses onto physical NAND pages and hides wear leveling, garbage collection, and bad-block remapping. It is invisible to the host but determines write amplification and latency variance. When it cannot free blocks fast enough, host writes hit the write cliff.

FTL

NAND media

Many NAND dies across channels, operating in parallel. Cells have finite program/erase cycles; as they wear out the controller retires them to spare blocks. When spares are exhausted the drive goes read-only, and uncorrectable <code>media_errors</code> begin to climb.

NAND

SMART / health log

The drive grading its own homework: <code>critical_warning</code> bits, <code>percentage_used</code>, <code>available_spare</code>, <code>media_errors</code>, temperature. Self-reported and polled — it can lag, reset after a controller reset, or (on cheap drives) be fabricated. Trust it, but corroborate with kernel and block-layer observations.

HEALTH

thermal + power envelope

Controllers throttle at <code>WCTEMP</code>/<code>CCTEMP</code> to protect themselves — performance drops with no error. Power-loss protection (enterprise drives) guards in-flight writes; consumer drives rely on a volatile cache, so every unsafe shutdown risks losing acknowledged writes.

THERMAL

Why this matters: 'the disk is slow' or 'the drive is failing' can come from a thermal throttle, a PCIe link trained down, a garbage-collection write cliff, a controller reset stall, NAND wear, a deep queue, or a firmware bug. The symptom rhymes, but each layer has a different signal — and a different fix.

The failures you'll actually see

Most NVMe incidents fall into a small set of recurring patterns. Recognise the shape, and triage gets dramatically faster.

CRITICAL

The controller reset loop

The controller stops answering, the driver waits out the io_timeout, logs Resetting controller, and replays in-flight commands — then the same firmware bug hangs it again. Each cycle is a 5-30 second I/O stall that cascades into database timeouts and request failures. No environmental correlation (temperature normal, PCIe clean, SMART clean) points at firmware, not hardware.

nvme nvme0: I/O QID N timeout, Resetting controller in the kernel log
I/O latency spiking to the 30s timeout before each reset
Brief recovery, then another stall — repeating
SMART and PCIe AER clean; temperature normal

Investigate →

CRITICAL

Sudden I/O errors in the log

The block layer starts returning blk_update_request: I/O error, dev nvme0n1. A single one can be a latent bad block surfaced on a cold read; a storm means a dying controller or a device on its way off the bus. Applications see EIO, mounts go read-only, and filesystems may need a check. The kernel log, not the application log, holds the root cause.

blk_update_request: I/O error, dev nvme0n1 in dmesg
Filesystem remounting read-only or throwing EIO
Errors clustered around a controller reset or removal event
SMART media_errors climbing in step (if NAND is the cause)

Investigate →

CRITICAL

The read-only cliff

Spare NAND blocks run out, the next bad block finds no replacement, and the controller sets critical_warning bit 3 — placing the media in read-only mode to preserve data. Every write is rejected. This is binary and cannot be transient: it is the drive self-protecting at end of life, usually after available_spare has been declining for weeks.

critical_warning bit 3 (read_only) set in nvme smart-log
All writes failing while reads still succeed
available_spare at or below the vendor threshold
media_errors elevated; percentage_used high

Investigate →

CRITICAL

The vanishing drive

The device disappears from the PCIe bus entirely — nvme0: Removing in the log, /dev/nvmeXnY gone, often preceded by a fatal PCIe AER error. Physical failure, a fatal transport error triggering OS isolation, loose seating, or power delivery loss. All I/O fails and any mounted filesystem is now damaged. In hot-swap chassis, an intentional pull looks identical.

nvme nvme0: Removing in the kernel log
/dev/nvmeXnY and /sys/class/nvme/nvmeX gone
pcieport: AER: Uncorrectable (Fatal) just before removal
All I/O to the device failing with EIO

Investigate →

ACTIVE

Silent data degradation

NAND cells lose retention and reads need multiple retry passes — some fail. media_errors climbs, the controller sets critical_warning bit 2 (reliability degraded), and read latency turns sporadically spiky. Errors land on cold data while the device stays responsive, so nothing pages until the corruption is discovered downstream. The data may be recoverable from RAID or backup now, not later.

media_errors rate rising, especially on read-heavy paths
critical_warning bit 2 (nvm_subsystem_reliability) set
Sporadic high read latency on specific ranges
available_spare declining, percentage_used high

Investigate →

IMMINENT

Wearing out unwatched

The drive is consuming its endurance faster than anyone is tracking. available_spare drifts toward the vendor threshold and percentage_used climbs, but because both look fine today nobody projects the trajectory. Spare depletes non-linearly — the last 10% goes fastest — so a drive that looked healthy at a quarterly check can hit the read-only cliff weeks later.

critical_warning bit 0 (available_spare) set, or spare near threshold
available_spare consumption rate accelerating
percentage_used past 80% and gaining >1% per week
No procurement or replacement scheduled

Investigate →

Choosing a tool

Best NVMe & SSD Health Monitoring Tools: 9 Options Ranked

A ranked review of the tools teams actually shortlist here, what each one is genuinely good at, and how the pricing behaves as you scale.

Read the buyer's guide →

NVMe monitoring maturity levels

NVMe observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much the data on the drive matters. Most production hosts should land at the second level.

Level 1: Survival

Know that the drive is in trouble

Survival monitoring is the floor. With these signals you can answer one question: is the drive present, accepting writes, and not throwing I/O errors? You will not learn why it is degrading, but you will learn that it broke before the application does. Survival is enough for scratch disks and non-critical hosts.

Controller state / device present Is /dev/nvme* there and is the controller state live, not dead or resetting?
Read-only mode (critical_warning bit 3) Writes rejected — the drive has self-protected at end of life.
Controller resets in the kernel log Each Resetting controller is a multi-second I/O stall.
I/O errors (blk_update_request) The block layer is surfacing EIO from the device.
SMART critical_warning nonzero Any bit set means the drive is signaling a hardware problem.

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production hosts should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident on its own: wear-out, thermal throttling, media errors, power-loss events, and the difference between each critical-warning bit.

Per-bit critical warnings Each bit has its own severity and response; never one blanket alert.
Percentage used (endurance consumed) Rated endurance burned; procure at 80%, replace at 90%.
Available spare vs vendor threshold How much wear runway is left before read-only mode.
Media errors rate New uncorrectable errors mean active NAND degradation.
Composite temperature Throttling drops performance with no error to show for it.
Unsafe shutdowns rate Power-loss events and, without PLP, silent corruption risk.
Warning / critical temperature time Cumulative thermal-stress history even if you missed it live.
Error log entries rate Broader error activity than media errors alone.

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone up. A spare-consumption rate bending upward, thermal transitions starting, a busy controller delivering low throughput, PCIe correctable errors trickling in, tail latency creeping. None of these page you on day one. They become the page-out incident on day thirty.

Available spare consumption rate The trajectory predicts premature failure months ahead.
Thermal management transitions (TMT1/2) Active throttling before critical_warning bit 1 fires.
Controller busy time vs throughput Busy but low IOPS means internal contention, not load.
PCIe AER counters Signal-integrity faults invisible at the I/O layer.
I/O latency and the tail p99/p999 outliers break SLAs, not the average.
GC write-cliff detection Sudden throughput drop at normal temperature.
Power-on hours and power cycles Fleet age and lifecycle correlation.
Data units written trend Catch a runaway writer accelerating wear.

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. A silently down-trained PCIe link, one namespace starving another, per-I/O latency histograms, vendor log pages, firmware version drift. Most hosts never need every signal here. Add the ones your incident history says you do.

PCIe link speed / width vs maximum A silent bandwidth cap with zero errors reported.
Per-namespace I/O statistics One namespace's abuse starving another's spare and GC.
Latency distribution via eBPF The percentile detail /sys/block averages cannot give.
Vendor log pages (0xC0-0xCF) DRAM cache, GC duty cycle, and true NAND-write telemetry.
Firmware version tracking Known-bad versions and reset-loop triggers.
Write amplification estimation Real NAND wear beyond host-visible writes.
Sector size / LBA format (512e vs 4K) A format mismatch is invisible I/O amplification.
Power-loss-protection baseline Whether the drive protects in-flight writes at all.

Operating mistakes worth avoiding

The traps NVMe teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

One blanket alert on critical_warning != 0

The <code>critical_warning</code> field is a bitmask, and the bits are not equal. Bit 3 (read-only) is an immediate outage; bit 1 (temperature) can be a transient throttle under legitimate load; bit 0 (spare below threshold) is a replacement signal, not a 3am page. A single alert on 'nonzero' either pages on everything or buries the one bit that mattered. Alert per bit, with the right severity for each.

Watching CPU temperature but not NVMe temperature

Teams monitor CPU and chassis temperature and never look at the drive. An M.2 SSD with no heatsink can hit 80C under a sustained write while the CPU sits idle, throttling database performance with no error anywhere. The <code>warning_temp_time</code> counter reveals thermal-stress history even when the live spike was missed.

Checking available spare but not its consumption rate

A drive at 40% spare looks fine; a drive at 40% spare that was 70% last month is weeks from the read-only cliff. Spare depletes non-linearly — the last 10% goes fastest — so the current value is not the signal. The rate of change is one of the strongest leading indicators of premature failure, and it gives months of warning.

Ignoring the PCIe transport layer

SMART can be spotless while a loose M.2 connector generates thousands of PCIe retransmissions and latency is 10x worse. AER correctable errors and a link that has trained down to a lower speed or width are invisible at the I/O layer and are not in the SMART log — they need sysfs. It is the diagnostic teams consistently skip.

Not alerting on unsafe shutdowns

Each unsafe shutdown is a power-loss event, and on a drive without power-loss protection it risks losing acknowledged-but-unpersisted writes. Teams ignore <code>unsafe_shutdowns</code> because 'the server came back up fine' — but the damage is silent filesystem or application corruption discovered days later, long after the power event scrolled off the logs.

Not knowing whether drives have PLP

The presence or absence of power-loss protection is invisible in SMART. A consumer drive without PLP in a write-intensive database silently risks data loss on every power event, and there is no bit that says 'PLP absent' — bit 4 only fires when an existing capacitor fails. Baseline PLP capability at provisioning, not during the post-mortem.

Monitoring average latency instead of the tail

NVMe mean latency is almost always fine. The operational damage comes from p99 and p999 — the 10ms+ outliers that cause query timeouts and deadline misses. The standard <code>/sys/block</code> interface reports only averages; without eBPF or blk-mq histograms the tail that actually breaks your SLA is invisible.

Treating an application timeout as an application problem

A controller reset causes a 5-30 second I/O stall that applications see as timeouts and I/O errors. Teams investigate the database or the request layer and never check the kernel log for the underlying <code>Resetting controller</code> that caused the cascade. The application error is the symptom; the NVMe reset is the cause.

NVMe runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

Controller availability and resets

▸

SMART critical warnings

▸

Endurance and wear

▸

Media errors and data integrity

▸

Temperature and thermal throttling

▸

PCIe transport and link health

▸

Latency, queues, and performance

▸

Power loss and data integrity

▸

Firmware, namespaces, and formatting

WHERE TO GO NEXT

Setting up NVMe monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.

> Start with the checklist > Back to Operations Guides