How NVMe actually works in production: a mental model for operators

Most storage incidents on NVMe are misdiagnosed for the same reason: operators debug them as if NVMe were a faster SATA disk. It is not. NVMe is a host-to-controller communication protocol that exposes flash storage over PCIe, and the failure modes live in places generic disk monitoring never looks: the PCIe link, the controller firmware, and the flash translation layer sitting between your filesystem and the NAND.

The mental model that makes NVMe behavior predictable has three layers: the PCIe transport, the controller, and the flash media. Once you hold this model, most “mystery slowness” and “healthy SMART, dead drive” incidents stop being mysterious. Scope here is local PCIe-attached NVMe. NVMe-oF, ZNS, and SPDK change the monitoring model fundamentally and are not covered.

What it is and why it matters

A SATA disk is mostly a passive device: the OS sends commands, the disk obeys, and SMART is a thin reporting layer. An NVMe device is a small embedded computer. It has its own firmware, its own DRAM (or borrowed host RAM), its own scheduler, and an internal mapping layer that constantly relocates your data without telling you. It manages wear, heat, and error correction autonomously, and it makes decisions, like throttling or going read-only, that your OS only learns about after the fact.

This is why NVMe failure archetypes look the way they do:

1. It gets slow before it dies. Latency rises as FTL overhead increases, long before outright failure.
2. It runs out of spare blocks. Write amplification exhausts over-provisioned space and performance collapses.
3. It thermal-throttles silently. No error, just gradually decreasing performance until thermal equilibrium.
4. The controller hangs. A firmware bug or internal fault stops all I/O until the kernel resets it.
5. It wears out predictably but silently. SMART shows declining life but nothing alerts until critical.
6. The PCIe link degrades. Signal integrity issues cause retransmissions that look like high latency.

Every one of these maps to a specific layer. Debugging NVMe well means knowing which layer you are looking at.

How it works: the three layers

flowchart TD
  APP[Application / filesystem] --> BLK[Linux block layer]
  BLK -->|writes commands into SQ| SQ[Submission queues in host RAM]
  SQ -->|DMA fetch| CTRL[NVMe controller]
  CTRL -->|completion entries| CQ[Completion queues]
  CQ -->|interrupt / MSI-X| BLK
  CTRL -->|FTL: LBA to NAND page map| FTL[Flash translation layer]
  FTL --> NAND[NAND dies across channels]
  PCIE[PCIe link: Gen x width, AER error reporting] -.transport.- CTRL

Layer 1: the PCIe transport

The NVMe device is a PCIe endpoint. It negotiates a link with a specific generation (Gen3/Gen4/Gen5) and width (x1/x2/x4), and that negotiation is not guaranteed to hold.

The critical operational fact: the link can degrade silently. A Gen4 x4 device running at Gen3 x2 delivers one-quarter of its rated bandwidth with zero errors visible to the filesystem, zero SMART changes, and nothing in dmesg. The drive looks healthy. Everything is just slower.

PCIe also has its own error reporting layer, AER (Advanced Error Reporting), that operates below NVMe entirely. Correctable errors are retransmitted transparently, so there is no data loss, but each retransmission is latency. A slightly loose M.2 connector can cause thousands of retransmissions: the drive reports healthy, and latency is 10x worse. This is the diagnostic signal teams most consistently miss.

Layer 2: the controller

The controller is where almost all interesting behavior lives. Its key components:

• Submission and completion queues. The host writes commands into submission queue (SQ) entries in host memory; the controller fetches them via DMA, processes them, and writes completion entries to the completion queue (CQ). Each CPU core typically gets its own queue pair, and queues can hold up to 64K entries each. (The Linux driver defaults to 1024 entries per I/O queue, settable via the io_queue_depth module parameter.) Queue depth and queue count determine how much parallelism the drive can exploit. A dedicated admin queue (always queue 0) handles management commands like identify, log pages, and firmware operations.
• DRAM or HMB. Enterprise drives carry their own DRAM for mapping tables and write buffering. Consumer drives are often DRAM-less and borrow host RAM via HMB (Host Memory Buffer). HMB is used mainly as an L2P table cache, so the performance penalty shows up mainly on random I/O.
• The flash translation layer (FTL). Maps logical block addresses to physical NAND pages and handles wear leveling, garbage collection, and bad block management. The FTL is entirely opaque to the host, and its behavior under pressure is the single largest source of latency variance in NVMe.
• Garbage collection. NAND must erase entire blocks before rewriting. GC moves valid pages out of partially used blocks to free them. When the drive is full or worn, GC competes directly with host I/O for controller resources, causing latency spikes.
• Wear leveling. Distributes writes across blocks so hot blocks do not die early. Consumes background bandwidth.
• Write buffer / SLC cache. Most controllers program part of the NAND in fast pseudo-SLC mode to absorb write bursts. When sustained writes exhaust it, throughput drops off a cliff, 50 to 80 percent in a step function, with no errors. This is by design, not a defect.
• Thermal management. The controller throttles itself at vendor-defined thresholds (WCTEMP for warning, CCTEMP for critical). Throttling is invisible to the OS unless you watch temperature and throughput at the same time.

Layer 3: the flash media

Modern devices have multiple NAND dies operating in parallel across channels. Write latency depends on how well the controller distributes writes across that parallelism, which is an FTL problem again.

NAND cells have finite program/erase cycles, varying enormously by cell type (on the order of 100K cycles for SLC down to roughly 1K for QLC). As blocks wear out, the controller remaps them to spare blocks. When spares run out, the error rate accelerates and the drive approaches read-only mode. Two more media behaviors matter operationally: read disturb (repeated reads to a block can flip bits in adjacent cells, forcing the controller to relocate data in the background) and retention (stored charge leaks over time, faster at high temperature and high P/E counts, so cold data can rot).

Where it shows up in production

Map each symptom to a layer first, then confirm with the layer-specific signal:

A few deployment facts that change which layer dominates:

• Consumer vs enterprise. Consumer drives lack DRAM (HMB instead), lack power-loss protection, throttle aggressively, and have lower endurance. Enterprise drives have full DRAM, PLP capacitors, higher DWPD, and more consistent latency. The same thresholds cannot be applied to both.
• M.2 vs U.2/U.3/EDSFF. M.2 drives lack active cooling and thermal-throttle much faster. An M.2 drive without a heatsink under sustained write load will throttle while the CPU sits idle.
• Drive fullness is a performance variable. Below roughly 80% fill the FTL has room to work; past that, GC pressure rises and write performance degrades. Maintaining 15-20% free space is operational headroom, not waste. TRIM matters for the same reason: without it, the FTL thinks the drive is fuller than it logically is.

Why SMART is not ground truth

Every SMART value is self-reported by controller firmware. That has three consequences.

First, firmware can be wrong or silent. Controllers can stop updating SMART counters during internal error states, and cheap drives may never assert critical warning bits even while failing. If SMART values have not changed in 24+ hours on an active drive, the data is stale, not stable.

Second, the fields are vendor estimates, not measurements. percentage_used is the vendor’s prediction of consumed endurance; the spec allows values above 100 and drives routinely run at 150% or more. The critical_warning field is a single bitmask where bit 2 (“reliability degraded”) is asserted at the vendor’s discretion about what counts as significant media or internal errors.

Third, SMART only covers the media and controller layers. The PCIe transport is invisible to SMART entirely. A drive at Gen3 x2 with accumulating AER retransmissions reports perfect health.

The fix is cross-referencing: treat SMART as the drive’s opinion, and check it against OS-level observations (block layer I/O counters, kernel logs, sysfs link status, AER counters) before trusting it.

Signals to watch in production

Note the split: SMART fields come from nvme smart-log, but controller state, link status, and AER counters are sysfs and kernel-log only. A monitoring setup that only polls smart-log sees roughly two of the three layers.

How Netdata helps

• Netdata’s NVMe collector breaks critical_warning into per-bit dimensions (read_only, nvm_subsystem_reliability, available_spare, temp_threshold, volatile_mem_backup_failed), so you can alert per bit with the right severity instead of one blanket alert for the whole byte.
• Lifetime SMART counters are exposed as rates where the rate is the signal: media errors, error log entries, and unsafe shutdowns only matter when they increment.
• Endurance signals (percentage_used, available spare) are charted over time, which makes the consumption trajectory visible. The rate of spare depletion is a stronger leading indicator than the current value.
• Composite temperature is charted alongside throughput, so the telltale thermal-throttling correlation (temperature up, throughput down, zero errors) is visible on one dashboard instead of requiring manual cross-referencing.
• Warning and critical composite temperature time counters are collected, so thermal stress that happened between polls is not lost.

Two gaps to cover yourself: the PCIe transport layer (link speed/width, AER counters) is not collected by the NVMe collector and requires sysfs access, and controller resets live only in kernel logs.

Related guides

• NVMe monitoring checklist: the signals every production SSD needs
• NVMe monitoring maturity model: from survival to expert
• nvme nvme0: I/O timeout, Resetting controller: what an NVMe controller reset means
• blk_update_request: I/O error, dev nvme0n1: reading NVMe I/O errors in the kernel log
• NVMe device disappeared: nvme0: Removing and a drive that fell off the PCIe bus
• NVMe controller state not live: reading resetting, deleting, and dead from sysfs
• NVMe controller reset loop: repeated resets from a firmware hang
• NVMe critical_warning is nonzero: decoding the SMART critical warning bitmask
• NVMe drive in read-only mode: critical warning bit 3 and rejected writes
• NVMe available spare below threshold: critical warning bit 0 and end-of-life wear
• NVMe temperature threshold exceeded: critical warning bit 1, WCTEMP, and CCTEMP
• NVMe NVM subsystem reliability degraded: critical warning bit 2