NVMe monitoring checklist: the signals every production SSD needs

Most NVMe monitoring collects a handful of SMART counters and stops there. That works until a controller hangs without touching SMART, a PCIe link silently retrains to half speed, or a drive sets a critical warning bit that your single blanket alert treats as noise. NVMe failures rarely arrive as a clean “disk error.” They arrive as latency, throttling, stalled queues, or a device that vanishes from the bus.

Scope: local PCIe-attached NVMe devices (M.2, U.2, U.3, EDSFF). NVMe-oF, ZNS, and SPDK userspace configurations have different monitoring models and are out of scope.

The checklist is four maturity levels. Adopt Level 1 everywhere, Level 2 on anything serving production traffic, and Levels 3 and 4 where NVMe is on the critical path. Each signal lists its source and the severity it deserves.

flowchart TD
  PCIe[PCIe transport layer] -->|AER counters, link speed and width via sysfs| Host
  Ctrl[NVMe controller] -->|SMART log 0x02: critical_warning, spare, temp, errors| Host
  Ctrl -->|state via /sys/class/nvme/nvmeX/state| Host
  Media[NAND flash media] -->|media errors, spare consumption| Ctrl
  Host[Host monitoring] --> Alerts[Alerting: PAGE / TICKET / PLAN]

Level 1: survival

The minimum to know the drive is alive and not refusing I/O.

Controller state is live. The single most direct availability signal. Read /sys/class/nvme/nvmeX/state; expected values include live, resetting, connecting, deleting, dead, and new. Any non-live state sustained more than 30 seconds in production is a PAGE. Brief resetting during a host-initiated reset (driver reload, firmware activation) is expected; sustained states are not. This is not in nvme smart-log and most metrics agents do not collect it.
Critical warning bit 3 (read-only mode). The drive has placed its media in read-only mode; all writes are rejected. This is a binary hardware fault, typically spare block exhaustion. PAGE unconditionally. Manual check: nvme smart-log /dev/nvme0 | grep critical_warning and test for bit 3 (0x08).
Device presence. /dev/nvme* accessible, /sys/class/nvme/ populated. Disappearance means PCIe loss, fatal uncorrectable error, or physical failure. PAGE.

Level 2: operational

What anyone running NVMe in production should be embarrassed to miss. This level turns SMART data into per-bit alerts with correct severity.

Per-bit critical warning alerts. Do not alert on critical_warning != 0 as one blanket rule. The bits have different severity and response:
- Bit 0 (available spare below threshold): TICKET. A replacement signal, not a 3 a.m. emergency. The drive still works.
- Bit 1 (temperature threshold exceeded): TICKET alone, since WCTEMP can trip transiently under sustained load. Escalate to PAGE if sustained over 5 minutes AND critical_comp_time is actively increasing.
- Bit 2 (NVM subsystem reliability degraded): TICKET alone (vendor definitions vary, and drives past 100% endurance can set this while still functional). PAGE when combined with a rising media_errors rate.
- Bit 3 (read-only): PAGE, always.
- Bit 4 (volatile memory backup failed): TICKET. The drive works, but in-flight writes are now unprotected against power loss. Consumer drives without PLP never set this bit.
Media errors rate. media_errors is a lifetime counter of uncorrectable data integrity failures. TICKET on any rate of increase above zero; PAGE when paired with critical warning bit 2. Rate of change is the actionable signal, not the absolute value. Single-digit errors over years on enterprise drives can be within tolerance.
Percentage used (endurance). A monotonically increasing counter that can exceed 100%. PLAN at > 80% (procure), TICKET at > 90% (schedule swap). Never PAGE on this alone; the real failure signals have their own alerts. A rate above roughly 1% per week suggests write amplification problems or a workload mismatched to the drive class.
Available spare vs spare threshold. When avail_spare approaches spare_thresh (typically 10%, vendor-dependent), the drive is running out of good blocks. Warning at or below 2x threshold, critical at or below threshold. Track the consumption rate: spare depletion accelerates non-linearly, and the last 10% goes much faster than the first.
Composite temperature. NVMe controllers throttle at vendor-defined thresholds (WCTEMP/CCTEMP), and throttling looks like “the application got slow” with zero errors. Typical WCTEMP is around 70-85C depending on drive class; the actual values live in Identify Controller data (nvme id-ctrl), not the SMART log, so do not hardcode universal thresholds. Watch the trend and correlate with throughput.
Unsafe shutdowns rate. Each increment is a power event without proper shutdown notification. On drives without power-loss protection, each one carries data loss risk. TICKET on any increase during normal operation; investigate power infrastructure, not the drive.

Level 3: mature

Leading indicators and composite patterns. This is where you catch failures weeks early instead of minutes.

Warning and critical composite temperature time. Cumulative minutes above WCTEMP and CCTEMP. A rate of change above zero means the drive is spending time in thermal stress even if you missed the live event. Some drives do not implement these counters (always 0).
Thermal management transitions. thm_temp1_trans_count and thm_temp2_trans_count count entries into light and heavy throttling. Rising transition counts with normal-looking current temperature means chronic undercooling.
Error log entries rate. num_err_log_entries covers more than media errors: admin command errors, I/O command errors, internal faults. A rising rate without rising media errors points at firmware or driver issues. Read nvme error-log /dev/nvme0 early; the log is circular and high error rates overwrite root cause.
Controller busy time vs host throughput. controller_busy_time (minutes resolution) compared against power-on time gives internal utilization. High busy time with low host IOPS means the controller is burning cycles on garbage collection, remapping, or throttling. Rough rule: busy ratio above 90% with low delivered IOPS means a sick drive.
Available spare consumption trajectory. Fit a rate to the spare decline and project the threshold crossing. Accelerating consumption is one of the strongest failure predictors NVMe gives you.
PCIe AER error counters. See the next section; these live in sysfs, not SMART.

Level 4: expert

The signals operators add after their third storage incident.

PCIe link speed/width degradation detection, continuously, not just at provisioning.
Per-namespace I/O statistics on multi-namespace drives, where one namespace’s abuse can starve the others.
Latency distribution (p99/p99.9 via eBPF or blktrace). Block layer stats only give averages, and NVMe mean latency is almost always fine; the 10ms tail outliers are what break SLAs.
Vendor-specific log pages (0xC0-0xCF) for DRAM cache state, GC duty cycle, and NAND write counts. This is the only way to estimate true write amplification; data_units_written counts host-visible writes, not NAND writes.
Firmware version tracking across the fleet, so vendor advisories map to actual drives.

The signals most collectors miss

The Level 1 controller state check and the entire PCIe transport layer do not come from the NVMe SMART log. Netdata’s nvme collector, for example, covers endurance, available spare, composite temperature, I/O volume, power cycles, critical warnings, unsafe shutdowns, media errors, and power-on time, but not controller state, PCIe AER, link negotiation, or controller resets. Those need sysfs and dmesg:

# Controller state (PAGE if not "live" for >30s)
cat /sys/class/nvme/nvme0/state

# PCIe AER counters (any sustained non-zero rate is abnormal)
cat /sys/class/nvme/nvme0/device/aer_dev_correctable
cat /sys/class/nvme/nvme0/device/aer_dev_fatal
cat /sys/class/nvme/nvme0/device/aer_dev_nonfatal

# Link negotiation: current should match max
cat /sys/class/nvme/nvme0/device/current_link_speed
cat /sys/class/nvme/nvme0/device/max_link_speed
cat /sys/class/nvme/nvme0/device/current_link_width
cat /sys/class/nvme/nvme0/device/max_link_width

# Controller resets and timeouts (TICKET on one, PAGE on 2+ per hour)
dmesg | grep -i "nvme.*reset\|nvme.*timeout"

Why each matters:

Controller state: a controller stuck in resetting is a firmware crash or PCIe problem in progress. dead means the kernel gave up.
AER counters: correctable errors (RxErr, BadTLP, BadDLLP) mean marginal signal integrity; they are silently retransmitted, so there is no data loss, but latency suffers. Uncorrectable errors are severe. In healthy systems these counters are zero; AER is the correct source for transport errors. Root access is typically required, and containers may not see sysfs.
Link speed/width: a Gen4 x4 drive retrained to Gen3 x2 delivers one quarter of its bandwidth with zero errors anywhere. Compare current against max; any mismatch is a TICKET. Check for deliberate bifurcation before assuming a fault.
Controller resets: each reset is a 5-30 second I/O stall that cascades into application timeouts. Host-initiated resets (driver reload, VM reboot) are expected; repeated spontaneous resets are a dying controller or firmware bug.

What most teams get wrong

One blanket alert for critical_warning != 0. Bit 3 is an outage, bit 0 is a procurement task. Per-bit severity is the whole point.
Ignoring NVMe temperature while monitoring CPU temperature. An M.2 drive can throttle a database to half speed while every other metric looks healthy.
Missing the PCIe transport layer. A slightly loose connector causes thousands of retransmissions and 10x latency while SMART reports perfect health.
No unsafe shutdown alerting, because “the server came back fine.” The corruption shows up later at the filesystem or application layer.
Treating data_units_written as NAND writes. True write amplification requires vendor-specific telemetry; SMART-only math underestimates wear.
Not baselining PLP capability at provisioning. Consumer drives without power-loss protection look identical to enterprise drives in SMART until the first power event.

How Netdata helps

Netdata’s nvme collector polls the SMART/Health log and exposes the critical warning bitmask as individual dimensions in nvme.device_critical_warnings_state, so you can alert per bit with the severity split this checklist describes instead of one noisy rule.
nvme.device_media_errors_rate and nvme.device_error_log_entries_rate convert lifetime counters into rates, which is the actionable form; correlating a media error rate increase with critical warning bit 2 is the PAGE condition for active media degradation.
nvme.device_estimated_endurance_perc, nvme.device_available_spare_perc, and nvme.device_composite_temperature give the endurance, spare, and thermal trends for Levels 2 and 3, including spare consumption trajectory.
nvme.device_unsafe_shutdowns_count makes power events visible as a rate, and the warning/critical composite temperature time charts catch thermal stress you missed live.
The gaps are explicit: controller state, PCIe AER counters, link speed/width, and controller resets are not in the nvme collector. Pair Netdata with sysfs scraping and kernel log monitoring for the Level 1 availability signal and the transport layer.