NVMe monitoring maturity model: from survival to expert

Most teams discover their NVMe monitoring gaps during an incident: a drive throttling silently at 80 C while every dashboard shows green, or a Gen4 x4 device running at Gen3 x2 for weeks with zero errors. NVMe fails differently from SATA and SAS. It has a PCIe transport layer with its own error reporting, a controller running a flash translation layer that causes latency variance invisible to block-layer averages, and wear signals that only matter as rates of change. Generic disk monitoring misses all of this.

This article lays out a four-level maturity model for monitoring local PCIe-attached NVMe devices. Each level lists the signals to collect, the commands to check them manually, and the criteria for graduating to the next level. Use it to audit your current posture or to sequence improvements after an incident review.

Scope note: this covers local PCIe-attached NVMe. NVMe-oF, Zoned Namespaces, SPDK userspace drivers, and NVMe behind hardware RAID have fundamentally different monitoring models and are out of scope.

flowchart TD
  L4["Level 4 - Expert: link speed/width, per-namespace stats, p99 latency, vendor logs, WAF"]
  L3["Level 3 - Mature: thermal transitions, error-log rate, busy time, spare rate, PCIe AER"]
  L2["Level 2 - Operational: per-bit warnings, wear, spare, media errors, temp, unsafe shutdowns"]
  L1["Level 1 - Survival: device present, controller live, read-only bit"]
  L1 --> L2 --> L3 --> L4

Level 1: survival

The bare minimum: know the drive exists, the controller is alive, and the drive has not locked itself read-only.

Signals:

• Device presence. /dev/nvmeXnY block devices exist and are readable. Disappearance from /sys/class/nvme/ means the device left the bus: hardware failure, fatal PCIe error, or physical removal.
• Controller state. cat /sys/class/nvme/nvme0/state returns one of live, resetting, connecting, deleting, dead, new. Any state other than live sustained for more than 30 seconds is abnormal in production. Brief resetting during firmware-initiated sanitize or self-test is expected; check nvme sanitize-log /dev/nvme0 to distinguish.
• Critical warning bit 3 (read-only mode). The drive has refused writes, almost always because spare blocks are exhausted. This is a PAGE unconditionally: binary hardware fault, never transient, never load-dependent.
• Kernel log for resets and removal. A controller reset stalls all I/O for 5-30 seconds while commands replay. One reset is a TICKET; two or more in an hour with failed recovery is a PAGE.

# Level 1 checks
ls /sys/class/nvme/
cat /sys/class/nvme/nvme0/state
nvme smart-log /dev/nvme0 | grep critical_warning   # bit 3 (0x08) = read-only
dmesg | grep -i "nvme.*reset\|nvme.*timeout"

Graduate when: alerts fire on non-live controller state, read-only mode, and device disappearance, and someone has verified those alerts actually page.

Level 2: operational

What a professional running NVMe in production would be embarrassed to miss. The core shift from Level 1 is per-bit alerting on critical_warning instead of one blanket alert, plus trending on wear and thermal counters.

Per-bit critical warning alerts

A single critical_warning != 0 alert is the most common NVMe monitoring mistake. The bits have different meanings and different response requirements:

Wear, errors, and thermal counters

All from nvme smart-log /dev/nvme0:

• percentage_used (endurance consumed). Monotonic, can exceed 100. PLAN above 80, TICKET above 90. Never PAGE on this alone; enterprise drives routinely run past 100. Rate of change above about 1% per week suggests write amplification problems or a workload mismatch (swap on NVMe, excessive logging).
• available_spare vs spare_thresh. TICKET at or below 2x the vendor threshold, TICKET again at or below the threshold itself (where bit 0 also fires). Spare consumption accelerates non-linearly: the last 10% depletes faster than the first 10%.
• media_errors rate. Lifetime counter of uncorrectable errors; the rate of increase is the signal. Any new error during operation is a TICKET for same-shift investigation. A single error during a cold-start scan or backup may just be a latent bad block surfacing.
• Composite temperature. SMART reports it in Kelvin in JSON (subtract 273.15); hwmon at /sys/class/nvme/nvme0/hwmon*/temp1_input reports millidegrees Celsius. The actionable threshold logic lives in bit 1, since WCTEMP and CCTEMP are in Identify Controller data, not the SMART log.
• unsafe_shutdowns rate. Each increment is a power-loss event without shutdown notification. TICKET per increment. On drives without PLP, every one carries data loss risk.
• warning_temp_time and critical_comp_time. Cumulative minutes above WCTEMP and CCTEMP. A rising rate on either means ongoing thermal stress, even if you missed the live event. Some drives do not implement these counters and always report zero.

# Level 2 baseline pull
nvme smart-log /dev/nvme0
nvme smart-log /dev/nvme0 --output-format=json   # for automation

One tooling caveat if you script against JSON: nvme-cli 2.11 changed nvme list --output-format=json from a flat Devices[] array to a nested subsystem/controller/namespace structure, which broke existing automation. Pin and test your nvme-cli version when upgrading.

Graduate when: each warning bit pages or tickets independently, wear and spare trends feed replacement procurement, and unsafe shutdowns are investigated rather than shrugged off.

Level 3: mature

Level 3 adds leading indicators and cross-signal correlation. The goal is to catch degradation weeks before it becomes a Level 1 or Level 2 event.

• Thermal management transitions. thm_temp1_trans_count, thm_temp2_trans_count, and the matching total-time counters tell you how often and how long the drive has throttled. TMT1 is light throttle, TMT2 is heavy. Rising transition counts under steady workload mean the thermal environment is degrading. Not all drives implement these; unsupported drives report zero.
• Error log entries rate. num_err_log_entries covers more than media errors: admin command errors, I/O command errors, internal faults. A rising rate with flat media_errors points at firmware or driver issues, not NAND. Read details with nvme error-log /dev/nvme0, and do it early: the log is a circular FIFO, so a burst of errors overwrites the root cause.
• Controller busy time vs throughput. controller_busy_time (minutes, cumulative) divided by uptime gives internal utilization. The diagnostic combination: 100% busy with high IOPS is healthy saturation; high busy with low host throughput means the controller is burning time on internal work (garbage collection, remapping) or throttling. The resolution is minutes, so this cannot catch brief stalls.
• Spare consumption rate. Track the slope of available_spare, not just the current value. Accelerating consumption is one of the strongest leading indicators of premature failure, visible months ahead. Runway estimate: (current_spare - spare_thresh) / monthly consumption.
• Power-on hours and power cycles. Mostly fleet hygiene: rate denominators for errors and wear, and detection of phantom power cycling (power cycles increasing without matching reboots suggests slot power instability).
• PCIe AER counters. The signal most teams miss entirely. Correctable errors mean the link is retransmitting silently: no data loss, but latency and bandwidth impact.

# PCIe AER counters (requires root; not exposed to containers)
cat /sys/class/nvme/nvme0/device/aer_dev_correctable   # key: TOTAL_ERR_COR
cat /sys/class/nvme/nvme0/device/aer_dev_fatal          # key: TOTAL_ERR_FATAL
cat /sys/class/nvme/nvme0/device/aer_dev_nonfatal       # key: TOTAL_ERR_NONFATAL

In healthy systems all AER counters are zero. Any sustained correctable rate, or any uncorrectable error, is a TICKET for physical-layer investigation: connector seating, cables, retimers, thermal cycling.

Graduate when: spare consumption and thermal transition trends drive procurement and cooling work proactively, and an AER investigation has a defined owner.

Level 4: expert

The deep signals, usually added after the third major incident.

• PCIe link speed and width. Compare current_link_speed/current_link_width against max_link_speed/max_link_width under /sys/class/nvme/nvme0/device/. Any mismatch means silently capped bandwidth: a Gen4 x4 device at Gen3 x2 delivers one quarter of its bandwidth with zero errors anywhere. TICKET on any mismatch. Watch for deliberate bifurcation (x16 slot split to 4x x4) and hot-plug retrains before treating a mismatch as a fault.
• Per-namespace I/O statistics. /sys/block/nvmeXnY/stat gives per-namespace I/O counters. On multi-namespace drives with shared controller capacity, one namespace’s abuse starves the others, and controller-level aggregates hide it. These are cumulative counters with only average latency derivable; there are no percentiles here.
• Tail latency via eBPF. NVMe mean latency is almost always fine; the damage comes from p99 and p999 outliers that trip application timeouts. biolatency -d nvme0n1 10 1 (bcc filters a single disk; -D instead prints per-disk histograms for all disks) or equivalent bpftrace one-liners give the distribution. Be aware eBPF tracing is not free: on very high-IOPS systems, per-I/O tracepoint accounting can add measurable microseconds of latency to the monitored path. Baseline the overhead before running it continuously.
• Vendor-specific log pages. Standard SMART does not expose NAND-level writes, GC duty cycle, or DRAM cache statistics. Vendor log pages (0xC0-0xCF) do, via vendor plugins or nvme vs-smart-add-log style commands. For drives supporting the OCP datacenter NVMe spec, nvme ocp smart-add-log /dev/nvme0 exposes additional health telemetry. Older distro nvme-cli builds may lack these plugins.
• Firmware version tracking. nvme list and nvme fw-log /dev/nvme0. Track versions fleet-wide: controller firmware bugs are a real data-loss class, and without inventory you cannot tell which drives match a vendor advisory. Any unexpected version change is a TICKET.
• Write amplification estimation. data_units_written counts host-visible writes (units of 1000 x 512 bytes), not NAND writes. True WAF is media writes divided by host writes, and the numerator only comes from vendor-specific telemetry such as the OCP C0 log page. Computing WAF from standard SMART alone underestimates actual NAND wear. If percentage_used is climbing faster than host write volume explains, unmeasured WAF is the first suspect.

Graduate when: link degradation and tail latency regressions are caught by monitoring rather than by users, and firmware and WAF data feed capacity and refresh planning.

Using the model

Levels are cumulative: Level 4 signals do not substitute for Level 1 paging. Instrument the layers separately. SMART tells you what the controller thinks; sysfs and AER tell you what the transport is doing; eBPF tells you what applications experience. The failure patterns that hurt most (thermal throttling, GC write cliffs, link degradation) are only diagnosable by correlating across those layers, because each layer alone looks innocent.

How Netdata helps

• Netdata’s NVMe collector polls SMART data and exposes per-bit critical warning dimensions in nvme.device_critical_warnings_state, so Level 2 per-bit alerting works out of the box instead of one blanket alert.
• Wear and endurance signals (nvme.device_estimated_endurance_perc, nvme.device_available_spare_perc) are charted over time, which makes the Level 3 rate-of-change signals (spare consumption trajectory, percentage-used slope) directly visible rather than requiring manual sampling.
• Media errors and error log entries are presented as rates (nvme.device_media_errors_rate, nvme.device_error_log_entries_rate), matching the operational rule that increments, not absolute values, are actionable.
• Thermal history (nvme.device_warning_composite_temperature_time, nvme.device_critical_composite_temperature_time, thermal management transition charts) lets you confirm past throttling events even when the live window was missed.
• Correlating NVMe charts with system temperature, throughput, and block-layer latency on the same dashboard is what separates a thermal death spiral from a GC stall: both look like “the disk got slow” in isolation.
• Know the gaps: PCIe AER counters and link speed/width are not collected by the Netdata NVMe collector and require sysfs access, typically as root. Cover Level 3 and 4 transport signals with host-level checks.

Related guides

• NVMe operations guides hub