NVMe critical_warning is nonzero: decoding the SMART critical warning bitmask

Your monitoring fired because critical_warning in the NVMe SMART log is nonzero. Or you ran nvme smart-log /dev/nvme0 during an investigation and saw something like critical_warning : 0x04 where you expected 0x00. You need two things fast: which bit is set, and how bad it is.

The most common mistake here is treating critical_warning != 0 as one alert with one severity. It is a bitmask of six independent conditions with wildly different severity. Bit 3 means the drive has gone read-only and is refusing writes: a page-right-now outage. Bit 0 means spare capacity is below the vendor threshold: a procurement ticket, not a 3 a.m. incident. Bit 1 may be a transient thermal event during a backup run that clears itself.

What this means

critical_warning is a single byte in the NVMe SMART / Health Information Log (Log Page 0x02). Zero means no active warnings. Any nonzero value means the controller itself is reporting that at least one internal safeguard has been breached. This is not a prediction: the drive is asserting a current condition.

The bits:

Bits combine. 0x03 is bits 0 and 1 (spare low and hot). 0x0c is bits 2 and 3 (reliability degraded, now read-only). Decode the hex before you decide anything.

Three properties matter operationally:

1. It is current state, not history. The bits reflect what the controller believes right now. A bit set during a thermal event may be clear an hour later.
2. It is not persistent. On some controllers the field clears after a controller reset even if the underlying condition persists. Poll every few minutes, not once an hour, or you will miss transient assertions entirely.
3. Trigger behavior varies by vendor. Bit 2 in particular is vague: different vendors trip “reliability degraded” at different internal thresholds, and some drives set it preemptively.

Bit 5 only exists on NVMe 1.4+ devices with a Persistent Memory Region. Most drives, especially consumer drives, never set it.

Per-bit severity and response

This is the triage map. The blanket critical_warning != 0 page is wrong because these rows have almost nothing in common operationally.

flowchart TD
  A[critical_warning nonzero] --> B{Decode hex: which bits?}
  B --> C[bit 3 - read-only]
  B --> D[bit 2 - reliability]
  B --> E[bit 1 - temperature]
  B --> F[bit 0 - spare low]
  B --> G[bit 4 - PLP failed]
  C --> C1[PAGE: fail over, replace drive]
  D --> D1{media_errors rising?}
  D1 -->|yes| D2[PAGE: active degradation]
  D1 -->|no| D3[TICKET: watch, check percent_used]
  E --> E1{Sustained and critical_comp_time rising?}
  E1 -->|yes| E2[PAGE: thermal emergency]
  E1 -->|no| E3[TICKET: fix cooling, watch]
  F --> F1[TICKET: procure replacement]
  G --> G1[TICKET: power-loss risk, plan swap]

Quick checks

All read-only and safe on a production host. The nvme commands generally require root.

# Read the raw field
nvme smart-log /dev/nvme0 | grep critical_warning

# Bit-by-bit breakdown (human-readable)
nvme smart-log -H /dev/nvme0 | grep -A8 "Critical Warning"

# JSON form for scripting
nvme smart-log /dev/nvme0 -o json | jq '.critical_warning'

# Corroborating fields, in one shot
nvme smart-log /dev/nvme0 | grep -E "available_spare|spare_thresh|percentage_used|media_errors|num_err_log_entries|temperature|warning_temp_time|critical_comp_time|unsafe_shutdowns"

# If bit 1 or thermal suspicion: current temperature via hwmon (millidegrees C)
cat /sys/class/nvme/nvme0/hwmon*/temp1_input

# If bit 2: are media errors actually increasing? Take two samples
nvme smart-log /dev/nvme0 | grep media_errors; sleep 300; nvme smart-log /dev/nvme0 | grep media_errors

# Corroborate in the kernel log
dmesg | grep -i "nvme" | grep -i "error\|reset\|timeout\|read.only"

How to diagnose it

Work the bit that is set. If multiple bits are set, handle bit 3 first, then bit 2, then the rest.

1. Decode the value. Convert the hex to bits using the table above. 0x08 is bit 3. 0x04 is bit 2. 0x01 is bit 0. Do not skip this step: the response for 0x01 and 0x08 could not be more different.

2. Bit 3 (0x08, read-only): treat as an outage. The drive has autonomously decided it can no longer safely accept writes, typically because spare blocks are exhausted. Writes are rejected at the device level. Confirm with dmesg (expect I/O errors on write), verify replication or RAID health for the data on this drive, fail the workload over, and replace the drive. Filesystem repair tools do nothing here; this is a hardware state.

3. Bit 2 (0x04, reliability degraded): corroborate before escalating. Check media_errors twice a few minutes apart. If the counter is actively increasing, the drive’s self-assessment and observed behavior agree: active degradation, page-worthy, and the drive is on its way out. If media_errors is flat and percentage_used is above 100%, you are likely looking at a vendor that trips bit 2 preemptively on drives past rated endurance. That is a ticket and an accelerated replacement plan, not a page. Also check available_spare: bit 2 plus declining spare is a much stronger signal than bit 2 alone.

4. Bit 1 (0x02, temperature): check duration, not just presence. Some drives set bit 1 at the warning threshold (WCTEMP), well below the critical threshold (CCTEMP), and it can trip transiently during backups, batch jobs, or benchmarks. Check current composite temperature and the critical_comp_time counter. Bit 1 sustained for more than 5 minutes while critical_comp_time is actively increasing means the thermal condition is not self-resolving: escalate. Otherwise, fix airflow and watch. WCTEMP and CCTEMP live in Identify Controller data (nvme id-ctrl), not the SMART log, so the SMART log alone cannot tell you which threshold was crossed.

5. Bit 0 (0x01, spare below threshold): start the replacement clock. Compare available_spare to spare_thresh. The drive is still functional, but its pool of replacement blocks is below the vendor’s safety margin. Spare consumption accelerates non-linearly near the end, so do not assume you have months. Check the recent consumption rate if you have trend data, and correlate with percentage_used and media_errors.

6. Bit 4 (0x10, PLP failed): reclassify the drive’s power-loss risk. The power-loss protection capacitor on an enterprise drive has failed or can no longer hold charge. Performance is unaffected, which is exactly why this is dangerous: every in-flight write is now unprotected, and nothing will look wrong until a power event. Check unsafe_shutdowns history, treat any future unsafe shutdown as a data-loss event, and schedule replacement. Consumer drives without PLP never set this bit.

7. Bit 5 (0x20, PMR read-only): confirm PMR is actually in use. Only NVMe 1.4+ devices with a Persistent Memory Region can set this. If your workload does not use PMR, ticket it and move on. If it does, treat it as a media failure in that region.

8. Re-poll after any reset. If the controller was reset (kernel log shows Resetting controller) or the host rebooted, check critical_warning again. Some controllers clear the field on reset even when the condition persists. A cleared field after a reset is not an all-clear.

Metrics and signals to monitor

Fixes

There is no software fix for any of these bits. Every fix is environmental (thermal), procedural (failover, replacement), or risk mitigation.

Bit 3 (read-only): Fail the workload off the device, verify replica or backup integrity, replace the drive. Do not attempt to “repair” the filesystem in place; the device is rejecting writes by design.

Bit 2 (reliability degraded): If media errors are rising, follow the same path as bit 3 but with more time: verify redundancy, rewrite at-risk cold data if you can, and replace immediately. If media errors are flat and percentage_used is over 100%, schedule replacement on a normal maintenance cadence and tighten your polling interval.

Bit 1 (temperature): Restore cooling headroom. Check heatsink seating on M.2 drives, chassis airflow and fan health, and heat radiated by adjacent components (a GPU next to an M.2 slot is a classic). Reduce write load for immediate relief; the drive throttles to protect itself, so the condition often resolves when load drops.

Bit 0 (spare below threshold): Procure and schedule replacement. If the spare consumption rate is accelerating, move the date up; the last portion of spare capacity depletes faster than the first.

Bit 4 (PLP failed): No field repair. Reclassify the drive as “no power-loss protection,” verify your power infrastructure and UPS, and schedule replacement. Every unsafe shutdown from this point forward is a potential data-loss event.

Bit 5 (PMR read-only): If PMR is in use, treat as media failure in that region and engage the vendor.

Prevention

• Alert per bit, not per byte. Replace any blanket critical_warning != 0 alert with per-bit alerts at the severities in the table above. This is the single highest-value change.
• Poll frequently. The field is current-state and can clear on reset. Polling every few minutes catches transient assertions that hourly checks miss.
• Track percentage_used and available_spare trends so bits 0 and 2 never surprise you. Both decline monotonically and give months of warning if you watch the rate.
• Baseline PLP capability at provisioning. There is no SMART field that says “PLP absent,” so record which drives have it before they go into production.
• Monitor temperature continuously, not just the threshold bit, so thermal problems show up as a trend before bit 1 ever asserts.
• Correlate with the kernel log. NVMe I/O errors, resets, and read-only transitions in dmesg corroborate what the SMART byte claims. SMART is self-reported; treat it as one witness, not the verdict.

How Netdata helps

• Netdata splits the critical_warning byte into individual dimensions on the nvme.device_critical_warnings_state chart, so each bit can carry its own alert severity instead of one noisy page for everything.
• Correlating the bit dimensions with nvme.device_media_errors_rate is the fastest way to classify bit 2: rising media errors plus bit 2 is active degradation; flat media errors plus high percentage_used is usually vendor-preemptive signaling.
• nvme.device_available_spare_perc and nvme.device_estimated_endurance_perc provide the trend lines behind bits 0 and 2, turning a binary warning into a runway estimate.
• nvme.device_composite_temperature alongside nvme.device_critical_composite_temperature_time tells you whether a bit 1 assertion is transient or an ongoing emergency.
• nvme.device_unsafe_shutdowns_count becomes critical context once bit 4 asserts, because every unsafe shutdown on a drive with failed (or absent) PLP carries data-loss risk.
• Because Netdata polls the SMART log continuously, transient assertions that clear on a controller reset are captured in the time series instead of being silently missed.

Related guides

• blk_update_request: I/O error, dev nvme0n1: reading NVMe I/O errors in the kernel log
• nvme nvme0: I/O timeout, Resetting controller: what an NVMe controller reset means
• NVMe controller state not live: reading resetting, deleting, and dead from sysfs
• NVMe device disappeared: nvme0: Removing and a drive that fell off the PCIe bus
• How NVMe actually works in production: a mental model for operators
• NVMe monitoring checklist: the signals every production SSD needs
• NVMe monitoring maturity model: from survival to expert