NVMe NVM subsystem reliability degraded: critical warning bit 2

Your monitoring just reported critical_warning : 0x4 on an NVMe device, or smartctl printed “SMART overall-health self-assessment test result: FAILED! - NVM subsystem reliability has been degraded.” The drive is still serving I/O. Nothing in the kernel log looks catastrophic. Is the drive dying, or is this noise?

It depends on what else the SMART log says. Critical warning bit 2 is the vaguest bit in the critical warning byte. The specification language is broad: the controller has detected significant media-related errors, or some internal error, that degrades NVM subsystem reliability. What counts as “significant” and “degraded” is left to the vendor, and vendors interpret it very differently. Some drives set bit 2 only when the flash is genuinely failing. Others, notably several Samsung consumer models, set it preemptively the moment Percentage Used crosses 100 percent, which is a warranty-consumed signal, not a failure signal.

Your job when this bit fires is to determine which situation you are in before you either ignore a dying drive or panic over a healthy one.

What this means

The NVMe critical warning byte lives in the SMART / Health Information log page and is a bitmask. Each bit is an independent assertion by the controller. Bit 2, value 0x04, is defined as: the NVM subsystem reliability has been degraded due to significant media related errors or any internal error that degrades NVM subsystem reliability. The kernel constant is NVME_SMART_CRIT_RELIABILITY. nvme-cli labels the bit “NVM subsystem Reliability” in its smart-log breakdown, and Netdata exposes it as the nvm_subsystem_reliability dimension in the nvme.device_critical_warnings_state chart.

Two properties of this bit matter operationally:

It is a self-assessment, not a measurement. The controller is telling you its own conclusion about its health. The inputs to that conclusion (ECC correction rates, internal retry counts, spare consumption, endurance math) are vendor-specific and mostly invisible to the host. Different vendors set this bit at wildly different internal thresholds, from “elevated ECC corrections” to “imminent death.”
It can be set preemptively. Some drives assert bit 2 when the rated endurance is consumed, even with zero media errors and full spare capacity. On those drives the bit means “warranty is over,” not “data is at risk.”

This is why the correct severity for bit 2 alone is a ticket, not a page. It demands same-shift investigation. It becomes page-worthy only when a second, independent signal confirms active degradation: specifically, a rising media_errors rate. The drive’s own assessment plus ongoing observable media failures is the Silent Data Degradation pattern, and that combination means data you can still read today may not be readable next month.

For the full decoding of the critical warning byte, see NVMe critical_warning is nonzero: decoding the SMART critical warning bitmask.

Common causes

Cause	What it looks like	First thing to check
Genuine NAND degradation	Bit 2 set, media_errors incrementing, available_spare declining	`nvme smart-log` media_errors rate over hours
Endurance consumed (warranty expiry)	Bit 2 set, percentage_used >= 100, media_errors zero, available_spare 100%	Compare percentage_used against the other fields
Internal controller error	Bit 2 set, num_err_log_entries climbing, media_errors flat	`nvme error-log /dev/nvme0` entry status codes
Latent bad blocks surfaced by a scan	Small media_errors bump after a backup or scrub, bit 2 set once	Whether media_errors keeps increasing after the scan ends
Vendor-specific preemptive assertion	Bit 2 set on a drive past rated endurance with clean SMART otherwise	Vendor documentation or known model behavior

Quick checks

All of these are read-only.

# Show the full SMART health log for the device
nvme smart-log /dev/nvme0

# Isolate the critical warning byte (0x04 = bit 2)
nvme smart-log /dev/nvme0 | grep "critical_warning"

# The corroborating fields: media errors, spare, endurance, error log count
nvme smart-log /dev/nvme0 | grep -E "media_errors|available_spare|percentage_used|num_err_log_entries"

# Inspect the actual error log entries (status codes, LBAs, namespaces)
nvme error-log /dev/nvme0

# Check the kernel log for corroborating I/O errors
dmesg | grep -i "nvme" | grep -iE "error|critical|fail"

A single snapshot of media_errors is a lifetime counter and tells you almost nothing by itself. Take two readings a few minutes apart under normal load. The delta is the signal.

How to diagnose it

The diagnostic question is: is the media actively failing right now, or did the controller set this bit for a reason that does not involve current data loss?

flowchart TD
  A[Bit 2 set: 0x04] --> B{media_errors increasing?}
  B -- yes --> C{available_spare declining or below threshold?}
  C -- yes --> D[PAGE: active degradation
verify redundancy, plan immediate replacement]
  C -- no --> E[PAGE: silent data degradation
errors without spare consumption still mean unreadable data]
  B -- no --> F{percentage_used >= 100%?}
  F -- yes --> G[Likely warranty-expiry assertion
TICKET: schedule replacement, keep watching]
  F -- no --> H{num_err_log_entries increasing?}
  H -- yes --> I[Inspect error-log entries
non-media errors: firmware or driver issue]
  H -- no --> J[Isolated or stale assertion
TICKET: baseline and monitor rates]

Read the full SMART log. Note critical_warning, media_errors, available_spare, available_spare_threshold, percentage_used, and num_err_log_entries.
Establish the media_errors rate. Poll every 60 seconds for 10 to 15 minutes under normal workload. Any increment means uncorrectable errors are occurring during operation. A flat counter means nothing is actively failing.
Check available spare against its threshold. If available_spare is at or below the vendor threshold, bit 0 will also be set and the drive is a replace-now candidate. If spare is 100 percent and the threshold is untouched, the flash block pool is intact.
Check percentage_used. If it is at or above 100 and media_errors is zero with full spare, you are very likely looking at the endurance-consumed assertion. Empirically, Samsung 960 EVO and 970 EVO Plus drives set bit 2 exactly when Percentage Used crosses 100 percent, with otherwise clean SMART data; field reports of 970 EVO Plus drives with Percentage Used above 100 and 100% available spare confirm the pattern. Other Samsung models may behave the same way. The NVMe specification itself notes that a Percentage Used value of 100 means the estimated endurance has been consumed but “may not indicate an NVM subsystem failure.” At least one major hosting provider’s standard guidance for 0x04 is: as long as Available Spare is above its threshold, the message reflects expired warranty and can be safely deprioritized.
Inspect the error log. Run nvme error-log /dev/nvme0 and look at the status codes of recent entries. Uncorrectable read errors on specific LBAs point at media. Admin command errors or internal errors without media involvement point at firmware or driver territory. The log is a circular buffer: at high error rates, the oldest entries, which often contain the root cause, get overwritten.
Classify and set severity. Bit 2 plus rising media_errors is the Silent Data Degradation pattern: page it. Everything else is a ticket with a monitoring follow-up.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`nvm_subsystem_reliability` (critical warning bit 2)	The controller’s own degraded-reliability assessment	Asserted at all; asserted together with rising media_errors
`nvme.device_media_errors_rate`	Uncorrectable errors are the ground truth of media health	Any rate above 0 per second; accelerating rate
`nvme.device_available_spare_perc`	Remaining block-replacement runway	Declining trend; at or below vendor threshold
`nvme.device_estimated_endurance_perc`	Explains preemptive bit-2 assertions	At or above 100 percent
`nvme.device_error_log_entries_rate`	Broader error activity beyond media	Increasing while media_errors stays flat (firmware or driver issue)
Read latency on the device	Retried reads show up as sporadic latency before hard errors	Sporadic high read latency, especially on cold data

Fixes

There is no software fix that clears bit 2 on a drive whose media is actually failing. The correct responses, by cause:

Active media degradation (bit 2 plus rising media_errors). Treat the drive as failed-but-still-serving. Verify RAID, replication, or backup health first, because the window where the data is still readable is closing. Force a rewrite or re-replication of at-risk data so redundant copies are current. Replace the drive immediately; do not wait for a maintenance window if media_errors is accelerating. If reads start failing at the block layer, you will see the kernel errors covered in blk_update_request: I/O error, dev nvme0n1.

Warranty-expiry assertion (bit 2, percentage_used >= 100, clean everything else). Schedule replacement on a normal procurement cycle. The drive is past its rated endurance and its failure signals deserve closer watching, but nothing is actively wrong. Adjust your alerting so this known-benign state does not page: alert on the bit-2-plus-media_errors combination rather than the bit alone. If smartd noise is the problem, smartmontools 7.5 added a -H MASK directive for smartd.conf that lets you ignore specific bits of the NVMe critical warning byte (MASK is a hex bitmask of the bits to keep monitoring; e.g. -H 0xf9 ignores the temperature and reliability bits, and -H 0xff is the full check) instead of the older workaround of excluding the device from monitoring entirely.

Internal or non-media errors. If num_err_log_entries is climbing without media_errors, classify the error log entries, check whether a firmware update addresses known issues for that model, and watch for escalation into controller instability. A controller whose internal errors progress to hangs will show up as resets; see NVMe controller reset loop: repeated resets from a firmware hang.

Prevention

You cannot prevent bit 2 from firing, but you can prevent it from being a surprise:

Alert per bit, not on the byte. A single critical_warning != 0 alert collapses a read-only emergency (bit 3), a thermal transient (bit 1), a replacement planning signal (bit 0), and this ambiguous bit 2 into one severity. Per-bit alerts with appropriate routing are essential.
Escalate on the combination, not the bit. The rule that matches reality: ticket on bit 2 alone, page on bit 2 plus a rising media_errors rate. This catches genuine degradation early and filters out preemptive vendor assertions.
Track percentage_used and available_spare trends. If you know a drive is approaching 100 percent endurance, a subsequent bit-2 assertion has context before anyone gets woken up.
Poll SMART at a sane cadence. SMART counters update on the controller’s schedule; polling faster than every 30 to 60 seconds buys nothing. Do poll continuously, because the critical warning bits report current state and are not persistent (per the NVM Express spec), so a controller reset can clear them even when the underlying condition persists.

How Netdata helps

Netdata decodes the critical warning byte into per-bit dimensions in nvme.device_critical_warnings_state, including nvm_subsystem_reliability for bit 2, so you can alert on this specific assertion instead of the raw byte.
It presents media_errors as an incremental rate (nvme.device_media_errors_rate), which is exactly the corroborating signal that decides whether bit 2 is a ticket or a page.
Available spare, spare threshold context, and estimated endurance (nvme.device_available_spare_perc, nvme.device_estimated_endurance_perc) are collected alongside, so the warranty-expiry explanation is visible in the same view as the warning itself.
The error log entries rate (nvme.device_error_log_entries_rate) separates media problems from firmware or command-level problems without pulling logs by hand.
Because all of these are per-second, per-device charts, the correlation this article describes (bit 2 plus media_errors rate plus spare trend) is a single-screen diagnosis rather than a cross-tool hunt.