$ guides / smartctl-disk-monitoring / smartctl-nvme-reliability-degraded ▌

Operations Guides

NVMe NVM subsystem reliability degraded: Critical Warning bit 2

smartctl -H reports FAILED on an NVMe drive that otherwise seems fine. I/O is normal, Available Spare is healthy, and there are no media errors. The Critical Warning byte is 0x04.

Bit 2 (0x04) in the NVMe Critical Warning byte means “NVM subsystem reliability has been degraded.” Unlike bit 0 (spare below threshold), bit 1 (temperature), or bit 3 (read-only mode), bit 2 has no single metric that directly explains why firmware set it. The NVMe specification defines it as triggered by “significant media related errors or any internal error that degrades NVM subsystem reliability,” but leaves the exact conditions vendor-defined.

This creates a recurring operational problem. Samsung 960/970 EVO and similar drives routinely set bit 2 when Percentage Used exceeds 100%, even at 100% Available Spare with zero Media and Data Integrity Errors. The drive is out of warranty, not failing. But smartctl reports overall health as FAILED whenever any Critical Warning bit is set, because it treats any nonzero Critical Warning as a health failure. The result is noisy alerts on drives that may have years of functional life remaining.

Treat bit 2 as a TICKET by default. Escalate to PAGE only when corroborated by signals that confirm active failure.

What this means

Bit 2 is a firmware-level reliability assessment. It can fire as a genuine early warning before media errors appear. It can also fire at warranty exhaustion with no accompanying degradation. The challenge is distinguishing the two.

The NVMe specification does not define what internal conditions trigger bit 2. Some vendors set it based on cumulative internal error rates that are not exposed in the SMART/Health log. Others appear to tie it to endurance consumption. The operator-visible signals do not always explain the bit, which is why corroboration matters.

Key behavioral notes:

The Critical Warning byte is a point-in-time snapshot. The NVMe spec states that bits in this field represent the state at the time the Get Log Page command is processed. The condition may have been transient. By the time you investigate, the bit may already be clear.
smartctl reports FAILED for any nonzero Critical Warning. This is not a drive failure verdict. It is smartctl’s blanket interpretation of the Critical Warning byte. A drive with bit 2 set and otherwise clean health is not necessarily dead.
Multiple bits can be set simultaneously. Always decode the full byte, not just the value you were alerted on. Bit 2 alongside bit 0 or bit 3 is a very different situation than bit 2 alone.

flowchart TD
    A["Critical Warning 0x04
bit 2 set"] --> B{"Media and Data
Integrity Errors
increasing from baseline?"}
    B -->|Yes| PAGE["PAGE
active media failure confirmed"]
    B -->|No| C{"Available Spare
below threshold
and falling?"}
    C -->|Yes| PAGE
    C -->|No| D{"Bit 3 also set?
Drive read-only?"}
    D -->|Yes| PAGE
    D -->|No| F{"Percentage Used
above 100%?"}
    F -->|Yes| TICKET_W["TICKET
warranty exhaustion
Plan replacement"]
    F -->|No| TICKET_O["TICKET
opaque firmware assessment
Monitor closely, trend daily"]

Common causes

Cause	What it looks like	First thing to check
Endurance exhaustion (warranty)	Percentage Used above 100%, Available Spare at or near 100%, zero media errors, bit 2 only	`smartctl -A /dev/nvme0n1 \| grep -E "Percentage Used\|Available Spare\|Media and Data"`
Active media degradation	Media and Data Integrity Errors increasing, Available Spare declining, bit 2 may accompany bit 0	Compare Media and Data Integrity Errors against baseline from prior collection
Controller or firmware internal error	Bit 2 set with no visible media errors and Percentage Used below 100%, possibly intermittent	Check NVMe error log for internal controller errors, check firmware version against vendor advisories
Cold-start transient	Bit 2 set immediately after power cycle, clears on next poll	Re-poll after 5 minutes; if bit clears, document and monitor

Quick checks

# Decode the full Critical Warning byte and all health fields
smartctl -A /dev/nvme0n1 | grep -E "Critical Warning|Available Spare|Percentage Used|Media and Data|Data Units Written|Composite Temperature"

# Check NVMe error log for recent internal errors
smartctl -l error /dev/nvme0n1

# Verify overall health assessment (FAILED may mean any bit is set, not just bit 2)
smartctl -H /dev/nvme0n1

# Check kernel logs for I/O errors, timeouts, or link issues
dmesg | grep -iE "nvme|i/o error|timeout" | tail -30

# Check whether bit 2 persists across multiple polls (it may be transient)
for i in 1 2 3; do sleep 5; smartctl -A /dev/nvme0n1 | grep "Critical Warning"; done

How to diagnose it

Decode the full Critical Warning byte. 0x04 means only bit 2 is set. 0x05 means bits 0 and 2 (spare below threshold and reliability degraded), which is far more serious. 0x0c means bits 2 and 3 (reliability degraded and read-only mode), which is an emergency.
Check Media and Data Integrity Errors. This is the NVMe equivalent of uncorrectable sector errors. If this counter has increased since your baseline, the drive has confirmed data corruption at the NAND level. Bit 2 corroborated by rising media errors reflects real physical degradation. Plan data evacuation.
Check Available Spare and Available Spare Threshold. If Available Spare is above threshold (typically 10%) and stable, the drive still has spare capacity for bad block remapping. If below threshold and declining, the drive is running out of repair capacity. Bit 2 alongside falling spare is a strong signal of active wear-out failure.
Check Percentage Used. Values above 100% indicate the drive has exceeded its rated endurance. The NVMe spec explicitly states that 100% “may not indicate an NVM subsystem failure.” Many Samsung drives set bit 2 at or shortly after 100% as a warranty notification, not a failure prediction. If Percentage Used is above 100% but Available Spare is healthy and media errors are zero, bit 2 is likely endurance-related.
Check the NVMe error log. Look for internal controller errors, timeout entries, or patterns that might explain the firmware’s reliability assessment. Note: on smartmontools 7.4, running self-tests on NVMe drives can fill the error log with spurious “Invalid Field in Command” entries. This was fixed in 7.5. Do not mistake these for real drive errors.
Re-poll the Critical Warning byte. Because the byte is a point-in-time snapshot, the condition may have cleared. Poll two or three times with a short interval. If bit 2 persists across polls, it represents a sustained condition. If it clears, it was transient.
Check kernel logs for corroborating I/O errors. A drive with bit 2 set but clean kernel logs is in a different risk category than one generating I/O errors, timeouts, or reset events visible to the host.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Critical Warning byte (full decode)	Distinguishes bit 2 alone from multi-bit conditions	Any additional bit set alongside bit 2
Media and Data Integrity Errors	Confirms whether degradation has progressed to actual data loss	Counter increasing from baseline
Available Spare vs threshold	Shows whether the drive has remaining repair capacity	Below threshold, or declining rapidly
Percentage Used	Distinguishes warranty-exhaustion bit 2 from degradation bit 2	Above 100% with healthy spare suggests warranty, not failure
Available Spare trajectory	Rate of spare consumption predicts time to exhaustion	Steady decline over weeks indicates wear-out
Data Units Written	Raw write volume drives endurance consumption	Accelerating write rate shortens runway
NVMe error log entries	Internal controller errors may explain opaque bit 2	New entries after baseline establishment
Kernel I/O errors and timeouts	Host-visible failures that SMART may not capture	Any I/O error or timeout on the device

Fixes

When bit 2 is warranty exhaustion (no corroboration)

This is the most common scenario. Percentage Used is above 100%, Available Spare is healthy, media errors are zero, and no other Critical Warning bits are set. The drive is functioning normally but the firmware has flagged it as out of warranty.

Actions:

Ticket for replacement planning. Do not page. The drive may continue operating for months or years beyond its rated endurance.
Track Available Spare trajectory. As long as it remains above threshold and is not declining rapidly, the drive has repair capacity remaining.
Monitor Media and Data Integrity Errors. If this counter begins increasing, reassess urgency.

When bit 2 is corroborated by active failure

If bit 2 is accompanied by any of the following, escalate to PAGE and begin data evacuation:

Media and Data Integrity Errors increasing from baseline
Available Spare below threshold and declining
Bit 3 set (read-only mode)
Kernel I/O errors on the device

This combination indicates real hardware degradation. Prioritize data migration and drive replacement.

Reducing smartd alert noise on warranty-exhaustion drives

For NVMe drives that are past rated endurance but healthy by all other measures, the persistent FAILED status from bit 2 generates ongoing alert noise.

smartmontools 7.5 and later introduced the -H MASK directive for NVMe devices. This limits the health check to specific bits in the Critical Warning byte. The MASK is an allowlist: only bits set in the MASK are checked. To check all bits except bit 2 (0x04), use:

# smartd.conf: check all Critical Warning bits except bit 2 (0x04)
/dev/nvme0n1 -d nvme -H 0xfb -a

The value 0xfb (binary 11111011) has bit 2 cleared, so smartd alerts on bits 0, 1, 3, 4, 5, 6, and 7, but not on bit 2 alone. This preserves monitoring for spare exhaustion, temperature, read-only mode, and volatile memory backup failure while suppressing the noisy reliability-degraded bit.

Limitations:

The -H MASK directive is marked as “NEW EXPERIMENTAL SMARTD 7.5 FEATURE” in the man page.
The ATA-only -I and -i directives (used to ignore specific ATA attributes) do NOT work for NVMe devices. They have no effect on NVMe Critical Warning handling.
On smartmontools 7.4 and earlier, there is no way to ignore specific NVMe Critical Warning bits. The only option is -d ignore, which disables ALL monitoring for the device. Do not use this as a workaround.
Suppressing bit 2 loses the early-warning value for drives where it indicates real degradation. Only suppress bit 2 on drives where you have confirmed the cause is warranty exhaustion, and continue monitoring Media and Data Integrity Errors, Available Spare, and other health signals independently.

Prevention

Baseline every drive at deployment. Capture a full smartctl -a snapshot when the drive enters service. Without baseline values for Media and Data Integrity Errors, Available Spare, and Percentage Used, you cannot distinguish historical accumulation from new growth.
Track Percentage Used against rated TBW. Compute the projected date the drive will reach 100% endurance. Plan replacement proactively rather than reacting to the bit 2 alert.
Monitor all Critical Warning bits independently. Do not collapse the byte into a single alert. Each bit has a different severity and response procedure. See NVMe Critical Warning bits: decoding the health-log bitmask for the full bitmask reference.
Alert on rate of change, not absolute values. Bit 2 with zero media errors and flat counters is a TICKET. Bit 2 with media errors increasing since baseline is a PAGE. The monitoring system must track growth, not just current state.
Decode before escalating. smartctl reports FAILED for any nonzero Critical Warning bit. Always decode the byte and corroborate with individual health fields before paging. See SMART says PASSED but the drive is failing for why the health self-assessment is a lagging summary indicator.

Monitoring with Netdata

Per-second SMART collection reveals whether bit 2 is transient or sustained. Transient bit sets that clear within seconds are visible without waiting for the next polling cycle.
Correlating bit 2 with Media and Data Integrity Errors rate of change distinguishes warranty-exhaustion noise from real degradation. If the media error counter is flat while bit 2 is set, the alert is lower priority. If it is climbing, the drive is failing.
Available Spare trajectory alongside Percentage Used provides the endurance runway picture. A drive at 150% Percentage Used with stable Available Spare is in a different risk class than one at 80% with rapidly declining spare.
Historical baselines handle first-observation bootstrap. When monitoring is first deployed on a fleet, Netdata establishes baselines for cumulative counters and alerts on growth from that baseline, not on absolute values that may reflect years of historical accumulation.
Anomaly detection on Available Spare trajectory can flag a declining spare consumption rate before it crosses a static threshold, giving lead time for replacement planning.