$ guides / smartctl-disk-monitoring / smartctl-nvme-media-data-integrity-errors ▌

Operations Guides

NVMe Media and Data Integrity Errors incrementing: confirmed NAND corruption

The NVMe SMART/Health Information log exposes a field called “Media and Data Integrity Errors.” When this value is zero, the controller has never returned data that failed integrity verification. When it increments, the controller detected an unrecovered data-integrity error in the NVM subsystem. NVMe groups failures such as write faults, unrecovered read errors, CRC/guard failures, and reference/application-tag mismatches here.

This counter is the NVMe equivalent of Offline Uncorrectable (SMART ID 198) on ATA drives. It is cumulative and monotonically increasing; it never decrements. Some enterprise NVMe drives have strong internal ECC that corrects many errors before they escalate to media errors. By the time this counter increments past that correction layer, the underlying media is failing.

The operational distinction is growth versus baseline. A non-zero value observed when you first deploy monitoring on an existing fleet may be historical. A value that has increased since your last poll means new data integrity failures are actively occurring.

What this means

Each increment represents a confirmed unrecovered media/data-integrity error. Many are NAND read or retention failures, but NVMe also counts write faults and end-to-end protection failures in this category; treat either as serious hardware or firmware trouble.

The severity model has two tiers:

TICKET: Non-zero on first observation of an existing drive. This may be a historical value from months or years of prior operation. Investigate, document, and establish a baseline for future comparison.
PAGE: The count has increased from the established baseline AND at least one corroborating signal is present.

Corroborating signals that justify paging:

Critical Warning bit 2 (value 0x04): NVM subsystem reliability degraded
Critical Warning bit 3 (value 0x08): media placed in read-only mode
Available Spare below the vendor-defined threshold (typically 10%)
SMART Overall Health self-assessment reporting FAILED

A related but distinct counter: “Error Information Log Entries” (NVMe Log 0x01) tracks all entries in the error log. That counter can increment on every boot or due to unsupported commands without indicating NAND corruption. Do not conflate the two. “Media and Data Integrity Errors” is the one that matters for data integrity.

flowchart TD
    A["Media and Data Integrity Errors: non-zero"] --> B{"Growth since baseline?"}
    B -->|"No, stable"| C["TICKET: historical value
document and monitor for change"]
    B -->|"Yes, increasing"| D{"Corroborating signal present?"}
    D -->|"Critical Warning bit 2 or 3"| E["PAGE: active NAND failure
evacuate data immediately"]
    D -->|"Available Spare below threshold"| E
    D -->|"SMART Health FAILED"| E
    D -->|"None found"| F["Investigate further:
error log, kernel logs,
rule out firmware reporting bug"]

Common causes

Cause	What it looks like	First thing to check
Progressive NAND wear-out	Media errors climbing alongside declining Available Spare and high Percentage Used	Available Spare and Percentage Used in full SMART dump
Batch or model-level defect	Media errors on a drive with low Percentage Used; possibly affecting multiple drives of same model and firmware	Firmware version; vendor advisories for the model
Firmware reporting bug	Counter shows an impossibly large value (for example, 10^20 or higher) with no corroborating signals and healthy drive	Verify value magnitude against physical plausibility; check vendor errata
PCIe transport issue masquerading as media error	Media errors alongside kernel I/O errors, link resets, or PCIe AER events in dmesg	Kernel logs for transport-level errors
Historical (not active)	Non-zero counter, stable across multiple polls, no corroborating signals	Compare against baseline snapshot from deployment

Quick checks

These commands are read-only and safe on a production system. Most require root.

# Read the counter itself
smartctl -A /dev/nvme0n1 | grep "Media and Data Integrity Errors"

# Check overall health self-assessment
smartctl -H /dev/nvme0n1

# Check Critical Warning byte (0x00 = clean, 0x04 = bit 2, 0x08 = bit 3)
smartctl -a /dev/nvme0n1 | grep "Critical Warning"

# Check Available Spare vs Available Spare Threshold
smartctl -a /dev/nvme0n1 | grep "Available Spare"

# Check Percentage Used (endurance consumption)
smartctl -a /dev/nvme0n1 | grep "Percentage Used"

# Check NVMe error log for error types and LBAs
smartctl -l error /dev/nvme0n1

# Check kernel logs for host-visible I/O errors and transport issues
dmesg -T | grep -iE "nvme|I/O error|medium error|reset|timeout|offline" | tail -20

<!-- TODO: verify that smartctl -l selftest works for NVMe drives on the target smartmontools version, or whether NVMe device self-test requires a different invocation -->
# Check self-test log for prior surface scan results
smartctl -l selftest /dev/nvme0n1

# Capture drive identity for firmware version and batch correlation
smartctl -i /dev/nvme0n1 | grep -iE "Model|Firmware|Serial"

# Full SMART dump for complete context
smartctl -a /dev/nvme0n1

How to diagnose it

Confirm the counter is actually growing. Compare the current value against your last baseline snapshot. If you do not have a baseline, capture one now and re-check after a short interval. A single non-zero reading without growth is a TICKET, not a PAGE.
Check for corroborating signals. Read the Critical Warning byte, Available Spare, and overall health assessment. If any of these corroborate (bit 2 or 3 set, spare below threshold, or health FAILED), escalate to PAGE.
Examine the NVMe error log. Run smartctl -l error /dev/nvme0n1 to see the most recent error entries. The NVMe completion queue status field uses Status Code Type (SCT) 2 for “media and data integrity errors,” with specific status codes such as unrecovered read error (SC 0x81) or write fault (SC 0x80). This confirms whether the logged errors are media-level or something else.
Check kernel logs for host-visible impact. Look for “I/O error,” device reset, or NVMe-specific messages in dmesg. If the host is seeing I/O errors on this device, the corruption is affecting production workloads, not just internal counters.
Rule out firmware reporting false positives. If the counter value is astronomically large (for example, 10^20 or more) and all other health signals are clean, suspect a firmware bug. Some drive models are known to report garbage in this field. Check whether a firmware update is available from the vendor. Do not page on an impossibly large value without corroboration.
Assess the wear context. Check Percentage Used and Available Spare. A drive at 120% Percentage Used with declining Available Spare is failing from wear exhaustion. A drive at 5% Percentage Used with media errors may have a manufacturing defect or firmware issue.
Check for fleet-wide patterns. If multiple drives of the same model and firmware version show the same symptom, this points to a batch defect or firmware bug rather than individual drive failure.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Media and Data Integrity Errors	Direct count of unrecovered NAND data corruption	Any growth from baseline
Critical Warning byte	Controller flagging critical conditions	Bit 2 (0x04) or bit 3 (0x08) set
Available Spare vs Threshold	Remaining spare blocks for remapping; threshold is vendor floor	Below threshold (typically 10%)
Percentage Used	Endurance consumption estimate	Above 100% means beyond rated life
SMART Overall Health	Controller binary self-assessment	FAILED
Error Information Log entries	Detailed error event history	Recent entries with SCT 2 status codes

Fixes

Active NAND failure (counter growing with corroboration)

No software fix exists for physical NAND degradation. The data stored in the affected cells is gone.

Evacuate data immediately. If the drive is part of a redundant array (RAID, replicated storage), initiate a controlled rebuild onto a healthy drive. Do not wait for the drive to fail completely. SSDs can transition from “slightly degraded” to “dead” within hours.
Replace the drive. Once data is safe, schedule replacement. Do not attempt to “repair” NAND by rewriting sectors or running secure erase. The underlying media defect remains.
If Available Spare is at 0%, the next bad block causes permanent data loss. The drive has no spare capacity left to remap. Treat this as an emergency.

Critical Warning bit 3 active (read-only mode)

The drive has placed itself in read-only mode to protect existing data. It will refuse all writes.

Copy all readable data off the drive immediately. Reads may still work, but the drive will not accept new writes.
Replace the drive. Read-only mode is typically permanent and non-recoverable.

Firmware reporting false positive

If the counter shows an impossibly large value with no corroborating signals:

Verify with smartctl -a. Check whether other health fields (Available Spare, Percentage Used, Critical Warning, overall health) are all clean. If they are, the media error counter value is likely garbage from a firmware bug.
Check for a firmware update. Some vendors have acknowledged and fixed reporting bugs in later firmware revisions.
Upgrade smartmontools if running an old version. Pre-7.0 versions had NVMe parsing issues on some platforms.
Do not page on this. Configure monitoring to require corroboration before alerting.

Historical non-zero (stable, no growth)

Establish the value as your baseline.
Monitor for growth. Any increase from this baseline is the signal that matters.
No immediate action required beyond continued monitoring.

Prevention

Baseline every drive at deployment. Capture a full smartctl -a snapshot when the drive enters service. Without a baseline, you cannot distinguish historical damage from active failure.
Alert on growth, not absolute value. This counter is cumulative and monotonically increasing. A drive that shipped with 2 media errors and has held at 2 for three years is stable. A drive that went from 0 to 2 this week is actively failing.
Require corroboration before paging. A single incrementing counter without Critical Warning, Available Spare, or health FAILED warrants investigation but should not wake someone at 3 a.m. unless corroborated.
Run smartd for automated monitoring. smartmontools 7.5 added NVMe checks for -p (Available Spare), -u (Percentage Used and Media and Data Integrity Errors), -t (both), and -H MASK (Critical Warning-bit filtering); -l selftest with -s schedules NVMe self-tests. On older versions, verify your release’s NVMe support before relying on these directives.
Schedule periodic extended self-tests. Self-tests proactively surface latent media defects before production I/O hits them. A weekly short test and monthly extended test is standard practice.
Track firmware versions fleet-wide. Known firmware bugs have produced false media error reports across multiple vendors. Without firmware tracking, you cannot correlate symptoms to a known-bad revision.

How Netdata helps

Per-second collection of NVMe SMART/Health metrics catches counter growth within seconds, not on a 5-minute polling delay. Rate-of-change alerting on cumulative counters distinguishes historical noise from active NAND failure automatically.
Correlated timeline view shows Media and Data Integrity Errors alongside Available Spare, Percentage Used, Critical Warning bits, and Composite Temperature on a single chart. The corroborating signals required for PAGE escalation are visible together, not scattered across separate tools.
Kernel log integration surfaces host-visible I/O errors next to SMART data. A “clean SMART, dirty kernel logs” pattern is visible without switching contexts.