$ guides / nvme / nvme-media-errors-increasing ▌

Operations Guides

NVMe media_errors increasing: uncorrectable data-integrity errors on NAND

Your monitoring shows the media_errors counter on an NVMe drive going up. In nvme smart-log output the field is labelled media_errors (smartctl calls it “Media and Data Integrity Errors”), and it is the one SMART field you should never explain away: each increment is a read or write where the controller could not maintain data integrity even after its internal ECC and retry mechanisms were exhausted.

The absolute value is almost meaningless on its own. A drive with 3 lifetime errors after four years of service can be perfectly healthy; a drive that went from 0 to 12 errors this week is failing. The rate of change, and what changes alongside it, is the diagnostic. A single new error during a backup run that scanned cold data can be one latent bad block finally being touched. A steady climb combined with critical_warning bit 2 (NVM subsystem reliability degraded) is the drive telling you it is dying.

This guide covers how to read the counter, how to classify what you are seeing, and how to decide between “investigate this shift” and “page someone now.”

What this means

NAND flash cells degrade. The controller fights this constantly with ECC, read retries at adjusted voltage thresholds, and background relocation of marginal blocks into the spare pool. media_errors increments only when all of that fails: the data at some LBA could not be recovered by the drive itself. The NVMe spec includes uncorrectable ECC failures, CRC checksum failures, and LBA tag mismatches in this counter.

Two things follow from that definition:

The counter is a lifetime total maintained by the controller. It never decreases, and the host does not write to it. Monitoring tools (including Netdata, which exposes it as the nvme.device_media_errors_rate chart) present it as a rate so that increments are visible instead of buried in a large historical number.
An increment does not automatically mean the host lost data. Depending on the event, the controller may still have returned correct data from a redundant internal copy, or the read may have failed up the stack and been served by your RAID mirror or replica. Either way, the primary copy on that NAND was unreadable, and that block will be retired.

The counter interacts with the rest of the SMART picture in specific ways. media_errors can climb while percentage_used is low and available_spare is at 100% (a bad NAND batch or a sudden failure, not wear). It can sit at a nonzero value for years without impact (isolated latent defects on an old drive). And it is a different counter from num_err_log_entries: the error log counts all error events, including non-media errors like invalid admin commands, so it can increment while media_errors stays flat. Do not conflate them.

Common causes

Cause	What it looks like	First thing to check
NAND wear-out (end of life)	Steadily accelerating error rate, `percentage_used` high or past 100%, `available_spare` declining toward threshold	`nvme smart-log`: compare `percentage_used` and `available_spare` against their trajectories
Latent bad block surfaced by a scan	One or two increments during a backup, scrub, or crash-recovery read of cold data; rate returns to zero afterwards	Timing: did the increment coincide with a full-volume read?
Retention failure or read disturb on cold data	Errors concentrated on reads of data written long ago; device otherwise healthy	`nvme error-log`: are the failing LBAs in old, rarely-touched regions?
Unsafe shutdown aftermath	A cluster of errors appearing shortly after `unsafe_shutdowns` incremented; may be a one-time event	Compare `unsafe_shutdowns` history against the error timeline
Manufacturing defect / bad NAND batch	Errors on a young drive with low `percentage_used` and full `available_spare`	Drive age (`power_on_hours`) and fleet cohort: are same-batch drives failing too?
PCIe transport or firmware issue masquerading as media errors	Errors correlate with link retraining, AER counters climbing, or a known-problematic firmware	AER counters in sysfs, current vs max link speed, `nvme fw-log`

Quick checks

All of these are read-only.

# Current media error count and the corroborating SMART fields
nvme smart-log /dev/nvme0 | grep -E "media_errors|num_err_log_entries|critical_warning|available_spare|percentage_used|unsafe_shutdowns"

# Individual error log entries: status codes, namespaces, LBAs
nvme error-log /dev/nvme0

# Drive age and lifetime write volume, for context
nvme smart-log /dev/nvme0 | grep -E "power_on_hours|data_units_written"

# PCIe transport health: in a healthy system these are all zero
cat /sys/class/nvme/nvme0/device/aer_dev_correctable
cat /sys/class/nvme/nvme0/device/aer_dev_fatal
cat /sys/class/nvme/nvme0/device/aer_dev_nonfatal

# Link negotiated down? current should equal max
cat /sys/class/nvme/nvme0/device/current_link_speed
cat /sys/class/nvme/nvme0/device/max_link_speed
cat /sys/class/nvme/nvme0/device/current_link_width
cat /sys/class/nvme/nvme0/device/max_link_width

# Kernel-side view: media errors surface as I/O errors with status codes
dmesg | grep -i nvme | grep -iE "error|sct"

On the kernel side, uncorrectable media errors surface in dmesg with Status Code Type 2 (media and data integrity). A line carrying sct 0x2 / sc 0x81 is an unrecovered read error: the kernel is reporting the same class of event the SMART counter is accumulating. See reading NVMe I/O errors in the kernel log for decoding these lines.

How to diagnose it

flowchart TD
  A[media_errors incremented] --> B{critical_warning bit 2 set?}
  B -->|Yes| C[PAGE: active degradation
verify redundancy, replace drive]
  B -->|No| D{Rate accelerating?}
  D -->|Yes, or errors frequent| E{TICKET: check percentage_used
and available_spare trajectory}
  D -->|Single isolated increment| F{Coincided with cold-data scan
or unsafe shutdown?}
  F -->|Yes| G[TICKET: likely latent defect
monitor rate, plan replacement]
  F -->|No| H{TICKET: check AER counters,
link speed, firmware, drive age}
  E --> I[End-of-life pattern:
procure and schedule swap]
  H --> J[Transport/firmware cause:
fix path, not the drive]

Work through it in this order:

Confirm the increment and its timing. Pull the current media_errors value and compare against your monitoring history. When did it move, and by how much? A drive that has been at 5 for two years and is still at 5 has no active problem.
Check critical_warning bit 2 immediately. This is the severity fork. Bit 2 means the controller itself has assessed its reliability as degraded. Bit 2 combined with a rising media_errors rate is the PAGE condition: active, confirmed degradation. See decoding the SMART critical warning bitmask and critical warning bit 2.
Read the error log for localization. nvme error-log /dev/nvme0 shows status codes, namespace IDs, and LBAs for individual events. Errors clustered in one LBA region point to localized damage or cold-data retention issues. Errors scattered randomly across the namespace point to general media degradation. Act fast: the log is a circular buffer, and a high error rate overwrites the oldest entries, which are usually the root cause.
Correlate with wear indicators. Pull percentage_used and available_spare. High and rising percentage_used plus declining spare plus rising media errors is the classic end-of-life pattern. Media errors on a drive with 3% used and 100% spare is a defect story, not a wear story.
Check for an unsafe-shutdown trigger. If unsafe_shutdowns incremented shortly before the media errors appeared, the cluster may be corruption from an incomplete write flush rather than ongoing degradation. This matters for the prognosis: one-time event versus progressive failure.
Rule out the transport layer. Check AER counters and link speed/width. PCIe link instability produces data-path errors that can present alongside or instead of genuine NAND errors, and reseating a connector is a very different fix from replacing a drive. Zero AER counters and full link speed close this branch.
Classify and set the response. Isolated increment during a cold-data scan: same-shift investigation, watch the rate. Accelerating rate with corroborating wear signals: replacement pipeline. Bit 2 plus rising errors: treat as an active incident.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`media_errors` rate (`nvme.device_media_errors_rate`)	Each increment is an unrecovered integrity failure	Any rate above zero on a previously stable drive
`critical_warning` bit 2	The controller’s own reliability assessment	Asserted while media errors are rising: page
`available_spare` vs `spare_thresh`	Remaining bad-block replacement runway	Declining toward threshold alongside rising errors
`percentage_used`	Endurance consumed; context for whether errors are wear-driven	High or past 100% with errors accelerating
`num_err_log_entries` rate	Broader error activity beyond media errors	Rising without media errors: firmware/driver path, not NAND
`unsafe_shutdowns`	Power-loss events that can cause one-time error clusters	Increment shortly before a media error cluster
PCIe AER counters	Transport-layer integrity that can masquerade as media errors	Any sustained nonzero rate
Read latency	Retry passes on marginal blocks show up as sporadic read latency before hard errors	Sporadic read spikes on cold data with rising media errors

Fixes

End-of-life wear (high percentage_used, declining spare, rising errors)

Replace the drive. There is no remediation for worn NAND. Verify RAID or replication health first, force a rewrite of at-risk data so it lands on fresh blocks (or on the surviving mirror), and schedule immediate replacement. Drives can operate past 100% percentage_used, but once media errors accelerate, the trajectory only goes one way. See available spare below threshold and percentage used at 100%.

Isolated latent defect (single increment during a scan)

No immediate action on the drive itself. The controller has already retired the bad block and mapped in a spare. Record the event, keep the rate alert in place, and confirm your redundancy would have covered the affected LBA if the read had failed upward. If isolated increments keep recurring on successive scans, treat it as the early stage of the wear pattern instead.

Unsafe shutdown aftermath

Fix the power path, not the drive: PSU, UPS, PDU, or whatever caused the ungraceful power loss. Then verify data integrity at the filesystem or application layer, because in-flight writes on a drive without power-loss protection may have been acknowledged but not persisted. Monitor the media error rate over the following days; if it returns to zero, it was a one-time event.

Young drive with errors (defect or batch issue)

Treat as a warranty case. Capture the full SMART log, error log, and firmware version (nvme fw-log /dev/nvme0) before contacting the vendor. Check fleet-wide: if other drives from the same batch and age show similar counters, escalate procurement for the whole cohort.

Transport or firmware cause

Reseat the drive, inspect connectors and cables, and confirm the link trains to full speed and width. Check for a firmware update addressing data-integrity issues before condemning the hardware. If the errors stop after the physical or firmware fix and the SMART rate stays flat, the NAND was never the problem.

Prevention

Alert on the rate, not the value. Any media_errors increment within a monitoring window should open a ticket. Reserve the page for the corroborated condition: rising errors plus critical_warning bit 2.
Trend available_spare and percentage_used continuously. Media errors rarely arrive without warning on a wearing drive; spare consumption and endurance rate are the leading indicators that give you weeks of runway.
Track unsafe_shutdowns and fix power infrastructure. Each ungraceful power loss on a drive without PLP is a dice roll on write-cache data.
Baseline PCIe health at provisioning. AER counters at zero and full negotiated link speed should be a deployment gate, so later deviations are visible.
Track firmware versions fleet-wide. Data-integrity bugs are fixed in firmware; you cannot act on a vendor advisory if you do not know which drives run the affected version.
Scrub cold data periodically. Regular full reads (or filesystem-level scrubs) surface latent blocks while your redundancy can still repair them, and let the controller retire marginal blocks on its own schedule instead of during an incident.

How Netdata helps

Netdata collects the NVMe SMART log per device and exposes media_errors as an incremental rate chart (nvme.device_media_errors_rate), so a single new error is visible as a spike instead of being lost in a lifetime total.
The critical warning bitmask is broken out per bit, so you can alert on bit 2 (reliability degraded) separately from bit 0 or bit 1 instead of firing one blanket alert on any nonzero value.
Available spare, endurance consumed, and unsafe shutdowns sit on the same per-device dashboard as media errors, which is exactly the correlation set this diagnosis depends on.
Error log entries have their own rate chart, making the “non-media errors rising, media errors flat” firmware-driver case distinguishable from genuine NAND failure at a glance.
Per-second collection means the timing correlation that classifies the event (backup window, power event, link retraining) is preserved rather than averaged away.