$ guides / nvme / nvme-silent-data-degradation ▌

Operations Guides

NVMe silent data degradation: media errors, reliability bit, and failing cold reads

The drive is still there. It answers I/O, the filesystem is mounted, latency looks mostly normal, and nothing in dmesg is screaming. But media_errors has been climbing for days, critical_warning now shows bit 2 set, and a few reads per hour are inexplicably slow. A backup verification job just failed a checksum on a file nobody has written to in months.

This is NVMe silent data degradation: NAND cells are losing their ability to hold charge, reads need multiple internal retry passes, and some reads fail outright even after ECC. The device stays responsive the whole time. There is no crash to page you, only a slow accumulation of uncorrectable reads, usually on data that is written once and rarely touched.

This article covers how to confirm the pattern, how to separate it from lookalikes, and what to do in the first hour.

What this means

NAND flash stores data as trapped charge in cells. That charge leaks over time, and leakage accelerates with cell wear (program/erase cycles), high operating temperature history, and read disturb from repeatedly reading neighboring cells. The controller’s ECC absorbs a certain number of flipped bits per read. When a cell degrades past what ECC can correct, the controller retries the read with shifted voltage thresholds. Each retry pass adds latency, sometimes tens of milliseconds. If all retry passes fail, the controller records a media error and returns an I/O error to the host.

Two properties of this mechanism explain the symptoms:

Errors appear on reads, not writes. Writes program fresh charge and succeed. Reads of old data find charge that has drifted. The drive can look healthy under a write-heavy workload while cold data rots underneath.
Cold data fails first. Recently written data still has strong charge. Data written months ago, at high temperature, on worn cells, is where retention failure shows up. Backup scans, quarterly reports, and archive reads are the workloads that trip over it.

The controller knows this is happening. When it assesses that media errors have degraded subsystem reliability, it sets critical_warning bit 2 (NVM subsystem reliability degraded). That bit plus a rising media_errors rate confirms degradation is active and ongoing, not historical.

flowchart TD
  A[NAND cells lose retention] --> B[Read needs ECC retry passes]
  B --> C[Sporadic high read latency]
  B --> D[Retries exhausted: uncorrectable read]
  D --> E[media_errors increments]
  D --> F[Error log entry with LBA]
  E --> G[critical_warning bit 2 set]
  A --> H[available_spare declining]
  A --> I[percentage_used high]
  G --> J[PAGE: active degradation confirmed]

Common causes

Cause	What it looks like	First thing to check
Drive beyond rated endurance	`percentage_used` at or above 100%, `available_spare` declining, media errors emerging	`nvme smart-log` for `percentage_used` and `available_spare` trend
High temperature history	`warning_temp_time` and `critical_comp_time` counters elevated; errors concentrated on old data	SMART thermal time counters vs. power-on hours
Read disturb	Errors on blocks adjacent to very hot read regions; drive otherwise healthy	`nvme error-log` for LBA patterns in failing entries
One-time power-loss event	Cluster of media errors surfacing after an unsafe shutdown, then stable	`unsafe_shutdowns` counter and when it last incremented
Latent bad blocks surfaced by a scan	Single-digit new errors during first full read in months, then quiet	Rate of new errors after the scan completes

A cluster of media errors right after an unsafe shutdown can be a one-time event: in-flight writes corrupted by the power loss, discovered on next read. The distinguishing test is whether the error rate keeps rising after the event.

Quick checks

All read-only and safe to run during an incident.

# Full SMART health snapshot
nvme smart-log /dev/nvme0

# The four fields that matter for this pattern
nvme smart-log /dev/nvme0 | grep -E "critical_warning|media_errors|available_spare|percentage_used"

# Detailed error entries: LBA, namespace, status code
nvme error-log /dev/nvme0

# Controller still alive and processing commands?
cat /sys/class/nvme/nvme0/state

# Kernel-visible I/O errors corroborating the SMART counters
dmesg | grep -i "nvme" | grep -iE "error|critical"

# Unsafe shutdowns, to test the power-loss theory
nvme smart-log /dev/nvme0 | grep unsafe_shutdowns

Interpretation notes:

critical_warning is a bitmask. Bit 2 set means the value has 0x04 in it. Decode all bits, not just this one; if bit 0 (spare below threshold) or bit 3 (read-only mode) is also set, your urgency just went up. See NVMe critical_warning is nonzero for the full decode.
media_errors is a lifetime counter and never decreases. The absolute number matters less than the rate of change. Single-digit errors over years of operation can be within normal tolerance for some enterprise drives.
nvme error-log entries tell you which LBAs failed and with what status. Each entry carries an internal error sequence count you can use to order events. The log is a circular buffer; at high error rates the oldest entries get overwritten, so capture it early.
The error log entry counter (num_err_log_entries) is broader than media_errors: it also counts admin command errors and invalid commands from tooling. If error log entries are rising but media_errors is flat, suspect software, not media.

How to diagnose it

Confirm the two corroborating signals. Check that critical_warning bit 2 is set AND media_errors is actively increasing. Take two nvme smart-log samples a few hours apart (or under normal workload) and compare. Bit 2 alone, with a flat media_errors counter, is a ticket-level signal: the vendor’s definition of “reliability degraded” varies, and some drives set it preemptively, for example when percentage_used crosses 100% with zero media errors. Bit 2 plus a rising media error rate is page-worthy: the drive’s self-assessment and observed behavior agree.
Classify the errors from the error log. Run nvme error-log /dev/nvme0 and look at status codes and LBAs. Uncorrectable read errors concentrated in specific LBA ranges point at specific data. Errors spread across the address space point at general media wear.
Rule out the one-time power-loss theory. Compare unsafe_shutdowns against your records of power events. If media_errors jumped around a known unsafe shutdown and has been flat since, you may be looking at residue, not active degradation. Keep watching the rate to confirm.
Check the wear context. Pull percentage_used, available_spare, and spare_thresh. High percentage used plus declining spare plus media errors is the classic end-of-life arc. Declining spare with moderate percentage used can indicate a bad NAND batch rather than normal wear. See NVMe available spare declining for reading that trajectory.
Check thermal history. warning_temp_time and critical_comp_time are cumulative minutes above the warning and critical temperature thresholds. A large fraction of power-on hours spent hot accelerates retention loss and explains why this drive is degrading ahead of its peers.
Corroborate at the kernel layer. Look for blk_update_request: I/O error, dev nvme0n1 lines in dmesg. These are the host-side view of the uncorrectable reads the SMART counter is recording. If the kernel log shows I/O errors but SMART shows nothing, treat SMART as stale and trust the kernel. See blk_update_request: I/O error, dev nvme0n1.
Map failing LBAs to data. If the error log gives you LBAs, determine which files or filesystem structures sit on them before you rewrite anything. The procedure is filesystem-specific; the goal is to know exactly what is at risk rather than treating the whole volume as uniformly suspect.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`media_errors` rate	Direct count of uncorrectable data integrity events	Any sustained rate above zero
`critical_warning` bit 2	Drive’s own assessment that reliability is degraded	Set, especially alongside rising media errors
Read latency (tail)	Retry passes show up as sporadic slow reads before they become errors	Occasional 10ms+ read outliers with no load explanation
`available_spare` vs `spare_thresh`	Remaining blocks for bad-block replacement	Declining trend, or at/below 2x threshold
`percentage_used`	Endurance consumed; context for whether wear explains the errors	Above 90%, or rate above ~1% per week
`warning_temp_time` / `critical_comp_time`	Thermal history accelerates retention loss	Growing as a fraction of power-on hours
`num_err_log_entries` rate	Broader error activity including non-media errors	Rising without media errors (points at firmware/driver)
Kernel I/O error lines	Host-side confirmation of uncorrectable reads	`blk_update_request: I/O error` in dmesg

Fixes

There is no fix that restores degraded NAND. Every action below is about protecting data and retiring the drive.

Protect the data first

Verify RAID or replication health. If the drive is part of a RAID set or a replicated volume, confirm the redundancy is intact and a resync or rebuild would succeed. The worst version of this incident is discovering a second degraded member during the rebuild of the first.
Verify backups are restorable. Cold data is what fails, and cold data is what backups hold. Do a restore test of the at-risk data, not just a job-status check.
Force-rewrite at-risk data. Reading and rewriting a block programs fresh charge into the cells, resetting the retention clock. For data that is still readable, a rewrite makes it durable again on the remaining life of the drive. A full-disk read pass (or a RAID scrub) also surfaces every latent error now, while you still have redundancy, instead of during the rebuild later. On Linux mdraid this is what a scrub does; on other stacks, use the equivalent. It is I/O-intensive and will add latency to a live workload, so schedule it, but do not skip it.

Do not run nvme format, sanitize, or any destructive command as a diagnostic. The drive’s remaining readable data is your recovery source until the replacement is in place.

Replace the drive

Treat confirmed silent data degradation (bit 2 plus rising media errors) as replace-immediately, not replace-at-next-window. The trajectory is monotonic: error rates accelerate as more cells cross the ECC threshold, and once spare blocks are exhausted the drive will go read-only (bit 3) or start returning hard failures. See NVMe available spare below threshold for the endgame of that arc.

When the replacement arrives, follow your stack’s standard procedure for failing out the old device, rebuilding, and verifying. Check firmware revision on the new drive against the rest of the fleet before it takes traffic.

If the errors turn out to be a one-time event

If diagnosis shows a flat error rate after a single unsafe shutdown, the drive may have years left. Keep it, but fix the power path, and put a tight alert on the media error rate so you catch it if the assumption proves wrong.

Prevention

Alert on the combination, not just the counter. Bit 2 alone pages too aggressively on some drives; a lone media error on a five-year-old drive pages too aggressively too. The pair, bit 2 plus rising media error rate, is the high-signal page. Individual signals stay at ticket severity.
Trend percentage_used and available_spare rates. Degradation is predictable months out if you watch the trajectory instead of the current value. See NVMe endurance runway.
Keep drives cool. Thermal history compounds retention loss. Track warning_temp_time as a fraction of power-on hours, not just live temperature.
Scrub cold data regularly. Scheduled full-read scrubs force the controller to detect and relocate marginal cells early, and force-refresh charge on aging data. A scrub finding errors is a success: it means you found them while redundant.
Track thermal and unsafe shutdown history per drive. Drives with high unsafe shutdown counts or heavy thermal time are the ones to watch hardest for retention failure.

How Netdata helps

Netdata polls the NVMe SMART log and exposes media_errors as an incremental rate (nvme.device_media_errors_rate), which is the form you actually alert on, since the raw counter is lifetime and monotonic.
Each critical_warning bit is broken out as its own dimension (nvme.device_critical_warnings_state), so you can alert on bit 2 combined with the media error rate rather than firing on any nonzero byte.
available_spare, spare_thresh, and percentage_used are charted together (nvme.device_available_spare_perc, nvme.device_estimated_endurance_perc), making the wear context for media errors visible in one place.
Thermal time counters (nvme.device_warning_composite_temperature_time, nvme.device_critical_composite_temperature_time) let you correlate degradation with heat history on the same dashboard.
Because collection is per-second, the sporadic latency outliers that come from read-retry passes are visible in block-device latency charts instead of being averaged away by slower pollers.