NVMe available spare below threshold: critical warning bit 0 and end-of-life wear

Your monitoring fired on an NVMe drive: critical warning bit 0 is set, or available_spare has dropped to (or below) the vendor threshold. The drive still works. Reads and writes complete, latency looks fine, nothing in the application layer is complaining. That is exactly what makes this signal easy to ignore and expensive to ignore.

Bit 0 means the controller’s pool of spare NAND blocks, the reserve it uses to transparently replace failed cells, has fallen below the safety margin the vendor baked into the firmware. The drive is telling you it is approaching end-of-life. It is not telling you it has failed.

This is a replacement-now signal, not a 3 a.m. page. The correct response is procurement plus a scheduled swap, with urgency scaled to how fast the spare pool is draining. The wrong response is either panicking (the drive is functional) or filing the alert away forever (the next bad block with no spare left is what pushes the drive into read-only mode, bit 3, and that one is a page).

What this means

Every NVMe controller keeps a pool of spare NAND blocks. As cells wear out or develop uncorrectable errors, the controller retires them and maps in spares. The SMART/Health Information log exposes two fields for this:

• available_spare: the percentage of the spare pool remaining. Starts at 100%, only goes down.
• spare_thresh: the vendor-set threshold below which the controller considers itself in a critical state (printed as available_spare_threshold by nvme-cli). Typically 10%, but it varies by vendor and model, so read it off the drive rather than assuming.

When available_spare drops below spare_thresh, the controller sets bit 0 of the critical_warning byte (value 0x01). In Netdata this surfaces as the available_spare dimension in the nvme.device_critical_warnings_state chart, alongside the raw percentage in nvme.device_available_spare_perc.

Two properties of this signal matter operationally:

1. Spare consumption is non-linear. The last 10% of the spare pool depletes far faster than the first 10%. A drive that took three years to go from 100% to 15% can go from 15% to 0% in weeks. Near the threshold, the runway is shorter than the history suggests.
2. Available spare and percentage used track different things. percent_used is the vendor’s endurance estimate against the rated TBW/DWPD. available_spare is the physical block reserve. You can have 50% used and 100% spare, or 100%+ used and 80% spare. A drive with moderate percent_used but collapsing spare is not wearing normally; suspect a NAND quality issue (a bad batch) rather than ordinary wear-out.

The failure cascade:

flowchart TD
  A[NAND cells wear out] --> B[Controller remaps to spare blocks]
  B --> C[available_spare declining]
  C --> D{available_spare below spare_thresh?}
  D -- no --> C
  D -- yes --> E[critical_warning bit 0 set: replace soon]
  E --> F[Spare pool keeps draining, non-linear]
  F --> G{Spares exhausted, next bad block?}
  G -- yes --> H[critical_warning bit 3: media read-only, PAGE]

Common causes

Quick checks

All of these are read-only and safe on a production host.

# The core reading: spare level, threshold, and the warning byte
nvme smart-log /dev/nvme0 | grep -i "available_spare\|critical_warning"

# Endurance context: is this normal wear or something faster?
nvme smart-log /dev/nvme0 | grep -i "percentage_used\|data_units_written\|power_on_hours"

# Active degradation check: media errors should not be climbing
nvme smart-log /dev/nvme0 | grep -i "media_errors\|num_err_log_entries"

# Error detail: which LBAs/namespaces, what status codes
nvme error-log /dev/nvme0 --log-entries=16

# Thermal history: heat accelerates NAND wear
nvme smart-log /dev/nvme0 | grep -i "temperature\|warning_temp_time\|critical_comp_time"

# Controller state: anything other than "live" means an ongoing reset or removal problem
cat /sys/class/nvme/nvme0/state

SMART data is per-controller, so nvme smart-log /dev/nvme0 is the usual form; against a namespace like /dev/nvme0n1 it also works with nvme-cli. The error log is a circular buffer, so capture it early before new entries overwrite the old ones.

How to diagnose it

1. Confirm which bit is actually set. Read the critical_warning hex value. 0x01 is bit 0 (spare below threshold). Do not confuse it with 0x04 (bit 2, NVM subsystem reliability degraded) or 0x08 (bit 3, read-only mode). Each bit has a different response. If the value is 0x09, both bit 0 and bit 3 are set: the drive has already gone read-only and this is no longer a planning exercise, it is an incident.
2. Read available_spare and spare_thresh off the drive. Do not assume the threshold is 10%. The gap between current spare and threshold tells you how far past the line the drive is.
3. Establish the trajectory. Pull historical available_spare values from monitoring. A single reading tells you where you are; the slope tells you how long you have. Estimate runway as (current_spare - threshold) / (spare consumed per month), then compress that estimate, because consumption accelerates as the pool shrinks.
4. Check the media_errors rate. Spare decline with flat media errors is wear-out in progress. Spare decline with rising media errors is active degradation, and if bit 2 is also set the drive itself has assessed its reliability as compromised. That combination escalates urgency from “schedule a swap” to “verify redundancy and swap this week.”
5. Check percentage_used for the wear pattern. High percent_used plus slow spare decline is textbook end-of-life. Moderate percent_used plus fast spare decline points at a NAND quality problem; check sibling drives from the same batch.
6. Verify redundancy before doing anything else. Confirm RAID, replication, or backup health for the data on this drive. The drive is functional today; your job is to make sure that stops mattering.
7. Rule out workload causes for the fleet. If multiple drives are draining spare early, look at write volume versus drive class, missing TRIM/discard, swap on NVMe, and thermal history. Replacing drives without fixing the workload just restarts the clock.

Metrics and signals to monitor

Fixes

There is no fix that restores spare blocks. Every path below is a variant of “get the data off and retire the drive”; the differences are timing and urgency.

Drive is below threshold, media errors flat, redundancy healthy

Procure a replacement and schedule the swap in a normal maintenance window. Keep monitoring the spare slope; if it steepens, pull the window forward. Do not postpone indefinitely: the non-linear tail means “it has been fine for months” is not evidence it will be fine next month.

Drive is below threshold and media errors are rising, or bit 2 is also set

Treat as urgent. Verify RAID/replication/backup integrity first, then migrate the data and replace the drive as soon as logistics allow. The controller has both exhausted its margin and detected active degradation.

Drive reports bit 3 (read-only, 0x08 or 0x09)

The drive has self-protected and rejects writes. This is a page, not a ticket. Fail over to redundant storage, copy data off read-only, and replace. There is no supported way to reverse read-only mode; it is a firmware-level decision.

Root cause is workload, not the drive

If the investigation shows write amplification or a workload-to-drive-class mismatch, fix that before the replacement drive inherits the same problem: enable TRIM/discard (the fstrim timer or mount-time discard), keep logical free space above roughly 20%, move swap and excessive logging off the NVMe device, and match the replacement’s endurance rating (DWPD) to the measured write rate. Sizing the replacement on capacity alone repeats the failure.

Prevention

• Alert per bit, not on the byte. A single critical_warning != 0 alert collapses bit 0 (replace soon), bit 1 (thermal, often transient), and bit 3 (read-only, page now) into one severity. Bit 0 should page no one; bit 3 should wake someone up.
• Track the spare trajectory, not just the current value. Rate-of-change of available_spare is one of the strongest leading indicators of premature failure. Alert at 2x threshold so procurement starts before bit 0 ever fires.
• Baseline endurance rate at deployment. Compute actual DWPD from data_units_written and compare it to the drive’s rating. A drive consuming rated endurance 3x faster than planned will hit bit 0 years early, and you can see that in month one.
• Match drive class to workload during provisioning. Consumer drives in write-intensive roles, without power-loss protection and with minimal overprovisioning, hit spare exhaustion on a completely different schedule than enterprise drives.
• Keep thermal headroom. Sustained operation near thermal thresholds accelerates NAND wear. Warning temperature time accumulating is an early sign the drive is aging faster than its write volume alone would cause.

How Netdata helps

• Netdata exposes each critical warning bit as its own dimension in nvme.device_critical_warnings_state, including available_spare (bit 0) and read_only (bit 3), so alerts can carry the right severity per bit instead of a single noisy byte check.
• The nvme.device_available_spare_perc chart gives you the current spare percentage continuously, which is what you need for slope and runway estimation rather than one-off nvme smart-log reads.
• nvme.device_estimated_endurance_perc (percent used) and nvme.device_media_errors_rate sit next to the spare chart, so the key diagnostic correlation, spare decline with or without rising media errors, is a single glance.
• Thermal context from nvme.device_composite_temperature and warning/critical temperature time charts lets you confirm or rule out heat as the wear accelerant during the same investigation.
• Because SMART data is polled, having these series captured continuously means you still have the full history when the drive later transitions from bit 0 to bit 3, which is exactly the window where post-incident questions get asked.

Related guides

• blk_update_request: I/O error, dev nvme0n1: reading NVMe I/O errors in the kernel log
• NVMe controller reset loop: repeated resets from a firmware hang
• nvme nvme0: I/O timeout, Resetting controller: what an NVMe controller reset means
• NVMe controller state not live: reading resetting, deleting, and dead from sysfs
• NVMe device disappeared: nvme0: Removing and a drive that fell off the PCIe bus
• How NVMe actually works in production: a mental model for operators
• NVMe monitoring checklist: the signals every production SSD needs
• NVMe monitoring maturity model: from survival to expert