NVMe available spare declining: watching the wear trajectory before the threshold

The drive reported available_spare at 100% for two years. This quarter it reads 94%, and last month it was 96%. Nothing has alerted: critical_warning is zero, I/O is clean, latency is normal. The question is whether you are watching normal aging or the early edge of a failure curve. The answer is almost never in the current value. It is in the rate of change.

Available spare is a gauge, not an event. By the time the drive asserts critical_warning bit 0 (available spare below threshold), you are inside the vendor’s safety margin and replacing the drive on the failure’s schedule instead of yours. An accelerating, non-linear spare decline is one of the strongest leading indicators of premature NVMe failure, and you can only see it if you track consumption months before the threshold crossing.

This article covers the trajectory: what the gauge measures, how to compute consumption rate and runway, what abnormal decline looks like, and where to set alerts. Once the threshold is crossed and bit 0 asserts, the response changes to replacement-now. That path is covered in NVMe available spare below threshold.

What this means

The controller keeps a pool of spare NAND blocks. As cells wear out or develop uncorrectable errors, the controller retires them and maps in replacements from the pool. Available spare is the percentage of that pool remaining. It starts at 100% and moves in one direction only. When the pool runs dry, the next failed block has no replacement, and the drive protects your data the only way it can: it places the media in read-only mode (critical_warning bit 3) and rejects all writes.

Two fields from the SMART log define your position: available_spare, the current level, and available_spare_threshold, the vendor-set floor. The threshold is typically 10% but varies by vendor and model, so read it from the drive rather than assuming. When available spare falls to or below that floor, the drive sets critical_warning bit 0 on its own. Your monitoring should get you moving well before the drive has to tell you.

Three properties of this gauge trip teams up:

• It is not percentage_used. The two track different things and move independently. Percentage used is a vendor estimate of consumed write endurance; available spare is the physical block reserve. You can see 50% used with 100% spare, or 100% used with 80% spare. Never infer one from the other.
• The decline is non-linear. Spare consumption accelerates as the drive ages: the last 10% of the pool depletes much faster than the first 10%. A linear extrapolation from early life overestimates your runway.
• Reporting can be stepped. Some controllers only report coarse values (100, 90, 80…). A single step down from 100 to 90 is not necessarily a failure event. Trajectory analysis needs multiple samples over weeks, not a reaction to one reading.

The diagnostic flow:

flowchart TD
  A["available_spare trending down"] --> B{"Decline rate"}
  B -->|"slow, linear"| C["Normal wear: project runway"]
  B -->|"accelerating"| D["Correlate: media_errors, temp, percentage_used"]
  C --> E{"Position vs spare_thresh"}
  D --> E
  E -->|"above 2x threshold"| F["PLAN: keep trending"]
  E -->|"at or below 2x threshold"| G["TICKET: schedule replacement"]
  E -->|"at or below threshold"| H["Bit 0 asserts: replace now"]
  H --> I["Spares exhausted: read-only, bit 3"]

Common causes

Quick checks

All of these are read-only SMART log queries. They typically need root privileges to access the device.

# Current spare level and vendor threshold (one grep catches both lines)
nvme smart-log /dev/nvme0 | grep "available_spare"

# Endurance context: is percentage_used moving with spare?
nvme smart-log /dev/nvme0 | grep "percentage_used"

# Active media degradation: uncorrectable errors retire blocks from the pool
nvme smart-log /dev/nvme0 | grep "media_errors"

# The drive's own warning state (bit 0 = spare below threshold)
nvme smart-log /dev/nvme0 | grep "critical_warning"

# Host write volume for the workload sanity check
nvme smart-log /dev/nvme0 | grep "data_units"

# Thermal state and history (SMART temperature is Kelvin; subtract 273.15)
nvme smart-log /dev/nvme0 | grep "temperature"
nvme smart-log /dev/nvme0 | grep "warning_temp_time"

# One-line snapshot for trajectory logging (run daily via cron or your collector)
nvme smart-log /dev/nvme0 -o json | jq '{spare: .avail_spare, thresh: .spare_thresh, pct_used: .percent_used, media: .media_errors}'

Spare moves over weeks and months, and SMART values update on the controller’s own polling schedule, not instantly. Daily snapshots are plenty for trajectory work.

How to diagnose it

1. Establish position against the threshold. Read both available_spare and available_spare_threshold from the drive. Compute position as spare divided by threshold. Above 2x the threshold is watch-and-trend territory. At or below 2x, schedule a replacement. At or below 1x, the drive asserts bit 0 itself.

2. Build the trajectory. A single smart-log reading is a snapshot. Pull the time series from your monitoring (Netdata charts this as nvme.device_available_spare_perc) over at least 90 days. If you have no history, start logging daily snapshots now. Even two weeks of data separates “steady” from “falling”.

3. Compute consumption rate and project runway. Runway in months equals (available_spare minus spare_thresh) divided by spare points lost per month. Example: 94% spare, 10% threshold, losing 2 points per month gives (94 - 10) / 2 = 42 months. Then check whether the monthly loss itself is growing. If the rate is accelerating, the linear answer overstates your runway. Apply a 2x safety factor to any projection that extends into the final 10% of the pool.

4. Classify the wear. Compare against percentage_used. If percentage_used is climbing in step with spare loss, endurance is being consumed by writes and the trajectory is workload-driven. If spare is falling while percentage_used is still low, suspect a NAND quality issue rather than normal wear.

5. Check for active media degradation. media_errors is a lifetime counter, so the rate of increase is the signal, not the absolute value. Any sustained increase means blocks are being retired right now. Pull nvme error-log /dev/nvme0 for details. Rising media errors combined with critical_warning bit 2 is the active-degradation pattern, and replacement becomes urgent regardless of current spare level.

6. Check the thermal leg. Heat accelerates NAND wear. Look at the composite temperature trend and whether warning_temp_time (cumulative minutes above the warning threshold) is still accumulating. A drive that has run hot will consume its spare pool faster than its write volume alone predicts.

7. Check workload and fill level. Compute your actual write rate from data_units_written deltas and compare it to the drive’s endurance rating. Look for write-amplifying configurations: swap on NVMe, excessive logging, journal modes that write data twice. Above roughly 80-90% full, garbage collection pressure rises and wear accelerates. Confirm TRIM/discard is enabled and running.

Metrics and signals to monitor

Fixes

You cannot add spare blocks back. The gauge only moves down. Every action below either slows the decline or lands the replacement before the cliff.

Normal wear on projection

If the decline is slow and linear, percentage_used tracks it, and temperatures are clean, there is nothing to fix. Put the projected runway date on the replacement calendar, re-check the slope monthly, and move on. Escalate to an active ticket only when position reaches 2x the threshold.

Workload-driven write amplification

The drive wears in proportion to NAND writes, which exceed host writes when the workload is small random writes, the drive is nearly full, or TRIM is not running. Find the writer first: swap on NVMe, chatty application logging, and double-write journal modes (ext4 data=journal writes every data block twice) are the usual suspects. Verify TRIM/discard is enabled. Bring fill level back under 80%. Tradeoff: changing journal mode or relocating swap has its own durability and performance implications, so test before rolling out.

Thermal-accelerated wear

Fix the cooling. M.2 drives without heatsinks run hot under sustained load, and a cheap heatsink plus verified airflow can change the wear slope materially. Check for adjacent heat sources (GPUs, other drives). There is no real tradeoff here; cooling is the cheapest wear reduction available.

Suspected NAND quality issue

Spare falling while percentage_used is low points at the NAND, not the workload. Compare same-model, same-age drives across your fleet; a matching trajectory across a cohort confirms a batch problem. Pursue warranty or RMA while the drive is still functional rather than waiting for the threshold to make the decision for you.

Active media degradation

media_errors rising alongside falling spare is the pattern that ends in read-only mode or data loss. Verify RAID or replication health first, rewrite at-risk cold data while the drive can still serve reads, and schedule an immediate replacement. Do not wait for the runway math on this one.

Prevention

• Baseline the threshold at provisioning. Record spare_thresh per drive model, because it is vendor-set and not always 10%. Alerts written against an assumed floor are wrong for part of your fleet.
• Alert on position, trend the rate. Warn at or below 2x spare_thresh, critical at or below spare_thresh. Between those lines, the slope is the signal; watching only the current value is the most common mistake teams make with this gauge.
• Track from day one. A trajectory needs history. You cannot reconstruct a decline curve after the drive starts dying.
• Match drive class to workload. Consumer QLC drives rate around 100-300 TBW; enterprise TLC rates 1-10 DWPD. Compute actual DWPD from data_units_written and compare it against the rating before deployment, not after the spare pool starts dropping.
• Keep free-space headroom. Maintain at least 20% logical free space so garbage collection has room to work. Fill level is a wear-rate knob.
• Manage thermals. Heatsinks on M.2, verified airflow, and attention to warning_temp_time keep heat from silently multiplying your wear rate.

How Netdata helps

• Netdata charts nvme.device_available_spare_perc continuously, which makes the trajectory visible instead of forcing you to reconstruct it from manual smart-log runs. The slope, not the snapshot, is the early-warning signal.
• Charting nvme.device_estimated_endurance_perc alongside spare separates normal endurance wear from abnormal spare loss in one view: the two gauges moving together is aging, spare moving alone is a hardware question.
• nvme.device_media_errors_rate next to the spare chart shows whether active degradation is driving consumption, which changes the response from “schedule a swap” to “expedite it”.
• nvme.device_critical_warnings_state breaks the critical warning bitmask into per-bit dimensions, so the bit 0 threshold crossing and the bit 3 read-only terminal state alert independently instead of hiding behind a single nonzero check.
• nvme.device_composite_temperature and nvme.device_warning_composite_temperature_time supply the thermal leg of the correlation, explaining wear that runs ahead of what write volume predicts.

Related guides

• NVMe available spare below threshold: critical warning bit 0 and end-of-life wear
• NVMe critical_warning is nonzero: decoding the SMART critical warning bitmask
• NVMe drive in read-only mode: critical warning bit 3 and rejected writes
• NVMe temperature threshold exceeded: critical warning bit 1, WCTEMP, and CCTEMP
• NVMe monitoring checklist: the signals every production SSD needs
• How NVMe actually works in production: a mental model for operators