NVMe write cliff: SLC cache exhaustion and garbage-collection stalls

Write throughput on an NVMe drive falls off a cliff: 50-90% below baseline, arriving as a step function rather than a gradual decline. Write latency jumps 3-10x at the same moment. The application layer starts reporting slow queries, stalled flush operations, or request timeouts, and the drive looks guilty.

The confusing part is what is absent. The drive is not hot. There are no media errors. The kernel log is quiet. SMART looks clean. Standard disk monitoring shows a busy device, which tells you nothing you did not already know.

This is the NVMe write cliff. It has two closely related causes: SLC cache exhaustion, which is normal controller behavior and mostly a workload-fit problem, and garbage-collection (GC) thrashing under space pressure, which is an operational problem you can fix. The diagnostic work is telling them apart, and telling both apart from a drive that is actually dying.

What this means

NAND flash cannot be overwritten in place. The controller must erase an entire block before rewriting it, and erases are slow. To hide this, the controller keeps a pool of pre-erased blocks and runs garbage collection in the background: it copies still-valid pages out of partially used blocks, erases the leftovers, and returns them to the free pool.

On top of that, most consumer and many entry datacenter drives program part of their TLC or QLC NAND in single-bit mode as a pseudo-SLC write cache. SLC writes are several times faster than native TLC/QLC writes, so burst write performance looks excellent until the cache fills. When it does, new writes go straight to TLC/QLC at native speed. That transition is the cliff.

GC pressure makes the cliff deeper and stickier. When the drive is nearly full (past roughly 80-90% of usable capacity) or heavily worn, the free-block pool is small, GC cannot keep up with incoming writes, and host writes wait on block erasure inline. If TRIM/discard is not reaching the drive, the FTL also believes deleted blocks are still live and wastes GC effort copying stale data forward, inflating write amplification and accelerating wear.

The signature that separates this from real hardware failure: high latency, a busy controller, normal temperature, and media errors not increasing. The drive is working hard, just not on your writes.

flowchart TD
  A[Write throughput drops 50-90%] --> B{Composite temperature elevated?}
  B -->|yes| C[Thermal throttling path]
  B -->|no| D{media_errors increasing?}
  D -->|yes| E[Media degradation path]
  D -->|no| F{Controller busy with low host IOPS?}
  F -->|yes| G{Drive >80-90% full or heavily worn?}
  G -->|yes| H[GC thrashing under space pressure]
  G -->|no| I[SLC cache exhaustion]
  F -->|no| J[Check host queuing and PCIe link]

Common causes

Cause	What it looks like	First thing to check
SLC cache exhaustion	Step-function throughput drop during sustained writes, recovers after idle time, no errors	Does throughput recover within minutes of stopping writes?
GC thrashing on a nearly full drive	Persistent degraded write performance, periodic latency spikes on a 30s-5min cycle, does not fully recover at idle	Filesystem fill level (`df -h`), drive capacity utilization
TRIM/discard not reaching the drive	Chronic write latency spikes, degradation worse than fill level alone would explain	Mount options, `fstrim.timer` status
Heavily worn drive with low over-provisioning headroom	Cliff arrives earlier and recovery is slower on a drive with high `percent_used`	`nvme smart-log` percent_used and avail_spare
Consumer drive in a sustained-write enterprise workload	Advertised burst speeds never sustained, cliff after a few GB to tens of GB of writes	Drive model’s cache behavior versus your write burst size

Quick checks

All read-only and safe to run during an incident.

# Snapshot the SMART fields that matter for the write cliff
nvme smart-log /dev/nvme0 | grep -E "critical_warning|temperature|percent_used|avail_spare|spare_thresh|media_errors|ctrl_busy_time|data_units_written"

# Check filesystem fullness on the affected mount
df -h /mountpoint

# Check whether discard is mounted or fstrim is scheduled
findmnt -o TARGET,OPTIONS /mountpoint
systemctl list-timers fstrim.timer --no-pager

# Check the drive's thermal thresholds (WCTEMP/CCTEMP live in Identify Controller, not SMART)
nvme id-ctrl /dev/nvme0 | grep -i "wctemp\|cctemp"

# Watch block-layer I/O for the cliff shape
iostat -xp nvme0n1 2 5

For average write latency, compute it from block stat deltas rather than trusting any single tool’s average: take two samples of /sys/block/nvme0n1/stat a known interval apart and divide the change in milliseconds-spent-writing (field 8) by the change in writes completed (field 5). Averages hide the tail. The write cliff shows up hardest in p99/p99.9, which needs BPF tooling such as biolatency if you have it.

One caveat on iostat: on some kernels the %util figure for NVMe devices is unreliable and can pin near 100% on nearly idle drives. Trust throughput and computed latency over %util.

How to diagnose it

Confirm the cliff shape. Look at write throughput over the incident window. A step-function drop of 50-90% that holds, with write latency jumping 3-10x at the same moment, is the write cliff. Gradual decline points elsewhere (thermal, link degradation).
Rule out thermal throttling. Compare composite temperature against WCTEMP from nvme id-ctrl. If temperature is normal and has been flat through the incident, this is not throttling. Thermal issues and the write cliff both drop throughput; temperature is the discriminator.
Rule out media degradation. Check media_errors and critical_warning. Both steady at baseline means the NAND is not failing and this is a performance event, not a health event. Rising media errors change the entire response: stop here and follow the degradation path instead.
Check controller busy versus delivered work. Read ctrl_busy_time (minutes) twice, a few minutes apart, and compare its growth against elapsed time and against host IOPS. The tell for GC thrashing is a controller busy nearly 100% of the interval while host-visible throughput is a fraction of baseline. The counter resolution is minutes, so it cannot catch brief stalls.
Check fullness and wear. df -h for filesystem fill, plus percent_used and avail_spare from SMART. A drive above roughly 80-90% full, or one with high endurance consumed, has a small free-block pool and reduced over-provisioning headroom. That is the GC thrashing profile.
Check TRIM delivery. If neither the discard mount option nor a working fstrim schedule exists, the FTL treats logically deleted data as live and GC copies it forward forever. This is a common root cause of chronic GC pressure on drives that are nowhere near full.
Run the recovery test. Stop the write workload for a few minutes. If throughput recovers fully at idle, you are looking at SLC cache exhaustion or GC that can still keep up. If recovery is slow or partial and the drive is full, GC is the bottleneck.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Write throughput vs baseline	Defines the cliff itself	Step drop of 50-90% without workload change
Write latency, especially p99	Tail latency is what applications feel	3-10x jump; individual I/Os taking tens of ms
ctrl_busy_time ratio	Separates a busy-healthy drive from a busy-sick drive	Near 100% busy with low host IOPS
Composite temperature	Rules thermal throttling in or out	Normal during the cliff confirms GC/SLC cause
media_errors rate	Rules media failure in or out	Any increase means stop and change response
Drive fill level	Primary driver of GC pressure	Sustained above 80-90%
percent_used / avail_spare	Worn drives have less GC headroom	High wear plus the cliff means replacement, not tuning
data_units_written rate	Feeds write-amplification suspicion	Growing faster than application writes explain

Fixes

SLC cache exhaustion

Accept it first: this is designed behavior, not a defect. No amount of tuning makes consumer TLC sustain pseudo-SLC speeds.

Shape the workload around the cache. Break large sustained writes into bursts with idle gaps so the controller can destage SLC data back to TLC. Batch jobs, WAL archiving, and backup streams are the usual candidates.
Leave unpartitioned space. Free capacity the controller can see enlarges the dynamic SLC cache area and gives GC headroom at the same time. Even 10-20% unallocated makes a measurable difference on consumer drives.
Match the drive class to the workload. If the workload genuinely needs sustained write throughput above native TLC/QLC speed, the fix is a drive with larger over-provisioning and sustained-write ratings, not configuration.

GC thrashing under space pressure

Stop or shed writes during the incident. Every write that arrives while GC is behind makes the hole deeper. Throttle the writer, pause the batch job, or drain the node if the architecture allows it.
Free space. Delete what you can, then tell the drive about it. Getting back under roughly 80% fill restores the free-block pool GC needs to get ahead.
Run fstrim once, deliberately:

# Reclaim logically deleted blocks so the FTL stops copying them
fstrim -v /mountpoint

Warning: discard operations can take 1-50ms each and on some controllers partially block concurrent I/O. On a latency-sensitive workload mid-incident, fstrim can briefly make tail latency worse before it makes things better. Run it during the write shed from the step above, not against live traffic.

Fix TRIM delivery permanently. Either mount with discard or keep a scheduled fstrim. The distro-default fstrim.timer is the lower-risk option on most controllers, since it batches discards instead of scattering them through live I/O.
Replace heavily worn drives. If percent_used is high and avail_spare is declining, the drive has permanently less GC headroom than it shipped with. Free space and TRIM buy time; replacement is the fix.

Prevention

Hold the fill line. Keep at least 15-20% free space on any NVMe drive doing meaningful write work. That is FTL operating headroom, not waste. Alert on fill level trending past 80%.
Guarantee TRIM. Verify discard or fstrim.timer at provisioning time and check it in configuration review. A missing one-line mount option is a chronic GC incident waiting to happen.
Baseline sustained write behavior per drive model. Know where each model’s cliff sits when empty and when full, so monitoring can alert on deviation rather than on the cliff itself.
Track wear trajectory. Drives with high percent_used or declining avail_spare hit the cliff earlier and recover slower. Fold wear state into capacity planning.
Alert on the busy-sick combination. Controller busy near 100% while host throughput is low is the earliest reliable indicator of internal contention. Catch it before the application does.

How Netdata helps

Netdata charts NVMe composite temperature (nvme.device_composite_temperature) next to block-layer throughput and latency, so ruling thermal throttling in or out takes seconds instead of a separate SSH session.
Media errors are tracked as an incremental rate (nvme.device_media_errors_rate), which makes “steady” versus “rising” explicit during the differential diagnosis.
Endurance and spare charts (nvme.device_estimated_endurance_perc, nvme.device_available_spare_perc) show whether the drive behind the cliff has the wear headroom to recover or is a replacement candidate.
Per-second disk I/O metrics capture the step-function shape of the cliff and the periodic spike pattern of GC thrashing, which polled SMART alone (30s+ update latency) will miss.
Correlating throughput, latency, temperature, and SMART health on one dashboard turns the four-question differential (hot? failing? busy? full?) into a single view instead of four command lines.