$ guides / smartctl-disk-monitoring / smartctl-ssd-write-cliff ▌

Operations Guides

The SSD write cliff: latency spikes when the pre-erased block pool empties

Write latency on an SSD jumps from sub-millisecond to hundreds of milliseconds or seconds. Applications time out. The kernel log may show command timeouts. But smartctl -H says PASSED, every SMART attribute looks clean, and the drive has plenty of endurance left.

The write cliff happens when an SSD exhausts its pool of pre-erased NAND blocks. Under normal conditions, the controller performs garbage collection (GC) in the background: it reads valid pages from partially invalidated blocks, writes them elsewhere, and erases the now-empty block to replenish the free pool. When the write rate outpaces background GC, the free pool empties. Every host write now requires a synchronous erase cycle before it can complete, and latency explodes by 100x to 1000x.

The drive is healthy and will recover once GC catches up. The operational challenge is distinguishing the write cliff from real drive problems (pending sectors, endurance exhaustion, thermal throttling) and from a related pattern on SMR HDDs that produces identical symptoms.

What this means

NAND flash cannot overwrite data in place. A page must be erased before it can be written again, and erase operations work at the block level, where a block contains many pages. The SSD controller hides this behind a flash translation layer (FTL) that maintains a pool of pre-erased blocks for fast host writes and reclaims invalidated pages through background GC. Write amplification (the ratio of NAND writes to host writes, always at least 1, often 2-10x) is the cost of this abstraction.

The write cliff is the moment the pipeline breaks down. Background GC cannot reclaim blocks fast enough to keep the pre-erased pool stocked, so the controller enters foreground GC. Host write commands queue behind synchronous erase cycles. The drive returns no errors. It simply takes orders of magnitude longer to acknowledge writes.

flowchart TD
    A["Normal: background GC replenishes free blocks"] -->|"sustained write pressure"| B["Free block pool depleting"]
    B -->|"pool exhausted"| C["Foreground GC: synchronous erase before write"]
    C --> D["Write latency spikes 100-1000x"]
    D --> E["Application timeouts, command stalls"]
    C -->|"write pressure drops"| F["GC catches up, pool refills"]
    E -->|"write pressure drops"| F
    F --> A

The write cliff is transient. Reduce the write rate or give the drive idle time, and performance recovers. This distinguishes it from endurance exhaustion, where Percentage Used is at or above 100% and the physical NAND is worn out permanently.

Common causes

Cause	What it looks like	First thing to check
Sustained writes without TRIM	Latency spikes under write load, improves when idle. Write amplification is high because the controller cannot distinguish stale from valid pages.	Run `fstrim -v /mountpoint` and check if latency improves
SSD nearly full (above 80%)	Progressive write degradation that worsens as the drive fills. Less free space means less room for GC to work efficiently.	`df -h` on the mounted filesystem
TRIM stripped by RAID controller	Write latency grows over time and never recovers on its own. Common behind LSI/Broadcom MegaRAID and similar controllers.	Check RAID controller documentation for TRIM passthrough support
High write amplification from random small writes	Percentage Used climbs faster than expected from host write volume. Latency spikes are intermittent.	Compare Data Units Written trend against application write volume
SMR HDD with CMR cache exhausted	Write speed collapses to single-digit MB/s on what appears to be an SSD-style write cliff, but the device is a hard drive.	Check drive model against manufacturer SMR/CMR datasheet
Firmware GC bug	Cumulative latency growth over weeks or months, even on low-write workloads. Not transient.	Check firmware version against vendor advisories

Quick checks

Run these commands to narrow the diagnosis. All are read-only except fstrim, which issues discard commands to the drive but does not modify user data.

# Check overall SMART health - should still say PASSED during a write cliff
smartctl -H /dev/sdX

# Check endurance metrics (NVMe)
smartctl -A /dev/nvme0n1 | grep -E "Percentage Used|Available Spare|Data Units Written"

# Check for command timeouts (ATA attribute ID 188, not universally implemented)
smartctl -A /dev/sdX | grep -i "Command_Timeout"

# Check write latency in real time (look at w_await and %util)
iostat -x 1

# Verify TRIM is supported and functional
lsblk -D
fstrim -v /mountpoint

# Check kernel logs for I/O timeouts or error recovery
dmesg | grep -iE "timeout|I/O error|reset" | tail -20

# Check SSD capacity utilization
df -h /mountpoint

# Check for media errors that would indicate real degradation, not GC
smartctl -A /dev/nvme0n1 | grep "Media and Data Integrity Errors"

# Check temperature (thermal throttling produces similar latency symptoms)
smartctl -A /dev/nvme0n1 | grep -i "Temperature"

How to diagnose it

Confirm the latency pattern. Use iostat -x 1 and look at w_await (write latency) and %util. The write cliff produces high write latency while read latency may remain normal. If read latency is also spiking, check Current Pending Sector count (ID 197) and Offline Uncorrectable (ID 198) instead. See the related guide on Current_Pending_Sector non-zero.
Verify SMART is clean. Run smartctl -A /dev/sdX (or /dev/nvme0n1). The write cliff should show clean media integrity: zero reallocated sectors, zero pending sectors, zero uncorrectable errors, zero media and data integrity errors (NVMe). If these are non-zero, you are looking at actual media degradation, not a write cliff.
Check endurance to rule out wear-out. On NVMe, confirm Percentage Used is below 100% and Available Spare is above threshold. If Percentage Used is at or above 100%, the problem is endurance exhaustion, not the write cliff. See NVMe Percentage Used at or above 100%.
Verify TRIM is working. Run fstrim -v /mountpoint. If it returns successfully and reports bytes trimmed, TRIM is functional. If it fails or reports zero bytes, the SSD is not receiving deallocation hints and GC efficiency is severely degraded. Behind RAID controllers, TRIM may be silently dropped.
Check drive fullness. If the filesystem is above 80% capacity, the SSD has less working room for garbage collection. This is the single most common amplifier of the write cliff in production.
Check for SMR if this is an HDD. SMR drives exhibit a similar performance collapse when their CMR cache fills. There is no standard SMART attribute to distinguish SMR from CMR. Check the manufacturer datasheet or use hdparm -I /dev/sdX and look for “Shingled Magnetic Recording” in the output.
Correlate latency with write rate. The write cliff is load-dependent. If latency spikes correlate with sustained write bursts and recover during idle periods, this confirms the diagnosis. If latency is high regardless of load, suspect a different problem.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Write latency (w_await in iostat)	Direct measure of the symptom. The write cliff produces write-specific latency spikes.	w_await jumping from sub-millisecond to tens or hundreds of milliseconds
NVMe Percentage Used	Distinguishes write cliff from endurance exhaustion. Write cliff occurs at any wear level.	Above 100% means endurance exhaustion, not GC pressure
NVMe Available Spare	Distinguishes write cliff from spare pool exhaustion.	Below threshold (typically 10%) means wear-out, not GC
Data Units Written / Total LBAs Written	Sustained high write rate is the root cause. Track the rate of change.	Write rate trending upward without corresponding workload change
Command Timeout (ID 188)	Some drives expose this. Foreground GC stalls produce timeouts.	Non-zero or increasing
Drive temperature	Thermal throttling produces similar latency symptoms but is caused by heat, not GC.	Temperature above WCTEMP (NVMe) or above 60C (HDD)
Media and Data Integrity Errors (NVMe)	Rules out actual NAND failure.	Any non-zero value indicates real media problems, not write cliff
Filesystem utilization	Above 80% full amplifies the write cliff.	Approaching or above 80% used

Fixes

Restore TRIM

The most common cause of chronic write cliff behavior is TRIM not reaching the drive. Without TRIM, the controller cannot distinguish pages that the filesystem has freed from pages that still hold valid data. Every garbage collection cycle must treat stale pages as valid, copying them unnecessarily and inflating write amplification.

Run fstrim -v /mountpoint to issue a one-time TRIM. Then verify that periodic TRIM is scheduled (typically via a systemd timer or cron job). Behind RAID controllers, check whether TRIM passthrough is supported and enabled. Some LSI/Broadcom MegaRAID firmware versions support TRIM passthrough but require explicit configuration.

Reduce drive fullness

Keep SSDs below 80% capacity. As the drive fills, the controller has fewer free blocks to work with, and GC efficiency drops sharply. If the filesystem is above 80%, freeing space or moving data to other volumes is the fastest path to recovery. Increasing over-provisioning (leaving unpartitioned space at the end of the drive) gives the controller more working room without affecting the filesystem.

Throttle the write workload

If the write rate is genuinely exceeding what the SSD can sustain, the long-term fix is to reduce write volume. Options include moving write-heavy workloads (database WAL, log ingestion, swap) to drives with higher sustained write ratings or more over-provisioning, tuning the application’s write batching behavior, or adding SSDs to spread the write load.

Let the drive recover

If the SSD has already hit the write cliff, reducing the write rate and leaving the drive powered on and idle allows background GC to catch up. The free block pool refills and latency returns to baseline. This is not a permanent fix. If the write rate that triggered the cliff resumes, the cliff will recur.

Address firmware bugs

Some SSD firmware versions have garbage collection bugs that cause cumulative latency growth independent of write pressure. If latency grows steadily over weeks or months on a low-write drive, check the vendor’s firmware advisories. Firmware updates typically require a drive power cycle and may briefly take the device offline. Plan accordingly.

Handle SMR drives

If the affected device is an SMR HDD, the write performance collapse is expected behavior once the CMR cache fills. There is no fix at the drive level. The operational response is to either avoid sustained write workloads (SMR drives are fine for archival reads) or replace the drive with a CMR model. SMR drives in ZFS resilver or RAID rebuild scenarios can turn an hours-long operation into one measured in days.

Prevention

Keep SSDs below 80% full. More free space means more working room for background GC.
Ensure TRIM reaches the drive. Verify TRIM works behind RAID controllers. Schedule periodic fstrim if the filesystem does not issue continuous discard.
Monitor write rate against rated endurance. Track Data Units Written or Total LBAs Written over time. A sudden increase in write rate without a corresponding workload change indicates a problem (misconfigured application, runaway logging).
Select the right drive for the workload. Enterprise SSDs with higher over-provisioning sustain higher write rates before hitting the cliff. Consumer SSDs in write-heavy enterprise workloads are a common mismatch.
Avoid SMR drives for write-intensive workloads. Check the datasheet before deployment. SMR drives are appropriate for sequential write-once, read-many workloads, not for databases, logs, or RAID/ZFS pools that require sustained random write performance.
Track firmware versions fleet-wide. Known GC bugs affect specific firmware versions. Monitoring firmware version lets you proactively identify and patch affected drives before latency starts growing.

How Netdata helps

Per-second disk latency metrics let you pinpoint the exact moment write latency spikes and correlate it with write throughput. The write cliff produces a distinctive pattern: write latency spikes while read latency stays flat, and the spike resolves when the write rate drops.
SMART attribute collection feeds Percentage Used, Available Spare, Data Units Written, and Media Errors into the same dashboard as disk latency. This makes it straightforward to distinguish the write cliff (clean SMART, high latency) from endurance exhaustion (Percentage Used above 100%) or media degradation (rising reallocated or pending sectors).
Temperature correlation helps separate thermal throttling from the write cliff. Both produce latency spikes, but only thermal throttling correlates with temperature approaching the vendor threshold.
Anomaly detection on write latency and write throughput can surface the write cliff before it triggers application timeouts. A sudden latency spike that deviates from the established baseline pattern is the earliest signal.
Historical correlation across the write cliff episode and recovery shows whether the condition is transient (GC catching up) or recurring (chronic workload mismatch), which drives the decision between a tactical fix (TRIM, free space) and a strategic fix (drive replacement, workload redistribution).