NVMe write amplification: why data_units_written understates real NAND wear

The usual incident goes like this: percentage_used on a drive is climbing faster than planned, someone pulls data_units_written from the SMART log, divides by power-on hours, and concludes the workload is well within the drive’s DWPD rating. Six months later the drive sets critical warning bit 0 and procurement is scrambling. The math was not wrong. The input was.

data_units_written counts what the host sent to the controller. It does not count what the controller wrote to the NAND. Between those two numbers sits the flash translation layer: garbage collection relocating valid pages, wear leveling shuffling cold blocks, metadata updates, SLC cache destaging. Every one of those operations programs NAND without appearing in a host-visible counter.

This article covers what the counter actually measures, why true write amplification factor (WAF) is not computable from standard SMART, where real NAND write telemetry lives, and what you can do with the signals you have to catch a runaway writer before it eats the drive’s endurance budget.

What this means

The NVMe spec defines data_units_written as the number of 512-byte data units the host has written to the controller, reported in thousands. One unit is 1000 x 512 = 512,000 bytes:

bytes_written_by_host = data_units_written * 512 * 1000

Netdata exposes this via the nvme.device_io_transferred_count chart (dimensions read and written, converted to bytes). It is a lifetime counter, and it is strictly host-visible volume.

WAF is defined as:

WAF = NAND writes / host writes

The numerator does not exist in the standard SMART / Health Information log (Log ID 0x02). True NAND write volume only appears in vendor-specific log pages, or in the OCP Datacenter NVMe SSD log page on drives that implement the OCP extension. Any WAF you compute from data_units_written alone silently assumes WAF = 1. On a random-write-heavy workload against a nearly-full drive, real WAF can be several times higher.

flowchart LR
  H[Host writes] --> C[NVMe controller]
  C --> DU["data_units_written (SMART, host-visible)"]
  C --> FTL[Flash translation layer]
  FTL --> GC[Garbage collection relocation]
  FTL --> WL[Wear leveling moves]
  FTL --> MD[FTL metadata updates]
  GC --> NAND[NAND program operations]
  WL --> NAND
  MD --> NAND
  NAND --> VP["Vendor log pages 0xC0-0xFF only"]

percentage_used is a vendor estimate of endurance consumed that does factor in actual NAND wear, including write amplification. That is why percentage_used can outrun the projection you made from data_units_written. The SMART log is not lying to you. You are comparing a host-side counter against a NAND-side estimate.

Common causes of elevated real WAF and fast endurance burn

Quick checks

All read-only.

# Current SMART counters: host write volume and endurance estimate
nvme smart-log /dev/nvme0 | grep -E "data_units_written|percentage_used|available_spare|power_on_hours|host_write_commands"

# Convert to host TB written
# data_units_written * 512000 / 1e12

# Is swap on NVMe?
cat /proc/swaps

# How full is the filesystem?
df -h /dev/nvme0n1*

# Is discard enabled or is fstrim running periodically?
findmnt -o TARGET,OPTIONS /your/mount | grep discard
systemctl list-timers | grep fstrim

# Average host I/O size: (data_units_written * 512000) / host_write_commands
# A very small average (a few KB) on a write-heavy drive is a WAF risk factor.

If your drive supports it, pull the real NAND write counter:

# OCP-compliant drives: physical media units written (log page 0xC0)
sudo nvme ocp smart-add-log /dev/nvme0n1

# Solidigm/Intel drives: nand_bytes_written and host_bytes_written (32MiB units)
sudo nvme solidigm smart-log-add /dev/nvme0n1

Availability depends on the drive vendor and your nvme-cli build; older distro packages may lack the OCP plugin, and some vendors expose this through their own log-page plugin or tooling instead. Units differ per vendor, so check the vendor’s documentation before doing arithmetic. These are read-only log page fetches.

How to diagnose it

1. Establish the host write rate. Sample data_units_written twice over a known window (an hour, a day) and compute bytes per day. This is your host-side floor.
2. Compute actual DWPD. (data_units_written_delta * 512000) / (drive_capacity_bytes * days). Compare against the drive’s rated DWPD. If you are above rating, the endurance budget drains faster than the warranty math assumes regardless of WAF.
3. Measure the endurance burn rate. Track percentage_used over weeks. Rule of thumb: more than 1% per week suggests write amplification problems or a workload/drive-class mismatch. If the endurance rate implies more wear than your host DWPD explains, the gap is real WAF.
4. Find the writer. If the host write rate itself is higher than expected, correlate with per-process I/O (iotop, /proc/<pid>/io) and with application behavior: swap activity, log volume, database checkpoint or compaction storms, ZFS small-recordsize writes.
5. Check fill level and TRIM. A drive past 80-90% full forces the GC to work harder per host write, which raises WAF. If discard is not configured, run fstrim on the mount and set up the periodic timer; without TRIM the FTL cannot know which blocks are free and WAF stays elevated at all fill levels.
6. Get the true number if you can. On OCP or vendor-supported drives, pull the NAND write counter from the vendor log page and compute actual WAF = NAND writes / host writes. Do this once per drive model and workload class; you only need the ratio, not continuous collection.

Metrics and signals to monitor

Fixes

Runaway host writers

• Swap on NVMe: move swap off the drive or reduce swappiness if the workload does not need it. Swap on a small-endurance drive is a classic silent endurance drain.
• Verbose logging: cut log level, add rate limiting, or ship logs off-box. This is host-visible volume, so it shows up directly in data_units_written growth.
• Small synchronous writes: batch at the application layer where possible. Review filesystem journal configuration (for example, avoid full data journaling unless you need it) and check ZFS recordsize/ZIL behavior on small-write workloads. Do not disable filesystem barriers or flush semantics to save wear unless you fully accept the data-loss risk on power failure.

Drive-side amplification

• Free up space and TRIM. Keep logical utilization below roughly 80%. Enable continuous discard or a periodic fstrim timer. This directly reduces GC pressure and therefore WAF.
• Overprovision. Leaving unpartitioned space gives the FTL more free blocks to work with, which lowers write amplification and stabilizes latency. The tradeoff is usable capacity.
• Right-size the drive class. If your measured workload exceeds the drive’s rated DWPD even at WAF = 1, no tuning fixes that. You need a higher-endurance drive, and the procurement conversation is easier at 30% percentage_used than at 95%.

Prevention

• Baseline at provisioning. Record DWPD rating, LBA format, and whether the drive exposes OCP or vendor NAND-write telemetry. Measure actual WAF once per drive model under your real workload so you have a defensible multiplier for projections.
• Trend, do not snapshot. Track the weekly rate of percentage_used and available_spare, not just current values. Runway estimate: (100 - percentage_used) / daily percentage increase gives days to rated end of life; apply a safety factor in the final 10%.
• Alert on the divergence. When percentage_used grows materially faster than host-write volume projects, that gap is your write amplification alarm. You do not need the true NAND counter to detect the problem, only to quantify it.
• Keep headroom. Maintain at least 15-20% free space for the FTL and keep discard configured. High fill level is the single most common amplifier of WAF in production.

How Netdata helps

• Host write volume: the nvme.device_io_transferred_count chart converts data_units_read and data_units_written to bytes, so you can trend host-side write rate directly and spot a runaway writer as a rate anomaly.
• Endurance burn: nvme.device_estimated_endurance_perc tracks percentage_used, the vendor estimate that already includes real write amplification. Comparing its slope against the host write rate exposes the WAF gap without vendor tooling.
• Spare runway: nvme.device_available_spare_perc plus its consumption rate gives the non-linear, late-stage signal that SMART-only WAF math misses.
• Wear confirmation: nvme.device_media_errors_rate turns the endurance question into a hard-failure question when wear starts producing uncorrectable errors.
• Internal overhead: correlating write throughput against controller behavior helps distinguish “the host is writing a lot” from “the drive is doing a lot of internal work per host write,” which is the operational signature of GC-driven amplification.

Related guides

• NVMe available spare below threshold: critical warning bit 0 and end-of-life wear
• blk_update_request: I/O error, dev nvme0n1: reading NVMe I/O errors in the kernel log
• NVMe controller reset loop: repeated resets from a firmware hang
• nvme nvme0: I/O timeout, Resetting controller: what an NVMe controller reset means
• NVMe controller state not live: reading resetting, deleting, and dead from sysfs
• NVMe critical_warning is nonzero: decoding the SMART critical warning bitmask
• NVMe device disappeared: nvme0: Removing and a drive that fell off the PCIe bus
• How NVMe actually works in production: a mental model for operators
• NVMe monitoring checklist: the signals every production SSD needs
• NVMe monitoring maturity model: from survival to expert
• NVMe NVM subsystem reliability degraded: critical warning bit 2
• NVMe drive in read-only mode: critical warning bit 3 and rejected writes