NVMe throughput collapse: busy controller, low IOPS, and internal contention

The drive is flat out. Whatever metric you look at says the controller is working constantly. And yet the application is starving: IOPS are a fraction of what the drive is rated for, write throughput has fallen off a cliff, and latency is climbing. Nothing in the kernel log looks broken. No media errors. No resets. The drive is busy, and it is delivering almost nothing.

The standard instinct (“the disk is saturated”) is both right and wrong here. The controller is saturated. The host-visible I/O path is not. The drive is burning its internal capacity on work you cannot see: garbage collection, FTL remapping, wear leveling, or simply running its clocks slower because it is too hot.

The diagnostic pivot is a single comparison: controller busy time against host-delivered IOPS. A drive that is 100% busy and delivering its rated IOPS is healthy and saturated. A drive that is 100% busy and delivering far less has an internal problem. This guide is about telling those two apart and finding which internal problem you have.

What this means

Every NVMe controller is a small embedded computer running a flash translation layer (FTL). The FTL maps logical block addresses to physical NAND pages, and it does housekeeping the host never sees: garbage collection (NAND must erase whole blocks before rewriting, so valid pages get moved out of partially used blocks), wear leveling, and read-disturb relocation. That housekeeping consumes the same controller resources as host I/O.

The SMART log exposes exactly one window into this: controller_busy_time, a cumulative counter in minutes. The controller counts itself as busy whenever it has at least one outstanding command. Sampled over time, the delta tells you the fraction of wall time the controller considered itself occupied.

The failure mode this article covers has a precise signature:

• Busy time ratio near 100%: controller_busy_time advances roughly one minute per real minute.
• Host throughput far below expectation: IOPS and bandwidth from block-layer counters are well under the drive’s capability for the workload.
• No error signals: media_errors steady, no critical warnings, no resets in dmesg.

That combination means the controller’s effort is not reaching the host. The usual suspects, in rough order of frequency: a write amplification or garbage collection storm (especially on a nearly full drive), SLC cache exhaustion on consumer-class drives, or silent thermal throttling. A degraded PCIe link is a fourth possibility that produces a similar “healthy but slow” profile without the busy-time signature.

flowchart TD
  A[Busy time ratio near 100 percent] --> B{Host IOPS at expected level?}
  B -->|yes| C[Healthy saturation: capacity limit]
  B -->|no| D{Composite temp near or above WCTEMP?}
  D -->|yes| E[Thermal throttling]
  D -->|no| F{Drive more than 80 percent full?}
  F -->|yes| G[GC storm / write amplification]
  F -->|no| H{Sustained sequential writes?}
  H -->|yes| I[SLC cache exhaustion]
  H -->|no| J[Check PCIe link and error log]

Common causes

Quick checks

All of these are read-only and safe to run during an incident.

# Host-side IOPS, bandwidth, and per-request latency (two 1-second samples)
iostat -xp nvme0n1 1 2

# Controller busy time: sample twice, 5 minutes apart
nvme smart-log /dev/nvme0 | grep controller_busy_time
sleep 300
nvme smart-log /dev/nvme0 | grep controller_busy_time

# Temperature and thermal management counters
nvme smart-log /dev/nvme0 | grep -i "temperature\|Thermal Management"

# Vendor thermal thresholds (not in the SMART log)
nvme id-ctrl /dev/nvme0 | grep -i temp

# Wear and error state
nvme smart-log /dev/nvme0 | grep -i "percentage_used\|available_spare\|media_errors\|num_err_log_entries"

# PCIe link: current vs maximum negotiated
cat /sys/class/nvme/nvme0/device/current_link_speed
cat /sys/class/nvme/nvme0/device/max_link_speed
cat /sys/class/nvme/nvme0/device/current_link_width
cat /sys/class/nvme/nvme0/device/max_link_width

# Filesystem fill level and discard configuration
df -h
findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS | grep nvme
cat /sys/block/nvme0n1/queue/scheduler

# Rule out a controller hang masquerading as slowness
dmesg | grep -i nvme | grep -iE "timeout|reset" | tail -20

Two reading notes on the output:

Do not trust iostat %util on NVMe. The kernel computes %util as “any I/O in flight at any moment,” a metric designed for single-queue spinning disks. A device that processes thousands of requests in parallel across hardware queues can sit at 100% %util while operating at a fraction of its real capacity. On NVMe, read r_await, w_await, and aqu-sz instead: if read latency stays around or under 1 ms and the average queue is comfortably below what the device can handle, the drive is not host-side saturated regardless of what %util says. The svctm column is deprecated in current sysstat versions; ignore it.

controller_busy_time has minute resolution. It cannot see sub-minute stalls. What it can see is exactly the condition this article is about: a controller that spends effectively all of its time occupied while the host gets little out of it.

How to diagnose it

1. Establish the busy ratio. Take two controller_busy_time samples a few minutes apart. If the delta roughly equals the wall-clock interval, the controller was busy 100% of the time. A busy ratio above 90% during a period of low host IOPS means the drive is sick, not loaded. You can also compute the lifetime ratio controller_busy_time / (power_on_hours * 60), but the sampled delta is what reflects the current incident.

2. Quantify what the host is actually getting. From the same window, compute IOPS and bandwidth from /sys/block/nvme0n1/stat deltas (or iostat -x). Compare against the drive’s datasheet rating for your I/O pattern, and against your own baseline. A drive rated for hundreds of thousands of random read IOPS that is delivering a few thousand while fully busy has internal contention.

3. Check temperature against vendor thresholds. Get composite temperature from the SMART log and WCTEMP/CCTEMP from nvme id-ctrl (they are in Identify Controller data, not the SMART log, so many monitoring tools never show them). If temperature is at or above WCTEMP, check thm_temp1_trans_count, thm_temp2_trans_count, and warning_temp_time: increasing values confirm the drive has been entering thermal management states. Thermal throttling is gradual and produces no errors, which is why it is so often misdiagnosed as an application problem.

4. If temperature is normal, check space pressure. GC competes directly with host I/O when free blocks run low. Check filesystem fill level and, critically, whether the filesystem is telling the drive about deleted blocks: no discard mount option and no fstrim schedule means the FTL treats the drive as fuller than it logically is, and GC pressure becomes chronic. Drives above roughly 80-90% fill enter the zone where this dominates. The classic pattern: throughput down 50-90%, latency spikes, controller_busy_time pinned, temperature normal, media_errors flat.

5. Match the write pattern to SLC cache behavior. If the collapse happens specifically during sustained sequential writes and the throughput drop is a step function rather than a slope, you are watching SLC cache exhaustion. This is by design, not a fault: the drive absorbs bursts in fast pseudo-SLC and destages to TLC/QLC during idle. The cliff arrives sooner as the drive fills, because the dynamic cache shrinks. Recovery after a few idle minutes confirms it.

6. Check wear state. A heavily worn drive spends more controller effort on error recovery and remapping. If percentage_used is high and available_spare is trending down, internal overhead rising with age is a plausible contributor, and the drive belongs on a replacement plan regardless of what else you find.

7. Eliminate lookalikes. A degraded PCIe link (current_link_speed or current_link_width below max) caps bandwidth with zero errors, but does not produce the busy-time anomaly. Repeated controller resets in dmesg are a different failure (a firmware hang, not contention); see the reset guide linked below. Also confirm the I/O scheduler is none: running mq-deadline or bfq on NVMe adds host-side queuing that muddies the picture.

One thing you cannot get from standard SMART: true write amplification factor. data_units_written counts host-visible writes only, not NAND writes. Real WAF requires vendor-specific telemetry (vendor log pages, exposed on some drives through nvme smart-log-add, with vendor-specific units), and not all drives expose it. Do not try to derive WAF from standard SMART counters; you will underestimate NAND wear.

Metrics and signals to monitor

Fixes

GC storm and space pressure

• Run fstrim on the mounted filesystems for immediate relief, then make sure deletion information reaches the drive continuously: a discard mount option or a scheduled fstrim job. Without this, the problem returns.
• Reduce fill level below 80%. This is not wasted capacity; it is FTL working room. If the workload genuinely needs the capacity, the drive is undersized.
• Increasing overprovisioning (leaving unpartitioned space) enlarges the pool the FTL has for GC and wear leveling, and improves write consistency.

SLC cache exhaustion

• This is hardware behavior, not a defect. The question is whether your workload requires sustained write throughput above native TLC/QLC speed.
• If it does, options are workload-side (smooth the write burst), provisioning-side (more free space grows the dynamic cache), or hardware-side (an enterprise drive with sustained-write ratings instead of a consumer drive relying on burst cache).

Thermal throttling

• Reduce write load to let the drive cool, then fix the thermal path: airflow, heatsink contact, adjacent heat sources like GPUs. M.2 drives with no heatsink throttle aggressively under sustained load.
• Watch for hysteresis: controllers often do not leave the throttled state until temperature drops well below the threshold, so throughput may stay depressed for a while after load is removed.

Worn drive

• No fix restores FTL headroom on a worn-out drive. Verify replication or backups, and schedule replacement. The endurance and spare guides below cover runway estimation.

PCIe link degradation

• Reseat the drive and power cycle to force renegotiation. If it persists, try another slot to isolate drive from slot. Check AER counters for a noisy physical layer.

Prevention

• Alert on the divergence, not the components. The actionable condition is busy ratio high AND host IOPS low. Either half alone is ambiguous.
• Baseline per drive model. “Low IOPS” is meaningless without the drive’s rated numbers for your I/O pattern and your own historical baseline at similar queue depths.
• Keep drives under 80% full and TRIM working. This removes the most common cause entirely.
• Track thermal history, not just live temperature. warning_temp_time and TMT counters reveal throttling that happened between your samples.
• Match drive class to workload. Consumer drives without power-loss protection, with small overprovisioning and aggressive cache cliffs, need different expectations and thresholds than enterprise drives.

How Netdata helps

• Netdata’s NVMe collector gathers controller_busy_time from the SMART log, the field at the center of this diagnosis. Note that it is collected but currently has no dedicated chart, so for the busy-ratio calculation use nvme smart-log samples as shown above.
• The nvme.device_composite_temperature chart plus thermal management transition and time charts (nvme.device_thermal_mgmt_temp*_transitions_rate, nvme.device_thermal_mgmt_temp*_time) let you confirm or eliminate throttling without being on the box during the event.
• Per-bit critical warning charts (nvme.device_critical_warnings_state) catch the thermal threshold bit and reliability degradation bit that sometimes accompany a collapse.
• nvme.device_estimated_endurance_perc and nvme.device_available_spare_perc trends tell you whether wear-driven FTL overhead is a contributing factor.
• Netdata’s block-device IOPS, bandwidth, and latency charts give the host-side half of the correlation: overlaying delivered throughput against temperature and busy state is what separates “healthy saturation” from “sick drive” in minutes instead of hours.

Related guides

• NVMe high I/O latency: reading block-layer latency and the outliers that matter
• NVMe endurance runway: projecting time-to-replacement from wear signals
• NVMe available spare declining: watching the wear trajectory before the threshold
• NVMe critical_warning is nonzero: decoding the SMART critical warning bitmask
• NVMe controller reset loop: repeated resets from a firmware hang
• How NVMe actually works in production: a mental model for operators