NVMe thermal throttling: the drive runs hot and performance quietly drops

The database is slow. CPU utilization is low, the query plan has not changed, and there are no I/O errors in dmesg, no media errors in SMART, no failed disk anywhere. Yet p99 query latency has doubled and IOPS are half of what they were last week.

This is the classic NVMe thermal throttling incident. The controller detected that it was running too hot and quietly reduced its own performance to protect itself. There is no error, no log entry, no kernel message. The drive simply gets slower until it reaches thermal equilibrium, and it gets faster again when the load drops. If you are only watching error counters and application logs, you will never see it.

An M.2 drive without a heatsink can hit 80C under sustained load, throttling database performance while the CPU is nearly idle. This is a common monitoring blind spot because teams watch CPU temperature but not NVMe temperature.

What this means

Every NVMe controller has vendor-defined temperature thresholds. The two that matter are:

• WCTEMP (Warning Composite Temperature Threshold): at or above this temperature, the controller throttles. Performance degrades, but the drive is self-protecting. Typical values are around 70-80C on consumer drives and 75-85C on enterprise drives.
• CCTEMP (Critical Composite Temperature Threshold): the emergency threshold, typically around 85C on consumer drives and 90-100C on enterprise drives. Sustained operation here risks controller shutdown.

Below WCTEMP, some controllers also implement lighter thermal management states (TMT1, a light throttle, and TMT2, a heavy throttle). WCTEMP and CCTEMP live in the Identify Controller data structure, not the SMART log, which is why most monitoring tools never show them and why teams do not know their own thresholds.

The throttle works by reducing internal clock speeds and NAND interface rates. From the host’s perspective: throughput declining, latency rising, queue depth growing because I/Os are backing up. The correlation between rising temperature and falling throughput is the signature. Garbage collection stalls and firmware problems cause similar performance drops but with no temperature correlation, which is exactly how you tell them apart.

Under a latency-sensitive workload there is a feedback loop worth understanding: throttling slows the drive, requests queue up, the queue keeps the drive busy, and the drive stays hot. It may stabilize at permanently reduced performance until you reduce the load.

flowchart TD
  A[Sustained I/O load] --> B[Composite temperature rises]
  B --> C{Temperature >= WCTEMP?}
  C -- no --> A
  C -- yes --> D[Controller throttles: clock and NAND rate reduced]
  D --> E[Throughput drops, latency and queue depth rise]
  E --> F{Load reduced or cooling fixed?}
  F -- yes --> G[Temperature falls, performance recovers]
  F -- no --> H{Temperature >= CCTEMP?}
  H -- no --> E
  H -- yes --> I[critical_warning bit 1 sustained, critical_comp_time increasing: escalate]

One more subtlety: the composite temperature reported in SMART is the controller’s own self-report, typically weighted across its sensors. It may not be the hottest NAND die. Individual temperature sensors are exposed separately via hwmon (temp2, temp3), and they can run meaningfully hotter than the composite value.

Common causes

Quick checks

All of these are safe and read-only.

# Current composite temperature (Kelvin in raw output; subtract 273.15 for Celsius)
nvme smart-log /dev/nvme0 | grep "^temperature"

# The drive's actual throttle thresholds (Kelvin; subtract 273.15 for Celsius)
nvme id-ctrl /dev/nvme0 | grep -i "wctemp\|cctemp"

# Per-sensor temperatures via hwmon (millidegrees Celsius; divide by 1000)
cat /sys/class/nvme/nvme0/hwmon*/temp1_input
cat /sys/class/nvme/nvme0/hwmon*/temp2_input
cat /sys/class/nvme/nvme0/hwmon*/temp3_input

# Cumulative time above WCTEMP and CCTEMP (minutes)
nvme smart-log /dev/nvme0 | grep -i "warning_temp_time\|critical_comp_time"

# Thermal management transition counts and total time
nvme smart-log /dev/nvme0 | grep -i "thm_temp"

# Is the drive currently flagging a temperature threshold? (bit 1 = 0x02)
nvme smart-log /dev/nvme0 | grep "critical_warning"

A few reading notes:

• If warning_temp_time or critical_comp_time is increasing between two samples taken minutes apart, the drive is spending time above threshold right now, not just historically.
• Some drives do not implement the time-above-threshold counters or the TMT counters; they read zero even when throttling. A zero does not prove the absence of throttling. Check current temperature against WCTEMP instead.
• If temp2_input or temp3_input does not exist, the drive exposes only the composite sensor.

How to diagnose it

1. Confirm the performance drop has no error signature. Check dmesg for NVMe errors or resets and check nvme smart-log /dev/nvme0 for media_errors growth. Thermal throttling presents with clean error counters. If you find PCIe AER errors or link retraining instead, you have a transport problem, not a thermal one.

2. Read the current composite temperature and the drive’s own thresholds. Run nvme smart-log for temperature and nvme id-ctrl for wctemp/cctemp. Convert Kelvin to Celsius. If the current temperature is at or above WCTEMP, throttling is either active or imminent. Do not assume generic thresholds; they vary widely by vendor and drive class.

3. Check the per-sensor temperatures via hwmon. If temp2 or temp3 runs well above the composite value, part of the drive (often a NAND die) is hotter than the headline number suggests. The mapping of sensor numbers to physical components is vendor-specific.

4. Establish that temperature and performance move together. Take a few samples of temperature, throughput, and latency during the slow window. Thermal throttling shows a tight correlation: temperature up, throughput down, latency up. A throughput drop with flat temperature points at garbage collection pressure or SLC cache exhaustion instead.

5. Check the cumulative counters for history. warning_temp_time, critical_comp_time, and the TMT transition counts tell you whether this is a first event or a chronic condition. A steadily growing warning_temp_time means the drive has been throttling repeatedly, possibly for weeks, even if you only noticed today.

6. Identify the heat source. If accessible, inspect the drive: heatsink present and seated, airflow path clear, no hot neighbor. Correlate the throttling windows with workload schedules. A drive that only throttles during the nightly backup has a workload-thermal mismatch; a drive that throttles under ordinary load has a cooling problem.

Metrics and signals to monitor

The single most useful habit: alert on warning_temp_time rate of change, not just on live temperature. The counter catches thermal stress events that happened between your samples or overnight when nobody was watching.

Fixes

Improve cooling (the actual fix for a cooling problem)

• Add or reseat a heatsink on M.2 drives. M.2 slots provide essentially no airflow. A bare M.2 drive under sustained load is the worst offender and can hit 80C; with a heatsink the same load may never reach the throttle threshold. Verify thermal pad contact.
• Restore airflow. Clear blocked intake paths, replace failed fans, check that nothing is exhausting hot air directly onto the drive. Drives mounted next to high-heat components like GPUs inherit that heat.
• Fix ambient temperature if the trend is environmental rather than local.

Reduce the thermal load (the fix for a workload-thermal mismatch)

• Reduce write intensity during thermal events. Writes generate more heat than reads. Shedding or deferring the heavy write job lets the drive cool and restores performance. This is also the correct first response during an active event.
• Move sustained heavy jobs off the hot drive or reschedule them to cooler windows, if the drive simply cannot sustain the required throughput within its thermal envelope.
• Right-size the hardware. If a workload legitimately needs sustained throughput that the drive cannot deliver without throttling, the fix is a drive with a higher thermal envelope (enterprise form factors with proper cooling), not more tuning.

Tradeoff to be aware of: reducing load works immediately but only treats the symptom. If the root cause is a missing heatsink or failed fan, the problem returns with the next heavy job.

Prevention

• Monitor NVMe temperature alongside CPU temperature. This is the mistake that makes this incident class invisible. Composite temperature plus the warning_temp_time counter should be baseline telemetry on every host with local NVMe.
• Know your thresholds at provisioning time. Record WCTEMP and CCTEMP from nvme id-ctrl for each drive model in the fleet. Alert at roughly 10C below WCTEMP under peak load so you get warning before throttling starts.
• Heatsink every M.2 drive that will see sustained load. Treat it as part of the build, not an upgrade.
• Trend temperature over weeks. A slow upward drift usually means degrading cooling (dust, fan wear) long before the first throttle event.
• Watch the TMT and time-above-threshold counters on drives that support them; they are the honest record of how much of the drive’s life has been spent throttled.

How Netdata helps

• Netdata collects composite temperature per device (nvme.device_composite_temperature) so the temperature trend sits on the same timeline as the block device throughput and latency charts, making the temperature-to-performance correlation visible in one view instead of two tools.
• The warning and critical composite temperature time charts (nvme.device_warning_composite_temperature_time, nvme.device_critical_composite_temperature_time) surface chronic thermal stress history, including events that happened while nobody was looking at a gauge.
• Thermal management transition and time charts (nvme.device_thermal_mgmt_temp1_transitions_rate, nvme.device_thermal_mgmt_temp2_transitions_rate, and the corresponding time charts) show when the drive is actively entering throttling states.
• The per-bit critical warning chart (nvme.device_critical_warnings_state) isolates the temperature bit from the other critical warning bits, so a transient thermal trip is not confused with spare exhaustion or read-only mode.
• Because temperature is a cumulative-counter and poll-based signal, per-second collection of the block device side (IOPS, throughput) fills in the live symptom picture between SMART updates.

Related guides

• NVMe critical_warning is nonzero: decoding the SMART critical warning bitmask
• NVMe monitoring checklist: the signals every production SSD needs
• How NVMe actually works in production: a mental model for operators
• NVMe available spare below threshold: critical warning bit 0 and end-of-life wear
• NVMe available spare declining: watching the wear trajectory before the threshold
• NVMe endurance runway: projecting time-to-replacement from wear signals
• NVMe media_errors increasing: uncorrectable data-integrity errors on NAND
• blk_update_request: I/O error, dev nvme0n1: reading NVMe I/O errors in the kernel log
• NVMe controller reset loop: repeated resets from a firmware hang
• nvme nvme0: I/O timeout, Resetting controller: what an NVMe controller reset means
• NVMe controller state not live: reading resetting, deleting, and dead from sysfs
• NVMe device disappeared: nvme0: Removing and a drive that fell off the PCIe bus