$ guides / smartctl-disk-monitoring / smartctl-nvme-thermal-throttling ▌

Operations Guides

NVMe thermal throttling: throughput dropping with no errors

NVMe throughput drops during sustained I/O. No I/O errors in dmesg, no command timeouts, SMART health says PASSED, Media and Data Integrity Errors at zero. The workload has not changed, but latency is up and throughput is down by 30, 50, or more percent. Hours later, performance recovers on its own. The pattern repeats: degradation during peak load, recovery during quiet periods or overnight.

This is NVMe thermal throttling. The controller reduces performance to stay below vendor-defined temperature thresholds. Nothing in the standard error path surfaces it. The drive does not log I/O errors, the kernel does not see transport errors, and the SMART health assessment stays PASSED because the drive is operating within its protective envelope.

The evidence is in the SMART/Health Information log, specifically the time-above-threshold counters that most monitoring setups never collect. This article covers how to read those counters, confirm the diagnosis, and distinguish thermal throttling from look-alike conditions.

What this means

NVMe controllers define two temperature thresholds in firmware:

WCTEMP (Warning Composite Temperature Threshold): typically 70-80C, vendor-defined. At WCTEMP, the controller may begin light throttling.
CCTEMP (Critical Composite Temperature Threshold): typically 80-85C, vendor-defined. At or above CCTEMP, the controller applies heavy throttling or shutdown.

When the composite temperature crosses WCTEMP, the controller starts reducing I/O throughput. The throttling mechanism is vendor-specific, but the observable effect is consistent: I/O latency increases and throughput decreases. The drive does not flag this as an error because it is a designed protection mechanism, not a fault.

The SMART/Health Information log (NVMe Log Page 02h) tracks time spent above these thresholds:

Warning Comp. Temperature Time: cumulative minutes the composite temperature was at or above WCTEMP but below CCTEMP. Cleared to zero if WCTEMP is not defined.
Critical Comp. Temperature Time: cumulative minutes the composite temperature was at or above CCTEMP. Cleared to zero if CCTEMP is not defined.

Non-zero values confirm throttling has occurred, even when current temperature is normal. These counters are cumulative.

The health log also tracks Host Controlled Thermal Management (HCTM) transitions. HCTM lets the host OS set two additional throttle thresholds (TMT1 and TMT2) via NVMe Set Features (Feature Identifier 0x10). When the composite temperature crosses a host-set TMT, the controller throttles and increments the corresponding transition count:

Thermal Management Temperature 1 Transition Count: increments each time TMT1 throttling engages.
Thermal Management Temperature 2 Transition Count: increments for TMT2 (heavier) throttling.
Total Time For Thermal Management Temperature 1/2: seconds spent in each throttled state.

flowchart TD
    A[Sustained I/O workload] --> B[Composite temperature rises]
    B --> C{Crosses WCTEMP?}
    C -->|No| A
    C -->|Yes| D[Controller throttles: throughput drops]
    D --> E[Temperature stabilizes or falls]
    E --> F{Below threshold?}
    F -->|No| D
    F -->|Yes| G[Throttle releases: throughput recovers]
    G --> A
    D --> H[Warning Comp Temp Time increments]
    B --> I{Crosses CCTEMP?}
    I -->|Yes| J[Heavy throttle or shutdown]
    J --> K[Critical Comp Temp Time increments]

A key subtlety: the composite temperature is a vendor-computed value, not a direct physical sensor reading. The NVMe specification states that the manner in which this value is computed is implementation-specific and may not represent the actual temperature at any physical point in the subsystem. The drive’s internal sensor may lag behind actual component temperature. By the time SMART shows 70C, the junction temperature may already be at the throttle point.

Common causes

Cause	What it looks like	First thing to check
Detached or missing M.2 heatsink	Single drive overheats, neighbors are fine. Common after server relocation or chassis work.	Physical inspection of heatsink mounting. Warning Comp. Temp. Time non-zero.
Inadequate chassis airflow	Multiple drives in the same area run hot. Throughput drops correlate across drives.	Compare temperatures across all NVMe drives. Check fan status and airflow path.
Adjacent GPU or CPU heat	One drive near a heat-generating component runs consistently hotter. Temperature tracks compute load, not disk I/O.	Check physical layout. Correlate temperature with GPU or CPU load.
Datacenter or room cooling failure	All drives in a rack or row show rising temperatures simultaneously.	Check ambient temperature at the rack intake.
Blocked vents or cable obstruction	One chassis or bay runs hotter than identical configurations.	Physical inspection of airflow path in the chassis.

Quick checks

All commands below are read-only and safe to run on production drives.

# Current temperature and all SMART/Health fields
smartctl -a /dev/nvme0n1

# Focus on temperature and time-above-threshold counters
smartctl -A /dev/nvme0n1 | grep -iE "temperature|warning|critical|thermal"

# Controller thresholds (WCTEMP/CCTEMP) from Identify Controller data
smartctl -c /dev/nvme0n1

# nvme-cli alternatives (if smartmontools is not installed)
nvme smart-log /dev/nvme0n1
nvme id-ctrl /dev/nvme0n1 | grep -iE "wctemp|cctemp"

# Real-time I/O throughput and latency to correlate with temperature
iostat -x 1 /dev/nvme0n1

# Kernel thermal or NVMe events (most drives do not log thermal AENs to dmesg,
# so absence of output does not rule out throttling)
dmesg -T | grep -iE "nvme|thermal|throttl" | tail -30

# Chassis ambient temperature (if lm-sensors is configured)
sensors

The most important single output is from smartctl -A. Look for these fields:

Temperature: the current composite temperature.
Warning Comp. Temperature Time: if non-zero, the drive has spent time above WCTEMP.
Critical Comp. Temperature Time: if non-zero, the drive has spent time above CCTEMP.
Available Spare: confirm this is not also declining, which would indicate wear-related degradation compounding the thermal issue.

How to diagnose it

Capture current temperature. Run smartctl -A /dev/nvme0n1 and note the composite temperature. Also note Temperature Sensor 1 through 8 if the drive exposes multiple sensors. Compare against WCTEMP from smartctl -c.
Check time-above-threshold counters. Non-zero Warning Comp. Temperature Time confirms the drive has been throttling. This is your primary evidence. The current temperature may be normal if the drive has already throttled and cooled.
Correlate with host-side I/O metrics. Run iostat -x 1 during a throughput degradation event. Look for increased await (latency) and decreased throughput (rkB/s and wkB/s) on the suspect drive. The degradation should correlate with temperature, not with a change in queue depth or workload pattern.
Rule out pSLC cache exhaustion. Consumer and prosumer NVMe drives use a pseudo-SLC write cache. Under sustained writes, this cache fills and write speed drops from several GB/s to hundreds of MB/s. This looks identical to thermal throttling from the operator’s perspective. Distinguishing heuristic: if the slowdown tracks cumulative write volume since the last TRIM, suspect pSLC exhaustion. If it tracks temperature, suspect thermal throttling.
Check Critical Warning bit 1. Run smartctl -A /dev/nvme0n1 | grep "Critical Warning". Bit 1 (0x02) indicates a temperature is currently above a threshold (WCTEMP or CCTEMP, depending on drive implementation). This bit reflects current state, not history. A drive that throttled hours ago shows 0x00 if it has since cooled.

Compare across drives in the same chassis. If multiple NVMe drives show elevated temperatures simultaneously, the root cause is environmental (airflow, ambient, fan failure), not drive-specific. If only one drive is hot, suspect a physical issue such as heatsink detachment or PCIe slot placement near a heat source.
Establish the throttle cycle pattern. If the degradation recurs on a daily cycle (worse during peak load, recovers overnight), thermal throttling is the likely explanation. Temperature has a time constant: the drive heats up under sustained load and cools when I/O subsides.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Composite Temperature	Current thermal state of the drive.	Trending toward WCTEMP under load.
Warning Comp. Temperature Time	Cumulative evidence of throttling history.	Non-zero value, especially if growing between polls.
Critical Comp. Temperature Time	Cumulative time above CCTEMP.	Any non-zero value. Sustained time at this temperature risks NAND damage.
Critical Warning bit 1 (0x02)	Temperature above WCTEMP or CCTEMP right now.	Currently set. May be transient but indicates the drive is at its thermal limit.
I/O throughput and await (iostat)	The user-visible symptom of throttling.	Throughput drops while queue depth stays constant. Await increases without corresponding media errors.
Available Spare	Thermal stress accelerates NAND wear.	Declining alongside elevated temperature confirms compounding damage.
Percentage Used	High temperature accelerates endurance consumption.	Rising faster than expected from write volume alone.

Fixes

Fix airflow and cooling

This is the most common and most effective fix. Thermal throttling means the drive cannot shed heat fast enough. Addressing the cooling path restores full throughput without changing any drive configuration.

Check fan status. A failed chassis fan is the most common root cause. Verify all fans are spinning and that airflow direction is correct (front-to-back in most server configurations).
Inspect the M.2 heatsink. On M.2 drives, the heatsink or thermal pad may have detached, especially after physical maintenance. A drive with no heatsink in a sustained-write workload will throttle quickly. Reseat the heatsink with proper thermal pad contact.
Check for airflow obstruction. Cables, drive cages, or dust accumulation can redirect airflow away from the NVMe slot. Ensure the airflow path to the drive is unobstructed.
Reduce ambient temperature. If the datacenter or room temperature is high, all drives suffer. Address the environmental issue rather than individual drives.

Reduce write workload

If cooling improvements are not possible or insufficient, reducing the write rate gives the controller more thermal headroom. This is a workaround, not a fix.

Move write-heavy workloads (logs, WAL, swap, temp files) to a different drive.
Batch writes to reduce sustained burst duration.
Enable TRIM to help the controller manage garbage collection efficiently, which reduces internal write amplification and heat generation.

Consider HCTM thresholds

Host Controlled Thermal Management lets the OS set TMT1 and TMT2 throttle points via NVMe Set Features (Feature Identifier 0x10). By default, the drive firmware uses its own WCTEMP and CCTEMP thresholds. HCTM gives the host control over when throttling begins.

Setting TMT1 lower than the firmware default causes the drive to throttle earlier and more gently, which can prevent the sharp throughput cliffs that occur when the firmware abruptly throttles at WCTEMP. The trade-off is that you are choosing to limit performance at a lower temperature to avoid hitting the harder thermal wall.

Warning: nvme set-feature applies changes immediately and affects live I/O performance. Test on a non-production drive first. HCTM thresholds may not persist across controller resets unless saved with the Save (SV) bit; check your drive’s behavior.

TMT1 and TMT2 thresholds are vendor-specific in their supported range and are not directly readable through standard smartctl output. Use nvme set-feature and nvme get-feature from the nvme-cli package to configure and query them.

Prevention

Monitor Warning and Critical Comp. Temperature Time. These counters are the only persistent evidence of throttling. Alert on any non-zero value that was previously zero. This is a Level 3 maturity signal in the SMART monitoring maturity model.
Track composite temperature trends. A rising baseline temperature over days or weeks indicates degrading cooling before throttling begins. A 10C increase from the established baseline without a workload change warrants investigation.
Establish temperature baselines per drive. NVMe drives in different chassis positions have different thermal profiles. Baseline each drive at deployment and alert on deviation, not on absolute thresholds alone.
Schedule periodic physical inspection. M.2 heatsinks detach. Fan bearings fail. Airflow paths get obstructed. These are mechanical issues that monitoring can hint at (rising temperature) but cannot confirm.
Do not page on temperature alone. Transient temperature spikes from burst I/O or ambient heat self-resolve with throttling. Temperature without corroborating signals (throughput drop, Warning Comp. Temp. Time incrementing, Critical Warning bit 1) is the drive operating within its protection envelope.

How Netdata helps

Per-second temperature correlation. Netdata collects NVMe SMART metrics at per-second resolution, making the correlation between temperature rise and throughput drop visible at the timescale where throttling actually occurs. Standard polling intervals of 5 to 10 minutes miss the throttle cycle entirely.
Throughput and latency alongside SMART data. Netdata surfaces disk I/O metrics (throughput, latency, queue depth, iowait) on the same dashboard as SMART health data, so the diagnostic pattern of temperature rising while throughput drops with no media errors is visible without switching tools.
Anomaly detection on temperature trends. Netdata’s ML-based anomaly detection flags unusual temperature patterns (gradual rise, new sustained-temperature baseline) before they reach the throttle threshold.
Rate-of-change alerting on cumulative counters. Warning Comp. Temperature Time and Critical Comp. Temperature Time are cumulative counters. Netdata’s rate calculation surfaces when these counters start incrementing, which is definitive evidence of throttling history.
Fleet-level temperature comparison. When multiple drives in the same chassis show temperature anomalies simultaneously, Netdata’s correlated dashboards make the environmental root cause immediately visible.