NVMe thermal management transitions: TMT1 and TMT2 as an early throttle signal

An NVMe drive that is thermally throttling rarely looks broken. There are no I/O errors, no kernel messages, no failed commands. Throughput drifts down, latency drifts up, and the application team files a ticket about “the database being slow.” By the time critical_warning bit 1 (temperature threshold exceeded) is set, the drive has already been protecting itself for a while.

The thermal management transition counters close that gap. They tell you the drive has entered a throttling state, how many times, and for how long, well before the warning or critical composite temperature thresholds are crossed. In the monitoring maturity model this is a Mature-level signal: it sits between “watch the composite temperature” and “the drive has declared a thermal problem.” See the NVMe mental model for where thermal throttling fits among the characteristic NVMe failure archetypes.

What TMT1 and TMT2 are

TMT1 and TMT2 are Thermal Management Temperature thresholds. When the controller’s temperature crosses TMT1, it enters a lighter throttling state: it reduces internal activity enough to shed heat, with a modest performance cost. When the temperature crosses TMT2, it throttles much more aggressively, accepting a large performance penalty to stay within its thermal envelope.

These thresholds come from the Host Controlled Thermal Management (HCTM) feature, introduced in the NVMe 1.3 specification. The host configures the thresholds via the Set Features command; the drive advertises its minimum and maximum allowable values in Identify Controller data. Not every drive implements them. On drives that do not, the counters stay at zero forever, which is indistinguishable from “never throttled” unless you check feature support.

The drive tracks four counters in the SMART / Health Information log:

All four are cumulative lifetime counters and persist across reboots. A non-zero value tells you the drive has throttled at some point in its life; only the rate of change tells you whether it is throttling now.

How the throttle states fit together

TMT1 and TMT2 sit below the warning and critical composite temperature thresholds (WCTEMP and CCTEMP) in the thermal escalation ladder. WCTEMP and CCTEMP are the thresholds behind critical_warning bit 1; exceeding them is the drive declaring a thermal problem. TMT1/TMT2 transitions are the drive quietly managing heat before it gets there.

stateDiagram-v2
    [*] --> Normal
    Normal --> TMT1 : temp crosses TMT1 threshold
    TMT1 --> Normal : temp falls back
    TMT1 --> TMT2 : temp crosses TMT2 threshold
    TMT2 --> TMT1 : temp falls back
    TMT2 --> WarningCritical : temp crosses WCTEMP/CCTEMP
    WarningCritical --> TMT2 : temp falls back
    WarningCritical --> Shutdown : sustained at CCTEMP
    note right of TMT1 : light throttle, thm_temp1 counters increment
    note right of TMT2 : heavy throttle, thm_temp2 counters increment
    note right of WarningCritical : critical_warning bit 1 set

Two operational consequences follow:

• The counters are the earliest throttle evidence you get. A drive oscillating between Normal and TMT1 under load is already giving up performance, but nothing is “wrong” yet: no warning bits, no errors. The transition count is the only record.
• TMT2 activity is a much stronger signal than TMT1. Brief TMT1 excursions under heavy sustained writes can be normal for a passively cooled drive. Repeated or lengthy TMT2 residency means the thermal design is losing, and the next step up the ladder is critical_warning bit 1. See NVMe critical_warning is nonzero for decoding that bitmask.

Reading the counters

The counters live in the SMART log, so nvme smart-log (or smartctl) is the way to read them. They are not exposed through hwmon sysfs; hwmon gives you temperature sensors, not throttle bookkeeping.

# Read thermal management transition counters from the SMART log
nvme smart-log /dev/nvme0 | grep -i "thm_temp"

# Same fields, JSON form (useful for scripting)
nvme smart-log /dev/nvme0 -o json | \
  jq '.thm_temp1_trans_count, .thm_temp1_total_time,
      .thm_temp2_trans_count, .thm_temp2_total_time'

# Check the current composite temperature for context
nvme smart-log /dev/nvme0 | grep "^temperature"

On a drive that has never throttled, all four counters are zero. On a drive with a history of thermal pressure you might see a TMT2 transition count in the tens with a handful of accumulated minutes: each transition was brief, but they add up.

To confirm whether a drive supports HCTM at all, inspect the Identify Controller data. Unsupported drives report zeros even while throttling internally.

# Check controller thermal capabilities and thresholds
nvme id-ctrl /dev/nvme0 | grep -i "temp\|hctm"

If the feature is absent, zeros in the transition counters carry no information, and you should rely on composite temperature plus warning_temp_time and critical_comp_time instead.

Interpreting what you see

All zeros, drive supports HCTM. The drive has never needed to throttle. Healthy baseline, no action.

Non-zero but static values. The counters are lifetime cumulative. A transition count that has not moved in weeks records a past event: a failed fan, a benchmark run, a hot aisle incident. Compare accumulated TMT time against power_on_hours to see what fraction of the drive’s life it has spent throttled. A static non-zero value is history, not a current problem.

Transition counts increasing. The drive is actively and repeatedly entering a throttle state. This is the early warning. Rising transition counts or total time warrant a ticket: the drive is routinely hitting its thermal limits, and the fix is environmental (cooling, airflow, heatsink seating, adjacent heat sources), not a drive replacement.

Many transitions per hour. An acute thermal event: likely a cooling failure or a sudden workload shift. Treat as urgent. If the TMT2 counters are the ones climbing, the drive is already in heavy throttle and critical_warning bit 1 is the next thing you will see.

Counts increasing but performance looks fine. Trust the counters. Thermal throttling is deliberately subtle: the drive trades performance for survival without emitting errors. Correlate with block-layer throughput and latency before concluding the throttle is harmless. The distinguishing feature of a thermal problem versus, say, a garbage-collection stall is exactly this correlation: performance dips that track temperature and transition events, with no media errors and recovery when load is removed.

Zero transitions on a drive that cannot count them. The trap is assuming “zero transitions” means “no throttling” on a drive without HCTM support. Check id-ctrl first; on unsupported drives the counters never move regardless of thermal behavior.

Where this shows up in production

The classic setting is M.2 drives without heatsinks, or with heatsinks making poor contact, in dense chassis. M.2 slots provide essentially no airflow of their own, and M.2 drives throttle much faster than U.2/U.3/EDSFF parts; consumer drives throttle more aggressively than enterprise ones. Under sustained write load (backups, compaction, log ingestion, RAID resync) these drives ride the TMT1 boundary, and the workload experiences it as unexplained latency variance and a throughput ceiling well below the drive’s rated numbers.

Other recurring triggers in the thermal death-spiral pattern: blocked airflow, adjacent high-heat components (a GPU next to the M.2 slot), ambient temperature drift from a cooling problem, and sustained 100 percent write load. The cascade is temperature rising, transition counts climbing, throughput declining, latency increasing, and finally critical_warning bit 1 if nothing changes. Catching it at the transition-count stage means fixing a fan or a heatsink during business hours instead of debugging a “slow application” during an incident.

One interaction worth knowing: throttling can become self-sustaining. If the workload is latency-sensitive and keeps queueing requests, the drive stays busy and stays hot, so it may not cool down even at the throttled rate. Reducing the write load is often the fastest way to break the loop while you fix the cooling.

Signals to watch in production

The correlation pattern that confirms root cause: transition counts and composite temperature rising together while throughput falls and latency rises, with zero media errors. That combination is thermal. If throughput falls with normal temperature and no transitions, look elsewhere (GC pressure, PCIe link degradation).

How Netdata helps

• Netdata’s NVMe collector reads the SMART log and exposes the four counters as nvme.device_thermal_mgmt_temp1_transitions_rate, nvme.device_thermal_mgmt_temp2_transitions_rate (transitions per second), and nvme.device_thermal_mgmt_temp1_time, nvme.device_thermal_mgmt_temp2_time (accumulated time; the underlying SMART fields are in minutes). The rate views turn lifetime counters into a current-state signal, which is exactly what you need for “is it throttling now.”
• Because the same collector also exposes composite temperature, warning and critical temperature time, and the per-bit critical warning states, you can see the full escalation ladder on one screen: transitions first, temperature time next, bit 1 last.
• Correlating the transition-rate charts with block-layer throughput and latency on the same dashboard is how you confirm the throttle is the cause of a performance complaint rather than a coincidence.
• Per-device charts make fleet comparison easy: one drive with climbing TMT2 transitions next to nine idle siblings points at a local cooling or seating problem, not a workload issue.
• Alerting on a non-zero transition rate catches the chronic undercooling case at the Mature level, before it escalates into a critical_warning bit 1 incident.

Related guides

• NVMe critical_warning is nonzero: decoding the SMART critical warning bitmask
• How NVMe actually works in production: a mental model for operators
• NVMe available spare below threshold: critical warning bit 0 and end-of-life wear
• NVMe available spare declining: watching the wear trajectory before the threshold
• NVMe endurance runway: projecting time-to-replacement from wear signals
• NVMe media_errors increasing: uncorrectable data-integrity errors on NAND
• NVMe error log entries growing: num_err_log_entries beyond media errors
• blk_update_request: I/O error, dev nvme0n1: reading NVMe I/O errors in the kernel log
• NVMe controller reset loop: repeated resets from a firmware hang
• nvme nvme0: I/O timeout, Resetting controller: what an NVMe controller reset means
• NVMe controller state not live: reading resetting, deleting, and dead from sysfs
• NVMe device disappeared: nvme0: Removing and a drive that fell off the PCIe bus