NVIDIA GPU thermal throttling: clocks dropping under heat

Training throughput fell 25% overnight. No errors in the application logs. nvidia-smi shows every GPU at 100% utilization. Nothing looks broken, and yet the job is measurably slower.

This is what NVIDIA GPU thermal throttling looks like in practice: the GPU keeps working, keeps reporting full utilization, and silently delivers 10-40% less real work because its SM clocks have dropped. Utilization measures time busy, not work done. A kernel running at half clock speed still occupies the SMs for the whole sampling window, so utilization.gpu stays pinned at 100% while tokens/sec or samples/sec collapse.

The GPU tells you exactly why it throttled, through clock event reasons. Temperature is the leading signal, throttle reasons are the confirmation, and clock speed is the severity gauge. This guide walks the diagnostic path from symptom to root cause, and covers the multi-GPU case where one hot GPU drags down an entire training job.

What this means

NVIDIA GPUs protect silicon with a staged thermal cascade. Thresholds are model-specific and only visible via nvidia-smi -q -d TEMPERATURE or NVML on the actual hardware, but typical ranges from the field:

SW thermal slowdown at GPU Max Operating Temp (datacenter around 83 C, consumer around 88-90 C). Mild clock reduction.
HW thermal slowdown at GPU Slowdown Temp (datacenter around 87 C, consumer around 92 C). Aggressive clock reduction, a factor of 2 or more.
Emergency shutdown at GPU Shutdown Temp (datacenter around 90 C, consumer around 95 C). The GPU powers off.

Two properties of this cascade trip up operators. First, the system finds equilibrium at degraded performance: clocks drop, heat output falls, temperature stabilizes, and the GPU runs indefinitely at terrible speed. There is no error because from the driver’s perspective, nothing is wrong. Second, the temperature you read during throttling can be below the threshold that triggered it, because the GPU cooled itself down by throttling. A GPU reading 82 C with hw_thermal_slowdown active is not a contradiction. The hotspot crossed the threshold, the clocks dropped, and the reported die temperature recovered.

flowchart TD
  A[Heat exceeds cooling capacity] --> B[Die temp crosses Max Operating Temp]
  B --> C[sw_thermal_slowdown active - mild clock cut]
  C --> D{Cooling recovers?}
  D -- yes --> E[Clocks restore - transient event]
  D -- no --> F[Temp crosses Slowdown Temp]
  F --> G[hw_thermal_slowdown active - 2x or more clock cut]
  G --> H[Throughput collapses - utilization still 100 percent]
  G --> I{Still heating?}
  I -- yes --> J[Emergency shutdown at Shutdown Temp]
  I -- no --> K[Equilibrium at degraded performance]
  K --> L[In multi-GPU jobs: every GPU waits for the throttled one]

Common causes

Cause	What it looks like	First thing to check
Local cooling failure (fan, seating, blocked airflow)	One GPU hot, siblings normal	Fan speed on that GPU, physical airflow, chassis slot
Environmental (HVAC failure, hot aisle, inlet temp)	All GPUs in the node or rack hot together	Ambient/inlet temperature, chassis fans via IPMI
Dust accumulation or degraded thermal paste	Baseline temperature creeping up over weeks or months	Long-term temperature trend vs the GPU’s own history
NVLink-adjacent heat transfer	Two adjacent GPUs hotter than the rest	Slot layout, whether the hot pair are neighbors
HBM thermal limit (memory-bound workloads)	Die temp fine, memory temp high, memory clock throttled	`temperature.memory` on HBM GPUs
Power or other throttle confused for thermal	Clocks low but temperature moderate	Throttle reasons, not temperature alone
Driver regression lowering the thermal target	Throttling at unusually low temperature after a driver update	Driver version, target temperature vs known baseline

Quick checks

All read-only and safe to run during production workloads.

# 1. Current die temperature for all GPUs
nvidia-smi --query-gpu=index,temperature.gpu --format=csv,noheader,nounits

# 2. Full temperature report: thresholds and current readings for GPU 0
nvidia-smi -i 0 -q -d TEMPERATURE
# Look for: GPU Current Temp, GPU Slowdown Temp, GPU Max Operating Temp,
# GPU Shutdown Temp, GPU Target Temperature

# 3. Active throttle reasons (the confirmation step)
nvidia-smi --query-gpu=index,clocks_event_reasons.active --format=csv,noheader

# 4. Individual thermal/power reasons in one shot
nvidia-smi --query-gpu=clocks_event_reasons.hw_thermal_slowdown,clocks_event_reasons.sw_thermal_slowdown,clocks_event_reasons.hw_slowdown,clocks_event_reasons.sw_power_cap,clocks_event_reasons.hw_power_brake_slowdown --format=csv,noheader

# 5. Current vs max clocks (severity of the throttle)
nvidia-smi --query-gpu=index,clocks.current.sm,clocks.max.sm,clocks.current.memory,clocks.max.memory --format=csv,noheader,nounits

# 6. HBM memory temperature (datacenter GPUs only; N/A on GDDR cards)
nvidia-smi --query-gpu=index,temperature.memory --format=csv,noheader,nounits

# 7. Fan speed where applicable (passively cooled datacenter cards report N/A)
nvidia-smi --query-gpu=index,fan.speed --format=csv,noheader,nounits

# 8. Power draw vs enforced limit (rule out power throttling)
nvidia-smi --query-gpu=index,power.draw,enforced.power.limit --format=csv,noheader,nounits

# 9. Recent driver-reported errors
dmesg -T | grep -i "NVRM: Xid" | tail -20

Two interpretation notes. temperature.gpu.tlimit is a margin (degrees remaining before the limit), not an absolute threshold. And clocks_throttle_reasons is the old field name; clocks_event_reasons is current, but both aliases work. Counter variants clocks_event_reasons_counters.* report cumulative time spent in each state, useful for catching throttling that happened between your samples.

How to diagnose it

Confirm throughput actually dropped. Check the application-level metric: tokens/sec, samples/sec, step time. GPU utilization cannot tell you this. If throughput is fine, you are chasing a phantom.
Read the throttle reasons before touching anything. Run check 3 or 4 above. If hw_thermal_slowdown or sw_thermal_slowdown is Active, it is thermal. If sw_power_cap is Active with moderate temperature, this is power throttling, a different problem with different fixes. If sync_boost is active, the GPU is being held back by its sync boost group: all GPUs in the group boost to the minimum clock achievable across the group, so one hot GPU can pull down healthy siblings. Check throttle reasons on every GPU, not just the one that looks slow.
Quantify the clock drop. Compare clocks.current.sm to clocks.max.sm under load. Below 80% of max during heavy compute is significant throttling. HW thermal slowdown cuts clocks by a factor of 2 or more, so a GPU at half clock is consistent with that state. Also check the memory clock: HBM thermal throttling reduces memory clock independently of SM clock, and it hits memory-bound workloads (LLM inference especially) hard.
Compare across GPUs in the same node. One hot GPU with cool siblings means a local cooling problem: fan, heatsink seating, blocked airflow, degraded thermal paste on that card. All GPUs hot together means environmental: HVAC, inlet temperature, rack airflow. NVLink-adjacent GPUs in neighboring slots heat each other, so a hot pair can be a placement problem, not two independent failures.
Check the cooling response. On air-cooled cards, fan speed pinned at 100% with temperature still rising means cooling capacity is exceeded. Fan at 0% with temperature rising means fan failure (though 0% is normal at idle on cards with fan-stop). On passively cooled datacenter GPUs, fan speed reports N/A and chassis fans are the cooling system: check them via IPMI or the BMC.
Rule out lookalikes if temperature is moderate. If clocks are low, utilization is low (1-3%), and throughput is catastrophically bad, that is not thermal throttling. Severe IOMMU/VT-d DMA translation overhead on some Intel systems can produce a similar signature; the fix is a kernel parameter change such as iommu=pt (sourced from vendor troubleshooting documentation, not from NVIDIA’s own Xid guidance). Also check recent driver updates: a driver regression on some consumer/Turing cards lowered the GPU Target Temperature and caused premature throttling, with GSP firmware toggling (NVreg_EnableGpuFirmware) reported as a factor; reported cases are consumer cards, and datacenter-GPU impact has not been documented.
In multi-GPU training, find the straggler. AllReduce synchronizes every GPU at each step, so the slowest GPU gates the whole job. Compare temperature, clocks, and throttle reasons across all GPUs. The “normal” GPUs are not healthy bystanders; they are waiting.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
GPU die temperature	Leading signal: rises before clocks drop	Sustained within 5 C of Max Operating Temp
`clocks_event_reasons.hw_thermal_slowdown`	Confirms severe throttling; 2x+ clock cut	Active for >60s during compute: page-worthy
`clocks_event_reasons.sw_thermal_slowdown`	First-stage throttle, earlier warning	Active during production compute
`clocks.current.sm` vs `clocks.max.sm`	Severity of the throttle	<80% of max under heavy load
`temperature.memory` (HBM GPUs)	Independent memory thermal limit	High while die temp is fine, memory clock dropped
Fan speed	Cooling response vs capacity	100% with rising temp, or 0% with rising temp
Application throughput (tokens/sec, step time)	The only ground truth for impact	Declining while utilization stays at 100%
Cross-GPU temperature spread	Local vs environmental cause	One outlier vs uniform elevation
Temperature baseline trend (weeks)	Dust and paste degradation creep	Peak-load temp rising month over month

Do not alert on raw temperature alone. 80 C is normal under full load for many GPUs, and the threshold that matters varies by model. Alert on throttle reasons, use temperature for severity and trend.

Fixes

Local cooling failure (one hot GPU)

Clear airflow obstructions: cables blocking intake, missing slot baffles, adjacent hot components.
Reseat the GPU and verify heatsink contact. Degraded thermal paste shows up as a baseline temperature that crept up over months and no longer responds to fan speed.
Replace a failed fan. A fan at 0% under load with rising temperature is a hardware fix, not a configuration fix.

Environmental (all GPUs hot)

Check datacenter HVAC and inlet temperature at the rack. This is the most common cause of a whole node throttling simultaneously.
Verify chassis fans via IPMI on passively cooled datacenter GPUs; the server fans are the GPU cooling system.
Short term, reduce load on the affected node or migrate workloads while cooling is restored. Throttling is self-limiting, but sustained HW slowdown eventually risks shutdown and accelerates long-term silicon aging.

Reducing thermal load on the GPU itself

Lower the power limit. Power and heat are directly coupled, and a modest power cap reduction often costs little throughput while buying real thermal headroom. This is a deliberate tradeoff: you are choosing a smaller, bounded performance reduction over an uncontrolled thermal one.
For memory-bound workloads hitting HBM limits, reduce memory intensity (batching changes, kernel fusion) rather than compute intensity.

What not to do

Do not restart the driver or reboot as a first response. Throttling is the GPU protecting itself; the reset fixes nothing and the next thermal excursion will recur.
Do not treat sw_power_cap as a thermal fault. Datacenter power capping is often intentional. Check power.limit vs power.default_limit before blaming cooling.
Do not raise thermal targets with nvidia-smi -gtt blindly. The command is unsupported on many GPUs, and overriding thermal protection trades hardware longevity for throughput; check nvidia-smi -q -h for support on your model before attempting it.

Prevention

Trend the thermal baseline. Track peak-load temperature per GPU over weeks. A shrinking gap to the Slowdown Temp is your early warning for dust buildup and paste degradation, and it is a maintenance ticket, not an incident.
Monitor throttle reasons, not just temperature. Sample clocks_event_reasons at 10-second resolution or better; thermal excursions can start and resolve within a minute-resolution gap.
Correlate utilization with application throughput. Define a per-workload baseline for tokens/sec or step time. A utilization-versus-throughput divergence is the signature of silent throttling.
Compare across GPUs continuously. Cross-GPU comparison catches both the single straggler and the fleet-wide environmental drift that per-GPU thresholds miss.
Verify thresholds after hardware or driver changes. Thermal limits are model-specific and driver behavior around them can change. After any driver update, spot-check nvidia-smi -q -d TEMPERATURE against your known baseline.

How Netdata helps

Netdata collects per-second GPU temperature, SM and memory clocks, power draw, and fan speed, which catches thermal excursions that minute-resolution polling misses entirely.
Clock throttle reason states are collected alongside temperature, so you see the confirmation signal (hw_thermal_slowdown active) on the same timeline as the temperature rise, instead of correlating across tools.
Per-GPU charts make the one-hot-GPU versus all-hot-GPU distinction immediate: local cooling fault versus environmental failure is a glance, not an investigation.
Correlating GPU clocks and temperature with host-level signals (chassis fans, ambient sensors where exposed) closes the loop between the GPU symptom and the cooling cause.
ML-based anomaly detection on per-GPU temperature baselines flags the slow upward drift from dust and paste degradation before it reaches throttle thresholds.