NVIDIA GPU temperature too high: die temperature and thermal limits

A GPU reporting 85 °C is not necessarily a GPU with a temperature problem. Under sustained load, datacenter GPUs like the A100 and H100 are designed to run hot, and 80 °C is routine under full load. The number alone tells you almost nothing.

What matters is where the temperature sits relative to the SKU’s own thermal limits, whether the GPU has started throttling, and whether the pattern is one hot GPU or a whole chassis running hot. A GPU at 85 °C with no throttle reasons active is fine. A GPU at 75 °C with hw_thermal_slowdown active has a cooling system that is already failing.

There is also a timing problem. Die temperature is a lagging indicator: by the time the GPU reports 85 °C on a card whose software slowdown threshold is 83 °C, it has already been shedding clocks. The throttle reason flags lead the temperature reading, not the other way around. That is why the correct alert is on throttle state, with temperature informing severity.

This guide covers how to read die temperature against the actual limits, how to tell a local cooling fault from an environmental one, and what to do about each.

What this means

NVIDIA GPUs manage thermals as a staged cascade, not a single threshold. The stages, per the driver’s own reporting:

1. SW thermal slowdown at GPU Max Operating Temp. The driver starts optimizing clocks for thermal conditions. Datacenter cards: roughly 83 °C. Consumer/workstation: roughly 88-90 °C.
2. HW thermal slowdown at GPU Slowdown Temp. The hardware takes over and cuts clocks aggressively, by a factor of 2 or more. Datacenter: roughly 87 °C.
3. Emergency shutdown at GPU Shutdown Temp. The GPU powers off. Datacenter: roughly 90 °C.

These numbers are model-specific and not published in a single table. The authoritative values for your card come from nvidia-smi -q -d TEMPERATURE on the actual hardware, or from NVML’s nvmlDeviceGetTemperatureThreshold(). Do not hardcode thresholds from another SKU.

flowchart TD
  A[Load increases, heat rises] --> B{Temp reaches Max Operating Temp}
  B -->|yes| C[SW thermal slowdown: driver trims clocks]
  C --> D{Cooling recovers?}
  D -->|yes| E[Clocks restore, equilibrium]
  D -->|no| F{Temp reaches Slowdown Temp}
  F -->|yes| G[HW thermal slowdown: 2x+ clock cut]
  G --> H{Still rising?}
  H -->|yes| I[Shutdown Temp: GPU powers off]
  H -->|no| J[Equilibrium at degraded performance]

Two properties of this cascade matter operationally. First, the system finds equilibrium: an overheating GPU does not usually crash, it settles into running your workload 2-10x slower while utilization still reads 100%. Second, utilization lies during throttling. A throttled GPU shows high utilization because kernels take longer, not because it is doing more work. Correlate with clocks and actual throughput.

One reporting quirk: temperature.gpu.tlimit is a margin (degrees of headroom remaining), not an absolute threshold. Treating it as an absolute limit inverts your alerting logic.

Common causes

The most important split is scope: one hot GPU means a local cooling problem (that card’s heatsink, seating, paste, or the chassis fan feeding it). All GPUs hot means an environmental problem (HVAC, intake temperature, rack airflow). Everything after that first branch is different.

Quick checks

All read-only.

# Current die temperature for every GPU
nvidia-smi --query-gpu=index,name,temperature.gpu --format=csv

# Full temperature picture for GPU 0: current, target, and this SKU's thresholds
nvidia-smi -i 0 -q -d TEMPERATURE

# HBM memory temperature (datacenter GPUs only; returns N/A on GDDR cards)
nvidia-smi --query-gpu=index,temperature.memory --format=csv,noheader,nounits

# Is the GPU throttling right now, and why
nvidia-smi --query-gpu=index,clocks_event_reasons.active --format=csv

# Individual throttle flags
nvidia-smi --query-gpu=clocks_event_reasons.sw_thermal_slowdown,clocks_event_reasons.hw_thermal_slowdown,clocks_event_reasons.sw_power_cap --format=csv,noheader

# How far have clocks actually fallen
nvidia-smi --query-gpu=index,clocks.current.sm,clocks.max.sm,clocks.current.memory,clocks.max.memory --format=csv

# Power state: is the GPU also pinned against its power limit
nvidia-smi --query-gpu=index,power.draw,power.limit,enforced.power.limit --format=csv

# Fan speed (N/A on passively cooled datacenter cards)
nvidia-smi --query-gpu=index,fan.speed --format=csv,noheader

For datacenter SXM hardware, fan.speed returns N/A because the cards are passively cooled. They have no fans of their own and depend entirely on chassis airflow. The fan status you need lives in the BMC, not in nvidia-smi:

# Chassis fan and intake temperature via IPMI (read-only)
ipmitool sensor | grep -i -E "fan|inlet|intake|temp"

How to diagnose it

1. Establish scope first. Pull temperature.gpu for every GPU in the node. If one GPU is hot and the rest are normal, you have a local cooling fault on that card. If all GPUs are elevated together, stop looking at the GPUs and look at the environment: intake temperature, HVAC, rack airflow.

2. Check throttle state before trusting the temperature. Run the clocks_event_reasons queries above. sw_thermal_slowdown = Active means the GPU crossed its Max Operating Temp and the driver is trimming clocks. hw_thermal_slowdown = Active means the hardware has taken over with an aggressive 2x+ clock cut: this is a paging condition if it sustains beyond 60 seconds during production compute, because at that point the cooling system is demonstrably not keeping up. If no thermal throttle reasons are active, the temperature is within design and the alert was miscalibrated.

3. Quantify the performance impact. Compare clocks.current.sm against clocks.max.sm. Below 80% of max under load is significant throttling. Utilization stays high while clocks fall, so the clock ratio, not utilization.gpu, is the truth about lost capacity.

4. Rule out power capping as the primary cause. sw_power_cap active alongside thermal flags often means the GPU hit its power limit first and the thermal state is secondary. Check enforced.power.limit and whether power.limit < power.default_limit (someone capped it deliberately). Power-throttled GPUs can run warm without any cooling fault.

5. For a single hot GPU, check cooling delivery. On SXM/HGX/DGX chassis, pull fan speeds from the BMC. A failed chassis fan leaves a passively cooled SXM module with zero independent cooling, and the failure pattern is one or two GPUs overheating while siblings stay cool. On actively cooled PCIe cards, fan.speed = 0 with rising temperature is a dead fan; fan.speed = 100% with rising temperature means cooling capacity is exceeded (dust, blockage, ambient).

6. Check idle temperature as a baseline. Let the node drain or look at a quiet window. An idle datacenter GPU typically sits in the 30-50 °C range. One GPU idling 20 °C hotter than its siblings on the same baseboard has a local cooling problem: blocked airflow to that slot, degraded thermal paste, or a heatsink issue. This signal is clean because it removes workload as a variable.

7. Look at the trend, not the point. Compare today’s peak-load temperature against the same workload weeks ago. A baseline creeping upward over months with the same workload points to dust accumulation or paste degradation, not a sudden fault.

Metrics and signals to monitor

Threshold guidance: alert relative to the SKU’s own GPU Max Operating Temp from nvidia-smi -q, not absolute numbers. A warning near 90% of max and a critical near 95% of max tracks the hardware’s design intent. Raw rules like “>80 °C” either false-positive on healthy datacenter cards under load or fire too late, after throttling has already been running.

Fixes

Failed or degraded chassis fan (SXM systems)

Replace the fan module. On most HGX/DGX-class chassis fans are hot-swappable, but confirm your platform’s service procedure before pulling anything. Until the fan is replaced, reduce or migrate workload off the affected GPUs: a passively cooled SXM module with compromised airflow has no fallback cooling and will ride the throttle cascade up to shutdown. Do not wait for the shutdown threshold to make the decision for you.

Fan failure on an actively cooled card

Replace the fan or the card. There is no software fix for a dead fan. As a stopgap, reducing the power limit lowers heat output, at the cost of clocks. That tradeoff keeps the card below its thermal thresholds but does not restore cooling capacity, so treat it strictly as a bridge to hardware replacement.

Blocked airflow and dust

Schedule a maintenance window and clean the heatsinks and airflow path. Check cable management inside the chassis for obstruction. After cleaning, verify by comparing peak-load temperature against the pre-incident baseline for the same workload. If the baseline does not come back down, look at thermal paste next.

Degraded thermal paste

Repasting requires removing the card and, on SXM modules, is typically not a field service task; coordinate with the vendor or hardware support. The diagnostic that justifies the ticket is the month-over-month temperature creep at constant workload plus an idle delta versus sibling GPUs. Plan replacement or vendor service rather than improvising on a production node.

HVAC / ambient failure

This is a facilities incident, not a GPU incident. Escalate to whoever owns the CRAC units and rack airflow. On the GPU side, your option is shedding load: drain jobs from the hottest nodes first. Datacenter GPUs will protect themselves via the throttle cascade, but sustained operation near HW slowdown costs you 2x or more performance anyway, so the GPUs are effectively degraded whether or not you act.

Power cap masquerading as thermal throttling

If sw_power_cap is the active reason and temperatures are moderate, confirm whether power.limit < power.default_limit was an intentional datacenter power-budgeting decision. If it was, the throttling is expected behavior, not a fault. If it was not, restore the default limit after confirming PSU and PDU headroom. Do not raise power limits on a node that is already thermally marginal: more power is more heat.

Workload reduction as a general mitigation

Migrating or pausing jobs is always available and always works as a bridge. The tradeoff is capacity. Use it to keep hardware below HW thermal slowdown while the real fix (fan, cleaning, HVAC) is in progress, not as the fix.

Prevention

• Alert on throttle state, not raw temperature. Ticket when temperature sustains within 5 °C of the SKU’s Max Operating Temp. Page when hw_thermal_slowdown is active and sustained beyond 60 seconds during production compute. Raw temperature alerts without throttle context generate false positives at values that are normal under load.
• Set thresholds relative to the hardware. Read GPU Max Operating Temp per SKU from nvidia-smi -q -d TEMPERATURE and set warning near 90% and critical near 95% of that value. Do not copy thresholds across GPU models.
• Monitor the cooling delivery, not just the die. For passively cooled datacenter cards, chassis fan health and intake temperature via IPMI/BMC are the leading signals. The GPU temperature is what happens after the cooling has already failed.
• Track the thermal baseline as a capacity signal. Peak-load temperature trending upward over months, or fan speed required for a given workload increasing, means cooling capacity is degrading. Maintain at least 10 °C of headroom between peak workload temperature and the throttle threshold; if that gap shrinks quarter over quarter, schedule cleaning or service before the incident.
• Compare across GPUs. A cross-GPU view catches the one hot card pattern and the straggler it creates in multi-GPU training. Per-GPU dashboards that never get compared side by side hide exactly the failure mode that matters most.
• Sample fast enough. Temperature spikes and throttle events can happen and resolve in seconds. Minute-resolution collection misses them; sample at 10 seconds or better.

How Netdata helps

• Netdata’s NVIDIA GPU collector surfaces die temperature, HBM memory temperature, fan speed, power draw, clocks, and throttle reasons per GPU at per-second resolution, which is the sampling rate short-lived thermal events require.
• Seeing temperature.gpu and the clock throttle reason flags on the same dashboard collapses the “is it throttling or just warm” question into one view, which is the most common triage mistake.
• Per-GPU breakdowns make the one-hot-GPU versus all-hot-GPUs split immediate, routing you to local cooling versus environmental causes on the first look.
• Correlating power.draw against the enforced power limit next to the thermal flags separates power capping from genuine cooling failure without an SSH session.
• Long retention at high resolution lets you compare this week’s peak-load temperature against last month’s baseline for the same workload, which is how you catch dust and paste degradation before the incident.
• Pairing GPU thermal signals with node-level metrics in the same view helps confirm when the “GPU problem” is actually a chassis or facilities problem upstream.

Related guides

• How an NVIDIA GPU actually works in production: a mental model for operators
• NVIDIA GPU monitoring checklist: the signals every production GPU fleet needs
• NVIDIA GPU monitoring maturity model: from survival to expert
• NVIDIA GPU ECC errors: corrected, uncorrected, volatile, and aggregate
• NVIDIA GPU HBM progressive failure: from single-bit errors to a dead GPU
• nvidia-smi hangs or is unresponsive: a wedged GPU or a stuck driver