NVIDIA GPU thermal cascade in dense servers: one hot GPU heats its neighbours

Three of the eight GPUs in a node are thermal throttling. Training step time has doubled. The instinct is to suspect the cooling system as a whole, or the workload. But line up per-GPU temperatures and one unit stands out: GPU 5 is running 12 degrees hotter than everything else, and the GPUs downstream of it in the airflow path are the ones throttling.

This is the thermal cascade pattern, specific to dense GPU servers (DGX, HGX, and similar trays) where GPUs share a cooling path. One GPU’s exhaust air is another GPU’s inlet air. A single unit with a local cooling problem raises the intake temperature of its neighbours, pushing them past their thermal limits. A throttled GPU still draws substantial power, so it keeps generating heat while producing less useful work. The system settles into a stable, badly degraded equilibrium: several GPUs throttled, all because of one.

The fix is almost never “reduce the workload.” Find the originating unit and its local cooling fault, then break the cascade there.

What this means

GPU thermal protection works in stages. As die temperature rises past the GPU Max Operating Temp, SW thermal slowdown reduces clocks. If temperature keeps climbing to the GPU Slowdown Temp, HW thermal slowdown applies an aggressive 2x or greater clock reduction. Past the Shutdown Temp, the GPU powers off. Exact thresholds are model-specific; read them from the hardware with nvidia-smi -q.

In a standalone server, thermal throttling is a single-GPU story. In a dense tray, the physics change:

1. A GPU develops a local cooling problem: degraded thermal paste, a poorly seated heatsink, an airflow blockage, or a tray position that receives pre-heated air.
2. Its die temperature climbs, and it begins to throttle.
3. A throttled GPU does not stop consuming power. It typically continues drawing most of its power budget while delivering a fraction of its former throughput. Heat output stays high; useful work drops.
4. That heat leaves the card as exhaust and becomes inlet air for the GPUs downstream in the chassis airflow path.
5. Those GPUs start warmer, hit their own thermal limits earlier, and throttle too.
6. The system reaches equilibrium at a degraded performance level. Every downstream GPU looks symptomatic, but only one is the cause.

The distinguishing feature is the shape of the temperature map: one GPU significantly hotter than the rest, usually in a specific physical position (middle or end of the tray), with the throttling GPUs clustered downstream of it. An all-GPUs-hot pattern is a different problem: it points to the environment (HVAC, ambient, blocked rack airflow), not a single unit.

flowchart TD
    A[GPU 5: local cooling fault] --> B[GPU 5 throttles but still draws power]
    B --> C[Exhaust heat raises inlet temp of downstream GPUs]
    C --> D[GPU 6 hits SW thermal slowdown]
    C --> E[GPU 7 hits SW thermal slowdown]
    D --> F[GPU 6 continues heating shared airflow]
    E --> F
    F --> G[GPU 7 escalates to HW thermal slowdown]
    G --> H[Equilibrium: multiple GPUs throttled, one root cause]

Common causes

The critical split: one hot GPU means local cooling, all hot GPUs means environmental. Everything in diagnosis flows from that.

Quick checks

All of these are read-only and safe on a production node.

# Per-GPU die temperature, one line per GPU
nvidia-smi --query-gpu=index,temperature.gpu --format=csv,noheader,nounits

# Which throttle reasons are active right now, per GPU
nvidia-smi --query-gpu=index,clocks_event_reasons.active --format=csv,noheader

# Individual thermal and power throttle flags per GPU
nvidia-smi --query-gpu=index,clocks_event_reasons.sw_thermal_slowdown,clocks_event_reasons.hw_thermal_slowdown,clocks_event_reasons.sw_power_cap --format=csv,noheader

# Current vs maximum SM clocks: the gap shows throttle severity
nvidia-smi --query-gpu=index,clocks.current.sm,clocks.max.sm --format=csv,noheader,nounits

# Power draw vs the enforced limit: throttled GPUs still draw power
nvidia-smi --query-gpu=index,power.draw,enforced.power.limit --format=csv,noheader,nounits

# Detailed temperature view, including thresholds, for one suspect GPU
nvidia-smi -i 5 -q -d TEMPERATURE

# HBM temperature (datacenter GPUs only; returns N/A on GDDR cards)
nvidia-smi --query-gpu=index,temperature.memory --format=csv,noheader,nounits

On the chassis side, check inlet temperature and fan speeds through the BMC:

# Chassis inlet temperature and fan speeds (datacenter GPUs report fan N/A; the chassis fans matter)
ipmitool sensor list | grep -i -E "inlet|fan|temp"

And check whether the driver has logged anything relevant:

# GPU error events in the kernel log
dmesg -T | grep -i "NVRM: Xid"

How to diagnose it

1. Build the temperature map. Pull per-GPU die temperatures and lay them out in physical slot order, not just index order. (Index-to-slot mapping is system-specific; check your platform documentation.) You are looking for the shape: one outlier versus a uniform rise.

2. Read the throttle reasons, not just temperatures. A GPU at 83C with sw_thermal_slowdown active is throttling; a GPU at 80C with no throttle reasons active is not. hw_thermal_slowdown active for a sustained period during production compute is a paging condition. Note which GPUs are throttling and which are merely warm.

3. Trace the cascade upstream. The throttling GPUs are victims. The originating unit is the hottest GPU upstream of them in the airflow path, typically middle or end of the tray in front-to-back designs. If the hottest GPU is also throttling hardest and its downstream neighbours are next worst, you have the cascade. If temperatures are uniform and high across all GPUs, skip to step 6: this is environmental.

4. Check whether the origin GPU is still drawing power. Compare power.draw against enforced.power.limit. A throttled GPU still drawing most of its limit confirms the amplifier mechanism: it is heating the shared airflow without delivering throughput.

5. Correlate with chassis data. Pull inlet temperature and fan speeds from IPMI/BMC. Inlet temp above spec across the node points to rack or room issues. Normal inlet temp with one GPU 10C or more above peers points to local cooling on that unit: paste, heatsink seating, or a blockage.

6. Rule out the environmental case. If all GPUs are hot, check inlet air temperature against spec, HVAC status, and rack-level recirculation (missing blanking panels, hot-aisle air looping back to intakes). No amount of per-GPU work fixes a hot room.

7. Physically inspect the originating unit. With the node drained and powered down per your platform’s procedure, check heatsink seating, thermal paste condition, airflow baffles and shrouds, and cable routing that might block airflow. If the same physical position runs hot across several identical nodes, suspect the tray position or a shared assembly fault rather than one bad card.

8. Confirm the cascade is broken after the fix. After remediation, put the node back under representative load and re-map temperatures and throttle reasons. Downstream GPUs should return to full clocks within minutes once inlet air normalizes. The repaired unit should hold temperature within a few degrees of its peers.

Metrics and signals to monitor

Do not alert on raw temperature alone. 80C under full load is normal for many GPUs. Alert on throttle reasons, and use temperature for severity and triage context.

Fixes

Local cooling fault on one GPU

Reseat the heatsink and replace thermal paste. The most common root cause of the one-hot-GPU pattern, especially on hardware older than a couple of years. It requires draining the node and following your platform’s service procedure. Tradeoff: node downtime, but it is the actual fix rather than a workaround.

Clear airflow obstructions. Reroute cables, reinstall missing baffles or shrouds, clear dust. Often zero-cost and immediately effective.

Replace the chassis fan or repair the cooling loop if IPMI data shows a fan at 0 or a fan at 100% that is not moving air. On liquid-cooled systems, pump degradation can present identically: one unit hot while others are fine.

Environmental causes

Restore rack airflow discipline. Install blanking panels in unoccupied rack units and seal obvious recirculation paths. Hot air looping from the exhaust side back to intakes raises inlet temperature for every node in the rack and can start cascades on nodes that were previously fine.

Fix the cooling plant issue. If inlet temperature is above spec across rows, escalate to facilities. No GPU-level action compensates for hot intake air.

Buying time while you fix the root cause

Migrate or shed workload on the affected node. If the platform supports it, move jobs off the node while it is serviced. This is the cleanest stopgap.

Reduce the power limit on the originating GPU as a temporary measure. Capping power reduces heat output at the cost of performance on that unit, which can relieve the downstream neighbours while you schedule repair. Treat it strictly as a bridge: a permanently power-capped GPU masking a paste problem will resurface later, and power capping is easy to forget. If you do this, track it as configuration drift.

Do not just restart the workload or reboot the node. The cascade is a physical equilibrium; it will re-form as soon as load returns.

Prevention

• Monitor the temperature spread, not just the maximum. The spread between the hottest and coolest GPU in a node is the earliest cascade indicator. Alert on the spread and on throttle reasons, not on absolute temperature thresholds.
• Trend the thermal baseline per GPU over months. A slowly rising baseline under constant workload indicates paste degradation or dust accumulation while there is still time to schedule maintenance calmly. Rule of thumb: keep peak workload temperature at least 10C below the throttle threshold.
• Watch inlet temperature and fan duty per node. Rising inlet temps or fans working harder for the same workload are rack-level leading indicators.
• Sample fast enough. Thermal excursions and throttle events can start and resolve in seconds. Minute-resolution sampling will miss the onset of a cascade; collect temperature, power, clocks, and throttle reasons at 10 seconds or faster.
• After any service that involves reseating cards or heatsinks, re-baseline. Compare the repaired unit’s temperature against its peers under load before returning the node to the production pool.
• In distributed training, treat asymmetric thermals as a straggler risk. One throttled GPU in a data-parallel job gates every collective. Cross-GPU comparison within the job catches the cascade from the workload side too.

How Netdata helps

Netdata shortens this diagnosis mostly by making the per-GPU comparison and the time correlation trivial:

• Per-second per-GPU temperature, power draw, and SM clocks on one screen, so the one-hot-GPU outlier and its downstream victims are visible at a glance rather than after a dozen nvidia-smi -i N queries.
• Clock throttle reasons collected alongside temperature and clocks, so you can confirm a hot GPU is actually throttling (sw_thermal_slowdown / hw_thermal_slowdown) rather than just warm.
• The throttled-but-still-drawing-power amplifier is directly visible: power draw staying near the enforced limit while SM clocks collapse.
• Long retention at high resolution, which is what makes the months-long thermal baseline drift (paste degradation, dust) visible before the first throttle event.
• Correlation with host-level metrics and hardware sensor data, so chassis inlet temperature and fan behaviour sit next to GPU temperatures when you separate local faults from environmental ones.
• In multi-GPU training, per-GPU side-by-side views expose the asymmetric pattern that turns one throttling GPU into a job-wide straggler.

Related guides

• NVIDIA GPU monitoring checklist: the signals every production GPU fleet needs
• How an NVIDIA GPU actually works in production: a mental model for operators
• NVIDIA GPU monitoring maturity model: from survival to expert
• NVIDIA GPU HBM progressive failure: from single-bit errors to a dead GPU
• NVIDIA GPU ECC errors: corrected, uncorrected, volatile, and aggregate