Most GPU fleets monitor one temperature: temperature.gpu, the die temperature. Alerts, dashboards, and throttle runbooks are built around it. On datacenter GPUs with HBM (A100, H100, H200), that is only half the picture, and for memory-bound work it is often the wrong half.

The HBM stacks sit on the same package as the die but have a worse heat path. Under LLM inference, embedding-heavy workloads, and memory-bound training, HBM temperature commonly runs 10 to 20 C hotter than the die. HBM also has its own throttle point. When it trips, the GPU reduces memory bandwidth with no Xid, no error, and no log entry most teams watch. The job keeps running. It just gets slower while die temperature still looks healthy.

Sustained high HBM temperature also raises DRAM leakage, which shows up later as higher correctable ECC error rates, row remapping, page retirement, and wear-out. It is both a performance signal and a reliability signal.

What HBM temperature is and why it matters

Datacenter GPUs use High Bandwidth Memory: stacked DRAM dies next to the GPU die on an interposer, connected through dense vertical interconnect. The stacking is what delivers bandwidth, but heat generated inside the stack has a poor path out. HBM temperature can therefore stay high even when fans, cold plates, and die temperature look normal.

Two consequences matter operationally:

  1. HBM has an independent throttle threshold. The die has the familiar cascade: software slowdown near max operating temperature, stronger hardware slowdown above that, then shutdown. The memory subsystem has its own limit, and the GPU can cut memory clocks and bandwidth while the die remains below its own threshold. On H100-class parts, operators commonly see die limits in the low 80s C and HBM slowdown points in the mid-90s C, but read the limits from the hardware rather than hardcoding them.

  2. Heat ages DRAM. Higher operating temperature increases charge leakage in DRAM cells. The early symptom is usually a rising correctable single-bit ECC error rate; the later symptoms are row remaps, page retirement, and reduced spare capacity. A GPU that lives at 90+ C HBM is consuming error-correction headroom. See NVIDIA GPU HBM progressive failure for that failure arc.

The broader thermal and power model is in how an NVIDIA GPU actually works in production. This article stays on the memory temperature signal.

Why HBM throttling is invisible

Teams miss this because the throttle event produces none of the usual artifacts:

  • No Xid. Memory thermal throttling does not log an Xid to dmesg. Xid-centric alerting stays quiet.
  • No die-temperature excursion. The die can remain under its own limit, so die-temperature alerts do not fire.
  • Utilization stays high. SM utilization is time-based. Memory-bound kernels stalled on throttled HBM still count as busy, so utilization.gpu can read 100% while delivered bandwidth and application throughput fall.
flowchart TD
  A[Memory-bound workload
LLM inference, embedding lookups] --> B[HBM temp rises
10-20C above die temp]
  B --> C{HBM temp vs memory
max operating temp}
  C -- below --> D[Full memory bandwidth]
  C -- above --> E[Memory clock reduced
bandwidth throttled]
  E --> F[Throughput drops
utilization.gpu still 100%]
  E --> G[No Xid, no error logged]
  B -. sustained high temp .-> H[Higher charge leakage
accelerating SBE rate]
  H --> I[Row remapping / page retirement
long-term wear-out]

The useful indicators are memory temperature itself, memory clock versus max memory clock, thermal violation counters, and the downstream divergence between high controller utilization and falling delivered throughput.

Reading HBM temperature

nvidia-smi

# Check die and memory temperature together
nvidia-smi --query-gpu=temperature.gpu,temperature.memory --format=csv,noheader,nounits

On supported datacenter GPUs this returns both values. Caveats: temperature.memory returns N/A on consumer GPUs with GDDR, and on some driver or GPU combinations it returns N/A even on datacenter parts. If you get N/A on an A100 or H100, try DCGM before assuming there is no sensor. V100 generally does not expose a memory temperature sensor.

DCGM

DCGM is the better path on datacenter hardware and the one to use for continuous monitoring. Field names and availability vary by DCGM version, so verify against your installed dcgmi or exporter config.

Commonly used fields:

FieldWhat it reports
DCGM_FI_DEV_MEMORY_TEMPHBM or memory temperature, Celsius
DCGM_FI_DEV_GPU_TEMPGPU die temperature, Celsius
DCGM_FI_DEV_MEMORY_MAX_OP_TEMPMemory max operating temperature; slowdown occurs above this
DCGM_FI_DEV_GPU_MAX_OP_TEMPGPU die max operating temperature
DCGM_FI_DEV_THERMAL_VIOLATIONCumulative time spent in thermal violation
DCGM_FI_DEV_CLOCKS_EVENT_REASON_SW_THERM_SLOWDOWN_NSCumulative software thermal slowdown time; combines die and memory constraints
DCGM_FI_DEV_CLOCKS_EVENT_REASON_HW_THERM_SLOWDOWN_NSCumulative aggressive hardware thermal slowdown time
# Sample die and memory temperature, 5 samples at 1s
dcgmi dmon -e 140,150 -c 5 -d 1000

# Read the per-SKU thermal limits from the hardware
dcgmi dmon -e 151,152 -c 1

# Check cumulative thermal violation and thermal slowdown counters
dcgmi dmon -e 241,1422,1423 -c 1

Treat 241, 1422, and 1423 as cumulative counters. Track deltas, not absolute values. A rising delta on the software thermal slowdown counter while memory temperature sits near the memory max operating temperature is strong evidence of memory thermal throttling, even if nothing else in the system says so.

Field 1422 is not memory-only. If it is accumulating while die temperature is far from the die limit, memory is the likely constraint by elimination. Confirm with memory clock and workload behavior.

dcgm-exporter

If you run dcgm-exporter for Prometheus, the default metric set often includes memory temperature and GPU temperature, but dashboards frequently chart only the die sensor. Check whether memory temperature is already collected and simply not alerted on. Add the max-operating-temperature fields to the exporter config so alerts can be relative to the hardware-reported limit instead of a hardcoded number.

Version gotchas

  • Sensor exposure depends on GPU generation, driver branch, NVML, and DCGM version. Test on each fleet image.
  • Blank or N/A memory temperature values can be a tooling issue rather than proof that the GPU lacks a sensor.
  • DCGM field names have changed across releases. Do not copy field IDs or names between major DCGM versions without checking the local documentation and exporter output.
  • If your DCGM version supports thermal health watches, use them as a backstop, but still alert on temperature margin and violation-counter deltas.

Thresholds by SKU

NVIDIA does not consistently publish datacenter GPU thermal specifications for operators; OEMs often receive them as system integrators. The programmatic source of truth is the max-operating-temperature fields read from the GPU. Community-reported values below are useful for sanity checks only.

GPUHBM typeDie max op tempHBM slowdown pointMemory temp sensor
H100HBM3~83 C class~95 C classUsually yes via DCGM
A100HBM2e~83 C class~95 C classUsually yes via DCGM
H200HBM3e~83 C classReported lower on some partsUsually yes via DCGM
V100HBM2~83 C classNot readable on many stacksGenerally no

Operational rules:

Alert relative to the hardware-reported memory limit. Read the memory max operating temperature per GPU and alert when memory temperature approaches it. A ticket threshold around 10 C below the limit is reasonable; at 5 C below, one warm aisle event, failed fan, or clogged filter can put the GPU into throttle.

Do not reuse die thresholds for memory. A “die temp > 80 C” style alert is miscalibrated for HBM. The die and memory limits differ, and memory temperature can move quickly under memory-bound load. The delta between die and HBM is often the tell.

Also check OEM BMC views carefully. BMC sensor thresholds have shipped with vendor-specific quirks, and a BMC label may not match what DCGM reports. If BMC and DCGM disagree, trust DCGM for paging and investigate the OEM mapping before wiring BMC thermal sensors into alerts.

What silent HBM throttling looks like in production

The symptom is a memory-bound workload that gets slower with no matching fault. Use this sequence:

# 1. Compare die vs memory temperature under load
nvidia-smi --query-gpu=temperature.gpu,temperature.memory --format=csv,noheader,nounits

# 2. Compare memory clock against its maximum
nvidia-smi --query-gpu=clocks.current.memory,clocks.max.memory --format=csv,noheader,nounits

# 3. Check which throttle reasons are active
nvidia-smi --query-gpu=clocks_event_reasons.active --format=csv,noheader

# 4. Check memory controller utilization: is the workload actually memory-bound?
nvidia-smi --query-gpu=utilization.memory --format=csv,noheader,nounits

The signature is: temperature.memory near the memory limit, clocks.current.memory below clocks.max.memory, high utilization.memory, high utilization.gpu, and no Xids in dmesg. If clocks_event_reasons.active shows sw_thermal_slowdown while the die is comfortably below its limit, suspect memory thermal throttling and confirm with DCGM deltas. nvidia-smi throttle reasons alone do not cleanly separate die-thermal from memory-thermal causes.

Separate this from power capping and die thermal throttling, which share the “busy but slow” shape:

  • Power cap: sw_power_cap active, power draw pinned near the enforced limit, temperatures moderate.
  • Die thermal: die temperature near die limit, thermal slowdown active, memory may or may not be hot.
  • Memory thermal: memory temperature near memory limit, memory clock depressed, die below its own limit.

For the long-term dimension, correlate HBM temperature history with ECC. A GPU that has run hot on HBM and later shows an accelerating correctable error rate is moving along the progressive-failure path. See NVIDIA GPU ECC errors: corrected, uncorrected, volatile, and aggregate.

Signals to watch in production

SignalWhy it mattersWarning sign
HBM or memory temperatureThe actual limiter on memory-bound workloadsWithin 10 C of memory max op temp; rising week over week
Memory max op tempHardware-reported denominator for sane alertsMissing from monitoring; alerts hardcoded to die thresholds
Die temp minus memory tempShows which subsystem is closer to its constraintDelta widening over months on the same workload
Thermal violation counterCumulative record of time over thermal limitsDelta above zero during production windows
SW/HW thermal slowdown countersConfirms throttling occurred and how muchRising deltas aligned with throughput regressions
clocks.current.memory vs clocks.max.memoryDirect evidence of memory clock reductionSustained gap under memory-bound load
utilization.memory plus application throughputSeparates throttled from idleHigh controller utilization with declining delivered throughput
ECC corrected error rateWear-out readout for hot HBMRate acceleration after periods of high memory temp
Thermal rise rate under constant loadLeading indicator of cooling degradationSame workload reaches higher memory temp than last month

How Netdata helps

  • Where NVIDIA GPU metrics are enabled, Netdata’s per-second collection is useful because HBM thermal excursions and recovery cycles can complete inside a one-minute scrape interval.
  • Charting die temperature and memory temperature together makes the usual 10 to 20 C delta visible and shows which subsystem is riding its limit when throughput drops.
  • Correlating memory temperature with memory clocks, memory controller utilization, power draw, and throttle reasons on one timeline separates memory thermal throttling from power capping and die thermal throttling without manual cross-referencing.
  • Tracking ECC counters beside HBM temperature history connects the immediate performance symptom to the long-term reliability cost of running memory hot.
  • Because memory thermal throttling produces no Xid and no log event, metric-based alerting on memory temperature relative to the hardware limit is the proactive detection path. Build that alert on memory temperature margin, thermal violation deltas, and memory clock depression, not on die temperature alone.