$ guides / nvidia-gpu / nvidia-gpu-throttle-reasons ▌

Operations Guides

NVIDIA GPU clock throttle reasons: why the GPU isn't running at full speed

A training job that used to finish an epoch in 40 minutes now takes 90. nvidia-smi shows 100% GPU utilization, memory usage looks normal, and the workload is running, just slowly. This is the classic throttling symptom: the GPU is executing kernels at reduced clock speeds, so everything takes longer while utilization stays high.

The mistake most teams make here is guessing. They check temperature, see 82C, and debate whether that is “too hot.” They see power draw pinned at the limit. None of that says why the clocks are down. The GPU already knows, and it reports the answer directly through the clock event reasons bitmask.

This guide covers reading that bitmask correctly, quantifying the performance loss, and fixing the specific cause.

What this means

Every NVIDIA GPU continuously balances performance against power draw, temperature, and administrative clock settings. When a limiter engages, the driver reduces SM or memory clocks and sets the corresponding bit in the clock event reasons bitmask, exposed as clocks_event_reasons.active in nvidia-smi and as DCGM field 112 (DCGM_FI_DEV_CLOCKS_EVENT_REASONS).

Two properties of this signal matter operationally:

It is a bitmask. Multiple reasons can be active at once; the most restrictive limiter wins. Never compare the raw value for equality against a single reason. Test individual bits.
Some reasons are normal. gpu_idle and applications_clocks_setting are expected states. Any hw_* reason active during production compute on a datacenter GPU is abnormal.

This one query replaces hours of correlating temperature, power, and clocks by hand. You read the cause directly.

flowchart TD
  A[Workload slow, clocks below max] --> B{Query clocks_event_reasons.active}
  B --> C[gpu_idle only: workload not feeding GPU]
  B --> D[sw_power_cap: hitting power limit]
  B --> E[sw/hw_thermal_slowdown: cooling problem]
  B --> F[hw_power_brake_slowdown: chassis/PSU power event]
  B --> G[sync_boost: another GPU in group is slower]
  B --> H[applications_clocks_setting: admin/app clock cap]
  D --> I{power.limit below default?}
  I -->|yes| J[Intentional cap or misconfiguration]
  I -->|no| K[Workload genuinely power-bound]
  E --> L{One GPU or all GPUs hot?}
  L -->|one| M[Local cooling, paste, airflow]
  L -->|all| N[Ambient / HVAC issue]

Common causes

Cause	What it looks like	First thing to check
Power limit reached (`sw_power_cap`)	power.draw pinned at enforced.power.limit, clocks reduced, temperature moderate	Is power.limit below power.default_limit (intentional cap)?
Thermal throttling (`sw_thermal_slowdown`, `hw_thermal_slowdown`)	temperature.gpu near or above Max Operating Temp, clocks dropping, fans at max	One hot GPU or all GPUs? Local cooling vs HVAC
Chassis power brake (`hw_power_brake_slowdown`)	Aggressive clock cut, may be brief, often coincides with load spikes across the node	PSU/PDU capacity and recurrence pattern
Application clocks set (`applications_clocks_setting`)	Clocks capped at a fixed value below max, consistent across workloads	Was `nvidia-smi -ac` or equivalent set intentionally?
SyncBoost (`sync_boost`)	One or more GPUs held at lower clocks in a multi-GPU job	Throttle reasons on the other GPUs in the sync group
Workload not feeding the GPU (`gpu_idle` during “active” job)	Low clocks, low power, utilization gaps	Host CPU, data pipeline, kernel launch gaps
HBM thermal limit	Memory-bound workload slow, die temperature fine	temperature.memory (datacenter GPUs only)

Quick checks

All read-only and safe to run on production nodes.

# The single most important query: which throttle reasons are active right now
nvidia-smi --query-gpu=clocks_event_reasons.active --format=csv,noheader

# All individual reasons as Active/Not Active flags
nvidia-smi --query-gpu=clocks_event_reasons.gpu_idle,clocks_event_reasons.applications_clocks_setting,clocks_event_reasons.sw_power_cap,clocks_event_reasons.hw_slowdown,clocks_event_reasons.hw_thermal_slowdown,clocks_event_reasons.hw_power_brake_slowdown,clocks_event_reasons.sw_thermal_slowdown,clocks_event_reasons.sync_boost --format=csv,noheader

# Quantify the impact: current clocks vs maximum
nvidia-smi --query-gpu=clocks.current.sm,clocks.max.sm,clocks.current.memory,clocks.max.memory --format=csv,noheader,nounits

# Power state: is the draw pinned at the enforced limit
nvidia-smi --query-gpu=power.draw,power.limit,power.default_limit,enforced.power.limit --format=csv,noheader,nounits

# Thermal state: die and HBM (memory temp returns N/A on consumer cards)
nvidia-smi --query-gpu=temperature.gpu,temperature.memory --format=csv,noheader,nounits

# How long the GPU has spent in each throttled state (cumulative counters)
nvidia-smi --query-gpu=clocks_event_reasons_counters.sw_power_cap,clocks_event_reasons_counters.hw_thermal_slowdown --format=csv

# Same signal via DCGM field 112, if you run DCGM
dcgmi dmon -e 112 -c 5 -d 1000

Notes on interpretation:

clocks_throttle_reasons.* is the older field name; clocks_event_reasons.* is current. Both aliases work.
Verify field availability on your driver branch with nvidia-smi --help-query-gpu before scripting against it.
On DCGM 3.x, DCGM_FI_DEV_CLOCK_THROTTLE_REASONS is deprecated; use DCGM_FI_DEV_CLOCKS_EVENT_REASONS. Both resolve to field ID 112.

How to diagnose it

Confirm throttling is actually happening. Compare clocks.current.sm to clocks.max.sm while the workload is running. Below roughly 80% of max under sustained load means significant throttling. Idle GPUs run at low clocks normally, so only judge clocks during active compute.
Read the reason, not the symptoms. Query clocks_event_reasons.active per GPU (-i N on multi-GPU nodes). Treat the result as a bitmask and test each bit; the named boolean fields (clocks_event_reasons.hw_thermal_slowdown and friends) do this for you.
Classify the reason:
- gpu_idle or applications_clocks_setting alone: normal or configured. If gpu_idle appears during a job that should be compute-bound, the GPU is starved, not throttled. Look upstream at the host CPU and data pipeline.
- sw_power_cap: the GPU is at its power limit. Check whether power.limit < power.default_limit (someone capped it) or the workload is simply power-bound at the stock limit.
- sw_thermal_slowdown / hw_thermal_slowdown: cooling is not keeping up. hw_ means the aggressive clock cut (2x or more) is active. One hot GPU points at local airflow, thermal paste, or fan; all GPUs hot points at ambient or HVAC.
- hw_power_brake_slowdown: an external chassis-level power brake asserted. Brief spikes during checkpoint saves or model loading can occur on cards with aggressive board power limits; sustained or recurrent activation with real throughput impact points at PSU or PDU capacity.
- sync_boost: this GPU is held back by a slower GPU in its sync group. Check throttle reasons on the other GPUs to find the actual limiter.
Quantify the cost. The ratio clocks.current.sm / clocks.max.sm during load is a rough performance multiplier. A GPU at 50% of max SM clock delivers roughly half the throughput regardless of what utilization says. Use the clocks_event_reasons_counters.* cumulative counters to measure how much wall-clock time the GPU has spent in each state.
Corroborate before acting. For power: confirm power.draw is sustained at or above about 95% of enforced.power.limit. For thermal: confirm temperature.gpu (and temperature.memory for memory-bound work on HBM cards) is near the model-specific thresholds from nvidia-smi -q -d TEMPERATURE. Throttle reason plus corroborating signal is a confirmed diagnosis; throttle reason alone is a lead.
Filter the known false positives. A brief hw_slowdown blip during PState or clock transitions (workload start, GPU reset, mode change) is normal driver behavior and self-corrects. Only act on reasons that persist during steady-state compute.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
clocks_event_reasons.active (per reason)	The direct cause of any clock reduction	Any hw_* reason Active during production compute for >60s
clocks.current.sm vs clocks.max.sm	Quantifies throughput loss	<80% of max under sustained load
clocks.current.memory vs clocks.max.memory	Memory clock cuts hit memory-bound workloads (LLM inference) hardest	Reduced memory clock during inference serving
power.draw vs enforced.power.limit	Confirms power capping	Sustained ratio >0.95 with sw_power_cap Active
temperature.gpu	Leading indicator for thermal throttling	Sustained within 5C of Max Operating Temp
temperature.memory	HBM has its own thermal limit, independent of the die	Near model-specific memory limit during memory-bound work
clocks_event_reasons_counters.*	Cumulative time in each throttled state	Growing hw_* counters day over day
pstate	Effect of throttling at a glance	Above P0 during sustained heavy compute

Severity guidance: hw_thermal_slowdown sustained during production compute pages. hw_power_brake_slowdown is a ticket unless it is sustained or recurrent with corroborating workload impact, in which case it pages. sw_power_cap and sw_thermal_slowdown are tickets. gpu_idle, applications_clocks_setting, and sync_boost are informational. Do not page on sw_power_cap in fleets where power capping is intentional datacenter policy.

Fixes

Power capping (sw_power_cap)

If power.limit < power.default_limit and nobody owns that decision, restore the default or get the cap documented as policy. Check with the infrastructure team first; rack-level power budgeting is a legitimate reason.
If the GPU is at its stock limit under heavy load, the hardware is working as designed. Raising power.limit toward power.max_limit is possible on some models but increases thermal load and may trip chassis-level limits. Treat it as a capacity decision, not a fix.
Longer term, reduce the workload’s power intensity: lower-precision compute, better kernel efficiency, or batching changes.

Thermal throttling (sw/hw_thermal_slowdown)

One hot GPU: inspect that card’s local cooling. Blocked airflow, failed fan (air-cooled cards), degraded thermal paste, or heat soak from an NVLink-adjacent neighbor.
All GPUs hot: environmental problem. Check intake temperature, HVAC, and rack airflow. There is no GPU-side fix for hot aisles.
Do not read temperature.gpu.tlimit as an absolute threshold; it is the margin in degrees remaining.

Chassis power brake (hw_power_brake_slowdown)

This is the PSU or chassis telling the GPU to back off. Check PSU redundancy status, PDU load, and whether multiple GPUs spike simultaneously (checkpoint saves, synchronized all-reduce bursts).
Recurring activation means the node’s power delivery cannot sustain the workload’s peak draw. That is a hardware capacity issue, not a GPU tuning issue.

Application clocks (applications_clocks_setting)

Someone set fixed clocks with nvidia-smi -ac or an equivalent API call, or an application requested them. If unintentional, reset to default clocks: nvidia-smi -rgc resets GPU clocks and nvidia-smi -rmc resets memory clocks (both supported on Volta+). Note that -ac itself is deprecated in favor of -lgc/-lmc for locking clocks; -rac resets application clocks.
Changing clocks on a production GPU mid-job is disruptive and can crash running contexts; schedule it.

SyncBoost (sync_boost)

The throttle reason on the affected GPU is caused by a peer. Find the slowest GPU in the group and diagnose its throttle reasons. Fixing the straggler fixes the group.

Starved GPU (gpu_idle during active work)

This is not a GPU fault. Profile the host: CPU-bound preprocessing, slow storage, too few data loader workers, NUMA-remote memory. See the host-starved GPU pattern in how an NVIDIA GPU actually works in production.

Prevention

Monitor the reason, not just the temperature. Alerting on “>80C” produces false positives and misses power and chassis events. Alert on hw_* throttle reasons sustained >60s during active compute, with temperature as context.
Baseline clocks per workload. Record the normal clocks.current.sm under load for each workload class. Deviations stand out immediately.
Track cumulative throttle counters. clocks_event_reasons_counters.* growth over weeks reveals slow cooling degradation (dust, paste aging) before it becomes an incident.
Document intentional caps. Power limits and application clocks that exist on purpose should be written down per node class so they stop generating mystery incidents.
Sample fast enough. Throttle events can start and resolve in seconds. Sub-10-second sampling is the floor for this signal; minute-resolution monitoring misses short events entirely.
Cross-check the fleet. In multi-GPU and distributed jobs, compare throttle reasons across GPUs. One throttled GPU is a straggler that gates the whole collective.

How Netdata helps

Netdata collects per-GPU clock event reasons, current vs max SM and memory clocks, power draw vs enforced limit, and die and memory temperatures at per-second resolution, fast enough to catch short throttle events.
With these signals on one timeline, the causal order is visible directly: temperature rising first, then sw_thermal_slowdown, then clocks dropping, then power falling as a consequence.
Cross-GPU views make the straggler case obvious: one node with hw_thermal_slowdown active while its peers run clean, or a whole rack showing sw_power_cap at the same moment (a power budget event, not a cooling event).
Anomaly detection on clock speed and temperature flags slow baseline drift, such as a GPU whose peak-load temperature creeps up month over month as cooling degrades.
Alerts can key off sustained hw_* reasons rather than raw temperature, which removes the most common source of thermal false positives.