NVIDIA GPU HW Power Brake Slowdown: the chassis is cutting GPU power

You are looking at a GPU that is running, reachable, not overheating, and still delivering a fraction of its normal throughput. nvidia-smi -q -d PERFORMANCE shows HW Power Brake Slowdown : Active (and usually HW Slowdown : Active alongside it), SM clocks are at half their rated speed or lower, and the temperature looks fine. The workload did not change. The GPU did not change. Something outside the GPU decided it gets less power.

This is the throttle reason that sends teams in the wrong direction. Unlike SW Power Cap or thermal slowdown, HW Power Brake Slowdown is not a GPU-internal decision. It is an external signal asserted by the baseboard or chassis management controller when the platform’s power budget is exceeded. GPU telemetry will tell you that it happened. It will not tell you why. The why lives in the BMC event log, the PSU inventory, and the PDU.

Brief transients are normal. A training step’s power spike, checkpoint saves, or model loading can momentarily assert the brake on cards with aggressive board power limits. Sustained or recurrent power brake across one or more GPUs is a power-infrastructure problem, and on a multi-GPU node it can be a PDU overload in progress.

What this means

The power brake is a hardware signal path from the chassis to the GPU. When the baseboard management controller (BMC, iDRAC, XCC, iLO depending on vendor) determines the platform power budget is exceeded, or when a PSU reports a warning condition, it asserts the brake. The GPU firmware reacts immediately and independently of the driver: core clocks drop by 2x or more. The driver then reports the condition through the throttle reason flags.

Three properties define this failure mode:

• The GPU is the victim, not the cause. Temperatures can be normal, ECC clean, PCIe link healthy. The GPU is doing exactly what the platform told it to do.
• DCGM will not show you the cause. DCGM surfaces this as HW throttling. The reason for the assertion exists only in chassis management telemetry: BMC system event logs, PSU status, power supply redundancy state, PDU load.
• It can hit multiple GPUs simultaneously. A chassis-level or PDU-level budget breach asserts the brake on every GPU in the affected power domain at once. If two or more GPUs in the same node show HW Power Brake Slowdown in the same time window, stop looking at individual GPUs and look at the power feed.

HW Slowdown showing Active at the same time is expected. HW Slowdown is the composite parent flag; it is active whenever HW Thermal Slowdown or HW Power Brake Slowdown is active. The distinguishing check is temperature: if the die is well below its slowdown threshold and hw_power_brake_slowdown is the active reason, this is power, not heat.

flowchart TD
  A[PSU warning or budget breach] --> C[BMC asserts power brake]
  B[PDU or circuit overload] --> C
  D[Unsupported GPU power table entry] --> C
  E[Chassis firmware bug] --> C
  C --> F[GPU cuts core clocks 2x or more]
  F --> G[nvidia-smi shows HW Power Brake Slowdown active]
  C --> H[Evidence in BMC event log and PSU status]

Common causes

Quick checks

All read-only. Run per-GPU with -i N on multi-GPU nodes so one slow GPU does not hang the query.

# Which throttle reasons are active right now
nvidia-smi --query-gpu=clocks_event_reasons.active --format=csv,noheader

# The specific reasons that matter here
nvidia-smi --query-gpu=clocks_event_reasons.hw_power_brake_slowdown,clocks_event_reasons.hw_thermal_slowdown,clocks_event_reasons.sw_power_cap,clocks_event_reasons.hw_slowdown --format=csv,noheader

# Human-readable performance section
nvidia-smi -q -d PERFORMANCE

# Clocks: how deep is the cut
nvidia-smi --query-gpu=clocks.current.sm,clocks.max.sm,clocks.current.memory,clocks.max.memory --format=csv,noheader,nounits

# Power: what is the GPU drawing vs what is enforced
nvidia-smi --query-gpu=power.draw,power.limit,power.default_limit,enforced.power.limit --format=csv,noheader,nounits

# Temperature: rule out thermal as the real cause
nvidia-smi --query-gpu=temperature.gpu,temperature.memory --format=csv,noheader,nounits

# Watch it live during a workload burst
watch -n 1 'nvidia-smi --query-gpu=temperature.gpu,power.draw,clocks.sm,clocks_event_reasons.active --format=csv,noheader'

Then leave the GPU alone and go to the chassis layer:

# BMC system event log (vendor-neutral where IPMI is available)
ipmitool sel list | tail -50

# PSU and power sensors
ipmitool sensor | grep -i "power\|psu\|watt"

On vendor platforms use the native interface instead: iDRAC on Dell, XClarity Controller (XCC) on Lenovo, iLO on HPE. On Lenovo ThinkSystem, for example, a real assertion appears in the XCC event log as a PCIe power brake event for the affected slots. If the BMC log is silent while nvidia-smi claims the brake is active, you are dealing with a reporting bug, not a power event.

How to diagnose it

1. Confirm the reason and the depth of the cut. clocks_event_reasons.hw_power_brake_slowdown = Active plus clocks.current.sm at roughly half of clocks.max.sm or less is the signature. hw_slowdown will also read active; that is the parent flag, not a second problem.

2. Rule out thermal. Check temperature.gpu against the GPU’s slowdown threshold (nvidia-smi -q reports thresholds on the device) and check whether hw_thermal_slowdown is also active. Normal temperature with power brake active means this is a power event, full stop.

3. Check the blast radius. Query every GPU in the node. One GPU braking while others are clean points toward that GPU’s slot, riser, or power cabling. Several GPUs braking in the same window points at the chassis power budget or the PDU feed. If multiple nodes on the same PDU brake together, you have your answer.

4. Check the BMC event log for the same timestamp. A genuine assertion leaves evidence at the chassis layer: PSU warnings, redundancy loss, power budget events, PCIe power brake entries. This is the authoritative record. Correlate its timestamps with when the throttle reason went active.

5. Check PSU inventory and redundancy. A failed or derated PSU shifts the load to the survivors. Under peak GPU draw the remaining capacity may not cover the platform, and the BMC asserts the brake to protect the hardware. This is the most common sustained cause.

6. Check power draw vs budget. Sum power.draw across all GPUs under peak workload and compare against the node’s power supply capacity and the PDU circuit rating. Sustained total draw above roughly 80% of the circuit rating, once CPU, storage, networking, and cooling are included, is the overload scenario.

7. Rule out reporting artifacts before scheduling hardware work. Two documented cases matter. Lenovo has documented an H100 reporting bug where nvidia-smi shows the power brake active continuously when it is not, and a separate driver bug (fixed in 2023 driver releases) where nvidia-smi failed to report a brake the chassis confirmed had fired. Dell has documented false HW Power Brake Slowdown on the XE8640 platform resolved by BIOS, CPLD, and iDRAC firmware updates. If nvidia-smi and the BMC log disagree, trust the BMC log and pursue firmware.

8. Assess workload impact. Check application-level throughput (training step time, inference latency) for the affected window. A brake that fires for a single sample during a power spike and clears is expected. A brake that is sustained, or recurs every step, with measurable throughput loss, is the incident.

Metrics and signals to monitor

Alerting guidance: a single transient assertion with no throughput impact is not actionable. Open a ticket when the brake is sustained or recurrent across multiple samples and corroborated by workload impact. Escalate to page-level urgency when several GPUs on the same node or feed brake together, because that is a PDU or chassis power event, not a GPU event.

Fixes

PSU degradation or lost redundancy

Replace the failed or derated PSU and restore redundancy. Until then, expect the brake under peak load. If you must keep running degraded, reducing GPU power limits (power.limit below default, set deliberately and documented) lowers the peak the surviving PSUs must cover. Tradeoff: you are capping performance to match damaged infrastructure; treat it as a bridge, not a resolution.

PDU or circuit overload

Rebalance load across circuits or PDUs, or cap GPU power limits to bring aggregate node draw under the circuit’s safe operating level (below ~80% of rating with all node consumers included). Do not just raise the GPU power limit back after a breaker trip; the budget exists because the feed cannot supply more.

Unsupported GPU in the chassis

If the BMC cannot find the GPU’s power table entry (vendor event logs show the lookup failure explicitly), the only clean fixes are moving the GPU to a platform that supports it or checking whether the vendor has added support in a newer BMC firmware. Running unsupported cards with the brake masked is running without the platform’s overload protection.

Firmware and reporting bugs

Update BIOS, CPLD, and BMC firmware per the vendor advisory for your platform when the chassis log and nvidia-smi disagree. Some platforms also expose PCIe power brake policy in the BIOS. Lenovo, for example, documents Reactive, Proactive, and Disabled modes on certain systems, where Proactive throttles when total GPU power exceeds a threshold derived from active PSU wattage. Changing the policy to Disabled removes a protection mechanism; do it only as a vendor-endorsed diagnostic step, never as the permanent fix.

Transient spikes

No fix needed. If the brake fires briefly during step power ramps and throughput is unaffected, this is the power management system working as designed. Record the baseline so future regressions are visible.

Prevention

• Track the counter, not just the flag. The cumulative clocks_event_reasons_counters values let you trend time-spent-braked per GPU per week. A rising trend is a power infrastructure problem developing, visible before users notice.
• Baseline aggregate node power under peak workload and compare it against PSU capacity and PDU rating whenever hardware or workload mix changes. New model architectures change power draw profiles; a fleet that was comfortable last quarter may not be after the next training framework upgrade.
• Monitor PSU redundancy as a first-class signal. Lost redundancy with zero workload impact today becomes sustained power brake the day a big job lands.
• Correlate BMC events with GPU telemetry in one place. The diagnosis above fails when the two data sources live in different tools with different clocks.
• Keep chassis firmware current on GPU platforms, and verify GPU SKUs against the server’s supported-hardware list before racking new cards.

How Netdata helps

• Per-second throttle-reason visibility. HW Power Brake events can last seconds. Netdata’s per-second collection of clocks_event_reasons.* catches assertions that minute-resolution polling misses entirely, and the counter variants show cumulative time spent braked.
• Clock, power, and temperature on one timeline. Seeing clocks.current.sm collapse while temperature stays flat and aggregate power draw spikes is the three-signal correlation that separates power brake from thermal throttling in one glance.
• Multi-GPU comparison. Charting throttle reasons across all GPUs in a node makes the blast-radius question (one slot vs whole chassis vs whole PDU) immediate instead of a manual per-GPU query loop.
• Workload correlation. Overlaying application throughput (step time, inference latency) against brake windows tells you whether you are looking at a benign transient or a real performance incident.
• Anomaly detection on power draw. ML-based anomaly flags on per-GPU and aggregate power draw surface the slow drift toward PDU capacity before the brake starts firing.

Related guides

• NVIDIA BAR1 memory exhaustion: mapping failures with free framebuffer
• NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver
• CUDA out of memory with free memory available: GPU memory fragmentation
• CUDA out of memory: diagnosing NVIDIA GPU framebuffer exhaustion
• NVIDIA GPU ECC disabled: the silent data-corruption risk
• NVIDIA GPU ECC errors: corrected, uncorrected, volatile, and aggregate
• NVIDIA GPU fan at 0%: fan failure on air-cooled cards
• NVIDIA GPU HBM (memory) temperature: the thermal limit most teams miss
• NVIDIA GPU HBM progressive failure: from single-bit errors to a dead GPU
• How an NVIDIA GPU actually works in production: a mental model for operators
• NVIDIA GPU memory leak: framebuffer usage climbing without a plateau
• Reading NVIDIA GPU memory correctly: nvidia-smi vs the PyTorch caching allocator