$ guides / nvidia-gpu / nvidia-gpu-sw-power-cap-throttling ▌

Operations Guides

NVIDIA GPU SW Power Cap: performance capped by the power limit

nvidia-smi reports SW Power Cap: Active under Clocks Event Reasons, the workload is running slower than it should, and the GPU is not hot. This is the power-limit throttle: the GPU’s power scaling algorithm is holding clocks below what the workload requested because the board is consuming as much power as it is allowed to.

Two things make this symptom confusing. First, SW Power Cap is not always a fault. A GPU at full load sitting exactly at its power limit is doing the maximum work its TDP allows, and datacenter operators often cap GPUs below the factory limit on purpose. Second, the number that actually matters is enforced.power.limit, not power.limit, and the two can disagree. Reading the wrong field sends you chasing a misconfiguration that does not exist, or missing one that does.

This guide covers how to read the power-limit fields correctly, decide whether the cap is intentional, and fix it when it is not.

What this means

Every NVIDIA GPU enforces a power ceiling. When the workload would draw more power than the ceiling allows, the driver reduces clocks until draw fits inside the budget. The workload keeps running, just slower. This is fundamentally different from thermal throttling: with sw_power_cap, temperature can be well inside limits. The GPU is not overheating; it is out of power budget.

There are four distinct limit values, and conflating them is the most common diagnostic error:

Field	Meaning
`power.draw`	Current board power consumption (on Ampere and later, a 1-second average)
`power.limit`	The configured ceiling, settable with `nvidia-smi -pl`
`power.default_limit`	The factory default for this board
`enforced.power.limit`	The ceiling actually being applied right now: the minimum of all limiters

enforced.power.limit is the ground truth. It folds in the software-set limit plus any external constraints (chassis, PSU, out-of-band management policy). On DGX H100/H200 systems there are three independent sources of power policy: the VBIOS (the maximum the board can do), nvidia-smi (host-set), and SMBPBI (set out-of-band via the BMC). The GPU’s power management unit applies the most conservative of them. A GPU can show power.limit at the factory default and still be capped lower because the BMC said so.

Form factor sets expectations. An H100 SXM is a 700W part; an H100 PCIe is a 350W part. A 350W draw pinned at the limit is expected behavior on the PCIe card and a serious anomaly on the SXM card.

flowchart TD
    A[sw_power_cap Active] --> B{power.draw near enforced.power.limit?}
    B -->|No| C[Idle or light load - usually benign]
    B -->|Yes| D{enforced.power.limit below power.default_limit?}
    D -->|Yes| E{Intentional cap?}
    E -->|Yes| F[Expected - verify against fleet policy]
    E -->|No| G[Find who set it: host, BMC, leftover maintenance]
    D -->|No| H[At factory TDP - normal under full load]
    H --> I{Performance acceptable?}
    I -->|No| J[Workload exceeds board budget - optimize or recheck expectations]

Common causes

Cause	What it looks like	First thing to check
Normal operation at full load	`power.draw` pinned at `enforced.power.limit`, clocks slightly below max, temp fine	Whether clocks and throughput are actually degraded
Intentional datacenter power capping	`power.limit` < `power.default_limit`, consistent across the rack or fleet	Fleet power policy, change records
Leftover limit from maintenance or testing	One GPU or one node capped below default while peers are not	Compare `power.limit` across identical nodes
Out-of-band cap (BMC/SMBPBI)	`enforced.power.limit` < `power.limit` even though `power.limit` is at default	BMC power policy on DGX/HGX systems
Limit lost and reset after reboot	Cap you set is gone; GPU back at default (or the reverse: a service reapplies an old cap)	systemd units or startup scripts that call `nvidia-smi -pl`
Driver regression reporting garbage	Power fields show N/A or implausible values	Driver version against known issues

Quick checks

All read-only and safe to run during production workloads.

# The four numbers that matter, side by side
nvidia-smi --query-gpu=power.draw,power.limit,power.default_limit,enforced.power.limit --format=csv

# Which throttle reasons are active right now
nvidia-smi --query-gpu=clocks_event_reasons.active --format=csv,noheader

# Individual throttle reason flags (useful when several bits are set)
nvidia-smi --query-gpu=clocks_event_reasons.sw_power_cap,clocks_event_reasons.sw_thermal_slowdown,clocks_event_reasons.hw_thermal_slowdown,clocks_event_reasons.hw_power_brake_slowdown --format=csv,noheader

# Current clocks vs max: how hard is it throttling
nvidia-smi --query-gpu=clocks.current.sm,clocks.max.sm,clocks.current.memory,clocks.max.memory --format=csv

# Rule out thermal throttling
nvidia-smi --query-gpu=temperature.gpu,temperature.memory,pstate,utilization.gpu --format=csv

# Full power readout (includes min/max settable range)
nvidia-smi -q -d POWER

Two interpretation rules before you conclude anything:

SW Power Cap active at idle or light load is usually benign. The cap is always enforced; the flag can assert at low draw. The real question is whether the GPU can reach its expected power draw and clocks under full load.
Sustained power.draw >= 95% of enforced.power.limit only means throttling when corroborated by sw_power_cap = Active or measurable clock/throughput degradation. Heavy compute near the limit with normal clocks is a healthy, well-fed GPU.

How to diagnose it

Confirm the throttle reason and rule out thermal. Check clocks_event_reasons.active and temperature.gpu. If temperature is near the model’s Max Operating Temp and sw_thermal_slowdown or hw_thermal_slowdown is active, you have a cooling problem, not a power problem. With a pure SW Power Cap, temperature is unremarkable.
Read the limit stack. Run the four-field power query. Compute the ratio power.draw / enforced.power.limit under load. If draw is pinned at the enforced ceiling and clocks are below max, power capping is confirmed as the cause of the slowdown.
Compare the enforced limit to the factory default. If enforced.power.limit < power.default_limit, something capped this GPU. Three suspects, in order: a host-set limit (power.limit will match the enforced value), an out-of-band cap from the BMC (power.limit looks normal but enforced.power.limit is lower), or a leftover setting from maintenance.
Compare across the fleet. Query the same fields on peer nodes of the same class. If every node in the rack is capped identically, this is almost certainly intentional rack power budgeting. If one node or one GPU diverges from its peers, suspect a local misconfiguration or leftover setting.
Check what reapplies limits at boot. A limit set with nvidia-smi -pl does not survive a driver reload or reboot; it resets to the default. Persistence is typically done with a systemd unit or startup script. That same mechanism is also how stale caps survive: look for units that call nvidia-smi -pl and check whether the value matches current policy.
Check form factor expectations before declaring a problem. Verify the GPU model (nvidia-smi --query-gpu=name --format=csv,noheader) and its expected TDP. 350W at the limit on an H100 PCIe is the card working as designed. 350W on an H100 SXM means half the power budget is missing.
Validate the reading itself. If any power field reports N/A or an implausible value, suspect a driver issue rather than a real cap. Driver 530.41.03 broke power-limit reporting (fields showing N/A) on some systems; the regression was fixed in 535.54.03. Separately, driver 530 and later removed the ability to change power limits on many laptop GPUs (the “Changing power management limit is not supported” error), which matters if you are trying to uncap a workstation card rather than a datacenter GPU.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`power.draw` vs `enforced.power.limit`	The ratio that tells you the cap is binding	Sustained ratio > 0.95 under load with `sw_power_cap` active
`enforced.power.limit`	The actual ceiling, including out-of-band constraints	Below `power.default_limit` without a policy explaining it
`power.limit` vs `power.default_limit`	Reveals host-set capping	Divergence from peers of the same node class
`clocks_event_reasons.sw_power_cap`	Confirms the throttle is power, not thermal	Active during production compute with degraded throughput
`clocks.current.sm` / `clocks.max.sm`	Quantifies throttle severity	Below ~80% of max during heavy load
`temperature.gpu`	Separates power cap from thermal throttle	Rising toward model-specific limits (different failure, different fix)
Node-level total GPU power vs PDU capacity	Fleet-level power budget	Aggregate GPU draw approaching circuit rating

Fixes

The cap is intentional and correct

Do nothing to the GPU. Continuous SW Power Cap at full load is the GPU doing maximum work within its budget. If the throttled performance is unacceptable, the fix is workload-side: improve efficiency, reduce batch power spikes, or revisit the rack power budget with the facilities team. Do not page on sw_power_cap alone in capped deployments; it is expected state.

A host-set limit is wrong or stale

Raise or reset it with nvidia-smi -pl:

# Set the power limit (requires root; value must be within the min/max range from nvidia-smi -q -d POWER)
sudo nvidia-smi -pl 700

Notes:

The value must be between the Min and Max Power Limit shown by nvidia-smi -q -d POWER. Values outside the range are rejected.
The setting does not survive a reboot or driver reload. If the correct state is a non-default limit, persist it via a systemd unit at boot; if the correct state is the default, remove any stale unit or script that reapplies an old cap.
On laptop and many workstation GPUs with driver 530 and later, this command fails with “not supported.” That is a known driver regression, not a syntax problem.
This changes live power policy on a production device. Coordinate with whoever owns the rack power budget before raising a cap; the limit may exist because the PDU cannot supply more.

The cap is enforced out-of-band

If enforced.power.limit is below power.limit on a DGX/HGX-class system, the host cannot override it. The BMC-side (SMBPBI) policy wins. Escalate to whoever manages the platform firmware and BMC configuration; nvidia-smi -pl will not fix this.

The GPU is at factory TDP and still too slow

The board is at its design limit. This is not a misconfiguration. Options are workload optimization, spreading work across more GPUs, or accepting the throughput. Check memory clock too: memory-bound workloads (LLM inference in particular) lose disproportionate performance when memory clocks are throttled.

Prevention

Baseline power policy per node class. Record the expected power.limit and enforced.power.limit for each class (training, inference, capped racks) and alert on drift. A cap that differs from peers is a finding even when it causes no immediate incident.
Treat power limits as configuration, managed and audited. Any nvidia-smi -pl in a systemd unit or startup script should be in version control with a reason attached. Most “mystery caps” are yesterday’s experiment that survived a reboot.
Alert on the corroborated condition, not the flag. Sustained power.draw >= 95% of enforced.power.limit AND sw_power_cap active AND clocks below expectation. The flag alone fires constantly in healthy, fully-loaded fleets.
After driver updates, verify power fields still report. Driver regressions have produced N/A power readings and broken limit-setting. A monitoring gap in power telemetry looks exactly like a healthy GPU at first glance.
Include power headroom in capacity planning. Aggregate peak GPU draw should stay comfortably below PDU circuit rating, or the “fix” for one node’s cap becomes the whole rack’s outage.

How Netdata helps

Power draw against the enforced limit, per GPU, per second. Power capping is a dynamic condition that minute-resolution sampling smooths away; per-second power.draw charted against enforced.power.limit shows exactly when the cap binds.
Throttle reasons as first-class metrics. clocks_event_reasons.sw_power_cap charted alongside thermal throttle reasons makes the power-vs-thermal distinction immediate, which is the first fork in this diagnosis.
Clock and throughput correlation. SM and memory clocks next to power draw and utilization show whether the cap is actually costing performance or just sitting at a healthy full-load equilibrium.
Configuration drift detection. Tracking power.limit, power.default_limit, and enforced.power.limit over time catches the leftover-maintenance cap and the post-reboot reset without anyone having to remember to check.
Cross-GPU comparison. Same-model GPUs on the same dashboard make the one oddly-capped GPU in a fleet obvious at a glance.