$ guides / nvidia-gpu / nvidia-gpu-utilization-misleading ▌

Operations Guides

Why NVIDIA GPU utilization is misleading: 100% doesn't mean saturated

A GPU dashboard pinned at 100% looks like proof that the hardware is fully used. It is not. During an incident, that distinction matters: a job can miss its throughput target while every utilization chart says the GPU is completely busy.

NVIDIA’s common GPU utilization metric is a time-domain measurement. It reports the fraction of a sampling period during which at least one kernel was executing. It does not report how many streaming multiprocessors were active, how many warps were resident, whether tensor cores were used, or how much memory bandwidth was consumed.

On a GPU with 108 SMs, one busy SM and 107 idle SMs can still produce 100% utilization if a kernel remained active through the sampling window. Conversely, a well-optimized workload reporting 60% utilization can complete more useful work than a poorly structured workload reporting 100%.

What this means

Treat utilization.gpu and DCGM field 203 as “the GPU was busy for this much time,” not “the GPU’s capacity was fully consumed.” The sample period is approximately one second on many products and can be shorter on others. Short kernels can also fall between samples and produce misleadingly low readings.

This creates two common operational errors:

False confidence: Utilization is 100%, so the team assumes the GPU is saturated and looks for more hardware.
False blame: Utilization is low, so the team assumes the GPU is faulty when the host, storage, network, synchronization, or application cannot supply work quickly enough.

The missing dimension is saturation. A GPU can be limited by SM execution, warp occupancy, tensor-pipe activity, DRAM bandwidth, PCIe or NVLink bandwidth, power, thermals, or host input. A single busy-time percentage cannot identify which one is limiting throughput.

flowchart TD
  A[GPU utilization at 100 percent] --> B{Useful throughput at baseline?}
  B -->|Yes| C{SM Active and occupancy high?}
  C -->|Yes| D[Compute is probably well used]
  C -->|No| E[Time busy but parallel capacity idle]
  B -->|No| F{Throttle reason active?}
  F -->|Yes| G[Clock-limited despite busy GPU]
  F -->|No| H{DRAM Active high?}
  H -->|Yes| I[Memory-bandwidth bound]
  H -->|No| J[Check host input, synchronization, or hang]

The stronger efficiency signals are DCGM’s profiling fields:

SM Active, field 1002: Whether SMs had at least one warp assigned.
SM Occupancy, field 1003: Resident warps relative to the theoretical maximum.
Tensor Core Active, field 1004: Whether tensor pipes were active.
DRAM Active, field 1005: Whether the device memory interface was transferring data.

High GPU utilization with low SM Active is the classic workload-invisible waste pattern: the GPU spends time running something, but little of its parallel compute capacity participates.

For the broader GPU subsystem model, see How an NVIDIA GPU actually works in production: a mental model for operators.

Common causes

Cause	What it looks like	First thing to check
Tiny or serialized kernels	High GPU utilization, low SM Active or low occupancy	Compare DCGM fields 203, 1002, and 1003 with application throughput
Memory-bandwidth limit	High utilization with high DRAM Active and moderate SM activity	Compare field 1005 with field 1002 and workload throughput
Power or thermal throttling	Utilization stays high while clocks and throughput fall	Check current clocks against maximum clocks and inspect throttle reasons
Application hang or spin	Stable 100% utilization, stable memory, no completed steps or tokens	Check application progress, process state, and recent XID events
Host-starved GPU	Bursty or low utilization while the job is supposed to be running	Correlate GPU idle gaps with host CPU, disk I/O, and network activity
Wrong aggregation level	Device-level metrics look healthy while one MIG instance is constrained	Inspect per-instance metrics instead of only physical-card aggregates
Unsupported profiling data	Profiling fields remain zero even under known load	Validate the DCGM fields against a known busy workload before alerting

Profiling-field availability varies by GPU generation and DCGM configuration. Unsupported fields can return blank or zero without a clear error. Do not build an alert on a profiling field until you have seen it move under a known workload.

Quick checks

These checks are read-only.

# Capture utilization with clock, power, and throttle context
nvidia-smi --query-gpu=index,name,utilization.gpu,utilization.memory,power.draw,enforced.power.limit,clocks.current.sm,clocks.max.sm,clocks_event_reasons.active --format=csv

# Capture time-domain utilization and DCGM profiling signals together
dcgmi dmon -e 203,1002,1003,1004,1005 -c 5 -d 1000

# Identify which host processes are using each GPU
nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv,noheader

# Show per-process SM and memory-controller activity
nvidia-smi pmon -s um -c 1

# Inspect the specific throttle reasons
nvidia-smi --query-gpu=index,clocks_event_reasons.active,clocks_event_reasons.sw_power_cap,clocks_event_reasons.sw_thermal_slowdown,clocks_event_reasons.hw_thermal_slowdown,clocks_event_reasons.hw_power_brake_slowdown --format=csv

# Check for recent driver or hardware errors
dmesg -T | grep -i "NVRM: Xid"

Do not interpret utilization.memory as framebuffer capacity. It is memory-controller activity over time. Framebuffer occupancy comes from memory.used, memory.free, and memory.total.

How to diagnose it

1. Define useful work before judging utilization

Use the workload’s own unit of progress:

Tokens per second for LLM inference.
Samples per second for training.
Training step time.
Requests completed per second for serving workloads.

If throughput is healthy, 100% utilization may be acceptable even if it is not a perfect efficiency measure. If throughput is falling while utilization remains 100%, the GPU is busy but not necessarily productive.

2. Compare utilization with the profiling fields

Collect field 203 beside fields 1002 through 1005. The combinations are more useful than any one value.

Observation	Likely meaning	Next move
High 203, high 1002, healthy throughput	Compute is being used effectively	Check clocks and power before adding capacity
High 203, low 1002	Time is consumed but SM capacity is idle	Inspect kernel structure, batching, and serialization
High 203, high 1005, moderate 1002	Memory bandwidth is likely limiting throughput	Reduce data movement or improve locality
High 203, low clocks, active throttle reason	The GPU is busy but clock-limited	Diagnose thermal, power, or chassis limits
High 203, flat throughput, stable memory	Possible hang, spin, or deadlock	Check application progress and XID events
Low or bursty 203 during active work	GPU may be waiting for input or synchronization	Inspect host CPU, storage, network, and data pipeline

3. Check whether clocks, not utilization, explain the slowdown

A GPU can remain 100% busy while running at reduced clocks. The kernel simply takes longer, so the time-domain utilization stays high.

Compare clocks.current.sm with clocks.max.sm. Under sustained compute, clocks below roughly 80% of maximum indicate meaningful throttling. Confirm the cause with clocks_event_reasons.active.

Power draw near the enforced limit is not automatically a fault. Treat it as confirmed power throttling only when sw_power_cap is active or when clocks and throughput degrade.

4. Attribute the activity to a process

Use nvidia-smi pmon -s um -c 1 to see which PID owns SM and memory-controller activity. This distinguishes a real training or inference process from an unexpected process, a stale CUDA context, or one tenant monopolizing the device.

In containers, nvidia-smi reports the host PID, not the container PID. Translate it through the container runtime or scheduler before terminating anything.

5. Verify that the application is making progress

A CUDA kernel can run continuously without completing useful work. Check the framework’s step counter, request completion rate, or tokens-per-second metric. For a suspected hang, look for recent XID events before assuming the hardware is healthy.

Do not start with a GPU reset. Attempt normal application termination first, preserve logs, and escalate only if the GPU remains occupied or unresponsive.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Workload throughput	Measures useful output rather than busy time	Tokens/sec or samples/sec falls while GPU utilization stays high
GPU utilization, DCGM 203	Shows how often any kernel was active	Used alone as a saturation or purchasing signal
SM Active, DCGM 1002	Shows whether SMs had active warp work	High 203 with low 1002
SM Occupancy, DCGM 1003	Shows how much warp capacity was resident	Persistent low occupancy during supposedly compute-heavy work
Tensor Core Active, DCGM 1004	Shows whether tensor pipes are being used	Tensor-heavy workload shows little tensor activity
DRAM Active, DCGM 1005	Shows device-memory interface activity	High DRAM activity with only moderate SM activity
SM clock versus maximum clock	Reveals effective compute capacity	Current clock below about 80% of maximum under load
Throttle reasons	Identifies why clocks were reduced	`sw_power_cap`, thermal slowdown, or power-brake slowdown active
Power draw versus enforced limit	Shows whether the board is power constrained	Sustained ratio above 0.95 with `sw_power_cap` active

Fixes

If SM Active or occupancy is low

Increase parallel work per kernel where the workload allows it. Common directions include larger or better-shaped batches, fewer tiny kernel launches, kernel fusion, and using optimized library kernels.

The tradeoff is memory and latency. Larger batches can improve occupancy but may increase request latency and framebuffer pressure. Validate with throughput and memory metrics, not utilization alone.

If DRAM Active is the bottleneck

Reduce memory movement and improve data locality. Depending on the workload, that can mean changing data layout, reducing host-device copies, improving access patterns, or increasing arithmetic intensity.

Adding another GPU may not fix a single workload that is already limited by each device’s memory interface. A memory-bound workload can show high GPU utilization while its real ceiling is bandwidth.

If the GPU is throttled

Fix the reason reported by the throttle flags:

For thermal slowdown, inspect cooling, airflow, ambient temperature, and adjacent-GPU heat.
For software power capping, compare power.limit, power.default_limit, and enforced.power.limit.
For hardware power-brake slowdown, investigate chassis or power-supply constraints.

Raising a power limit can restore clocks, but it can also exceed rack or PDU budget. Treat intentional power caps as capacity policy, not automatically as a fault.

If the application is not making progress

Preserve application logs, scheduler state, and recent XID output. Terminate the affected job gracefully first. If it does not exit, use a stronger process termination after confirming the host PID.

A GPU reset, driver reload, or node reboot is disruptive and should be reserved for a GPU that remains occupied or unresponsive after the workload is stopped.

If the host is starving the GPU

Measure data-loading and preprocessing time against GPU compute time. Increase loader parallelism, remove CPU-heavy preprocessing from the critical path, improve storage or network throughput, and verify NUMA locality between the host CPUs and the GPU.

A low-utilization GPU is not evidence that the GPU is underused by choice. It may be waiting for the host.

Prevention

Throughput-first dashboards. Chart tokens/sec, samples/sec, or step time beside GPU metrics so “busy” is always compared with useful output.
Profiling-field collection. Collect DCGM fields 1002 through 1005 alongside field 203 where the GPU and DCGM deployment support them.
Composite alerts. Alert on combinations such as falling throughput plus active throttling, or high utilization plus no application progress. Avoid paging on utilization percentage alone.
Workload-specific baselines. Training, inference, transcoding, and batch jobs have different normal utilization shapes. Baseline each workload class separately.
Short sampling intervals. Use sampling at ten seconds or faster for GPU health signals. Minute-level collection misses bursts, short throttling events, and rapid stalls.
MIG-aware views. Monitor MIG instances individually as well as the physical device. Aggregate utilization can hide one constrained or idle instance.

How Netdata helps

Netdata places GPU utilization beside memory usage, power draw, clocks, temperature, and throttle state, making it easier to see when “busy” diverges from effective capacity.
High-frequency collection catches short-lived utilization bursts and clock reductions that minute-level dashboards hide.
Host CPU, disk, network, and process metrics on the same timeline help distinguish GPU saturation from host starvation.
Per-process GPU activity can identify whether one PID is spinning or monopolizing the device while useful workload throughput stalls.
If DCGM profiling fields are exported into your metrics pipeline, charting fields 203, 1002, 1003, 1004, and 1005 together exposes the high-utilization, low-SM-Active waste pattern directly.
Application throughput metrics can be correlated with GPU signals so alerts are based on lost output, not just a busy-looking device.