Your training job or inference service is running, nvidia-smi shows high GPU utilization, power draw looks healthy, and throughput is far below what the hardware should deliver. The profiling metrics show the pattern: the device memory interface is active nearly 100% of cycles while the streaming multiprocessors sit mostly idle. The GPU is memory-bandwidth bound. It is not short on compute. It is waiting on HBM.

This is one of the most misdiagnosed GPU performance problems in production. Time-based utilization metrics make the GPU look busy, so teams conclude they need more GPUs, higher clocks, or a bigger power limit. None of those help. The SMs are starving because data cannot move from HBM fast enough, and the only real fixes are algorithmic: fewer bytes moved per FLOP.

This article covers how to confirm the memory-bound pattern, how to rule out the lookalikes (host starvation, thermal throttling, power capping), and which fixes actually move the needle.

What this means

Every kernel moves data between HBM and the SMs. Each workload has an arithmetic intensity: floating-point operations per byte transferred. When arithmetic intensity is low, the kernel finishes its math long before the next chunk of data arrives, and the SMs stall on memory while the memory controllers run flat out.

A large fraction of real ML workloads live in this regime: elementwise operations, reductions, normalization layers, unfused attention, large embedding lookups, and most inference serving at moderate batch sizes. On an A100 80GB with roughly 2 TB/s of HBM bandwidth, a DRAM active ratio of 0.9 means about 1.8 TB/s is flowing. The memory subsystem is saturated; the compute units are not.

The diagnostic signature is the combination, not any single metric:

  • High DRAM activity: the device memory interface is sending or receiving data nearly every cycle.
  • Low SM and Tensor activity: SMs rarely have a warp doing useful work; tensor pipes are mostly idle.
  • High reported “utilization”: time-based busyness metrics can still read 100%, which is why this gets misdiagnosed.

One subtlety: there is no NVML API that returns memory bandwidth in GB/s. The tools give you a time-based activity ratio. On HBM parts, where the memory clock is essentially fixed, multiplying that ratio by theoretical peak gives a reasonable bandwidth estimate. On GDDR cards with dynamic memory clocks, the estimate is less reliable.

Common causes

CauseWhat it looks likeFirst thing to check
Unfused kernel chains (elementwise ops, norms, activations)DRAM active pinned, SM active low, many small kernels back to backKernel timeline in a profiler; count distinct small kernels per step
Standard attention without FlashAttentionDRAM active high and scales quadratically with sequence length; SM active moderateSequence length sensitivity: does throughput collapse as sequences grow?
Low-batch inference servingDRAM active near 1.0 during decode, tensor active near zero, utilization.memory highBatch size and per-request throughput vs. latency curve
Large embedding lookupsBursty DRAM active, low SM active, irregular accessWhich layers dominate step time in a profile
Poor memory layout (strided or uncoalesced access)High DRAM active but low effective throughput; more bytes moved than neededProfiler memory analysis for uncoalesced access
HBM thermal throttling degrading bandwidthMemory clock reduced, temperature.memory near its limit, throughput downtemperature.memory and clocks.current.memory vs. max

Quick checks

These are read-only and safe on a production node.

# Time-based utilization (the misleading one): GPU and memory controller busyness
nvidia-smi --query-gpu=utilization.gpu,utilization.memory --format=csv,noheader,nounits -l 2

# DCGM profiling ratios: SM active (1002), Tensor active (1004), DRAM active (1005)
dcgmi dmon -e 1002,1004,1005 -c 10 -d 1000

# Memory (HBM) temperature and memory clock vs max
nvidia-smi --query-gpu=temperature.memory,clocks.current.memory,clocks.max.memory --format=csv,noheader,nounits

# SM clock and throttle reasons, to rule out throttling as the cause of low SM activity
nvidia-smi --query-gpu=clocks.current.sm,clocks.max.sm,clocks_event_reasons.active --format=csv,noheader

# Power draw vs enforced limit
nvidia-smi --query-gpu=power.draw,enforced.power.limit --format=csv,noheader,nounits

Notes on interpretation:

  • utilization.memory measures the percentage of time the memory controller was reading or writing. It is a busyness metric, not a bandwidth-saturation metric. A kernel issuing many small transactions can pin it at 100% while moving far less than peak bandwidth. Treat it as a hint, not proof.
  • The DCGM profiling fields are ratios of active cycles. DRAM active (field 1005) near 1.0 with SM active (1002) and Tensor active (1004) well below it is the memory-bound signature. DRAM active of 0.9 on an A100 80GB implies roughly 1.8 TB/s in flight against about 2 TB/s theoretical.
  • Profiling fields are not available on every GPU and require the DCGM profiling module loaded and running with sufficient privileges. If dcgmi dmon -e 1002,1004,1005 returns blanks or errors, verify profiling support for your GPU model before concluding anything.
  • clocks_event_reasons.active replaced the older clocks_throttle_reasons.active field name in recent drivers; if the query errors, try the older name.

How to diagnose it

flowchart TD
  A[Throughput below expectation] --> B{utilization.gpu high?}
  B -->|No| C[Check host starvation: CPU, dataloader, PCIe]
  B -->|Yes| D[Read DRAM active vs SM active]
  D -->|DRAM high, SM low| E[Memory-bandwidth bound]
  D -->|Both high| F[Compute bound or healthy saturation]
  D -->|Both low| G[Stalls: sync, launch gaps, CPU bound]
  E --> H{temperature.memory high?}
  H -->|Yes| I[HBM thermal throttle: fix cooling first]
  H -->|No| J[Algorithmic fix: fusion, FlashAttention, batching, layout]

  1. Confirm the symptom is real. Compare achieved throughput (samples/sec, tokens/sec) against the established baseline for this workload on this hardware. Utilization metrics alone never prove a bottleneck.

  2. Collect the profiling trio. Run dcgmi dmon -e 1002,1004,1005 during a representative workload window, at least 10 to 30 seconds. You want sustained ratios, not a single sample, because utilization oscillates between compute and data phases in normal training.

  3. Classify the ratio pattern. DRAM active high (above roughly 0.8) with SM and Tensor active low is memory-bound. Everything high is healthy saturation. Everything low points elsewhere: host starvation, synchronization stalls, or kernel launch gaps.

  4. Rule out the lookalikes before touching the code. Check temperature.memory against the memory thermal limit, and clocks.current.memory against max. HBM bandwidth degrades at high memory temperature, so a memory-bound reading caused by thermal throttling is a cooling problem first, not an algorithm problem. Also confirm clocks are not reduced by sw_power_cap or thermal slowdown reasons, which can depress SM activity independently.

  5. Rule out host starvation. Low SM activity with low DRAM activity plus bursty PCIe throughput and high host CPU means the GPU is waiting on the host, not on HBM. That is a different playbook (data loading, preprocessing, NUMA locality).

  6. Attribute the bytes. With the pattern confirmed, profile the workload to find which kernels move the most memory traffic. The usual suspects are unfused elementwise chains, standard attention at long sequence lengths, and embedding-heavy stages. The fix targets those kernels, not the whole model.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
DRAM active (DCGM field 1005)True time-based saturation signal for the HBM interfaceSustained above 0.8-0.9 during active work
SM active (DCGM field 1002)Whether compute units have warps doing workLow while DRAM active is high
Tensor active (DCGM field 1004)Whether tensor pipes are engagedNear zero during GEMM-heavy phases
utilization.memory (nvidia-smi)Coarse memory controller busyness100% with low delivered throughput (misleading if read alone)
temperature.memoryHBM bandwidth degrades at high memory temperatureApproaching the memory max operating threshold
clocks.current.memory vs maxDetects memory clock throttlingBelow max during sustained memory-bound load
Achieved throughput (app-level)The metric that actually proves the bottleneckDeclining samples/sec or tokens/sec while DRAM active is pinned

A practical alert pattern: page nobody, but ticket when the DRAM-to-SM activity ratio stays heavily skewed (memory interface saturated, compute idle) during active production windows. That is a capacity and efficiency problem worth an engineering task, not a 3 a.m. incident, unless throughput has collapsed below the SLO.

Fixes

No operational knob adds HBM bandwidth. Raising SM clocks, raising the power limit, or adding GPUs running the same kernel shape does nothing for a memory-bound workload. The fixes reduce bytes moved per unit of math.

Fuse kernels

Chains of elementwise operations (scale, bias, activation, dropout, residual add) each do a full HBM round-trip: read the tensor, write the tensor, next op reads it again. Fusing them into one kernel keeps intermediates in registers or shared memory and touches HBM once. This is the highest-leverage fix for elementwise-heavy graphs. Framework-level fusers and compilers (graph fusion, torch.compile-style codegen) get you partway; hand-fused kernels get the rest.

Tradeoff: fused kernels are harder to debug and reduce flexibility during model iteration. Fuse the hot path, keep the experimental path unfused.

Use IO-aware attention (FlashAttention)

Standard attention materializes the full attention matrix in HBM, which is why its memory traffic explodes with sequence length. FlashAttention tiles the computation into SRAM-sized blocks and fuses the whole attention step, eliminating the HBM round-trips for the intermediate matrix. If your profile shows attention dominating memory traffic, this is the fix.

Tradeoff: requires a supported kernel path for your model architecture and hardware; numerical behavior is equivalent but validate outputs when switching.

Raise arithmetic intensity with batching

Decode-phase inference is classically memory-bound: weights are read from HBM for every request, and small batches amortize that read over almost no compute. Larger batches reuse each weight read across more work, shifting the balance back toward compute-bound. Continuous batching and larger per-step batch sizes are bandwidth optimizations as much as latency optimizations.

Tradeoff: larger batches raise per-request latency and increase memory capacity pressure. You are trading the bandwidth wall for the capacity wall; watch VRAM headroom as you tune.

Fix memory layouts and access patterns

Strided or uncoalesced access moves more bytes than the math requires. Padding, transposing layouts so inner loops read contiguous memory, and aligning embedding tables to cache-friendly boundaries all reduce effective traffic for the same computation. These show up as high DRAM active with mediocre delivered throughput even before the ratio hits saturation.

Tradeoff: layout changes ripple through the codebase and can conflict with framework defaults. Profile first; change layout only where the profile shows wasted traffic.

Fix the cooling if HBM is hot

If temperature.memory is near the memory thermal threshold and memory clocks are reduced, bandwidth itself is degraded. No code change recovers bandwidth the hardware is no longer delivering. Address airflow, chassis cooling, or ambient temperature first, then re-measure the DRAM/SM ratios.

Prevention

  • Baseline the ratios per workload. Record typical DRAM, SM, and Tensor active for each production model. Memory-boundness is a per-workload property; the baseline tells you when a change pushed a workload into the bandwidth wall.
  • Profile before you scale. Any proposal to add GPUs for a throughput problem should come with evidence the workload is compute-bound. If DRAM active is the pinned metric, more GPUs running the same kernels buys you the same wall, replicated.
  • Watch memory temperature as a leading indicator. HBM thermal headroom shrinks with dust, aging thermal interfaces, and ambient drift. Track the margin between peak-load memory temperature and the throttle threshold over months.
  • Gate efficiency reviews on the DRAM/SM ratio. A sustained DRAM-to-SM activity skew during production windows is a standing optimization ticket. Catching it in review is cheaper than discovering it when the capacity bill arrives.
  • Sample fast enough. Memory-bound phases can be sub-second within a step. Sampling at minute resolution averages away the very pattern that identifies the bottleneck.

How Netdata helps

  • Netdata collects per-second NVIDIA GPU metrics, so memory-bound phases inside a training step or inference burst are visible instead of being averaged away at minute resolution.
  • SM utilization and memory controller utilization side by side expose the skew directly: busy memory interface, idle compute.
  • Correlating memory temperature with memory clock and throughput separates a genuinely memory-bound workload from HBM thermal throttling, which demand completely different responses.
  • Power draw and throttle-reason context rule out the lookalikes: low SM activity caused by power capping or thermal slowdown reads differently once those signals are on the same timeline.
  • Per-GPU views across a node make it easy to spot a single memory-thermal-throttled card dragging a data-parallel job while its peers run clean.