$ guides / nvidia-gpu / nvidia-gpu-tensor-cores-idle ▌

Operations Guides

NVIDIA GPU Tensor Cores idle: mixed precision not being used

Your training job is running, nvidia-smi shows high SM utilization, power draw looks healthy, and step time is 5 to 10 times worse than the hardware spec says it should be. DCGM profiling shows Tensor Core Active pinned at or near zero. The GPU is busy, but it is doing the work on CUDA cores instead of Tensor Cores.

Nothing errors. Nothing crashes. The job just runs at a fraction of the throughput you paid for, and because SM utilization still reads 90%+, it looks healthy in every dashboard that only tracks utilization.gpu.

For a mixed-precision workload (FP16, BF16, TF32 on Ampere and later, FP8 on Hopper), Tensor Core Active should be well above 30% during the compute phases. Near-zero during a known matrix-heavy workload is a fault. Near-zero during a non-matrix workload (data preprocessing, embedding lookups, small reductions) is correct behavior, so the first diagnostic question is always whether the workload should be using Tensor Cores at all.

What this means

Tensor Cores are the matrix-multiply units inside each SM, present on Volta and later. They execute mixed-precision GEMM and convolution operations at many times the throughput of the general-purpose CUDA cores. When a framework runs a matmul or convolution, cuBLAS or cuDNN selects a kernel. If the inputs, shapes, data types, and library configuration all line up, it picks a Tensor Core kernel. If anything is off, it silently falls back to a CUDA core kernel.

The fallback is silent. cuBLAS does not warn you. PyTorch does not warn you. The only externally visible symptom is Tensor Core Active (DCGM field 1004, DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, the ratio of cycles the tensor pipe is active) sitting at zero while the job crawls. The same operation on CUDA cores is often 10 to 20 times slower.

A common trap: operators see utilization.gpu at 100% and conclude the GPU is fully used. As how an NVIDIA GPU actually works in production covers, SM utilization measures time busy, not efficiency. A CUDA-core GEMM keeps SMs 100% busy while delivering a small fraction of Tensor Core FLOPS.

flowchart TD
  A[Tensor Core Active near 0%] --> B{Workload matrix-heavy?
matmul / conv dominant}
  B -- No --> C[Expected behavior.
Non-matrix ops use CUDA cores]
  B -- Yes --> D{AMP / mixed precision
actually enabled?}
  D -- No --> E[Framework running FP32.
Enable AMP or TF32]
  D -- Yes --> F{Dims multiples of 8?
M, N, K, channels}
  F -- No --> G[Pad batch, channels,
sequence lengths]
  F -- Yes --> H{Library versions and
compile flags correct?}
  H -- No --> I[Fix cuBLAS/cuDNN version,
compute capability target]
  H -- Yes --> J[Profile kernel selection:
cuBLAS/cuDNN heuristic miss]

Common causes

Cause	What it looks like	First thing to check
Framework running full FP32	Tensor Core Active at 0%, step time several times slower than expected, memory usage higher than the mixed-precision plan	Is AMP actually enabled in the training script, or was it lost in a refactor?
TF32 disabled (PyTorch default on Ampere+)	FP32 matmuls on A100/H100 show 0% Tensor Core activity; no warning from PyTorch	`torch.get_float32_matmul_precision()` returns `highest`
Matrix dimensions not multiples of 8	Tensor Core activity near zero or erratic for specific layers, especially with odd channel counts, vocab sizes, or batch shapes	Inspect layer shapes: M, N, K, and conv channel counts
cuBLAS/cuDNN version or shape heuristic miss	Specific ops run on CUDA cores while others hit Tensor Cores; behavior changes after a library upgrade	Compare behavior across library versions; profile per-op
Wrong compute capability at compile time	Custom CUDA extensions or source-built frameworks fall back to generic kernels	Check the `TORCH_CUDA_ARCH_LIST` or equivalent used at build time
Workload is genuinely not matrix-heavy	Tensor Core Active at 0% but step time matches expectations	Confirm the op mix: mostly elementwise, reductions, embeddings

Quick checks

All of these are read-only and safe against a production job.

# Watch Tensor Core Active (DCGM field 1004) for 10 samples at 1s intervals
dcgmi dmon -e 1004 -c 10 -d 1000

# Combine with SM activity, DRAM activity, and power for context
dcgmi dmon -e 1002,1004,1005,155 -c 10 -d 1000

# Confirm the GPU is busy doing something (rules out a stalled job)
nvidia-smi --query-gpu=utilization.gpu,utilization.memory,power.draw --format=csv,noheader

# Inside the training environment, check what PyTorch will actually do
import torch
print(torch.get_float32_matmul_precision())      # 'highest' = TF32 off (default on Ampere+)
print(torch.backends.cuda.matmul.allow_tf32)     # legacy API, deprecated after PyTorch 2.9
print(torch.backends.cudnn.allow_tf32)           # legacy API, deprecated after PyTorch 2.9
print(torch.cuda.is_bf16_supported())
print(torch.cuda.get_device_capability())        # e.g. (8, 0) for A100, (9, 0) for H100

Confirmed: PyTorch documentation states that after PyTorch 2.9 a new API set (fp32_precision) replaces the older allow_tf32 flags, which are deprecated after that version.

Tensor Core Active during a known matmul-heavy phase. Near zero with high SM utilization is the smoking gun: the GPU is executing the math on CUDA cores.
The AMP code path in the training script. Check that the autocast context actually wraps the forward pass and that a recent refactor did not remove it.
Layer shapes. List the M, N, K of the dominant GEMMs and the channel counts of the dominant convolutions. Anything not a multiple of 8 is a fallback candidate.

How to diagnose it

Confirm the workload should use Tensor Cores. If the job is dominated by elementwise ops, reductions, normalization, or embedding gathers, 0% Tensor Core activity is correct. Framework AMP keep-lists deliberately run some ops in FP32 for numerical stability. Only proceed if the job is matmul/conv dominant and slower than expected.
Measure Tensor Core Active over a full step. A single sample can land between kernels. Sample dcgmi dmon -e 1004 across several complete training or inference steps. Sustained near-zero across compute phases is the fault condition; above ~30% during matmul phases is healthy.
Verify mixed precision is enabled in the framework. In PyTorch, that means autocast or an explicitly enabled TF32 path. On Ampere and newer, PyTorch defaults torch.set_float32_matmul_precision() to highest, which disables TF32 for FP32 matmul, and it does not warn you about this outside of torch.compile. If your “FP32” model on an A100 shows 0% Tensor Core activity, this default is the likely cause.
Check matrix dimensions. For FP16/BF16 Tensor Core kernels, the M, N, and K dimensions of GEMMs and the input/output channel counts of convolutions should be multiples of 8 (multiples of 16 for INT8 on Turing). An odd vocab size, an unpadded sequence length, or a 3-channel input conv can push a hot layer onto CUDA cores. This is the most common cause when AMP is correctly enabled but Tensor Core activity is still low.
Check library versions and compile targets. cuBLAS 11.0+ selects Tensor Core kernels automatically when shapes and precisions allow, so explicit opt-in flags are no longer needed (the old CUBLAS_TENSOR_OP_MATH math mode is deprecated). But pedantic compute modes explicitly disable Tensor Cores, and kernels compiled for the wrong compute capability can miss the Tensor Core path entirely. If you build PyTorch, TensorFlow, or custom extensions from source, verify the arch list covers the GPUs you run.
Profile per-op if the above is inconclusive. A profiler that shows per-kernel activity will identify which specific ops fall back. Note the profiling constraint below before combining this with DCGM collection.

Tooling caveat: on Ampere and older GPUs, DCGM’s profiling fields (including field 1004) share hardware PMC slots and cannot be collected concurrently with Nsight Systems or Nsight Compute. DCGM exposes dcgmProfPause()/dcgmProfResume() to hand the counters over; these are C API functions only, with no equivalent dcgmi CLI subcommand. On Hopper and newer, profiling metrics can be watched concurrently without these conflicts.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Tensor Core Active (DCGM 1004)	Direct measure of Tensor Core pipe engagement	Near zero during known mixed-precision matmul phases
SM activity (DCGM 1002)	Confirms the GPU is busy, not stalled	100% SM activity with 0% Tensor Core Active
DRAM activity (DCGM 1005)	Distinguishes compute-bound from memory-bound	High DRAM activity masking a compute misconfiguration
Power draw (DCGM 155)	Tensor Core GEMMs draw heavily; a CUDA core fallback often shows lower sustained power	Power well below expectation for a “saturated” GPU
Training step time	The user-visible cost of the misconfiguration	Step time several times slower than the hardware baseline

The correlation that matters: high SM utilization plus near-zero Tensor Core Active plus slow step time equals “busy but doing the math the slow way.”

Fixes

Enable mixed precision properly

In PyTorch, wrap the forward pass in autocast with the right dtype for your hardware (BF16 on Ampere+ is usually the safer default than FP16 because it needs no gradient scaler). For FP32 models on Ampere+, explicitly enable TF32 if the precision is acceptable:

# Current API (PyTorch 2.9+)
torch.backends.cuda.matmul.fp32_precision = 'tf32'
torch.backends.cudnn.conv.fp32_precision = 'tf32'

# Older API (deprecated after PyTorch 2.9)
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True

Tradeoff: TF32 and reduced-precision training change numerics. For most training workloads this is fine and is what the hardware was designed for, but verify convergence against your FP32 baseline before rolling it out fleet-wide.

Fix the shapes

Pad dimensions to multiples of 8 where you control them: vocab size rounded up to the next multiple of 8, batch sizes and sequence lengths bucketed to friendly sizes, conv channel counts aligned in the model definition. A single odd-shaped hot layer can dominate step time even if every other layer uses Tensor Cores.

Fix the build

If you compile frameworks or extensions from source, set the CUDA arch list to cover the compute capabilities you deploy on (for example 8.0 for A100, 9.0 for H100). A build targeting only an older architecture can leave newer Tensor Core paths unused. Also check that nothing in your stack selects a pedantic cuBLAS compute mode, which explicitly disables Tensor Cores even on cuBLAS 11.0+.

Upgrade libraries deliberately

cuBLAS and cuDNN kernel selection heuristics are version- and architecture-sensitive. There are documented cases of a cuDNN upgrade fixing mixed-precision behavior on one GPU generation while regressing another. Pin known-good library versions per GPU generation, and treat library upgrades as changes that need a Tensor Core Active regression check, not just a “does it run” check.

Know when to do nothing

If the workload is not matrix-heavy, 0% Tensor Core Active is correct. Do not chase it. Spend the effort on the actual bottleneck, which for memory-bound inference is usually HBM bandwidth, not compute.

Prevention

Baseline per workload. For every production training or inference job, record the expected Tensor Core Active range during compute phases. Alert when a known mixed-precision workload drops to near zero for a sustained window.
Regression-check config changes. Framework upgrades, container image rebuilds, cuDNN/cuBLAS bumps, and model shape changes all deserve a Tensor Core Active comparison before and after.
Make AMP part of code review. The failure mode is usually a refactor that drops the autocast context or a new environment that resets TF32 defaults. Treat precision configuration as infrastructure, not an implementation detail.
Track it per MIG instance if you partition GPUs. DCGM profiling fields can be collected per group (for example dcgmi dmon -e 1001,1002,1003,1004 -g <mig-group>), so the check works on shared A100/H100 partitions too.

How Netdata helps

Netdata collects per-second GPU metrics including SM utilization, memory controller utilization, power draw, and clocks, so the “100% busy but slow” pattern is visible without waiting for a step-time complaint.
Correlating SM utilization against power draw exposes the classic signature: a GPU at full utilization drawing less power than a healthy Tensor Core workload should.
With the DCGM collector, Tensor Core Active sits in the same per-second view as utilization and power, so step-phase oscillation is not averaged away at minute resolution.
Historical baselines per node and per workload let you catch the regression the day a framework upgrade or image rebuild silently turns off mixed precision, rather than at the next capacity review.
ML-based anomaly detection on step-adjacent signals (utilization, power, memory throughput) flags the throughput collapse even when no static threshold fires.