NVIDIA GPU utilization low during training: a data-pipeline bottleneck

Your training job is running, the loss is decreasing, and nvidia-smi shows the GPU at 20-40% utilization, or oscillating between 0% and 100% in a sawtooth. The job finishes, so it looks healthy. It is not: you are paying for a GPU that is idle most of the time because the host cannot feed it.

Low utilization during active training is almost never a GPU fault. It means the GPU is starved: waiting on CPU preprocessing, disk or network data loading, pageable-memory copies across PCIe, or a synchronization barrier. The hardware is fine; the pipeline upstream of it is the bottleneck.

The diagnostic distinction that matters is between three states: transfer-bound (copy engines and PCIe busy while SMs wait), host-starved (everything on the GPU idle while host CPU or storage is saturated), and busy-but-throttled (high utilization, low throughput, a thermal or power problem, not covered here). This guide covers the first two.

What this means

utilization.gpu from nvidia-smi measures the percentage of time over a sampling period where at least one kernel was executing on any SM. It is a time-busy metric, not an efficiency metric, and it says nothing about why the SMs are idle. Critically, host-to-device memory copies run on dedicated copy engines, not SMs, so GPU utilization reads 0% during data transfer. A training loop that spends 60% of each step loading and copying a batch will show a sawtooth: high utilization during forward/backward, zero during copy and preprocessing.

Two patterns confirm the mechanism:

• Sustained low utilization (30-60%) during an active epoch, with high host CPU and saturated disk or network I/O: the classic host-starved GPU. Data loading time per batch exceeds compute time per batch.
• Sawtooth utilization with high copy-engine activity and bursty PCIe throughput: transfer-bound. The GPU waits on pageable host memory copies that cannot overlap with compute.

The distinguishing signal set: memory-copy utilization and PCIe throughput high while SM activity is low means transfer-bound. Everything low on the GPU while host CPU, disk, or network is high means the bottleneck is upstream of PCIe entirely.

flowchart TD
  A[Low GPU utilization during training] --> B{Memory-copy engine and PCIe active?}
  B -- "Yes: copies busy, SMs idle" --> C[Transfer-bound]
  B -- "No: GPU fully idle" --> D{Host CPU, disk, or network saturated?}
  D -- Yes --> E[Host-starved: CPU preprocessing or slow storage]
  D -- No --> F[Synchronization barrier or framework stall]
  C --> G[Fix: pin_memory, non_blocking copies, overlap transfer with compute]
  E --> H[Fix: more dataloader workers, prefetch, faster storage, NUMA locality]
  F --> I[Check multi-GPU collectives, GIL contention, worker hangs]

Common causes

Quick checks

All read-only, safe to run on a live training node.

# 1. SM utilization and memory controller utilization, sampled per second
nvidia-smi --query-gpu=utilization.gpu,utilization.memory --format=csv -l 1

# 2. PCIe throughput in MB/s per direction (rxpci = host to device)
nvidia-smi dmon -s t -d 1

# 3. Per-process GPU utilization and memory (which process, how busy)
nvidia-smi pmon -s um -c 10

# 4. Power draw: a starved GPU sips power relative to its limit
nvidia-smi --query-gpu=power.draw,enforced.power.limit --format=csv,noheader,nounits

# 5. Confirm it is NOT throttling: all throttle reasons should be idle/absent
nvidia-smi --query-gpu=clocks_event_reasons.active --format=csv,noheader

# 6. NUMA topology: which CPU cores and memory are local to this GPU
nvidia-smi topo --matrix

Interpretation of the combinations:

• utilization.gpu low, utilization.memory low, rxpci bursty with long idle gaps, host CPU high: host-starved.
• utilization.gpu low, copy activity and rxpci high while SMs idle: transfer-bound. Note that utilization.memory measures the GPU memory controller, not the copy engine, so it can stay low during host-to-device copies.
• utilization.gpu high but throughput poor, with clocks_event_reasons showing sw_thermal_slowdown or sw_power_cap: that is throttling, a different problem. Stop here and chase the throttle reason instead.

How to diagnose it

1. Confirm the workload is actually running. A stalled job and a starved job can both show low utilization. Check that training step time is nonzero and steps are completing. A job at 100% utilization with no step progress is a hang, not starvation.

2. Rule out throttling first. Run check 5 above. If any slowdown reason is active during compute, the GPU is busy but clocked down. Low SM clocks with high utilization is the throttled case, not data starvation.

3. Sample utilization over a full training step, not one instant. The nvidia-smi utilization sampling period is roughly 1 second and can miss sub-second phases. Collect 30-60 seconds of -l 1 samples and look at the pattern: a sawtooth with a regular period matching step time is the signature of alternating compute and load phases.

4. Split the step: data time vs compute time. Instrument the training loop to time the batch fetch (the next(iter(loader)) call) separately from the forward/backward/optimizer section. If fetch time exceeds compute time, the diagnosis is confirmed regardless of what the metrics show.

5. Classify the bottleneck with PCIe counters. During the GPU-idle gaps, watch nvidia-smi dmon -s t. If rxpci saturates during gaps, the pipeline is transfer-bound: fix the copy path (pinned memory, overlap). If rxpci is also idle during gaps, the data has not reached the copy stage: the bottleneck is CPU preprocessing or storage.

6. Check the host side during GPU idle gaps. Look at per-core CPU, iowait, disk throughput, and NIC throughput. CPU pegged means preprocessing-bound. High iowait or a saturated mount means storage-bound, common with network filesystems holding training data.

7. Check NUMA locality. nvidia-smi topo --matrix shows CPU affinity per GPU. If the training process is scheduled on cores remote from the GPU’s NUMA node, host-to-device copies pay cross-socket latency. Bind the process to the local node and re-measure.

8. For multi-GPU jobs, rule out a straggler. If one GPU in the node shows a different utilization pattern than the others, the rest may be idle waiting at collectives. Check that GPU for thermal throttle, PCIe link downgrade, or NVLink errors before concluding the data pipeline is at fault.

Metrics and signals to monitor

Two caveats on the copy-engine metric. DCGM’s DCGM_FI_DEV_MEM_COPY_UTIL only counts the dedicated copy engine; CUDA kernel-based memory copies bypass it, and it does not work on MIG partitions. Where available, the profiling fields are more precise: DCGM_FI_PROF_GR_ENGINE_ACTIVE for “is any kernel running” (higher precision than utilization.gpu and MIG-compatible) and DCGM_FI_PROF_DRAM_ACTIVE for true memory-interface utilization. Use DCGM_FI_PROF_SM_ACTIVE to see what fraction of SMs are active when kernels do run.

Fixes

Grouped by cause, in the order you should try them.

Host-starved: the CPU cannot produce batches fast enough

More dataloader workers. PyTorch’s default is num_workers=0, which loads data synchronously in the main process: the GPU stalls for the full load time of every batch. Setting num_workers in the range of your available CPU cores parallelizes loading across worker processes. This is the single most common fix.

Prefetch and persistent workers. prefetch_factor (default 2) controls how many batches each worker loads ahead; raise it when compute is fast relative to loading. persistent_workers=True avoids shutting down and respawning worker processes at every epoch boundary, which removes a recurring stall.

Fix worker hangs on Python 3.12+. DataLoader with num_workers>0 uses fork() by default on Linux, and Python 3.12+ emits a DeprecationWarning when fork is called from a multithreaded parent, which can deadlock real workloads. If workers stall, set multiprocessing_context="spawn" or "forkserver" in the DataLoader. Spawn is slower to start but safe.

Faster storage. If iowait dominates, move training data from a network filesystem to local NVMe, or stage hot shards locally. No dataloader tuning compensates for a mount that cannot deliver bytes.

NUMA pinning. Pin the training process to the CPU cores and memory node local to the GPU, per nvidia-smi topo --matrix. Cross-socket copies add latency to every batch.

Transfer-bound: copies are serializing with compute

Pinned (page-locked) host memory. Pageable host memory must be staged through a pinned bounce buffer by the driver before DMA to the GPU, which serializes the copy. In PyTorch, set pin_memory=True in the DataLoader; it pins memory on a background thread so the main training thread does not block. Do not call .pin_memory() manually on tensors in the training loop: the pinning call itself blocks the main thread and is slower than a plain .to(device).

Non-blocking device transfers. Issue .to(device, non_blocking=True) so the host-to-device copy overlaps with compute instead of blocking the stream. One sharp edge: non_blocking=True on a device-to-host copy can return before data lands, and reading the host tensor without synchronization yields garbage. Synchronize before consuming results on the host.

Preprocess on the GPU. Where the augmentation pipeline allows it, move transforms to the GPU so the CPU sends raw samples and the GPU spends its idle gaps doing useful work.

Do not do these

• Do not add GPUs. A starved GPU scales the starvation; step time does not improve.
• Do not blame the GPU or reload the driver. Nothing here is a driver fault.
• Do not chase throttling fixes (cooling, power limits) when clocks_event_reasons is clean.

Prevention

• Instrument the step. Permanently log data-fetch time vs compute time per step, or at least per N steps. A starved loss curve is indistinguishable from a healthy one; step-time decomposition is the only reliable early warning.
• Baseline per workload. Record the utilization pattern, PCIe throughput, and step time of a healthy run per model and batch size. Alert on deviation from that baseline, not on absolute utilization thresholds, which are workload-dependent.
• Never ship num_workers=0 to production. Make DataLoader configuration part of training-job review: workers, pinned memory, prefetch, persistent workers.
• Track host metrics alongside GPU metrics. GPU dashboards alone cannot see starvation; you need host CPU, iowait, disk, and NIC on the same timeline.
• Sample at 1 second or faster. Starvation gaps and transfer bursts live at sub-second scale. Minute-resolution monitoring averages the sawtooth into a misleading flat line.

How Netdata helps

• Per-second GPU and host correlation. Netdata samples GPU utilization, memory controller activity, PCIe throughput, and power draw at 1-second resolution on the same timeline as host per-core CPU, iowait, disk, and network metrics, so the sawtooth and the host-side cause line up without stitching tools together.
• Starvation detection. Low utilization.gpu concurrent with high host CPU or saturated disk during an active job is the host-starved signature; the GPU-plus-system view surfaces that coincidence directly.
• Transfer-bound detection. PCIe RX/TX throughput next to SM utilization makes the copies-busy-SMs-idle pattern obvious.
• Throttling ruled out fast. Clock throttle reasons, SM clocks vs max, temperature, and power draw vs limit are collected together, so you can confirm in one screen that this is starvation and not a slowdown cascade.
• Anomaly detection on step-time proxies. ML-based anomaly flags on utilization and power patterns catch a workload drifting from its established baseline before anyone notices the epoch time creeping up.

Related guides

• How an NVIDIA GPU actually works in production: a mental model for operators
• CUDA out of memory: diagnosing NVIDIA GPU framebuffer exhaustion
• NVIDIA GPU memory leak: framebuffer usage climbing without a plateau
• NVIDIA BAR1 memory exhaustion: mapping failures with free framebuffer
• NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver