$ guides / nvidia-gpu / nvidia-gpu-hbm-progressive-failure ▌

Operations Guides

NVIDIA GPU HBM progressive failure: from single-bit errors to a dead GPU

HBM failure on a datacenter GPU is almost never a surprise if you are watching the right signals. It is a staged degradation: single-bit ECC errors rise over days or weeks, the GPU burns through its self-repair capacity (row remaps on Ampere and later, page retirement on pre-Ampere), the first uncorrectable double-bit error lands, and eventually the GPU can no longer repair itself or falls off the PCIe bus entirely.

The early stages are silent. Workloads keep running, training loss looks fine, nvidia-smi looks fine. The first symptom most teams notice is the last stage: a dead CUDA context, a hung job, or Xid 79 in dmesg. Every stage before that was observable, often for weeks.

This article walks the full degradation curve, what each stage looks like in your telemetry and logs, and what to do at each point. For what the ECC, page retirement, and row remapping counters actually mean, see NVIDIA GPU ECC errors: corrected, uncorrected, volatile, and aggregate.

What this means

Datacenter GPUs (V100, A100, H100) ship with ECC enabled by default. A single-bit error (SBE) is corrected transparently. A double-bit error (DBE) is uncorrectable: data corruption has occurred, and whatever the GPU computed in that window is suspect.

The GPU has two self-repair mechanisms:

Page retirement (pre-Ampere): a page is permanently removed from the allocatable pool after 1 DBE or 2 SBEs at the same address. Hard limit of 64 pages (64 KiB each), tracked in InfoROM . Past that limit, future DBEs are unrecoverable.
Row remapping (Ampere and later): individual faulty DRAM rows are remapped to spare rows. More granular than page retirement. remapped_rows.failure = true means the spare capacity for a bank is exhausted and the GPU can no longer self-heal that region.

The degradation cascade:

flowchart TD
  A[SBE rate rising over days/weeks] --> B[Xid 63: remap or retirement events in dmesg]
  B --> C[Spare capacity consumed: remapped rows up / retired pages up]
  C --> D{Uncorrectable error?}
  D -->|repair succeeds| C
  D -->|first DBE: Xid 48 / 94 / 95| E[Data corruption; CUDA context killed]
  C -->|spares exhausted: Xid 64| F[remapped_rows.failure = true; RMA eligible]
  F --> G[Next DBE is unrecoverable]
  E --> H[Drain and replace]
  G --> H
  E -->|catastrophic| I[Xid 79: GPU fallen off the bus]

Two accelerants matter. First, high HBM temperature speeds up the underlying DRAM degradation, so a GPU that runs hot ages faster (the HBM temperature signal is separate from GPU die temperature; see NVIDIA GPU monitoring checklist). Second, time: error rate acceleration, not the absolute count, is the leading indicator. A GPU with a steady 2 SBEs per day for a year is background noise. A GPU that went from 2 per day to 20 per day over two weeks is dying.

Common causes

Cause	What it looks like	First thing to check
HBM manufacturing defect or aging silicon	SBE rate accelerating on one GPU over days/weeks, isolated to that unit	Compare corrected ECC rates across all GPUs in the node and fleet
Sustained high HBM temperature	Rising SBE rate correlating with `temperature.memory` near or above the GPU’s memory max operating threshold	`nvidia-smi --query-gpu=temperature.memory` history vs the SBE trend
Spare repair capacity exhausted	Xid 64 in dmesg, `remapped_rows.failure = true` (Ampere+), or retired pages approaching the limit (pre-Ampere)	`nvidia-smi -q -d ROW_REMAPPER` or `nvidia-smi -q -d PAGE_RETIREMENT`
ECC disabled (silent corruption instead)	Zero ECC errors on a production GPU that should have ECC on; corruption appears as NaNs or bad checkpoints with no hardware signal	`nvidia-smi --query-gpu=ecc.mode.current`
Driver hang triggered by ECC error patterns	Xid events followed by nvidia-smi becoming slow or unresponsive	Management path latency per GPU; check driver branch for known ECC-related hang fixes

A prior physical stress event (an overheating excursion, a power fault) can also start the clock on an otherwise healthy GPU, so a GPU that survived a thermal incident deserves closer ECC watching afterwards.

Quick checks

All read-only and safe to run on a production node.

# Volatile and aggregate ECC counters per GPU
nvidia-smi --query-gpu=index,ecc.errors.corrected.volatile.total,ecc.errors.uncorrected.volatile.total,ecc.errors.corrected.aggregate.total,ecc.errors.uncorrected.aggregate.total --format=csv

# ECC mode: must be Enabled on datacenter GPUs
nvidia-smi --query-gpu=index,ecc.mode.current --format=csv,noheader

# Row remapping state (Ampere and later)
nvidia-smi --query-remapped-rows=remapped_rows.correctable,remapped_rows.uncorrectable,remapped_rows.pending,remapped_rows.failure --format=csv

# Human-readable detail, including bank-level info
nvidia-smi -q -d ROW_REMAPPER

# Retired pages (pre-Ampere; not applicable on Ampere+)
nvidia-smi -q -d PAGE_RETIREMENT

# HBM temperature, right now
nvidia-smi --query-gpu=index,temperature.memory,temperature.gpu --format=csv

# Xid history from the kernel log
dmesg -T | grep -i "NVRM: Xid"
journalctl -k | grep -i "NVRM: Xid"

# Management path health per GPU (a dying GPU often gets slow to query before it dies)
time nvidia-smi --query-gpu=gpu_name --format=csv,noheader -i 0

Field names matter here. The corrected/uncorrected ECC fields are ecc.errors.corrected.* and ecc.errors.uncorrected.*; there is no single_bit_total field. The remapping display flag is -d ROW_REMAPPER, not -d REMAPPED_ROWS. Verify any field against nvidia-smi --help-query-gpu on your driver branch before scripting it.

How to diagnose it

Baseline every GPU now. Record per-GPU volatile and aggregate SBE/DBE counts, remapped rows (correctable, uncorrectable, pending, failure), retired pages, and HBM temperature. You cannot detect acceleration without a baseline. Store these outside the node; volatile counters reset on driver reload or GPU reset.
Identify whether the problem is one GPU or many. A single GPU with a rising SBE rate is a hardware unit failing. Many GPUs in one node rising together points at environment (thermals, power). Many GPUs across the fleet of the same batch rising together suggests a manufacturing lot issue.
Trend the SBE rate, not the count. Compute errors per hour per GPU and compare against the GPU’s own history and against peer GPUs running the same workload. Rate doubling over days, or a rate clearly above the peer cohort, is the signal. A short burst followed by nothing can be a single radiation event; require persistence before treating it as degradation.
Check repair capacity consumption. On Ampere+, look at remapped_rows.uncorrectable growth and whether remapped_rows.pending > 0 (a reset or power cycle is needed to commit pending remaps; on some platforms the hardware remap is only committed by a full node power cycle, not a VM-level reset ). On pre-Ampere, watch retired pages against the hard limit.
Correlate with Xid events in dmesg. Xid 63 is informational: a remap or retirement event was recorded, meaning self-healing is working but capacity is being consumed. Xid 64 means a remap or retirement recording failed: spares are exhausted. Xid 48 (DBE), Xid 94 (contained ECC error, A100+), and Xid 95 (uncontained ECC error, A100+) mean corruption occurred. Xid 79 means the GPU fell off the bus.
Correlate with HBM temperature. Pull the temperature.memory history for the affected GPU. A GPU that has been running near its memory thermal limit while its SBE rate climbs is confirming the mechanism; cooling remediation may slow the decline but will not reverse it.
Decide the stage and act. SBE-only with rising rate: planned replacement on a trend-derived timeline. Any new DBE: drain and replace, and treat recent outputs from that GPU as suspect. Xid 64 or failure = true: the GPU is RMA-eligible. Xid 79: node-level recovery, then hardware inspection.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`ecc.errors.corrected.volatile.total` rate	Leading indicator of HBM degradation	Rate accelerating vs own baseline or peers; require persistence before alerting on bursts
`ecc.errors.uncorrected.volatile.total` delta	Any new DBE is data corruption	Any new event (edge-triggered) on a GPU with recent production work: page
`remapped_rows.uncorrectable`	Consuming spare rows due to uncorrectable errors	Any nonzero value; increasing trend
`remapped_rows.pending`	Remap recorded but not committed	Greater than 0: schedule the reset or power cycle
`remapped_rows.failure`	Spares exhausted; GPU cannot self-heal	New transition to true; RMA the GPU (latched state, so alert on the transition, not the boolean)
Retired pages (pre-Ampere)	Permanent capacity loss, hard limit	Any increase; plan replacement well before the limit
Xid 48 / 94 / 95 in dmesg	Uncorrectable or uncontained ECC events	New event: page if the GPU had active production work
Xid 63	Self-healing events, informational	Increasing frequency alongside SBE acceleration
Xid 64	Remap or retirement failure	New event: spares exhausted
Xid 79	GPU off the bus	Any occurrence: page
`temperature.memory`	Accelerant for HBM degradation	Sustained near the GPU’s memory max operating threshold
nvidia-smi response latency per GPU	A wedging GPU often gets slow to query first	Sustained above ~2s per-GPU response

Two alert-design rules. First, ECC and remap alerting must be event- or transition-based, never level-based: aggregate counters and failure flags are persistent state stored in InfoROM, so a threshold on the raw value re-fires forever on a GPU you already know about. Second, never page on a historical nonzero counter after a reboot; use volatile counters or explicit delta tracking.

Fixes

SBE-only, rate rising

No immediate outage, so do not over-react, but do not ignore it either.

Raise monitoring cadence on that GPU (per-minute or faster SBE rate sampling).
Set a replacement timeline from the trend: extrapolate the SBE rate and the remap consumption rate, and schedule the swap before either reaches a terminal threshold. This trajectory is typically days to weeks, not hours.
Reduce thermal stress: if temperature.memory is high, fix the cooling contribution (airflow, fan behavior, workload placement). This slows the decline; it does not stop it.
Keep production running only with the explicit acceptance that the first DBE invalidates recent work. For long training runs, tighten checkpoint cadence on the affected node.

First DBE (Xid 48, or Xid 94/95 on A100 and later)

Treat all outputs from that GPU since the last known-good point as suspect. For training, roll back to a checkpoint before the event.
Drain the GPU. A single DBE on a GPU with a rising SBE history is not a random event; it is the curve reaching its next stage.
Reset the GPU before returning anything to service (dynamic page offlining on A100+ can mark the affected page unusable without a reset, but the GPU still warrants quarantine and replacement planning). Reset capabilities differ by architecture and NVSwitch topology; on some Ampere NVSwitch systems without a running Fabric Manager, individual GPU reset is not supported and all GPUs and NVSwitches must be reset together .
Schedule replacement. Repeated DBE-class events in a short window are progressive DRAM failure and grounds for RMA .

Remap or retirement failure (Xid 64, `remapped_rows.failure = true`, retired pages at limit)

The GPU has no self-repair capacity left. It is RMA-eligible.
Do not return it to production after a reset. The next uncorrectable error is unrecoverable.
This state is latched in InfoROM, so it survives reboots; your monitoring should treat it as a known-quarantined unit, not a recurring alert.

GPU fallen off the bus (Xid 79)

Do not attempt a software fix. Minimum recovery is a node reboot.
After reboot, if the GPU does not reappear, it is dead. If it does reappear, run full diagnostics before trusting it; a GPU that falls off the bus once tends to repeat. Check whether it was the terminal stage of a visible ECC degradation curve you missed.

Prevention

You cannot prevent HBM wear, but you can make it boring:

Baseline and trend ECC per GPU from day one. The entire early-warning value lives in the rate-of-change data.
Watch HBM temperature as a first-class signal, not just GPU die temperature, and fix cooling problems before they age the memory.
Verify ECC is enabled on every datacenter GPU (ecc.mode.current), especially on cloud instances where it may ship disabled. Disabled ECC converts this whole detectable curve into silent data corruption.
Alert on transitions: new DBE events, new Xid 64, new transitions of remapped_rows.failure. Never alert on raw persistent counters.
Commit pending remaps promptly. If remapped_rows.pending > 0, schedule the required reset or power cycle instead of letting pending repairs accumulate.
Quarantine, don’t rehabilitate. A GPU with uncorrectable remaps, a remap failure flag, or repeated DBEs comes out of the production pool. The cost of a scheduled swap is trivial next to a corrupted multi-week training run.

How Netdata helps

Per-second collection of the ECC corrected and uncorrected counters per GPU, so SBE rate acceleration is visible as a trend days before the first DBE, not as a retrospective discovery.
Row remap and retired page state collected alongside the ECC counters, so you can see spare capacity being consumed (Xid 63 events, uncorrectable remaps climbing) in the same view as the error rate driving it.
HBM memory temperature correlated on the same dashboard as SBE rate, making the thermal-acceleration link directly visible per GPU.
Kernel log monitoring for NVRM Xid events, so Xid 48, 63, 64, 94, 95, and 79 surface as classified events rather than something you grep for after a job dies.
Management-path health signals (per-GPU query latency, reachability) that catch the “slow to respond, then wedged, then Xid 79” terminal pattern.
Per-GPU anomaly detection on the SBE rate, which flags acceleration against each GPU’s own baseline without hand-tuned thresholds per unit.