$ guides / nvidia-gpu / nvidia-gpu-single-bit-ecc-rate-rising ▌

Operations Guides

NVIDIA GPU single-bit ECC error rate rising: the leading indicator of HBM failure

A GPU in your fleet has started logging corrected ECC errors faster than it used to. Maybe you noticed a bump in ecc.errors.corrected.volatile.total during a routine check, or Xid 92 events started appearing in dmesg. Nothing is broken yet: jobs run, loss curves look normal, nvidia-smi reports a healthy card. This is exactly the moment most teams get wrong. They look at the absolute count, decide it is small, and move on.

The count is not the signal. The rate is. A GPU that accumulated 100 corrected single-bit errors over a year is fine. A GPU that went from 0 to 100 in a week is failing, and it will likely produce an uncorrectable double-bit error (Xid 48) or a remapping failure (Xid 64) within weeks. A rising single-bit ECC (SBE) rate is the earliest reliable indicator of HBM degradation, and it usually appears well before anything else goes visibly wrong.

This guide covers how to measure the SBE rate correctly, how to tell a failing GPU from background noise, and when to drain the node before the card takes a training run down with it.

What this means

Datacenter GPUs (V100, A100, H100) ship with ECC enabled by default. Single-bit errors in HBM are corrected transparently by the memory controller, so the workload never sees them. A low background rate of SBEs is normal: cosmic rays, aging silicon, and dense HBM produce a trickle of corrected errors on every card. Operational guidance for a healthy card is on the order of a few per day.

Failure looks different. As HBM cells physically degrade, the error rate climbs, often slowly at first and then accelerating. The GPU’s self-healing mechanisms absorb the damage for a while: on pre-Ampere hardware, bad pages are retired (up to a hard limit of 64); on Ampere and later, individual DRAM rows are remapped to spares. Once those mechanisms run out of headroom, or the degradation outpaces them, a double-bit error lands, the CUDA context dies with Xid 48 or 95, and whatever the GPU was computing is suspect.

The diagnostic cascade looks like this:

flowchart TD
  A[HBM cells begin degrading] --> B[SBE rate rises above baseline]
  B --> C[Xid 92: high SBE rate - informational]
  B --> D[Row remapping absorbs damage - Xid 63]
  D --> E{Self-healing headroom left?}
  E -->|yes| F[Rate keeps climbing, remapped rows grow]
  E -->|no| G[remapped_rows.failure or 64 retired pages]
  F --> H[Double-bit error: Xid 48 or 95]
  G --> H
  H --> I[CUDA context killed, data corruption risk]

The operator’s window is between B and H. Everything after H is incident response. Everything before it is scheduled maintenance. That window is often weeks long, which is why tracking the rate is worth the effort.

The counterintuitive part: the workload looks healthy during the entire window. Training converges normally on a GPU whose corrected error rate is accelerating, because the errors are corrected. This is the classic “looks normal but is silently catastrophic” pattern. The only place the degradation is visible is in the ECC counters and the kernel log.

Common causes

Cause	What it looks like	First thing to check
Progressive HBM degradation (manufacturing defect, aging)	SBE rate accelerating over days to weeks on one specific GPU; remapped rows or retired pages growing	Compare the GPU’s current rate against its own baseline from prior weeks
Thermal-accelerated degradation	SBE rate correlates with HBM temperature; errors cluster during sustained memory-heavy load	`nvidia-smi --query-gpu=temperature.memory` during the error bursts
Normal background soft errors	Steady trickle of a few SBEs per day, flat rate for months, no Xid 92, no remapping activity	Rate trend, not the count; a flat low rate needs no action
Prior hardware stress event (overheat, power event)	Step change in rate after a known thermal or power incident, then elevated plateau	Correlate the rate change against thermal and Xid history in logs
Fleet-wide or batch hardware defect	Multiple GPUs from the same purchase batch showing the same rising pattern	Compare rates across all GPUs of the same model and age

The distinguishing question in every case is the same: is this GPU deviating from its own baseline, or from its peers? A fleet-wide constant threshold will bury the signal.

Quick checks

All of these are read-only and safe to run during production workloads.

# Volatile (since last driver load) corrected and uncorrected totals
nvidia-smi --query-gpu=ecc.errors.corrected.volatile.total,ecc.errors.uncorrected.volatile.total --format=csv,noheader,nounits

# Aggregate (lifetime, survives reboots) totals
nvidia-smi --query-gpu=ecc.errors.corrected.aggregate.total,ecc.errors.uncorrected.aggregate.total --format=csv,noheader,nounits

# Human-readable breakdown by SRAM/DRAM, correctable/uncorrectable
nvidia-smi -q -d ECC

# Confirm ECC is actually enabled (zero errors with ECC off means nothing)
nvidia-smi --query-gpu=ecc.mode.current --format=csv,noheader

# Check kernel log for Xid 92 (high SBE rate) and any Xid 48/63/64
dmesg -T | grep -i "NVRM: Xid"

# Ampere and later: row remapping status
nvidia-smi --query-remapped-rows=remapped_rows.correctable,remapped_rows.uncorrectable,remapped_rows.pending,remapped_rows.failure --format=csv,noheader

# Pre-Ampere: retired pages
nvidia-smi --query-retired-pages=retired_pages.single_bit_ecc.count,retired_pages.double_bit_ecc.count,retired_pages.pending --format=csv,noheader

# HBM temperature right now
nvidia-smi --query-gpu=temperature.memory --format=csv,noheader,nounits

A note on field names: use exactly ecc.errors.corrected.* and ecc.errors.uncorrected.*. Names like single_bit_total do not exist, and monitoring scripts built on invented names silently return nothing. Verify any field against nvidia-smi --help-query-gpu before wiring it into a collector. Likewise, the display flag for row remapping is -d ROW_REMAPPER, not -d REMAPPED_ROWS.

How to diagnose it

Establish that the rate is real, not a counter artifact. Volatile counters reset on every driver reload. If the driver was reloaded yesterday, a “0 to 40 in a day” spike may actually be a healthy GPU’s normal background rate viewed over a short window. Check aggregate counters and dmesg history first to reconstruct the longer trend.
Compute the rate, not the count. Take repeated samples of ecc.errors.corrected.volatile.total (or the equivalent DCGM counter) and compute delta per hour. The tell is a sustained rate above roughly 10 per hour on a GPU that was previously near zero, or any clear acceleration, such as the rate doubling over days. Isolated bursts during a single job are less conclusive than a sustained shift.
Compare against this GPU’s own baseline. SBE rates are workload- and temperature-dependent: memory-heavy workloads and hotter HBM produce more corrected errors. A threshold that is an alarm for an idle inference card may be Tuesday for a card running saturated LLM training. Alert on deviation from what this GPU does under this workload, and use peer comparison only as a secondary check.
Check whether the driver agrees. Look for Xid 92 in dmesg. Xid 92 is the driver’s own “high single-bit ECC error rate” notification. It is informational on its own and should not page anyone, but its appearance confirms the driver sees the same elevated rate you computed. A rising frequency of Xid 92 events is itself a warning sign.
Check self-healing consumption. On Ampere and later, look at remapped_rows.correctable, remapped_rows.uncorrectable, remapped_rows.pending, and remapped_rows.failure. On older cards, check retired page counts against the 64-page hard limit. A rising SBE rate plus growing remapped rows means the GPU is actively spending its finite repair budget. remapped_rows.failure = true or pending = Yes moves this from “watch” to “act”.
Correlate with temperature. Pull temperature.memory over the same window. If error bursts track HBM temperature excursions, cooling improvement may slow the degradation, but treat that as buying time, not a fix. The cells that are failing stay failed.
Rule out the false positives. ECC disabled (check ecc.mode.current), a single isolated burst during one anomalous job, or a flat low rate that has been stable for months are not this failure mode. Do not drain a node over background noise.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
SBE rate (delta of `ecc.errors.corrected.volatile.total` per hour)	The leading indicator itself; count without rate is meaningless	Sustained >10/hour from a previously quiet GPU, or a rate accelerating week over week
SBE rate acceleration (rate of change of the rate)	Acceleration predicts failure earlier than any fixed threshold	Rate doubling over days, or clear upward inflection vs the GPU’s own baseline
Xid 92 events in dmesg	Driver-side confirmation of elevated SBE rate	First occurrence on a previously clean GPU; increasing frequency
`remapped_rows.correctable` / `.uncorrectable` (Ampere+)	Shows how much repair budget the card has spent	Steady growth; any uncorrectable remapped rows
`remapped_rows.pending` / `.failure`	Pending needs a reboot to apply; failure means spares exhausted (RMA-eligible)	Any transition to `failure = true`; pending rows waiting on a reboot
Retired pages (pre-Ampere)	Permanent capacity loss; hard limit of 64 pages	Any increase; approaching 64
`ecc.errors.uncorrected.volatile.total` delta	The failure you are trying to preempt: data corruption	Any new event (delta > 0). This pages
`temperature.memory` (HBM)	Errors are temperature-dependent; also drives degradation	Sustained high HBM temp coinciding with error bursts

On the DCGM side, the volatile SBE total is exposed as a counter you differentiate over time to get the rate; the source material references DCGM field 310 for this. The same rate computation applies regardless of collection path: sample the counter, compute delta over time, trend it.

Fixes

There is no software fix for degrading HBM. The “fixes” are scheduling decisions.

Baseline the card and set per-GPU alerts

The immediate action is instrumentation, not intervention. Record the current rate as the new baseline, set deviation-based alerting on acceleration, and make sure Xid monitoring is classified correctly: Xid 92 informational, Xid 48/95 page-on-new-event, Xid 64 ticket-on-transition. This converts a silent failure into a scheduled one.

Drain and replace proactively

If the rate is sustained above the card’s baseline (a GPU that sat near zero now running >10/hour), accelerating, or consuming remapping headroom, schedule the node for drain and GPU replacement at the next maintenance window. The economics are clear: a planned drain costs an hour of capacity; an Xid 48 mid-run costs the run, plus checkpoint validation, plus the incident. Track this like any other leading-indicator capacity decision, the same way you would track retired pages approaching the 64-page limit.

Address the thermal contributor

If error bursts correlate with HBM temperature, check airflow, chassis cooling, and whether memory clocks are being thermally throttled. Improving cooling can slow the rate of new damage. Do not let a temperature-driven rate reduction convince you the problem is solved; remapped rows and retired pages are permanent state.

After a double-bit error lands

If the escalation you were trying to preempt happens anyway: treat any new uncorrected error as data corruption. Quarantine the GPU, validate any recent work products (checkpoints, inference outputs) that passed through it, and follow the uncorrected-ECC response path. See the related guide on corrected vs uncorrected ECC errors for the full breakdown.

Do not paper over the signal

Do not reset volatile counters to make dashboards green. Resetting destroys your rate history and makes the next operator’s diagnosis harder. Volatile counters reset on driver reload anyway; build your trending so it survives that (use aggregate counters for lifetime context, volatile deltas for current rate).

Prevention

Trend SBE rate per GPU from day one. You cannot alert on deviation from baseline if you never captured a baseline. Start collecting ECC counters on every card when it enters service, not after the first Xid 92.
Keep ECC enabled on every datacenter GPU. Zero errors with ECC disabled is not health; it is blindness. Verify ecc.mode.current as part of configuration drift checks, since some cloud images ship with ECC off.
Sample fast enough. ECC counters and Xid events are cheap to collect at 10-second or faster intervals. Minute-resolution monitoring smooths away exactly the bursts that characterize this failure.
Alert on transitions, not state. New Xid events, deltas in uncorrected counts, transitions of remapped_rows.failure from false to true. Latched state that pages forever trains operators to ignore the channel.
Classify Xid codes correctly. Xid 63 is the page retirement/remapping event (informational), Xid 92 is the high SBE rate notification (informational), Xid 48 and 95 are the corruption events (page). Misrouted Xids are how teams both miss real failures and burn out on noise.
Correlate with HBM temperature and workload. Store temperature.memory alongside ECC counters so you can separate thermal contribution from pure degradation when you need to make the replace-or-keep call.

How Netdata helps

Netdata collects per-GPU ECC counters (corrected and uncorrected, volatile and aggregate) at per-second resolution, which makes rate-of-change computation and acceleration detection possible without building custom collectors.
Row remapping and retired page statistics are charted alongside ECC counters, so you can see repair budget consumption and error rate in the same view instead of correlating two separate tools.
HBM memory temperature is collected next to the error counters, letting you visually separate temperature-driven error bursts from monotonic degradation.
Deviation-from-baseline detection is the natural fit for this failure mode: Netdata’s per-metric anomaly scoring flags a GPU whose SBE rate is behaving abnormally relative to its own history, which is the correct alerting model here.
Because Xid events live in kernel logs rather than NVML counters, pairing Netdata’s GPU metrics with log-based Xid alerting closes the loop between “rate is rising” and “driver confirms it.”