$ guides / nvidia-gpu / nvidia-gpu-row-remapping-failure ▌

Operations Guides

NVIDIA GPU Remapping Failure Occurred: HBM spare rows exhausted (Ampere+)

You ran nvidia-smi -q -d ROW_REMAPPER (or your monitoring did) and saw Remapping Failure Occurred : Yes. This is not a driver problem, not a configuration problem, and not something a reboot clears. It is the GPU telling you that its hardware self-repair budget is spent.

On Ampere and later GPUs (A100, H100, H200, Blackwell), the memory subsystem does not retire whole pages the way Volta and Turing did. Instead, when HBM rows start failing, the GPU remaps them to a finite pool of spare rows built into each DRAM bank. While spares remain, the GPU heals itself transparently: correctable ECC events trigger remaps, XID 63 appears in the kernel log as an informational note, and workloads never notice.

Remapping Failure Occurred: Yes means at least one bank has run out of spare rows. The GPU can no longer self-repair. The next failing row on that bank will produce an uncorrectable error: XID 48 or XID 95, a killed CUDA context, and potentially corrupted outputs from whatever was running. This flag is the primary hardware-replacement trigger on modern datacenter GPUs, and most fleets do not monitor it.

What this means

Row remapping is a wear-out budget, not an error. Every remap consumes a small piece of a fixed hardware resource. The progression looks like this:

flowchart TD
  A[HBM row begins failing] --> B[ECC corrects errors transparently]
  B --> C[GPU remaps row to spare - XID 63 logged]
  C --> D{Spare rows remaining in bank?}
  D -->|Yes| B
  D -->|No| E[Remapping Failure Occurred = Yes - latched in InfoROM]
  E --> F[Next failing row = uncorrectable error - XID 48/95]
  F --> G[CUDA context killed - possible data corruption]
  E --> H[GPU is RMA-eligible per NVIDIA policy]

Three operational facts about the failure flag:

It is latched state, not an event. Once true, it stays true forever because it is stored in the GPU’s InfoROM. A reboot does not clear it. This has direct alerting consequences: paging on the raw boolean fires on every check cycle forever. Alert on the transition from false to true, or on a new XID 64, not on the state.

The GPU keeps running. A GPU with failure = true does not stop working immediately. It continues operating with zero remaining self-repair capacity. This is exactly what makes the signal dangerous: workloads look fine until the next row fails, and then you get an uncontained uncorrectable error in the middle of a multi-day training run.

NVIDIA’s RMA policy treats it as a replacement trigger. The documented criterion is the row-remapping failure flag set and validated by NVIDIA’s field diagnostic tool. Do not plan to “watch it for a while.” Plan the replacement.

A note on terminology before you start: the correct nvidia-smi display flag is -d ROW_REMAPPER. Several sources reference -d REMAPPED_ROWS, which does not exist and will fail or return nothing. Similarly, the pre-Ampere fields (Single Bit ECC, Double Bit ECC, Pending Page Blacklist under Retired Pages) were replaced on Ampere+ by Correctable Error, Uncorrectable Error, Pending, and Remapping Failure Occurred. If your monitoring scrapes the old field names against A100 or H100, it is silently collecting nothing.

Common causes

Row remapping failure is a hardware end-state, so “causes” here means the degradation paths that consumed the spare row budget:

Cause	What it looks like	First thing to check
Progressive HBM degradation	Correctable ECC rate accelerating over days/weeks, remap count climbing, then failure flag flips	`ecc.errors.corrected.volatile.total` trend vs. the GPU’s own baseline
Manufacturing defect / early-life failure	Failure flag appears on a young GPU, possibly with a burst of remaps in the first months	Aggregate ECC counters and remap counts vs. fleet peers of the same model and age
Sustained thermal stress	GPU with a history of running near thermal limits; memory degradation correlates with heat	Historical `temperature.memory` (HBM) and throttle reason history
Prior uncorrectable events consumed spares	`remapped_rows.uncorrectable` is high; each uncorrectable remap permanently consumed spares	`remapped_rows.uncorrectable` count and XID 48/63 history in logs

One thing that is not a cause: workload type. Row remapping failure is not triggered by a specific job, batch size, or framework. If the flag flipped during a particular training run, the run was the witness, not the culprit.

Quick checks

All of these are read-only and safe on a production node.

# Row remapper status (Ampere+ only; older GPUs return nothing useful)
nvidia-smi -q -d ROW_REMAPPER

# Machine-readable version for scripting
nvidia-smi --query-remapped-rows=remapped_rows.correctable,remapped_rows.uncorrectable,remapped_rows.pending,remapped_rows.failure --format=csv,noheader

# ECC counters: volatile (since driver load) and aggregate (lifetime)
nvidia-smi --query-gpu=ecc.errors.corrected.volatile.total,ecc.errors.uncorrected.volatile.total --format=csv,noheader,nounits
nvidia-smi --query-gpu=ecc.errors.corrected.aggregate.total,ecc.errors.uncorrected.aggregate.total --format=csv,noheader,nounits

# XID history: 63 = remap recorded (informational), 64 = remap failed
# dmesg is a ring buffer and rotates; use journalctl for older history
dmesg -T | grep -i "NVRM: Xid"
journalctl -k | grep -i "NVRM: Xid"

# Confirm ECC is actually enabled (disabled ECC masks the whole degradation path)
nvidia-smi --query-gpu=ecc.mode.current --format=csv,noheader

# Per-GPU query on multi-GPU nodes: one bad GPU can hang a full query
nvidia-smi -i 0 -q -d ROW_REMAPPER

What to look for in the ROW_REMAPPER output:

Remapping Failure Occurred: Yes - the trigger for this article. Spares exhausted on at least one bank.
Uncorrectable Error count - rows remapped due to uncorrectable errors. High values mean the budget was spent on real corruption events.
Correctable Error count - rows remapped due to correctable errors. This is wear-leveling consumption of the budget. Note that correctable remaps can be evicted and replaced by uncorrectable remaps when a bank’s reserved rows run out, so the raw count is less informative than it looks.
Pending: Yes - a remap has been recorded to InfoROM but not yet applied. It takes effect on GPU reset or reboot. In virtualized or managed bare-metal environments, a guest-level reset may not commit the hardware remap; a full node power cycle may be required.
Bank Remap Availability Histogram - the leading indicator. It classifies each bank’s remaining spare rows into availability buckets (Max, High, Partial, Low, None). Banks drifting from Max toward Partial and Low are the budget being consumed before the failure flag flips. This histogram is the signal almost nobody monitors, and it is the one that gives you weeks of warning.

How to diagnose it

Confirm the flag and scope it. Run nvidia-smi --query-remapped-rows=... --format=csv,noheader on every GPU in the node, per-GPU with -i N. Is this one GPU or several? One GPU points to a unit-level hardware defect. Multiple GPUs of the same age and batch failing simultaneously suggests a lot-level manufacturing issue or a shared environmental factor (heat, power).
Reconstruct the timeline from logs. Grep kernel logs for XID 63 (remap recorded) and XID 64 (remap entry failure) going back as far as your log retention allows. A long tail of XID 63 events ending in XID 64 and the failure flag is the classic exhaustion arc. XID 64 near the start of the GPU’s life, with few preceding XID 63s, is more consistent with an early-life hardware defect. Do not conflate the two XIDs: XID 63 means self-healing worked; XID 64 means recording a remap failed.
Check the ECC story. Compare volatile and aggregate corrected/uncorrected counters. An accelerating corrected-error rate in the volatile counters, combined with a climbing remap count, confirms active ongoing degradation rather than a one-time historical event. If ECC mode is disabled, stop here: everything you have seen so far is an undercount, because errors were never being detected or corrected in the first place.
Look at the bank availability histogram. If failure = true, at least one bank is at None. Check how many others are at Low or Partial. A GPU with one exhausted bank and the rest at Max is a different risk profile from one where half the banks are nearly spent, though both are RMA-eligible.
Correlate with thermals. Pull historical temperature.memory (HBM temperature) and throttle reason history for this GPU. Sustained elevated HBM temperature accelerates memory aging and helps explain why this GPU degraded faster than its peers. This matters for the replacement decision: if the cooling problem persists, the replacement GPU will age the same way.
Decide workload disposition. Check what is running on the GPU now (nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv,noheader). Any output produced since the last known-good state, especially since the most recent uncorrectable ECC event, should be treated as suspect. For training, that means validating or discarding recent checkpoints.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`remapped_rows.failure` (transition)	Spare rows exhausted; RMA trigger	false → true transition, or new XID 64
`remapped_rows.uncorrectable`	Real corruption events consumed spares	Any nonzero value; increasing count
`remapped_rows.correctable`	Budget consumption rate (wear)	Steady climb; acceleration
`remapped_rows.pending`	Remap waiting for reset to apply	Any pending remap; pending that survives a reset
Bank Remap Availability Histogram	Leading indicator, weeks of warning	Banks leaving Max for High/Partial/Low
`ecc.errors.corrected.volatile.total` (rate)	Degradation is accelerating before failure	Rate doubling vs. the GPU’s own baseline or fleet peers
`ecc.errors.uncorrected.volatile.total` (delta)	Corruption is happening now	Any new event (delta > 0)
XID 63	Self-healing working; budget being spent	Increasing frequency over time
XID 64	Remap failure event	Any occurrence; recurrence on a GPU already flagged
`temperature.memory` (HBM)	Heat accelerates memory aging	Sustained elevation near memory thermal limits

Two alerting rules worth stating explicitly, because getting them wrong is common:

Alert on transitions and deltas, never on raw state. failure = true is latched. Retired pages, remap counts, and aggregate ECC counters are monotonic persistent state in InfoROM. An alert keyed to “value is nonzero” will re-fire forever on a GPU that is already quarantined and awaiting RMA. Track what you have already seen and alert only on change.

Route severity by event type. A new XID 48 or XID 95 during active production work is a page: uncorrectable corruption just hit live outputs. A new XID 64 or a fresh failure transition during active work can justify a page. An already-known failure = true on a drained, quarantined card is a ticket. XID 63 is informational. If your alerting treats all of these the same, you will either drown in noise or miss the one that matters.

Fixes

There is no software fix for exhausted spare rows. The honest set of options:

Drain and quarantine immediately. Do not schedule new work on the GPU while awaiting replacement. The risk calculus is asymmetric: the GPU may run fine for days, or the next row may fail in an hour and corrupt a checkpoint three weeks into a training run. Cordon the device at your scheduler level so nothing lands on it accidentally.

Apply pending remaps, after draining. If Pending: Yes, the recorded remap is not yet in effect. Committing it requires a GPU reset or reboot, which is disruptive: it kills anything running on the GPU, and a node reboot obviously takes down every GPU on the host. Do this only after the node is drained. Verify afterward: if the remap does not clear pending after the appropriate reset cycle, that itself is evidence for the RMA case. In managed or virtualized environments, confirm with your provider whether a full bare-metal power cycle is needed to commit InfoROM-recorded remaps.

RMA the GPU. This is the fix. NVIDIA’s RMA policy criteria are met when the row-remapping failure flag is set and validated by the NVIDIA Field Diagnostic tool. Run the field diagnostic, capture the output, open the ticket. On cloud instances, escalate to the provider with the nvidia-smi -q -d ROW_REMAPPER output and the XID history; this is a recognized replacement condition.

Validate recent outputs. Any result produced on this GPU since the last clean state is suspect. For training: roll back to a checkpoint predating the first uncorrectable event. For inference or batch processing: consider re-running. This is the step teams skip because it is expensive, and it is the step that matters most.

Do not “fix” it with a driver reload or reset. A GPU reset will not restore spare rows, will not clear the latched flag, and will not reduce the risk. The only thing a reset does in this situation is commit pending remaps.

Prevention

You cannot prevent HBM wear, but you can stop being surprised by it:

Monitor the histogram, not just the flag. The Bank Remap Availability Histogram gives you the consumption curve. Alerting when banks leave Max buys you a planned replacement instead of a 3 a.m. incident.
Track corrected ECC rate acceleration per GPU. The rate of change, relative to the GPU’s own baseline and to same-model peers, is the earliest reliable degradation signal. Absolute counts are noise; acceleration is signal.
Make all row-remap alerting transition-aware. Build seen-state tracking so latched InfoROM state does not re-alert, and so a real false → true transition is never swallowed.
Watch HBM temperature as a longevity signal. Sustained memory thermal stress shortens the runway. Correlate temperature.memory history with remap consumption across the fleet.
Verify ECC is enabled fleet-wide. Disabled ECC turns this entire degradation path invisible until corruption surfaces in application outputs.
Baseline per GPU model and age. Early-life remap bursts and steady-state wear look different. Knowing which GPU is aging faster than its cohort is how you catch batch defects.
Include row remapping in acceptance checks for new hardware. A “new” GPU or replacement unit with existing remap counts or pending state should go straight back.

How Netdata helps

Row remapper fields per GPU: Netdata’s NVIDIA collector exposes remap counts and failure state per device, so the false → true transition is a chartable, alertable event rather than something you discover during a postmortem.
ECC counters with delta tracking: volatile and aggregate corrected/uncorrected counts per GPU, so rate acceleration is visible as a slope instead of a number you have to remember.
XID correlation: kernel log XID events alongside GPU metrics on one timeline, so you can see the XID 63 tail that preceded the failure flag instead of grepping dmesg after the fact.
HBM temperature history: memory temperature trends per GPU, letting you connect thermal stress to remap consumption when arguing for cooling fixes alongside the RMA.
Fleet-wide comparison: per-GPU remap and ECC data across nodes, which is how you spot one GPU aging faster than its cohort or a batch of same-age units degrading together.
High-frequency sampling: remap events and ECC deltas surface at second-level granularity, which matters because the useful signals here are transitions and deltas, not averages.