NVIDIA Xid 63 and 64: ECC page retirement and row-remap events

You found a line like this in the kernel log:

NVRM: Xid (PCI:0000:10:1c): 63, pid=1896, Row Remapper: New row marked for remapping, reset gpu to activate.

Or the less welcome variant, Xid 64. Both come from the GPU’s memory self-repair machinery: ECC detected a bad region of HBM or GDDR memory, and the driver tried to permanently remove it from service. Xid 63 means the repair was recorded successfully. Xid 64 means the repair/recording attempt failed, which is a more serious situation.

Neither event crashes your workload by itself. Xid 63 is informational: the GPU’s error correction did its job. But these events are hardware degradation telemetry, and the right response depends on the architecture, whether the repair is pending a reset, how often the events recur on the same device, and whether spare memory rows are running out.

What this means

Datacenter GPUs (V100, A100, H100 and later) ship with ECC enabled. Single-bit errors are corrected transparently. When errors in a memory region become persistent, the GPU removes that region from the allocatable pool so future allocations never touch it. The mechanism depends on the architecture generation:

• Pre-Ampere (Volta, Turing): dynamic page retirement. Whole memory pages (about 64 KB each) are retired. The framebuffer supports a maximum of 64 retired pages. Once that budget is spent, no further retirement is possible and future uncorrectable errors are unrecoverable.
• Ampere and later (A100, H100, and newer): row remapping. Individual DRAM rows are remapped to spare rows, which is far more granular. The framebuffer supports up to 512 remappings, so the self-repair budget is much larger. On these GPUs the “Retired Pages” section of nvidia-smi -q shows N/A; look at “Remapped Rows” instead.

Xid 63 means a retirement or remapping entry was recorded successfully to the GPU’s InfoROM (persistent on-board storage). Xid 64 means the recording or repair attempt failed; it is not, by itself, proof that the spare pool is full or that the InfoROM is bad.

A recorded repair is not necessarily an active repair. Retirements and remaps are recorded in the InfoROM but often stay in a pending state until a GPU reset. While pending, the bad memory is still in use, protected only by ECC. That is why the Xid 63 log line ends with “reset gpu to activate.”

flowchart TD
  A[Xid in kernel log] --> B{Which Xid?}
  B -->|63| C[Repair recorded to InfoROM]
  B -->|64| D[Recording failed]
  C --> E{Pending remap?}
  E -->|Yes| F[Schedule GPU reset to activate]
  E -->|No| G[Track event rate per GPU]
  G --> H{Multiple per week on same GPU?}
  H -->|Yes| I[Accelerating degradation - plan replacement]
  D --> J[Reset GPU - watch for recurrence]
  J --> K{Recurs or failure flag set?}
  K -->|Yes| L[RMA eligible]

What triggers these events

The last row deserves explanation. NVIDIA documents a race condition between logging the retirement to the InfoROM and tearing down a CUDA job while persistence mode is enabled, most often hit when shutting down in response to a double-bit error. The symptom is specific: you see Xid 48 but no Xid 63. The documented workaround is to exit persistence mode before rebooting (nvidia-smi -i <target GPU> -pm 0), after which Xid 63 should appear. NVIDIA’s current dynamic-page-retirement guidance still documents this workaround; verify the behavior on the installed driver branch before relying on it in automation.

Quick checks

All of these are read-only and safe on a production node.

# 1. Find the events and their exact GPU (PCI address)
dmesg -T | grep -i "NVRM: Xid" | grep -E ": 6[34],"
journalctl -k | grep -i "NVRM: Xid" | grep -E ": 6[34],"

# 2. Row-remap status, machine-readable (Ampere and later)
nvidia-smi --query-remapped-rows=remapped_rows.correctable,remapped_rows.uncorrectable,remapped_rows.pending,remapped_rows.failure --format=csv,noheader

# 3. Row-remap detail, human-readable, includes the bank remap availability histogram
nvidia-smi -q -d ROW_REMAPPER

# 4. Retired pages (pre-Ampere; shows N/A on Ampere+)
nvidia-smi -q -d PAGE_RETIREMENT

# 5. ECC counters to gauge the error rate behind the events
nvidia-smi --query-gpu=ecc.errors.corrected.volatile.total,ecc.errors.uncorrected.volatile.total --format=csv,noheader,nounits

Two interpretation notes: the remapped-row counts reflect entries recorded in the InfoROM, not necessarily rows already swapped in hardware; all pending entries are committed together on the next reset. And the counts persist across reboots, so track deltas, not absolute values. A GPU that remapped two rows eight months ago and has been quiet since is a different situation from one that remapped two rows this week.

How to diagnose it

1. Confirm which Xid you have and which GPU. The PCI address in the log line maps to a specific device. On multi-GPU nodes, everything downstream is per-GPU, so pin the address first.
2. Classify the event. Xid 63 is a successful repair record. Xid 64 is a failed one. If Xid 48 appears immediately before the Xid 63, the remap was triggered by a double-bit error and recent output from that GPU is suspect. See NVIDIA Xid 48: Double Bit ECC Error.
3. Check the pending state. remapped_rows.pending (or retired_pages.pending on older cards) tells you whether the repair is recorded but inactive. Pending = Yes means the bad memory is still addressable until a reset.
4. Check the failure flag. remapped_rows.failure = true means a row-remapping failure rule was hit and the affected region can no longer be repaired by that mechanism. This is latched, persistent state stored in the InfoROM: once true, it stays true. Per NVIDIA’s current row-remapping RMA policy, the flag is set when an uncorrectable-row attempt reaches a bank that already has eight uncorrectable rows remapped, when a row that was already remapped needs remapping again, or after 512 total uncorrectable remappings; the policy is written for Ampere-and-later row remapping, and Blackwell adds HBM channel repair when a spare channel is available.
5. Look at the rate, not just the event. A single Xid 63 on a GPU with an otherwise clean history is background wear. Several per week on the same GPU is accelerating HBM degradation. Compare the corrected ECC error rate across the node’s other GPUs to isolate whether this device is an outlier.
6. Check the availability histogram. The bank remap availability histogram in nvidia-smi -q -d ROW_REMAPPER shows how many spare rows remain per bank. A bank approaching zero is your early warning before a Xid 64.

Metrics and signals to monitor

Two alerting rules worth stating plainly, because teams get them wrong:

• Alert on transitions, not state. remapped_rows.failure and retired-page counts are persistent. If you alert on the raw value, the alert fires forever after the first event. Alert on the delta or the false-to-true transition, and on new Xid lines in the log.
• Do not page on Xid 63. It is the self-healing mechanism working. It belongs in a ticket queue with trend tracking. A new Xid 64 during active production work can justify a page, because the GPU could not isolate the bad memory.

What to do about it

A single Xid 63, no pending remap, no history

Record it and move on. Add the GPU to a watch list and trend the corrected ECC rate for that device. This is normal wear on a large fleet.

Xid 63 with pending = Yes

The recorded repair is inactive and the bad row or page is still in use. Schedule a GPU reset to activate it. nvidia-smi -r resets a GPU, but it requires the device to be idle: all CUDA contexts on it must be gone first, which usually means draining the node. In some virtualized environments a guest-level reset does not commit the repair and a hypervisor-level action or full node power cycle is required. If resets do not stick or the GPU misbehaves afterward, see Resetting a wedged NVIDIA GPU.

Recurring Xid 63 on the same GPU

A few per week on one device means degradation is accelerating. Check the bank remap availability histogram for remaining spare capacity, tighten ECC rate trending on that GPU, and start the replacement conversation before the failure flag forces it. The trajectory matters more than the count.

Xid 64

The recording failed. Reset the GPU (or reboot the node) promptly, then watch closely for recurrence. NVIDIA’s debug guidance treats a recording failure as requiring immediate action; if errors continue after the reset, the GPU needs an RMA. Do not keep running long training jobs on a GPU producing repeated Xid 64 events: the next uncorrectable error in that memory has no isolation mechanism left, and you will find out via corrupted results or Xid 48/95, not via a clean retirement.

If you see Xid 48 without a following Xid 63, suspect the persistence-mode race described above and retire the page by dropping persistence mode before the reboot.

Prevention

You cannot prevent memory wear, but you can stop being surprised by it:

• Baseline ECC behavior per GPU. Low single-digit corrected errors per day can be background noise. What matters is acceleration versus each GPU’s own history and versus its peers in the same node.
• Trend retirements and remaps as deltas. Persistent InfoROM counters will re-alert forever if you watch absolute values.
• Track pending state as its own alert. A pending remap that sits for weeks because nobody scheduled a reset is bad memory still serving production traffic.
• Drain and reset on a maintenance cadence. If your fleet accumulates pending remaps, fold GPU resets into regular maintenance windows instead of waiting for a wedge to force one.
• Keep ECC enabled. Datacenter GPUs ship with ECC on; if a provider or image has disabled it, none of these protections or signals exist and memory corruption is silent. Verify with nvidia-smi --query-gpu=ecc.mode.current --format=csv,noheader.
• Sample fast. Xid events and ECC spikes are short-lived. Minute-resolution collection misses them; this is one of the common fleet mistakes covered in the NVIDIA GPU monitoring checklist.

How Netdata helps

• Per-second GPU metrics catch the transient ECC and throttle behavior that minute-interval scrapers miss, which matters because retirement events are preceded by accelerating corrected-error bursts.
• ECC counter collection (volatile and aggregate, corrected and uncorrected) per GPU gives you the rate trend that turns a single Xid 63 from noise into a degradation signal.
• Remapped-row and retired-page visibility lets you watch pending state and delta changes instead of raw persistent counters, so alerts fire on new events, not old history.
• Correlation across signals is the real shortcut: an Xid 63 next to a spike in uncorrected ECC and a dead CUDA context tells a complete story (double-bit error, successful remap, lost context) that no single metric shows alone.
• Fleet-wide comparison surfaces the one GPU whose error rate is diverging from its peers, which is the earliest practical replacement trigger.

Related guides

• NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver
• How an NVIDIA GPU actually works in production: a mental model for operators
• NVIDIA GPU monitoring checklist: the signals every production GPU fleet needs
• NVIDIA GPU monitoring maturity model: from survival to expert
• nvidia-smi hangs or is unresponsive: a wedged GPU or a stuck driver
• NVIDIA persistence mode: why the GPU keeps re-initializing and P-state flaps
• Resetting a wedged NVIDIA GPU: nvidia-smi –gpu-reset and when only a reboot works
• NVIDIA Xid 13: Graphics Engine Exception
• NVIDIA Xid 31: GPU memory page fault (invalid address)
• NVIDIA Xid 43: GPU stopped processing (the GPU hang)
• NVIDIA Xid 48: Double Bit ECC Error
• NVIDIA Xid 79: GPU has fallen off the bus