NVIDIA Xid 48: Double Bit ECC Error

You found NVRM: Xid (PCI:....): 48, Double Bit ECC Error in dmesg, a training job just died, or a monitoring alert fired on a new uncorrected ECC error. This error is unambiguous: data corruption has occurred in GPU memory, and whatever was running on that GPU when it happened cannot be trusted.

The CUDA context that triggered the error is typically killed, recent outputs (weights, checkpoints, inference results) may be corrupt, and the GPU needs a reset or node reboot before it returns to clean service. There is no “watch and see” path. The correct response is to drain the GPU, verify whether the faulting memory was retired or remapped, validate or discard recent work, and schedule replacement.

Xid 48 is also rarely the beginning of the story. It is usually the endpoint of a slow HBM degradation that the correctable (single-bit) ECC error rate was warning about for weeks. If your monitoring only pages on Xid 48, you are finding out at the worst possible time.

What this means

GPU memory on datacenter cards (V100, A100, H100) is protected by ECC. A single-bit error (SBE) is detected and corrected transparently. A double-bit error (DBE) is detected but cannot be corrected: the data in that memory word is gone. When the GPU hits one, the driver logs Xid 48 and typically kills the affected CUDA context.

What happens next depends on the architecture:

• Pre-Ampere (V100 and older): the driver attempts dynamic page retirement. The faulting page (about 64 KB) is blacklisted so it is never allocated again, logged as Xid 63. Retirement persists across reboots in the GPU’s InfoROM. There is a hard driver limit of 64 retired pages; after that, future DBEs are unrecoverable.
• Ampere and newer (A100, H100): the hardware uses row remapping instead of page retirement, remapping individual faulty DRAM rows to spare rows. Xid 48 arrives accompanied by either Xid 94 (the error was contained to one application) or Xid 95 (uncontained, meaning corruption may have propagated to other processes on the GPU). Xid 95 is strictly worse and should change how broadly you distrust recent outputs.

In multi-GPU training, a single Xid 48 also tends to cascade: the dead context causes NCCL collectives to hang or time out across every GPU in the job, so the symptom you see first may be a stuck job, not the Xid itself.

flowchart TD
  A[Accelerating SBE rate on one GPU] --> B[Xid 63: pages retired or rows remapped]
  B --> C[Xid 48: double bit ECC error]
  C --> D[CUDA context killed]
  C --> E{Error contained?}
  E -->|Xid 94, Ampere+| F[One application affected]
  E -->|Xid 95, Ampere+| G[Corruption may span processes]
  D --> H[Recent outputs suspect: weights, checkpoints]
  F --> I[Drain GPU, reset, verify retirement]
  G --> I
  I --> J{Recurs or retirement failing?}
  J -->|Yes| K[Quarantine and replace GPU]
  J -->|No, first event| L[Return after reset, watch SBE rate closely]

Common causes

Quick checks

All read-only and safe to run during an incident.

# 1. Confirm the Xid and look for companions (63, 64, 94, 95)
dmesg -T | grep -i "NVRM: Xid"

# 2. ECC counters, volatile (since driver load) and aggregate (lifetime)
nvidia-smi --query-gpu=ecc.errors.corrected.volatile.total,ecc.errors.uncorrected.volatile.total,ecc.errors.corrected.aggregate.total,ecc.errors.uncorrected.aggregate.total --format=csv,noheader,nounits

# 3. Retired pages: SBE-caused, DBE-caused, and pending
nvidia-smi --query-retired-pages=retired_pages.single_bit_ecc.count,retired_pages.double_bit_ecc.count,retired_pages.pending --format=csv,noheader

# 4. Human-readable detail including per-page addresses and causes
nvidia-smi -q -d PAGE_RETIREMENT

# 5. Row remapping status (Ampere and newer only)
nvidia-smi -q -d ROW_REMAPPER

# 6. What is still running on the GPU
nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv,noheader

Things to note in the output:

• A delta in ecc.errors.uncorrected.volatile.total is the confirming signal for a new DBE. Volatile counters reset on driver reload, so track deltas, not absolutes.
• retired_pages.pending = Yes means a retirement is recorded but needs a reboot or reset to take effect; the bad page is still addressable until then.
• On Ampere+, look at the uncorrectable and pending remapped row counts, and especially whether a remapping failure is flagged. A remapping failure means spare rows are exhausted and the GPU can no longer self-heal that bank.
• If Xid 48 appears with no matching Xid 63, the page may not have been retired. See the persistence mode note below.

How to diagnose it

1. Confirm the event is new. Check dmesg timestamps and your ECC volatile counters. Xid monitoring must be edge-triggered: a GPU with a historical DBE that was already handled should not re-page you at 3 AM.
2. Identify which work was running. Match the dmesg timestamp against scheduler records and --query-compute-apps history. Anything that ran on that GPU in the window before the context died is suspect.
3. Determine containment (Ampere+). Xid 94 with the 48 means the error was contained to one application. Xid 95 means treat all recent outputs from that GPU as potentially corrupt, not just the killed process.
4. Verify self-healing happened. Check retired pages (pre-Ampere) or remapped rows (Ampere+) for a new entry matching the event, and check for pending states that require a reboot.
5. Compare against fleet peers. Pull ECC counters and retired page counts for the other GPUs in the node and fleet. Degradation isolated to one GPU points to that unit; fleet-wide elevation suggests a batch or environmental issue.
6. Check the degradation history. If the corrected-error rate had been accelerating on this GPU for weeks, this Xid 48 is the predicted failure, not a fluke. That history is also your replacement justification.
7. Decide containment scope. For ML training, assume the current checkpoint and any checkpoint written since the last known-good validation may be corrupt. Roll back to a validated checkpoint.

Metrics and signals to monitor

Fixes

Drain the GPU and stop scheduling work to it

Remove the GPU from the scheduler’s allocatable pool before doing anything else. A GPU that has produced one DBE has demonstrated corrupting hardware; letting a new job land on it risks poisoning more work. Cordon at the orchestrator level (Kubernetes taint, Slurm drain, or your equivalent). If the GPU is idle when you discover the event, downgrade to an urgent ticket plus quarantine rather than a page, but still treat it as out of service.

Validate or discard recent work

For training, roll back to the last checkpoint that passed validation before the event window. Do not resume from a checkpoint written by a context that later died to a DBE. For inference, the blast radius is requests served during the window; there is no practical way to re-validate those retroactively, which is why the SBE-rate leading indicator matters so much.

Reset the GPU or reboot the node

Xid 48 requires a GPU reset at minimum; the killed context does not recover on its own. On systems where per-GPU reset is supported, nvidia-smi --gpu-reset after all processes have exited the device is the lighter option; otherwise reboot the node. A reset or reboot is also what applies pending page retirements and row remaps.

Warning: --gpu-reset fails if any process still holds the device, and on some platforms it is not supported at all. Do not run it on a GPU you have not drained. See resetting a wedged NVIDIA GPU for the reset decision tree and when only a reboot works.

Note on NVSwitch systems: on Ampere with NVSwitch, individual GPU reset is not supported unless Fabric Manager is running; otherwise all GPUs and NVSwitches must be reset together. On Hopper with NVSwitch, individual reset works regardless.

Handle the retirement bookkeeping

If Xid 48 fired but no Xid 63 followed, the page may not have been retired. NVIDIA documents a race between writing errors to the InfoROM and ending a CUDA job while persistence mode is enabled, most often hit when shutting down in response to a DBE. The documented workaround is to exit persistence mode on the affected GPU before rebooting, which forces pending InfoROM writes:

# Workaround for the persistence-mode InfoROM race, before reboot.
# Disabling persistence mode means the next job pays a one-time driver
# re-init cost; run this only on the affected, already-drained GPU.
nvidia-smi -i <gpu> -pm 0

If you run the persistence daemon (nvidia-persistenced) rather than kernel persistence mode, it is not susceptible to this race.

Replace the GPU

A single isolated DBE on a GPU with a clean history might be a soft error. Repeated events, any DBE following an accelerating SBE trend, a row-remapping failure, or retired pages climbing toward the 64-page limit means the memory is failing physically. Schedule replacement. A GPU that has produced one DBE from progressive degradation will produce more, and each one is a data-corruption event.

Prevention

You cannot prevent DRAM from degrading, but you can stop being surprised by it:

• Alert on the SBE rate, not just the DBE. Corrected-error acceleration relative to the GPU’s own baseline and its fleet peers is the early warning that lets you drain and replace before corruption happens.
• Edge-trigger your Xid alerting. Page on new Xid 48/95 events, never on raw nonzero counters or historical log entries.
• Track retirement and remapping deltas. Retired pages and row-remapping state are latched in InfoROM. Alert on increases and transitions, not the raw values.
• Keep drivers current. Older driver branches limit DRAM Xid messages to one per boot, so repeated DBEs between reboots were invisible. If you are on an older branch, your “first” Xid 48 may not be the first.
• Keep ECC enabled. Some cloud providers ship with ECC disabled. With ECC off, this entire failure mode becomes silent data corruption with no signal at all. Verify with nvidia-smi --query-gpu=ecc.mode.current --format=csv,noheader.

How Netdata helps

• Netdata collects per-GPU ECC corrected and uncorrected counters at per-second resolution, so a DBE shows up as an immediate counter delta instead of waiting for a log pipeline.
• SBE rate charting per GPU makes the weeks-long acceleration visible against the GPU’s own history, which is the signal that should have paged you before Xid 48 ever fired.
• Retired pages and row-remapping metrics are collected alongside ECC counters, so you can confirm in one view whether self-healing absorbed the event or replacement is due.
• Correlating the ECC delta with GPU utilization and per-process data in the same dashboard answers “what was running when it happened” without spelunking scheduler logs.
• Anomaly detection on the corrected-error series catches slow degradation that fixed thresholds miss, because the threshold that matters is “faster than this GPU last month,” not a fleet-wide constant.

Related guides

• How an NVIDIA GPU actually works in production: a mental model for operators
• NVIDIA GPU monitoring checklist: the signals every production GPU fleet needs
• NVIDIA GPU monitoring maturity model: from survival to expert
• NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver
• nvidia-smi hangs or is unresponsive: a wedged GPU or a stuck driver
• NVIDIA persistence mode: why the GPU keeps re-initializing and P-state flaps
• Resetting a wedged NVIDIA GPU: nvidia-smi –gpu-reset and when only a reboot works