NVIDIA Xid 94 and 95: contained vs uncontained ECC errors (Ampere+)

You are looking at a kernel log line like NVRM: Xid (PCI:0000:01:00): 94, pid=7194, Contained: ... or ...: 95, pid=7062, Uncontained: ... on an A100, H100, or newer GPU, and you need to know two things fast: how bad is it, and what do you drain.

Both Xid 94 and Xid 95 are uncorrectable (double-bit class) ECC error events. The distinction NVIDIA draws, starting with the A100, is containment: whether the corruption was confined to the faulting application context or may have spread across everything running on the GPU. That distinction drives the entire response.

Xid 94 means the driver believes the error was contained to the application that hit it. That application is dead and must be restarted; other applications on the GPU are considered unaffected. Xid 95 means the error could not be contained. Every workload on that GPU is potentially corrupted, and the GPU must be reset before anything restarts on it. Treat both as page-worthy events on a GPU with active production work.

What this means

Before Ampere, an uncorrectable ECC error on a datacenter GPU was always a whole-device event: every context on the GPU was suspect and the GPU needed a reset. Ampere introduced error containment hardware. When a memory fault fires, the GPU can often isolate the damage to the channel or context that triggered it and report that through the Xid payload.

The two codes describe the same underlying fault class (an uncorrectable memory error) with two different blast radii:

The Xid payload also carries a RST: flag that tells you whether the driver requires a GPU reset: RST: No for the contained case, RST: Yes for the uncontained case. Read the flag rather than assuming from the code number alone, especially on MIG systems where the log line also carries a GPU-I: instance identifier.

flowchart TD
  A[Uncorrectable ECC event in dmesg] --> B{Xid code?}
  B -->|Xid 94, Contained| C[Kill and restart faulting pid only]
  C --> D[Other jobs keep running]
  C --> E[Schedule GPU reset when convenient]
  B -->|Xid 95, Uncontained| F[Stop ALL workloads on the GPU]
  F --> G[Drain and reset GPU]
  G --> H[Validate or discard recent outputs]
  B -->|Xid 48| I[DBE, no containment info: treat like Xid 95]

On Volta or older hardware you will never see 94 or 95; containment does not exist there, and the equivalent event is Xid 48, which needs the full-drain response. See NVIDIA Xid 48: Double Bit ECC Error.

Common causes

The last row matters: a Xid 94 that arrives as a sympathetic error during a broader fault cascade is a consequence, not the root cause. Read the surrounding log before you conclude “memory error.”

Quick checks

All read-only.

# 1. Get the full Xid line(s) with payloads and RST flags
dmesg -T | grep -i "NVRM: Xid"
journalctl -k | grep -i "NVRM: Xid"

# 2. Widen the window: what else fired around the same time, on all GPUs?
dmesg -T | grep -iE "NVRM|xid|pcie|nvlink" | tail -50

# 3. ECC counters: volatile (since driver load) and aggregate (lifetime)
nvidia-smi --query-gpu=ecc.errors.corrected.volatile.total,ecc.errors.uncorrected.volatile.total,ecc.errors.corrected.aggregate.total,ecc.errors.uncorrected.aggregate.total --format=csv

# 4. Row remapping state (Ampere+ replacement for page retirement)
nvidia-smi --query-remapped-rows=remapped_rows.correctable,remapped_rows.uncorrectable,remapped_rows.pending,remapped_rows.failure --format=csv
nvidia-smi -q -d ROW_REMAPPER

# 5. Retired pages: meaningful on pre-Ampere; on Ampere+ these fields may report N/A
nvidia-smi -q -d PAGE_RETIREMENT

# 6. Who is on the GPU right now (for scoping the blast radius)
nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv

Two notes on interpretation. First, on Ampere and later, page retirement is replaced by row remapping, so the retired-pages query can show N/A even on a GPU with a real memory fault; use the row remapper output as the source of truth. Second, there are field reports of Xid 94 firing with no increment in any ECC or row-remap counter, with payloads naming non-DRAM units (for example an SM or LTC tag). The containment mechanism covers more than DRAM ECC, so a clean counter readout does not prove the event was spurious.

How to diagnose it

1. Confirm the code and the payload. Pull the exact line from the kernel log. Record the Xid number, the pid, the Contained/Uncontained unit string, the RST: flag, and on MIG systems the GPU-I: instance. The pid tells you which job to kill for a 94; the RST flag tells you whether a reset is mandatory.
2. Check for a cascade. Look at the full kernel log for the surrounding seconds on every GPU in the node. Xid 94 arriving alongside other Xids (NVLink, PCIe, or the same event on multiple GPUs) changes the diagnosis from “one bad row” to “node-level fault.” If you see Xid 79 anywhere, you are in a different incident: see NVIDIA Xid 79: GPU has fallen off the bus.
3. Scope the blast radius. For a 94: the faulting pid’s outputs since its last checkpoint are invalid; other contexts are considered clean by the driver, but if the workload is a tightly coupled distributed job, the killed rank has effectively poisoned the collective anyway. For a 95: every job on the GPU is suspect, full stop.
4. Assess degradation trend. Pull ECC counters and row remapping state. One isolated 94 with flat corrected-error history is likely a soft error. A 94 on a GPU whose corrected error rate has been accelerating, or that has pending or failed row remaps, is a memory device wearing out. Correlate with Xid 48 history and row remapping capacity.
5. Decide the response tier. Isolated 94, clean history: restart the job, watch. 94 with degradation signals, or any 95: drain, reset, quarantine, and start the RMA conversation.
6. Preserve evidence. Capture the dmesg section, nvidia-smi -q output, and the ECC/row-remapper state before any reset. Resets clear volatile counters and you will want the timeline for the vendor case.

Metrics and signals to monitor

Alert on Xid 94/95 as edge-triggered log events (new occurrence since last check), never on the historical presence of the string in logs. Route 95 as a page; route 94 as a page when the GPU had active production work, or as an urgent ticket when it was idle and drained.

Fixes

Xid 94: contained

• Restart the faulting application. The log gives you the pid; the driver has already torn down its context. Do not restart it in place assuming partial state survived. Resume from the last checkpoint.
• Leave other workloads running. That is the point of containment. If the workload is distributed training, though, killing one rank usually means the job restarts as a whole regardless.
• Schedule a GPU reset when convenient. The driver’s own guidance for the contained case is to reset at the next maintenance window, which also lets any pending row remapping be applied. The reset is disruptive to whatever is running at that point, so plan the window. See Resetting a wedged NVIDIA GPU: nvidia-smi –gpu-reset and when only a reboot works.

Xid 95: uncontained

• Drain the entire GPU now. Not just the job in the log line. Every context is potentially corrupted, and continuing to run on the GPU risks silent corruption of outputs you will ship.
• Reset the GPU before restarting anything. NVIDIA’s documented action is that the GPU must be reset before applications can restart. On MIG-enabled systems, drain all instances on the physical GPU first, then reset.
• Invalidate recent outputs. Anything produced by any workload on that GPU since its last known-good checkpoint must be treated as suspect: re-run, re-validate, or discard. This is the expensive part of a 95 and the reason it is the worst-case Xid in this family.
• If the reset fails or the GPU does not come back clean, you are in reboot-and-RMA territory.

Recurring 94s, or 94s escalating to 95

This is the degradation signature that justifies hardware replacement: an increasing rate of Xid 94, especially when a Xid 95 follows on the same GPU. Containment is the GPU buying you grace; a rising contained-error rate means the memory is degrading past what containment can absorb. Quarantine the GPU, open a vendor case with your captured logs and row-remapper state, and do not return it to production training while the rate is climbing.

Prevention

You cannot prevent soft errors, but you can shrink both the probability and the cost.

• Alert on corrected ECC rate acceleration, per GPU, peer-relative. The corrected-error slope is your early warning; by the time a 95 fires you are reacting, not preventing.
• Track row remapping headroom. Pending remaps mean a reset is owed; failure = true means the bank can no longer self-heal.
• Checkpoint frequently enough that invalidating one GPU-context of work is cheap. This is the single biggest cost lever for both 94 and 95.
• Monitor Xids as classified, per-code events. If 43, 48, 63, 94, and 95 all land in one “GPU error” alert, you will misroute the response. See NVIDIA Xid errors: reading NVRM Xid messages in the kernel log.
• Keep persistence mode enabled so monitoring does not go blind between jobs; see NVIDIA persistence mode.

How Netdata helps

• Netdata’s NVIDIA GPU collector tracks ECC corrected and uncorrected error counters per GPU, so the acceleration in corrected errors that precedes a Xid 94 or 95 is visible as a trend, not a surprise.
• Per-second sampling catches the counter deltas around an event that minute-resolution scraping misses.
• Correlating ECC trends with temperature and throttle history on the same dashboard helps separate thermally aggravated memory faults from baseline degradation.
• Row remapping and retired-page style health signals sit next to utilization and memory metrics, so a degrading GPU is visible in context before it takes a job down.
• Pairing metric anomalies with kernel-log Xid events in one view shortens the “which GPU, which job, how bad” loop during the incident itself.
• The broader signal set this article draws on is laid out in the NVIDIA GPU monitoring checklist.

Related guides

• NVIDIA Xid errors: reading NVRM Xid messages in the kernel log
• NVIDIA Xid 48: Double Bit ECC Error
• NVIDIA Xid 79: GPU has fallen off the bus
• NVIDIA Xid 43: GPU stopped processing (the GPU hang)
• Resetting a wedged NVIDIA GPU: nvidia-smi –gpu-reset and when only a reboot works
• nvidia-smi hangs or is unresponsive: a wedged GPU or a stuck driver
• NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver
• How an NVIDIA GPU actually works in production: a mental model for operators
• NVIDIA GPU monitoring checklist: the signals every production GPU fleet needs
• NVIDIA GPU monitoring maturity model: from survival to expert
• NVIDIA persistence mode: why the GPU keeps re-initializing and P-state flaps