$ guides / nvidia-gpu / nvidia-gpu-xid-92-high-sbe-rate ▌

Operations Guides

NVIDIA Xid 92: high single-bit ECC error rate

You found a line like this in the kernel log:

NVRM: Xid (PCI:0000:3b:00.0): 92, pid='<unknown>', name=<unknown>, High single-bit ECC error rate

Nothing crashed. No job failed. nvidia-smi still works, the GPU is still training or serving, and there is no Xid 48 or 79 demanding an immediate reboot. That is exactly the point of Xid 92: the driver has seen the correctable single-bit ECC error rate cross its internal threshold. Nothing is broken yet. Something will be.

This article covers what Xid 92 means, how to confirm the degradation is real and progressive, and how to decide when the GPU goes on the replacement list. For how Xid messages work in general, see NVIDIA Xid errors: reading NVRM Xid messages in the kernel log.

What this means

Datacenter GPUs run with ECC enabled by default. Single-bit errors (SBEs) are corrected transparently in hardware; a low background rate is normal noise. Double-bit errors (DBEs) are uncorrectable and mean data corruption, which is the Xid 48 / Xid 95 path.

Xid 92 is the driver’s own escalation of the SBE story. The driver tracks the rate of correctable errors and, when that rate crosses an internal threshold, emits Xid 92 to say: this is no longer background noise. NVIDIA does not publish the exact threshold, so you cannot predict when it will fire from raw counts alone. Treat any Xid 92 as confirmation of what your SBE rate trend should already have been showing, and put the GPU on a watch or replacement list.

Two things Xid 92 is not:

It is not an acute fault. The driver does not require a reset, and no running workload needs to be killed because of this message alone. The GPU keeps operating correctly because ECC is still correcting every error it sees.
It is not benign. “Corrected” does not mean “free.” An accelerating SBE rate is the leading indicator in the memory degradation pattern: corrected errors climb, rows get remapped, pages get retired, and eventually a double-bit error lands and corrupts whatever was in flight. Xid 92 tells you that you are somewhere along that curve.

flowchart TD
  A[SBEs accumulate on HBM] --> B{ECC corrects each error}
  B --> C[Xid 92 fires: SBE rate over driver threshold]
  C --> D{Is the rate still accelerating?}
  D -->|Yes| E[Rows remapped / pages retired consumed]
  E --> F{Spare rows or retired-page budget exhausted?}
  F -->|Yes| G[Next uncorrectable error: Xid 48 or 95 - data corruption]
  F -->|No| H[GPU on watch list, replacement scheduled]
  D -->|No, stable| H
  G --> I[Emergency response: validate recent work, reset, RMA]

Common causes

Cause	What it looks like	First thing to check
Progressive HBM degradation	SBE rate climbing over days or weeks on one GPU, aggregate corrected count growing steadily, remapped rows appearing	Compare corrected error rate against the GPU’s own history and against peer GPUs of the same model
Thermal-accelerated degradation	Elevated SBE rate coinciding with high or rising HBM temperature	`temperature.memory` trend on this GPU versus its baseline
Manufacturing defect in a batch	Multiple GPUs from the same purchase batch showing similar SBE acceleration at similar ages	Fleet-wide comparison of corrected error rates by GPU model and age
Transient environmental noise	A short burst of SBEs that does not repeat; rate returns to baseline	Check whether the rate delta is a one-time step or a sustained new slope

A low, flat rate of corrected errors (a handful per day) is expected background and does not produce Xid 92. The message means the driver saw a rate it considers abnormal for this hardware.

Quick checks

All of these are read-only and safe to run on a production node.

# 1. Confirm Xid 92 and check for worse Xids nearby
dmesg -T | grep -i "NVRM: Xid"

# 2. Corrected and uncorrected ECC counters, volatile (since driver load)
nvidia-smi --query-gpu=ecc.errors.corrected.volatile.total,ecc.errors.uncorrected.volatile.total --format=csv,noheader,nounits

# 3. Lifetime aggregate counters (persist across driver reloads, stored in InfoROM)
nvidia-smi --query-gpu=ecc.errors.corrected.aggregate.total,ecc.errors.uncorrected.aggregate.total --format=csv,noheader,nounits

# 4. Row remapping status (Ampere and later)
nvidia-smi --query-remapped-rows=remapped_rows.correctable,remapped_rows.uncorrectable,remapped_rows.pending,remapped_rows.failure --format=csv,noheader

# 5. Retired pages
nvidia-smi -q -d PAGE_RETIREMENT

# 6. HBM temperature right now
nvidia-smi --query-gpu=temperature.memory --format=csv,noheader,nounits

# 7. Verify ECC is actually enabled (zero counters with ECC off means blind, not healthy)
nvidia-smi --query-gpu=ecc.mode.current --format=csv,noheader

Things to note while running these:

The uncorrected (double-bit) count is the critical field in checks 2 and 3. If it is nonzero and new, you are no longer in Xid 92 territory; you are in Xid 48 territory. See NVIDIA Xid 48: Double Bit ECC Error.
Volatile counters reset on driver reload and GPU reset. Use them for rate tracking; use aggregate counters for lifetime history.
remapped_rows.pending greater than zero means a remap has been recorded but not applied; it takes effect on GPU reset. The degraded row is still in use until then.

How to diagnose it

The goal is not to “fix” Xid 92. The goal is to establish the trajectory of the degradation and decide where this GPU sits on the path to replacement.

Confirm the event is Xid 92 and not something worse. Grep dmesg for all recent Xids on that PCI address. If you also see Xid 48, 64, or 95, escalate immediately; Xid 92 is no longer the headline.
Establish the SBE rate, not the count. A corrected count of 5,000 means nothing without a time window. Sample ecc.errors.corrected.volatile.total twice, hours or a day apart, and compute errors per hour. Compare against the GPU’s own prior weeks and against peer GPUs running similar workloads. A rate that doubles over days is the warning sign.
Check row-remap consumption (Ampere+). Look at remapped_rows.correctable and remapped_rows.uncorrectable. Correctable remaps are the hardware doing wear leveling; uncorrectable remaps are more concerning. remapped_rows.failure true means a row-repair failure rule was hit and the affected region can no longer be self-healed by that mechanism. Per NVIDIA’s RMA policy, a validated remap-failure flag is the DRAM RMA criterion. Note that failure is latched state in InfoROM: once true it stays true, so alert on the transition, not the boolean.
Check retired pages. Retired pages persist in InfoROM across reboots and count against the pre-Ampere page-retirement limit of 64 pages. Any increase is permanent hardware degradation. Approaching the limit means future double-bit errors have nowhere to be retired to.
Correlate with HBM temperature. Pull the temperature.memory trend for the same period. Sustained high HBM temperature both accelerates degradation and independently throttles memory clocks. If the GPU runs hot under the same workload it used to run cool on, the cooling path (or the card) is aging too.
Quarantine the risk decision, not the GPU. Unless you also have uncorrected errors, the correct response is: keep the GPU in service if you must, tag it in inventory as a replacement candidate, tighten monitoring on it, and schedule replacement at the next maintenance window. If it runs long-horizon training where a mid-run DBE would corrupt a checkpoint, weight that into how soon “next window” needs to be.
Open a vendor/support case if the card is under warranty. Xid 92 with an accelerating SBE rate, growing remap counts, or any remap failure is the evidence package the vendor will ask for. Capture nvidia-smi -q output, the dmesg excerpt, and your SBE rate trend before rebooting anything.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Corrected ECC error rate (delta of `ecc.errors.corrected.volatile.total`)	The signal Xid 92 confirms; the earliest leading indicator of HBM failure	Rate accelerating week over week, or clearly above peer GPUs of the same model
Uncorrected ECC errors (`ecc.errors.uncorrected.volatile.total`, new Xid 48/95)	Data corruption has occurred; recent outputs may be invalid	Any new event (delta > 0) on a GPU with recent production work
Row remapping (`remapped_rows.*`)	Shows how much self-repair capacity has been consumed and whether a repair rule failed	Uncorrectable remaps increasing, `pending > 0`, or a new transition of `failure` to true
Retired pages (`retired_pages.*`)	Permanent capacity loss, bounded by a hard retirement limit	Any increase; trajectory toward the limit
HBM temperature (`temperature.memory`)	Heat accelerates degradation and throttles memory clocks independently of the die	Sustained approach to the memory thermal limit, or baseline drifting upward over months
Xid 63 / 64 in dmesg	Remap events (63, informational) and remap failures (64) show the self-healing system at work and at its limit	Xid 64 as a new event, or rapid repetition of Xid 63

The common failure in monitoring this is alerting on raw counter values. Corrected counts and remap counts are monotonically increasing persistent state. A GPU that had pages retired six months ago and has been stable since should not re-alert. Track deltas and transitions, and alert on acceleration, not accumulation.

Fixes

There is no fix that makes the HBM healthy again. The fixes are decisions about service life.

Keep in service under watch (default for Xid 92 alone)

The driver itself does not require action. If the SBE rate is elevated but stable, no uncorrected errors exist, remap consumption is low, and the workload is checkpoint-friendly or latency-insensitive, keeping the GPU in service with tightened monitoring is defensible. Tag it in inventory so the next oncall does not re-derive the whole investigation.

Apply pending remaps with a GPU reset

If remapped_rows.pending or retired_pages.pending is nonzero, the remediation is already recorded but not applied. A GPU reset (or reboot) applies it. This is disruptive: all processes on the GPU must be drained first. See Resetting a wedged NVIDIA GPU: nvidia-smi –gpu-reset and when only a reboot works for the procedure and its limits. Resetting does not reduce the SBE rate; it only moves the known-bad rows out of use.

Schedule replacement

The right trigger set for planned replacement, per the degradation path above:

SBE rate clearly accelerating over days to weeks
Any uncorrectable remapped row, or remapped_rows.failure transitioning to true
Retired page count climbing, especially double-bit retirements
Any double-bit ECC event, full stop

Drain the node, replace or RMA the card, and validate the replacement before returning capacity. A GPU that degrades along this curve does not recover; it only gets to the DBE faster or slower.

Do not “fix” it by disabling ECC

Disabling ECC stops the errors from being reported, not from happening. It converts a monitored degradation into silent data corruption. On a datacenter GPU doing production work, ecc.mode.current should be Enabled; if you find it disabled during these checks, that is a separate ticket.

Prevention

You cannot prevent HBM aging, but you can stop being surprised by it. Xid 92 should never be the first time you learn a GPU’s SBE rate is climbing.

Trend the corrected ECC rate per GPU. Sample the volatile corrected counter regularly and compute deltas. Compare each GPU against its own baseline and against same-model peers. Acceleration relative to peers catches bad hardware before the driver’s internal threshold does.
Watch HBM temperature as a degradation accelerant. Cards that chronically run near their memory thermal limit age faster. Track the temperature baseline per GPU over months; upward drift is a cooling or hardware problem in progress.
Alert on deltas and transitions, not levels. New Xid events, new uncorrected errors, failure transitioning to true, retired pages increasing. Raw persistent counters in InfoROM will page you forever about already-handled history if you alert on their absolute values.
Inventory the remap and retirement budget. Periodic fleet reports of remapped_rows.* and retired_pages.* turn “which GPUs are candidates for next quarter’s replacements” into a query instead of an incident.
Classify Xids correctly in your log pipeline. Xid 92 routes to INFO/ticket as a tracking signal; Xid 48 and 95 route to page-on-new-event; Xid 64 routes to ticket with transition awareness. Misrouted severities either page on noise or sleep through corruption. The NVIDIA GPU monitoring checklist covers the full signal set, and the monitoring maturity model places this kind of delta-aware ECC tracking at the mature level.

How Netdata helps

Netdata collects the NVIDIA ECC counters (corrected and uncorrected, volatile and aggregate) per GPU at high frequency, so the SBE rate trend that precedes Xid 92 is visible as a slope, not a post-incident reconstruction.
Row remapping and retired page metrics are charted alongside ECC errors, so you can see remap consumption rising in the same view as the corrected error rate that is driving it.
HBM memory temperature is collected next to die temperature, letting you correlate thermal drift with error rate acceleration on the degrading card.
Because collection is per-second and per-GPU, peer comparison across a node or fleet is straightforward: the outlier card stands out against same-model neighbors.
Anomaly detection on the corrected ECC series flags rate acceleration on a single GPU even before it crosses any static threshold, which is the earliest practical version of what Xid 92 eventually confirms.
Kernel log monitoring for NVRM: Xid lines closes the loop: the metric trend and the driver’s own escalation event land in one place.