NVIDIA GPU ECC errors: corrected, uncorrected, volatile, and aggregate

You opened nvidia-smi -q -d ECC because something looked off, and now you are staring at four sets of counters with non-zero values and no idea which ones matter. Maybe a monitoring alert fired on “ECC errors present”. Maybe a training run produced NaN loss on one node and someone told you to check ECC. Either way, the raw output does not tell you the two things you need to know: has data been corrupted, and is this GPU dying.

The answer depends on two independent axes. Corrected versus uncorrected tells you whether corruption occurred. Volatile versus aggregate tells you over what time window the count accumulated. Most bad operational decisions around ECC come from confusing these axes: paging on an aggregate count that has been stable for a year, or dismissing an accelerating corrected-error rate because “ECC fixed it”.

This article covers the counter model, what to check first, how to tell background noise from progressive hardware failure, and what to do at each severity level.

What this means

ECC (error-correcting code) on datacenter GPUs (V100, A100, H100 and later) detects and repairs memory bit flips. It is enabled by default on datacenter cards. The four counter types are combinations of two questions:

Corrected (single-bit error, SBE): one bit flipped in a memory word, ECC fixed it transparently. The workload saw correct data. A low background rate of SBEs is normal: cosmic rays, aging silicon, and dense HBM all produce soft errors. One to five corrected errors per day is background noise, not an incident.

Uncorrected (double-bit error, DBE): two bits flipped in the same word, ECC could not repair it. Data corruption has occurred. Any output produced by recent work on that GPU is suspect. A DBE typically surfaces in the kernel log as Xid 48, kills the affected CUDA context, and triggers page retirement.

Volatile: counters accumulated since the last driver load. They reset to zero on driver reload or reboot. This is your short-term window: “what has this GPU been doing lately”.

Aggregate: lifetime counters persisted in the GPU’s InfoROM. They survive reboots and driver reloads. This is your lifetime health record.

flowchart TD
  A[ECC counter non-zero] --> B{Corrected or uncorrected?}
  B -->|Corrected SBE| C{Rate stable or accelerating?}
  B -->|Uncorrected DBE| D[Data corruption occurred]
  C -->|Stable low rate| E[Background noise - log and watch]
  C -->|Accelerating| F[Memory degrading - ticket and plan replacement]
  D --> G{Active or recent production work?}
  G -->|Yes| H[Page - validate outputs and quarantine]
  G -->|No, drained or idle| I[Urgent ticket - quarantine GPU]

The operational rules that follow from the model:

• Always read aggregate counters for lifetime health. Volatile counters alone hide a GPU that has been degrading for months across reboots.
• Never reboot to “fix” ECC errors. A reboot zeroes the volatile counters and destroys exactly the short-term history you need to judge the current error rate. The aggregate record survives, but the “is it getting worse right now” signal is gone.
• Alert on deltas, not absolute values. A GPU with an aggregate DBE count of 2 from an incident eight months ago, now running clean, should not page anyone. A new DBE event, or a corrected rate that doubled week over week, should.

One more distinction that changes severity: where the errors are. DRAM/HBM errors are the common case and the hardware has self-healing for them (page retirement, row remapping on Ampere and later). SRAM errors in the register file or caches are more concerning: they sit in the compute path, and the remediation story is thinner. The per-location breakdown in nvidia-smi -q -d ECC tells you which you are dealing with.

Finally, the silence case. Consumer and GeForce GPUs have no ECC at all. Bit flips corrupt data silently with zero signal anywhere. If you run production inference on GeForce hardware, you have no memory-integrity observability; that is a fleet decision to make deliberately, not discover during an incident. Separately, ECC can be disabled on datacenter cards (some cloud providers ship this way for extra usable memory or performance). Disabled ECC looks exactly like a healthy GPU: zero errors, forever, because nothing is being detected. Verify the mode before trusting the counters.

Common causes

Quick checks

All of these are read-only and safe to run during production work.

# 1. Confirm ECC is actually enabled before trusting any counter
nvidia-smi --query-gpu=ecc.mode.current,ecc.mode.pending --format=csv,noheader

# 2. Volatile counters (since last driver load)
nvidia-smi --query-gpu=ecc.errors.corrected.volatile.total,ecc.errors.uncorrected.volatile.total --format=csv,noheader,nounits

# 3. Aggregate counters (lifetime, stored in InfoROM)
nvidia-smi --query-gpu=ecc.errors.corrected.aggregate.total,ecc.errors.uncorrected.aggregate.total --format=csv,noheader,nounits

# 4. Full human-readable dump with per-location breakdown
nvidia-smi -q -d ECC
# Look at whether errors are in device memory (DRAM/HBM) or SRAM
# locations: L1 cache, L2 cache, register file, texture memory

# 5. Retired pages (permanent capacity loss, limit 64)
nvidia-smi --query-retired-pages=retired_pages.single_bit_ecc.count,retired_pages.double_bit_ecc.count,retired_pages.pending --format=csv,noheader

# 6. Row remapping status (Ampere and later only)
nvidia-smi --query-remapped-rows=remapped_rows.correctable,remapped_rows.uncorrectable,remapped_rows.pending,remapped_rows.failure --format=csv,noheader

# 7. Correlate with kernel log events
dmesg -T | grep -i "NVRM: Xid"
# Xid 48 = double-bit ECC error, Xid 63 = page retirement/remap event,
# Xid 64 = remapping failure, Xid 92 = high SBE rate, Xid 95 = uncontained ECC

# 8. Check thermal correlation
nvidia-smi --query-gpu=temperature.gpu,temperature.memory,clocks_event_reasons.active --format=csv,noheader

Note on field names: the real queryable fields are ecc.errors.corrected.* and ecc.errors.uncorrected.*. Names like ecc.errors.single_bit_total do not exist. Verify any field against nvidia-smi --help-query-gpu before wiring it into monitoring.

How to diagnose it

1. Verify ECC is on. If ecc.mode.current is disabled, every counter reading is meaningless. Zero errors with ECC off is not health, it is blindness. If current and pending differ, someone toggled it and the change is waiting on a reset.

2. Read both time windows. Pull volatile and aggregate totals. Volatile high with aggregate only slightly higher means recent activity. Aggregate high with volatile at zero since driver load means old history, possibly already handled. You need both to place the GPU on the timeline.

3. Classify corrected versus uncorrected. Any new uncorrected (DBE) event means corruption occurred. Check when it happened (Xid 48 or 95 timestamps in dmesg) against what was running. If production training or inference was active, treat recent outputs and checkpoints as suspect.

4. Break down by location. In the nvidia-smi -q -d ECC output, look at whether counts sit in device memory or in SRAM locations (L1, L2, register file, texture). DRAM/HBM errors feed into page retirement and row remapping. SRAM uncorrectable errors are more serious and push harder toward replacement.

5. Compute the rate, not the count. Divide corrected volatile errors by driver uptime to get errors per day, and compare against the same GPU’s prior weeks and against peer GPUs of the same model running similar work. A stable 2 per day is noise. A rate that went from 2 per day to 30 per day is a memory device on its way out. Peer-relative comparison is the fastest way to isolate one bad card from an environmental cause.

6. Check the self-healing budget. Retired pages have a hard limit of 64 per GPU; at that point no further retirement is possible and future DBEs land on live memory. On Ampere and later, check remapped_rows.failure: if it is true, the spare-row pool for a bank is exhausted and the GPU is RMA-eligible per NVIDIA policy. remapped_rows.pending or retired_pages.pending = Yes means a reboot is needed to apply a recorded retirement or remap. The pending work waits for a maintenance window; do not reboot reactively just to apply it during an incident.

7. Correlate with temperature. Sustained high temperature accelerates bit flips and can produce ECC errors that mimic memory degradation. If the error rate tracks thermal excursions and drops when cooling recovers, fix the cooling problem before condemning the GPU.

8. Decide severity. New DBE with recent production work: page, quarantine the GPU, validate or discard recent outputs. Accelerating SBE rate or rising retired pages: ticket, plan replacement. Stable low SBE rate: log it, trend it, move on.

Metrics and signals to monitor

The single most important alerting rule: page on events (a new DBE, a new Xid 48/95), ticket on trends (SBE rate acceleration, retired-page growth), and never alert on the raw value of a persistent aggregate or latched state counter. A GPU sitting at remapped_rows.failure = true for three weeks awaiting RMA should not fire a page every check cycle.

Fixes

If you see a new uncorrected error (DBE)

Corruption has already happened; there is nothing to fix in place. Quarantine the GPU from the scheduler so no new work lands on it. Identify what ran since the last clean checkpoint and treat outputs and saved checkpoints from that window as suspect. The driver will typically retire the affected page (Xid 63 follows the DBE). After the GPU is drained, run diagnostics and plan replacement if this is a repeat event or if retired pages are climbing. Do not return the card to production training on the assumption that page retirement solved the underlying problem.

If the corrected error rate is accelerating

This is the early-warning phase and it buys you time. Trend the rate daily, compare against peers, and check the retirement and remapping budgets. If the rate is clearly accelerating, schedule the GPU for replacement on your terms rather than the hardware’s. Keep it running if the workload tolerates it, since corrected errors are still being repaired transparently, but keep it out of long-running uncheckpointed jobs.

If errors correlate with temperature

Fix the cooling, not the GPU. Check fan or chassis cooling, airflow, and ambient temperature. If the ECC rate returns to baseline after thermal remediation, the memory may be fine.

If retired pages or remaps are pending

retired_pages.pending = Yes or remapped_rows.pending > 0 means the action is recorded but applies on reset. Schedule a reboot in a normal maintenance window. This is one of the few cases where rebooting is the correct action, and it is planned, not reactive.

If ECC is disabled

Re-enable it and plan the reset that applies the change. Until then, treat the GPU as having no memory-integrity signal. If a cloud provider ships instances with ECC off, raise it with them; for financial, medical, or scientific workloads this is a hard requirement.

If you are on consumer hardware

There is no software fix. GeForce GPUs lack ECC entirely. If memory integrity matters for the workload, the answer is different hardware, not different monitoring.

Prevention

• Trend rates, not counts. Store volatile and aggregate counters per GPU over time. Rate-of-change and peer-relative comparison catch degradation weeks before the first DBE.
• Track the self-healing budget. Retired pages toward 64 and row-remap availability toward exhaustion are your runway metrics. Alert on new increases, not raw values.
• Alert on events for DBE, trends for SBE. Edge-triggered paging on new Xid 48/95 or uncorrected deltas; ticket-level alerts on corrected-rate acceleration. Never level-trigger on persistent counters.
• Verify ECC mode as configuration drift. Include ecc.mode.current in your per-node baseline checks. A GPU that silently lost ECC has lost all detection.
• Include ECC fields in exporter configuration. If you use dcgm-exporter for Prometheus, note that the ECC counters (DCGM_FI_DEV_ECC_SBE_VOL_TOTAL, DCGM_FI_DEV_ECC_DBE_VOL_TOTAL, DCGM_FI_DEV_ECC_SBE_AGG_TOTAL, DCGM_FI_DEV_ECC_DBE_AGG_TOTAL) and the retired-page fields are commented out of the default counter set. Uncomment them or you have no ECC telemetry at all. Check the default-counters.csv shipped with your dcgm-exporter version, as which fields are commented out varies by release.
• Baseline per GPU model and workload. SBE background rates differ across HBM densities and altitudes. A fleet-wide fixed threshold will either cry wolf or miss the one card drifting away from its peers.
• Correlate ECC with thermals. Keep temperature and ECC series side by side so thermal-induced flips do not trigger unnecessary RMAs.

How Netdata helps

• Both counter windows, per GPU. Netdata collects volatile and aggregate ECC totals per GPU, so you can see short-term rate and lifetime history on the same chart without manual nvidia-smi runs.
• Rate visualization instead of raw counts. Plotting corrected errors as a per-second or per-minute rate makes acceleration visible at a glance, which is the actual degradation signal, and flat baselines stay visibly flat.
• Per-location breakdown. DRAM versus SRAM error counts are charted separately, so the more severe register file and cache errors stand out instead of hiding in a single total.
• Correlation with temperature and throttle reasons. ECC series sit alongside GPU die and HBM temperature, power draw, and clock throttle events on one dashboard, which is exactly the cross-check needed to separate thermal-induced flips from hardware degradation.
• Anomaly detection on error rates. Per-GPU ML anomaly scoring flags a card whose corrected-error rate deviates from its own baseline and from fleet peers, catching the straggler before the first DBE.

Related guides

• NVIDIA GPU monitoring checklist: the signals every production GPU fleet needs
• How an NVIDIA GPU actually works in production: a mental model for operators
• NVIDIA GPU monitoring maturity model: from survival to expert
• Resetting a wedged NVIDIA GPU: nvidia-smi –gpu-reset and when only a reboot works
• NVIDIA Xid 43: GPU stopped processing (the GPU hang)
• nvidia-smi hangs or is unresponsive: a wedged GPU or a stuck driver
• CUDA out of memory: diagnosing NVIDIA GPU framebuffer exhaustion
• NVIDIA GPU memory leak: framebuffer usage climbing without a plateau