$ guides / nvidia-gpu / nvidia-gpu-ecc-disabled ▌

Operations Guides

NVIDIA GPU ECC disabled: the silent data-corruption risk

A datacenter GPU with ECC disabled does not crash when memory goes bad. It keeps running, keeps returning answers, and keeps writing checkpoints, while single-bit errors corrupt whatever it is computing. There is no error counter incrementing, no XID in dmesg, no page in the middle of the night. The usual discovery path is a training run that diverges to NaN on one specific node, or an inference service returning subtly wrong results, days or weeks after someone flipped the setting.

ECC on datacenter GPUs (V100, A100, H100 and later) is enabled by default, but it can be turned off with nvidia-smi -e 0 to reclaim the framebuffer used for parity, roughly 6% of total capacity. Cloud providers sometimes ship instances this way. Administrators sometimes do it deliberately to fit a model that otherwise will not fit. Either way, the trade is almost never worth it on production hardware, and it is frequently not tracked as the configuration change it is.

This article covers what ECC mode actually controls, why disabling it produces silent corruption rather than detectable failures, how to verify ECC state across a fleet, and how to re-enable it safely. For the error counters ECC produces when it is on, see NVIDIA GPU ECC errors: corrected, uncorrected, volatile, and aggregate.

What ECC mode actually controls

GPU framebuffer memory (HBM on most datacenter cards, GDDR on some datacenter cards and on workstation and consumer cards) is subject to the same soft-error physics as any DRAM: cosmic rays, aging silicon, thermal stress, marginal cells. ECC dedicates a portion of memory capacity and bandwidth to storing parity information so the memory controller can detect and correct errors in flight.

With ECC enabled:

A single-bit error is corrected transparently. The correctable error counter increments, the workload never notices.
A double-bit (uncorrectable) error is detected. The driver raises XID 48 (or XID 95 for uncontained errors on A100/H100), kills the affected CUDA context, and blocks further CUDA operations on that device until a GPU reset. The failure is loud.

With ECC disabled, neither path exists. The parity hardware is off. A bit flip in a model weight, an activation, or a gradient passes through silently and the computation continues with corrupted data. There is no counter, because there is no detection. That is what “silent data corruption” means operationally: the failure has no hardware-side signal at all.

One corollary trips up operators reading dashboards: if ECC is disabled, the ECC error fields report N/A or zero. A dashboard showing zero ECC errors on a GPU with ECC disabled is not a healthy GPU. It is a GPU that cannot see its own errors. Always interpret ECC counters alongside ecc.mode.current.

Why the failure is worse than it looks

Operators often assume silent corruption will surface as NaN or INF in training loss and therefore be self-announcing. That assumption is unsafe. Gate-level fault injection research on GPU workloads found that NaN and INF account for only about 1% of silent-corruption outcomes. The other 99% are numerically plausible wrong answers: slightly degraded model quality, mispredictions, corrupted checkpoints that pass a load check. Software-level NaN guards catch only the sliver of cases that produce non-numeric results.

The typical discovery pattern is training loss converging normally right up until it does not, on one specific GPU and node combination. By then the corruption window may span days of checkpoints.

With ECC enabled, the same underlying hardware degradation follows a detectable arc: correctable error rate accelerates, pages get retired, rows get remapped, and only then does an uncorrectable error occur. That arc is the early warning system. Disabling ECC deletes the early warning system and keeps the hardware degradation. See NVIDIA GPU HBM progressive failure: from single-bit errors to a dead GPU for what that arc looks like when ECC is on.

flowchart TD
  A[Memory bit flip in framebuffer] --> B{ECC mode?}
  B -->|Enabled, single-bit| C[Corrected transparently
SBE counter increments]
  B -->|Enabled, double-bit| D[XID 48 / 95
CUDA context killed
driver blocks device]
  B -->|Disabled| E[No detection, no counter
computation continues]
  E --> F[NaN loss or divergence
about 1% of cases]
  E --> G[Numerically plausible
wrong results: 99%]
  G --> H[Corrupted checkpoints
bad inference, no alert]
  C --> I[Trend SBE rate:
early warning of degradation]
  D --> J[Page: validate recent work
reset GPU]

The trade people are actually making

The motivation for nvidia-smi -e 0 is real: ECC consumes roughly 6% of framebuffer for parity, and disabling it can add a small amount of memory bandwidth as well. On a 48 GiB L40 that is about 3 GiB. When a model or batch size is 2 GiB over the line, disabling ECC looks like a free fix. The exact reclaim percentage varies by GPU generation, and NVIDIA does not publish a single authoritative figure — verify against the card’s own nvidia-smi -q -d MEMORY output (memory with ECC on versus off).

Reported performance impact of ECC enabled varies widely by workload (published numbers range from about 5% to 15%), and most published measurements are for rendering and gaming rather than CUDA compute. There is no widely reproduced ML-specific number; benchmark your own workload if the tradeoff matters.

The honest framing: disabling ECC converts a small, known, bounded capacity and bandwidth cost into an unbounded, undetectable correctness risk on every workload the GPU touches. On shared or long-running production infrastructure, that is a bad trade. The right fixes for “model does not fit” are capacity-side: smaller batch, sharding, quantization, MIG placement, or a bigger SKU.

Current vs pending: why the setting lies to you

ECC mode changes do not take effect immediately. The driver stages them:

ecc.mode.current is the mode in effect right now.
ecc.mode.pending is the mode that will apply after a GPU reset or reboot.

If someone ran nvidia-smi -e 0 yesterday but the node has not been reset, current is still Enabled and pending is Disabled. The compliance problem runs in both directions: a GPU can show ECC enabled right now and be scheduled to lose it at the next reboot, and a GPU you just re-enabled is still running unprotected until you actually reset it. Any verification or enforcement check must read both fields, and any drift check should flag a pending value that differs from current.

There are also platform-specific persistence quirks. On some driver versions the pending mode does not survive reboot as expected on workstation cards — community reports include RTX A4000 on driver 530.30.02 — so a setting you believed was staged can revert silently.

Which GPUs can even have ECC

Datacenter GPUs (V100, A100, H100, L40 and similar): ECC supported, enabled by default.
Consumer GeForce cards: no ECC hardware at all. ECC query fields return N/A. There is nothing to verify or enable.
Workstation cards vary by generation, and defaults differ from datacenter cards. Notably, the RTX 5090 (Blackwell generation) removed the user-facing ECC toggle entirely; GDDR7 carries on-die ECC per the JEDEC spec, but whether the controller uses it by default and whether NVIDIA will expose control via a driver or VBIOS update is not documented.

On a mixed fleet, treat “ECC query returns N/A” and “ECC disabled” as different conditions: the first is a hardware property, the second is a configuration decision.

Verifying ECC state across the fleet

This is a config-compliance check, and it belongs in the same drift-detection class as persistence mode, compute mode, and power limits. All commands are read-only.

# Current and pending ECC mode for every GPU on the node
nvidia-smi --query-gpu=index,ecc.mode.current,ecc.mode.pending --format=csv

# Full ECC detail for one GPU (mode plus error counters)
nvidia-smi -q -d ECC -i 0

What to look for:

Output	Meaning	Action
`current=Enabled, pending=Enabled`	Expected production state	None
`current=Disabled` on a datacenter GPU	Running unprotected now	Investigate origin, plan re-enable
`current=Enabled, pending=Disabled`	Staged to lose ECC at next reset	Find who changed it, revert pending
`current=Disabled, pending=Enabled`	Re-enable staged but not applied	Schedule the reset; GPU is still unprotected
N/A	Consumer card, no ECC hardware	Exclude from this check

For fleet-wide verification, wrap the query in whatever config-management or drift-detection tooling you already run (Ansible, a Prometheus textfile collector, or a scheduled job writing to your monitoring pipeline). The check is cheap; run it on a schedule, not once. Define the expected baseline per node class: a rendering workstation pool may legitimately differ from a training cluster, but the difference should be policy, not accident.

If you find ECC disabled unexpectedly, treat it as an investigation, not just a toggle to flip back. Plausible origins include a cloud provider default, a manual change to fit a workload, a driver update side effect, or an incomplete maintenance procedure. Until you know which, you do not know what else was changed. Also consider the exposure window: any checkpoints, results, or outputs produced while ECC was disabled on hardware with an underlying memory problem are suspect, though in most cases you will have no way to bound that retroactively.

Re-enabling ECC safely

# Stage ECC enable (per-GPU with -i, or all GPUs without it)
sudo nvidia-smi -i 0 -e 1

# Confirm the pending mode took
nvidia-smi --query-gpu=index,ecc.mode.current,ecc.mode.pending --format=csv

Then the disruptive part: the change only applies after a GPU reset. nvidia-smi -r resets the GPU and requires that no processes are using it. On a busy node that means draining workloads first, and on many systems a full reboot is the more reliable path. Do not run a GPU reset on a node with active production work; it will kill every CUDA context on the device.

One known rough edge: on some systems, particularly with distribution-packaged drivers, the staged change appears not to survive or apply cleanly. Reported fixes involve dropping to a non-graphical runlevel, unloading the NVIDIA kernel modules (nvidia-drm, nvidia-modeset, nvidia-uvm), and then resetting. If pending shows Enabled but current still shows Disabled after a reboot, that is the class of problem you are in. The module-unload route is an operator-reported workaround (reported on apt-installed drivers, including on Tesla L40), not an NVIDIA-documented procedure.

After the reset or reboot, verify before returning the node to service:

# Both fields must now read Enabled
nvidia-smi --query-gpu=index,ecc.mode.current,ecc.mode.pending --format=csv

# ECC counters should now be live (zero is fine; N/A is not)
nvidia-smi --query-gpu=ecc.errors.corrected.volatile.total,ecc.errors.uncorrected.volatile.total --format=csv,noheader

Signals to watch

Signal	Why it matters	Warning sign
`ecc.mode.current`	The actual protection state right now	Disabled on any datacenter GPU in production
`ecc.mode.pending`	Reveals staged changes before they bite	Differs from current
ECC volatile counters	Only meaningful when ECC is on	Zero or N/A alongside `current=Disabled` reads as “healthy” on naive dashboards
XID 48 / 95 in dmesg	Uncorrectable error events (only possible with ECC on)	Any new event on a GPU with recent production work
Training loss / inference quality per node	The user-visible symptom of corruption	Divergence or NaN localized to one node

The first two rows are the compliance check this article is about. The rest only work if the first two are right.

How Netdata helps

Netdata’s NVIDIA GPU collector surfaces ECC mode alongside the ECC error counters, so “zero errors” is never displayed without the context of whether detection is even on.
Per-second collection of ecc.errors.corrected.volatile.total and uncorrected totals lets you see rate acceleration in correctable errors, the leading indicator of the memory degradation ECC exists to expose.
Correlating ECC state and counters with XID events, retired pages, and row remapping on the same GPU distinguishes a healthy card from one in the early stages of progressive memory failure.
Fleet-wide views make ecc.mode.current drift visible as a per-node outlier, which is how a single misconfigured or cloud-defaulted node actually gets found.