NVIDIA GPU configuration drift: ECC, persistence, power limits, compute mode

A GPU that passes every health check can still be misconfigured. ECC turned off, persistence mode lost after a reboot, a power limit set below default, compute mode flipped to Exclusive. None of these produce an XID error or a crashed job on their own. They produce silent data corruption, unexplained throttling, scheduling failures, and monitoring gaps that surface days later as a different incident.

Configuration drift is a control-plane problem, not a hardware problem. The fix is comparing the live configuration of each GPU against a known-good per-model baseline and treating every unexplained difference as a signal. This guide covers the four knobs that matter most (ECC mode, persistence mode, power limit, compute mode), plus InfoROM integrity, and how to tell an unauthorized change from a failed automation run or a maintenance leftover.

What counts as configuration drift

Drift is any difference between the live GPU configuration and the baseline you expect for that node class. The fields that matter: persistence mode, compute mode, ECC mode, MIG configuration, power limits, and driver/firmware versions. A single read-only command captures most of it:

# Configuration snapshot for drift comparison
nvidia-smi --query-gpu=index,persistence_mode,compute_mode,ecc.mode.current,mig.mode.current,power.limit,power.default_limit,driver_version,vbios_version --format=csv,noheader

Store this output per node, per GPU model, at provisioning time. Drift detection is then a diff, not a judgment call. Two caveats:

• Some drift is intentional. Datacenter operators cap power below TDP for rack density. Compare against fleet policy, not factory defaults.
• Several of these settings do not survive a reboot. A node that was correct yesterday can be wrong this morning with no human involvement at all.

flowchart TD
  A[Scheduled config snapshot per GPU] --> B[Diff against per-model baseline]
  B -->|Match| C[No action]
  B -->|Drift detected| D{Change explained?}
  D -->|Approved maintenance or policy| E[Update baseline with change record]
  D -->|Failed automation run| F[Re-apply setting, fix automation]
  D -->|Maintenance leftover| G[Restore baseline, close the loop]
  D -->|No record anywhere| H[Treat as unauthorized: check processes, access logs]

ECC mode: the silent corruption knob

ECC is enabled by default on datacenter GPUs. When it is disabled, memory errors become silent data corruption: no detection, no correction, no alerting possible. ECC disabled on a GPU processing production data is a high-priority ticket.

Check current and pending state:

# ECC mode, current vs pending
nvidia-smi --query-gpu=ecc.mode.current --format=csv,noheader
nvidia-smi -q | grep -A1 "ECC Mode"

ECC changes take effect only after a reboot, which is why nvidia-smi -q reports both Current and Pending. Drift shows up in two ways: ECC disabled outright, or a Pending value that disagrees with Current because someone ran nvidia-smi -e and the reboot never happened (or happened and did not apply).

One trap: zero ECC errors on a production GPU can mean ECC is off, not that the GPU is healthy. Verify ECC mode before interpreting error counts. Relatedly, page retirement only occurs when ECC is enabled, so a disabled-ECC GPU also cannot self-heal bad memory pages.

For the failure mode when ECC errors do accumulate, see NVIDIA GPU ECC errors: corrected, uncorrected, volatile, and aggregate.

Persistence mode: reboot-erased by design

Persistence mode keeps the driver initialized when no CUDA contexts are active. Without it, the driver can unload between jobs, which causes seconds of delay on the first CUDA call, monitoring gaps, P-state flapping, and race conditions in job schedulers. Running production without persistence mode is one of the most common operational mistakes.

# Persistence mode per GPU
nvidia-smi --query-gpu=persistence_mode --format=csv,noheader

This is the most common drift finding on rebooted nodes because nvidia-smi -pm 1 does not persist across reboots. The durable fix is a boot-time mechanism: a systemd unit that re-applies the setting, or the nvidia-persistenced daemon, which holds the NVIDIA character device files open. If your baseline says Enabled and the node reboots, drift detection will catch the regression on the first post-boot snapshot. NVIDIA documentation describes legacy persistence mode as near end-of-life, to be eventually deprecated in favor of the Persistence Daemon; it does not name a specific driver branch where the warning first appears.

If your scheduler sees intermittent “GPU not ready” failures right after node reboots, or brief nvidia-smi unreachability between jobs, check this first.

Power limits: drift that looks like a hardware problem

A reduced power limit produces a confusing incident: clocks drop, throughput falls, temperatures look fine, and the throttle reason is sw_power_cap. Operators burn time suspecting cooling or silicon when the actual cause is a configuration change.

# Power limit fields worth baselining
nvidia-smi --query-gpu=power.draw,power.limit,power.default_limit,power.max_limit,enforced.power.limit --format=csv,noheader,nounits

How to read the comparison:

Power limits set with nvidia-smi -pl take effect immediately but do not survive a reboot, so drift runs in both directions: a cap that disappears after reboot (sudden power and thermal consumption the rack was not budgeted for) or a cap that appears (unexplained throttling). One cause worth taking seriously: crypto miners commonly cap power to raise efficiency, so an unexplained lowered power limit on a shared node warrants a security check, not just a config restore. Cross-reference with nvidia-smi pmon -c 1 for unexpected processes.

Some GPUs do not support changing the power limit at all and will refuse the setting. That is a hardware/firmware limitation, not a driver bug. Baseline per model so you know which nodes can even have this drift.

Compute mode: drift that blocks jobs outright

Compute mode controls which processes may run on the GPU:

# Compute mode per GPU
nvidia-smi --query-gpu=compute_mode --format=csv,noheader

The operational impact is abrupt and confusing: the GPU is healthy, reachable, and idle, yet jobs refuse to start. If a previously multi-tenant node suddenly rejects co-scheduled work, check compute mode before touching the scheduler. Compute mode resets to Default on reboot, so a node that legitimately runs Exclusive or Prohibited needs that re-applied at boot, and drift detection should treat an unexpected reset-to-Default as a finding too. Note that nvidia-smi --gpu-reset may not reliably apply pending configuration changes on Linux; the nvidia-smi manual explicitly warns that a full reboot may still be required for pending mode changes (including ECC mode) to take effect.

The older EXCLUSIVE_THREAD mode is removed on current drivers; the exclusive setting maps to EXCLUSIVE_PROCESS. If old automation references the thread mode, expect a deprecation warning and different behavior than the script intended.

InfoROM: the integrity check under everything else

The InfoROM stores the GPU’s persistent data: aggregate ECC counters, retired page history, serial and part numbers. If the InfoROM is corrupted, that history becomes unreliable. The dangerous part is the direction of the failure: a degrading GPU can look healthy because its aggregate error counts and retired-page records are unreadable or wrong, and previously retired pages may no longer be blacklisted.

# InfoROM checksum validation
nvidia-smi -q -d INFOROM

Treat a checksum failure or read error as a high-priority ticket. Corollary for drift detection: if InfoROM is suspect, do not trust the absence of aggregate ECC or retirement history as evidence of health. Cross-check with volatile ECC counters, which reset on driver load and do not depend on InfoROM state. Recovery from InfoROM corruption is a vendor-support or RMA conversation, not something to improvise on a production node.

Current vs pending: why “fixed” is not fixed

Most of these knobs apply through a two-stage model: the change registers as Pending and only becomes Current after a GPU reset or reboot. This creates a characteristic drift pattern:

1. An operator or script sets the correct value.
2. The Pending column shows the right thing; Current still shows the wrong thing.
3. The node never gets its reset window, or the reset does not apply the change on that model.
4. Weeks later, drift detection fires and nobody remembers the change.

When triaging drift, always look at both Current and Pending (ECC mode shows this explicitly in nvidia-smi -q). A Pending value sitting unapplied is itself a finding: a configuration change is in flight that nobody has finished. Note that a GPU reset generally requires no applications using the device, and nvidia-persistenced holding file descriptors can block a reset until the daemon is stopped.

Building and enforcing the baseline

Checklist for making drift detection real:

• Per-model baselines. Defaults differ across GPU models (ECC support, power limit ranges, MIG availability). A single fleet-wide baseline generates false positives.
• Boot-time snapshot. Because persistence mode, power limit, and compute mode reset on reboot, diff configuration right after boot, not just on a schedule.
• Pending-state diff. Capture Pending alongside Current so in-flight changes are visible.
• Change-record correlation. Every legitimate drift (power capping, exclusive mode for a dedicated node) should map to a ticket. Drift without a record is an investigation.
• Process cross-check. Unexplained power or compute-mode changes get a process-list review via nvidia-smi pmon before being written off as accidents.
• Read-only collection. All the checks above are nvidia-smi queries; nothing in the detection path mutates state.

Signals to monitor

How Netdata helps

• Netdata’s NVIDIA GPU collector (via NVML) continuously captures the fields this guide baselines: persistence mode, compute mode, ECC mode and error counters, power draw versus limit, and throttle reasons, so drift shows up as a chart deviation rather than a surprise at the next incident.
• Per-second sampling catches transient states that polling scripts miss, like a power cap applied and removed between cron runs.
• Correlating sw_power_cap throttle reasons against power.limit on one dashboard is the fastest way to confirm a throttling incident is configuration drift rather than thermal or hardware failure.
• Long retention lets you pin the exact window where a config metric flipped, which narrows the search through automation logs and access records.
• Alerts on ECC mode state and uncorrected ECC deltas close the silent-corruption gap that disabled ECC creates.

Related guides

• NVIDIA GPU ECC disabled: the silent data-corruption risk
• NVIDIA GPU ECC errors: corrected, uncorrected, volatile, and aggregate
• DCGM, nvidia-smi, and NVML: which NVIDIA GPU telemetry source to use
• NVIDIA DCGM exporter on Kubernetes: mapping GPU metrics to pods