NVIDIA GPU retired pages: page retirement, pending retirements, and end of life

Retired pages are the permanent record of a GPU’s memory hardware degrading. Every retirement means the driver found a framebuffer page it could no longer trust and removed it from the allocatable pool for good. The counts persist in the GPU’s InfoROM across reboots and driver reloads, which makes them one of the few genuinely cumulative hardware health signals nvidia-smi exposes.

Operators usually hit this signal one of two ways: monitoring flags retired_pages.pending = Yes and nobody knows whether to panic, or someone asks why a GPU with 40 GB of HBM shows slightly less memory than its siblings. Both require understanding the difference between SBE-retired and DBE-retired pages, what a pending retirement means for running workloads, and where the end-of-life line sits.

For the broader ECC picture that feeds into retirement, see NVIDIA GPU ECC errors: corrected, uncorrected, volatile, and aggregate.

What page retirement is and why it matters

Dynamic page retirement is the NVIDIA driver’s mechanism for quarantining framebuffer pages that have proven unreliable. When the ECC subsystem determines a page is bad, the driver records the page’s address and marks it so no future allocation can touch it. The GPU keeps running, workloads keep allocating, and the lost capacity per page is small (on the order of 64 KB per page). The mechanism exists so that memory degradation is survivable.

It matters for three reasons:

1. It is one-directional. A retired page never comes back. The counts only go up, they survive reboots, and they are stored in the GPU’s InfoROM, not on the host. Retired page counts are the closest thing a GPU has to a wear gauge.
2. The retirement table has a hard limit. The driver can track a maximum of 64 retired pages. Once the table is full, no further pages can be retired, and a subsequent uncorrectable error on a new address has no mitigation left. A GPU at or near 64 retired pages is at the end of its useful life.
3. Retirement is the evidence trail for replacement decisions. Fleet operators, and NVIDIA’s own RMA process, use retired page counts and causes to decide whether a board gets swapped.

Retired pages are not an acute fault. The GPU continues operating with reduced capacity, and ECC continues to protect active workloads. The job here is to read the signal correctly and make a good replacement decision, not to respond to an incident.

How page retirement works

There are two retirement causes, and they mean very different things about the hardware.

SBE retirement is preventive. When a page accumulates repeated single-bit (correctable) ECC errors at the same address, the driver concludes the page is marginal and retires it before it produces something worse. Background soft errors are normal at low rates, but repeated errors at one address indicate a physically weakening cell.

DBE retirement is reactive. A double-bit (uncorrectable) ECC error means data corruption has already occurred. The page that produced it is retired immediately, and the driver logs XID 48 to the kernel log. Any DBE-retired page is a serious event: whatever was in that page may have been corrupt, and any workload running at the time should have its outputs validated.

The full lifecycle:

stateDiagram-v2
  [*] --> Healthy: page in allocatable pool
  Healthy --> Suspect: repeated SBEs at same address
  Healthy --> BadPage: DBE (XID 48)
  Suspect --> PendingRetire: retirement recorded in InfoROM
  BadPage --> PendingRetire: retirement recorded in InfoROM
  PendingRetire --> Retired: driver reload / GPU reset / reboot (XID 63)
  Retired --> [*]: permanently blacklisted
  PendingRetire --> RetirementFailed: XID 64 (reset + escalate)

The intermediate state in that diagram is where most operational confusion lives.

Reading the counters

Retired page data is not available through --query-gpu. It has its own query path:

# Counts and pending-blacklist status
nvidia-smi -q -d PAGE_RETIREMENT

# Per-page CSV dump with addresses and cause per page
nvidia-smi --query-retired-pages=gpu_uuid,retired_pages.address,retired_pages.cause --format=csv

The -d PAGE_RETIREMENT output reports the SBE retirement count, the DBE retirement count, and a pending flag (Yes or No), plus per-page addresses and causes in the detailed view. Current --query-retired-pages fields expose the per-page address, timestamp, and cause (Single Bit ECC or Double Bit ECC); derive totals and pending state from the detailed display or DCGM counters. Older documentation refers to the pending field as “Pending Page Blacklist”; newer nvidia-smi output labels it “Pending”. Both mean the same thing.

If you collect through DCGM instead, the equivalent field IDs are:

Two interpretation rules save you from common mistakes:

• Alert on deltas, not on raw counts. These counters are monotonic and persistent. A GPU that retired two pages eight months ago and has been stable since is not an emergency; alerting on “count > 0” will page you forever on hardware that is fine. Track increases between samples and alert on new retirements.
• Volatile versus aggregate does not apply here. Unlike ECC error counters, retired page counts are already lifetime-persistent. There is no reset-on-driver-load behavior to account for. The count you read is the count since the board shipped.

Pending retirements: the dangerous state

A pending retirement means the GPU knows a page is bad, has written the address to InfoROM, but is still capable of allocating that page. The retirement does not take effect until the driver is reloaded. Until then, the bad page remains in the allocatable pool and can produce further errors, including another DBE, while workloads are running on it.

NVIDIA’s own documentation is blunt about this: pages that are retired but not yet blacklisted can still be allocated and may cause further reliability issues.

Applying a pending retirement requires one of:

# Option 1: GPU reset (disruptive - kills all CUDA contexts on the GPU)
nvidia-smi -i <gpu> -r

# Option 2: reboot the node
# If XID 63 is missing after XID 48, also exit persistence mode before
# reset/reboot to force the documented InfoROM race workaround:
nvidia-smi -i <gpu> -pm 0

These actions are disruptive to anything running on the GPU. nvidia-smi -r will fail while processes hold contexts, so the practical sequence is: drain the GPU or the node, confirm no compute apps remain (nvidia-smi --query-compute-apps=pid --format=csv,noheader), then reset, then verify. Remember to re-enable persistence mode afterward if you disabled it (nvidia-smi -i <gpu> -pm 1).

One more gotcha: there is a documented race in persistence mode between logging errors to InfoROM and ending a CUDA job. The symptom is seeing XID 48 in dmesg but never seeing the follow-up XID 63 (page retired). If you hit that pattern, exiting persistence mode forces the pending writes to flush. See NVIDIA persistence mode: why the GPU keeps re-initializing and P-state flaps for the broader persistence tradeoffs, and NVIDIA’s dynamic page retirement documentation for the mechanism.

After applying the retirement, verify with the query from the previous section: the pending-blacklist status should return to No and the SBE or DBE count should have incremented by the number of pages that were pending.

End of life: when retired pages mean replacement

The retirement table holds a maximum of 64 pages. That number drives the entire end-of-life model:

Two nuances on this table.

Any DBE retirement puts the GPU on the replacement queue, even at a low total count. A single uncorrectable error is a qualitatively different event from a pile of preventive SBE retirements: the hardware has already produced corruption once. The SBE count can grow to a dozen pages on a GPU that serves reliably for years; a single DBE-retired page is a replacement conversation.

Retirement rate matters as much as the absolute count. A GPU that retired 10 pages in its first week and none since is a different risk from one that retires a page a month on a steady slope. Extrapolate the slope against the 64-page limit to estimate runway, the same way you would for any consumable resource. Accelerating SBE retirement is the classic leading edge of the silent memory degradation pattern.

Ampere and later: row remapping changes the picture

On A100 and later datacenter GPUs, row remapping supplements page retirement as the memory error mitigation mechanism. Instead of blacklisting a whole page, the hardware remaps the faulty DRAM row to a spare row, which is finer-grained. Page retirement still exists, but on these GPUs you should also watch the remapper:

# Row remapping status (Ampere+ only; older GPUs return N/A)
nvidia-smi --query-remapped-rows=remapped_rows.correctable,remapped_rows.uncorrectable,remapped_rows.pending,remapped_rows.failure --format=csv,noheader

# Human-readable form - note the flag is ROW_REMAPPER, not REMAPPED_ROWS
nvidia-smi -q -d ROW_REMAPPER

The field to fear here is remapped_rows.failure = true, which means a row-remapping failure has occurred and the GPU can no longer self-heal the affected region. Like the pending flag, remapped_rows.pending > 0 requires a reset or reboot to apply. The same delta-tracking discipline applies: failure is latched state in InfoROM, so alert on the transition to true, not on the boolean, or you will re-alert on the same GPU forever. For row-remapped GPUs, NVIDIA’s DRAM RMA policy is different: the failure flag, validated by field diagnostics, is the DRAM RMA criterion. It can be set by a ninth uncorrectable-row attempt on a bank with eight already remapped, a retry of a row already remapped, or the 512-total uncorrectable-remap limit; Blackwell adds HBM channel repair when a spare channel is available. The 64-page retirement thresholds apply to the page-retirement table, not row remapping.

Signals to watch in production

For how these fit into a full monitoring stack, see NVIDIA GPU monitoring checklist: the signals every production GPU fleet needs.

How Netdata helps

• Netdata’s NVIDIA GPU collector tracks retired page counts and pending status per GPU over time, which is exactly the delta-tracking view this signal needs: trend lines on monotonic counters instead of raw values.
• Because retirement is the downstream product of ECC errors, having SBE rate, DBE events, and retired page counts on the same dashboard lets you watch a degradation story develop: corrected error rate accelerates, first SBE retirement lands, pending flag sets, retirement applies after reset.
• Per-second sampling catches the pending flag as soon as it sets, so the window where a known-bad page is still allocatable is measured in the time it takes you to schedule a drain, not the time it takes someone to notice.
• Long retention on these slow-moving counters is what makes runway extrapolation possible: a retirement every six weeks is invisible on a 24-hour graph and obvious on a 6-month one.
• Correlating XID events from system logs with counter transitions closes the loop on the persistence-mode race (XID 48 without XID 63) that pure counter polling would miss.

Related guides

• NVIDIA GPU ECC errors: corrected, uncorrected, volatile, and aggregate
• NVIDIA persistence mode: why the GPU keeps re-initializing and P-state flaps
• NVIDIA GPU monitoring checklist: the signals every production GPU fleet needs
• NVIDIA GPU monitoring maturity model: from survival to expert
• How an NVIDIA GPU actually works in production: a mental model for operators
• NVIDIA BAR1 memory exhaustion: mapping failures with free framebuffer
• NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver
• CUDA out of memory with free memory available: GPU memory fragmentation
• CUDA out of memory: diagnosing NVIDIA GPU framebuffer exhaustion
• NVIDIA GPU memory leak: framebuffer usage climbing without a plateau
• Reading NVIDIA GPU memory correctly: nvidia-smi vs the PyTorch caching allocator
• nvidia-smi hangs or is unresponsive: a wedged GPU or a stuck driver