Retired pages are the permanent record of a GPU’s memory hardware degrading. Every retirement means the driver found a framebuffer page it could no longer trust and removed it from the allocatable pool for good. The counts persist in the GPU’s InfoROM across reboots and driver reloads, which makes them one of the few genuinely cumulative hardware health signals nvidia-smi exposes.

Operators usually hit this signal one of two ways: monitoring flags retired_pages.pending = Yes and nobody knows whether to panic, or someone asks why a GPU with 40 GB of HBM shows slightly less memory than its siblings. Both require understanding the difference between SBE-retired and DBE-retired pages, what a pending retirement means for running workloads, and where the end-of-life line sits.

For the broader ECC picture that feeds into retirement, see NVIDIA GPU ECC errors: corrected, uncorrected, volatile, and aggregate.

What page retirement is and why it matters

Dynamic page retirement is the NVIDIA driver’s mechanism for quarantining framebuffer pages that have proven unreliable. When the ECC subsystem determines a page is bad, the driver records the page’s address and marks it so no future allocation can touch it. The GPU keeps running, workloads keep allocating, and the lost capacity per page is small (on the order of 64 KB per page). The mechanism exists so that memory degradation is survivable.

It matters for three reasons:

  1. It is one-directional. A retired page never comes back. The counts only go up, they survive reboots, and they are stored in the GPU’s InfoROM, not on the host. Retired page counts are the closest thing a GPU has to a wear gauge.
  2. The retirement table has a hard limit. The driver can track a maximum of 64 retired pages. Once the table is full, no further pages can be retired, and a subsequent uncorrectable error on a new address has no mitigation left. A GPU at or near 64 retired pages is at the end of its useful life.
  3. Retirement is the evidence trail for replacement decisions. Fleet operators, and NVIDIA’s own RMA process, use retired page counts and causes to decide whether a board gets swapped.

Retired pages are not an acute fault. The GPU continues operating with reduced capacity, and ECC continues to protect active workloads. The job here is to read the signal correctly and make a good replacement decision, not to respond to an incident.

How page retirement works

There are two retirement causes, and they mean very different things about the hardware.

SBE retirement is preventive. When a page accumulates repeated single-bit (correctable) ECC errors at the same address, the driver concludes the page is marginal and retires it before it produces something worse. Background soft errors are normal at low rates, but repeated errors at one address indicate a physically weakening cell.

DBE retirement is reactive. A double-bit (uncorrectable) ECC error means data corruption has already occurred. The page that produced it is retired immediately, and the driver logs XID 48 to the kernel log. Any DBE-retired page is a serious event: whatever was in that page may have been corrupt, and any workload running at the time should have its outputs validated.

The full lifecycle:

stateDiagram-v2
  [*] --> Healthy: page in allocatable pool
  Healthy --> Suspect: repeated SBEs at same address
  Healthy --> BadPage: DBE (XID 48)
  Suspect --> PendingRetire: retirement recorded in InfoROM
  BadPage --> PendingRetire: retirement recorded in InfoROM
  PendingRetire --> Retired: driver reload / GPU reset / reboot (XID 63)
  Retired --> [*]: permanently blacklisted
  PendingRetire --> RetirementFailed: XID 64 (reset + escalate)

The intermediate state in that diagram is where most operational confusion lives.

Reading the counters

Retired page data is not available through --query-gpu. It has its own query path:

# CSV query for retired page counts and pending status
nvidia-smi --query-retired-pages=retired_pages.single_bit_ecc.count,retired_pages.double_bit_ecc.count,retired_pages.pending --format=csv,noheader

# Human-readable detail, including per-page addresses and causes
nvidia-smi -q -d PAGE_RETIREMENT

# Per-page CSV dump with addresses and cause per page
nvidia-smi --query-retired-pages=gpu_uuid,retired_pages.address,retired_pages.cause --format=csv

The CSV query returns three fields per GPU: the SBE retirement count, the DBE retirement count, and a pending flag (Yes or No). The -d PAGE_RETIREMENT output additionally lists each retired page’s address and whether it was retired due to single-bit or double-bit errors. Older documentation refers to the pending field as “Pending Page Blacklist”; newer nvidia-smi output labels it “Pending”. Both mean the same thing.

If you collect through DCGM instead, the equivalent field IDs are:

DCGM fieldMeaning
390Total pages retired due to SBE
391Total pages retired due to DBE
392Pending retirement status

Two interpretation rules save you from common mistakes:

  • Alert on deltas, not on raw counts. These counters are monotonic and persistent. A GPU that retired two pages eight months ago and has been stable since is not an emergency; alerting on “count > 0” will page you forever on hardware that is fine. Track increases between samples and alert on new retirements.
  • Volatile versus aggregate does not apply here. Unlike ECC error counters, retired page counts are already lifetime-persistent. There is no reset-on-driver-load behavior to account for. The count you read is the count since the board shipped.

Pending retirements: the dangerous state

A pending retirement means the GPU knows a page is bad, has written the address to InfoROM, but is still capable of allocating that page. The retirement does not take effect until the driver is reloaded. Until then, the bad page remains in the allocatable pool and can produce further errors, including another DBE, while workloads are running on it.

NVIDIA’s own documentation is blunt about this: pages that are retired but not yet blacklisted can still be allocated and may cause further reliability issues.

Applying a pending retirement requires one of:

# Option 1: GPU reset (disruptive - kills all CUDA contexts on the GPU)
nvidia-smi -i <gpu> -r

# Option 2: exit persistence mode, which reloads the driver for an idle GPU
nvidia-smi -i <gpu> -pm 0

# Option 3: reboot the node

All three are disruptive to anything running on the GPU. nvidia-smi -r will fail while processes hold contexts, so the practical sequence is: drain the GPU or the node, confirm no compute apps remain (nvidia-smi --query-compute-apps=pid --format=csv,noheader), then reset, then verify. Remember to re-enable persistence mode afterward if you disabled it (nvidia-smi -i <gpu> -pm 1).

One more gotcha: there is a documented race in persistence mode between logging errors to InfoROM and ending a CUDA job. The symptom is seeing XID 48 in dmesg but never seeing the follow-up XID 63 (page retired). If you hit that pattern, exiting persistence mode forces the pending writes to flush. See NVIDIA persistence mode: why the GPU keeps re-initializing and P-state flaps for the broader persistence tradeoffs, and NVIDIA’s dynamic page retirement documentation for the mechanism.

After applying the retirement, verify with the query from the previous section: retired_pages.pending should return to No and the SBE or DBE count should have incremented by the number of pages that were pending.

End of life: when retired pages mean replacement

The retirement table holds a maximum of 64 pages. That number drives the entire end-of-life model:

StateInterpretationAction
Any new DBE-retired pageUncorrectable error occurred; corruption possiblePut the GPU on the replacement queue regardless of total count; validate recent workload outputs
Total approaching ~48 pages (~75% of limit)Majority of retirement budget consumedSchedule replacement at the next maintenance window
Total >= 60 pagesNearly at the 64-page hard limitUrgent replacement; NVIDIA’s RMA eligibility guidance starts here
Total = 64 pagesRetirement table fullNo further mitigation exists; any new bad page is unrecoverable. Remove from production

Two nuances on this table.

Any DBE retirement puts the GPU on the replacement queue, even at a low total count. A single uncorrectable error is a qualitatively different event from a pile of preventive SBE retirements: the hardware has already produced corruption once. The SBE count can grow to a dozen pages on a GPU that serves reliably for years; a single DBE-retired page is a replacement conversation.

Retirement rate matters as much as the absolute count. A GPU that retired 10 pages in its first week and none since is a different risk from one that retires a page a month on a steady slope. Extrapolate the slope against the 64-page limit to estimate runway, the same way you would for any consumable resource. Accelerating SBE retirement is the classic leading edge of the silent memory degradation pattern.

Ampere and later: row remapping changes the picture

On A100 and later datacenter GPUs, row remapping supplements page retirement as the memory error mitigation mechanism. Instead of blacklisting a whole page, the hardware remaps the faulty DRAM row to a spare row, which is finer-grained. Page retirement still exists, but on these GPUs you should also watch the remapper:

# Row remapping status (Ampere+ only; older GPUs return N/A)
nvidia-smi --query-remapped-rows=remapped_rows.correctable,remapped_rows.uncorrectable,remapped_rows.pending,remapped_rows.failure --format=csv,noheader

# Human-readable form - note the flag is ROW_REMAPPER, not REMAPPED_ROWS
nvidia-smi -q -d ROW_REMAPPER

The field to fear here is remapped_rows.failure = true, which means the spare row pool is exhausted and the GPU can no longer self-heal that bank. Like the pending flag, remapped_rows.pending > 0 requires a reset or reboot to apply. The same delta-tracking discipline applies: failure is latched state in InfoROM, so alert on the transition to true, not on the boolean, or you will re-alert on the same GPU forever. Whether the 64-page RMA guidance transfers directly to row-remapped GPUs is not something the documentation confirms; treat the replacement thresholds above as applying to the page retirement table specifically.

Signals to watch in production

SignalWhy it mattersWarning sign
retired_pages.single_bit_ecc.count (delta)Preventive retirements; slope predicts runwayAny increase; acceleration over weeks
retired_pages.double_bit_ecc.count (delta)Reactive retirements after corruptionAny increase; replacement queue regardless of total
retired_pages.pendingBad page still allocatable until resetYes; schedule a drain and reset promptly
Total retired pages vs 64Remaining mitigation budget~48 (schedule replacement), >= 60 (urgent)
remapped_rows.pending / failure (Ampere+)Row remapper needs reset; spares exhaustedpending > 0, transition of failure to true
XID 48 / 63 / 64 in dmesgThe event trail behind the counters48 without a following 63 (persistence-mode race); any 64
Corrected ECC error rateFeeds future SBE retirementsRate acceleration, per the ECC guide

For how these fit into a full monitoring stack, see NVIDIA GPU monitoring checklist: the signals every production GPU fleet needs.

How Netdata helps

  • Netdata’s NVIDIA GPU collector tracks retired page counts and pending status per GPU over time, which is exactly the delta-tracking view this signal needs: trend lines on monotonic counters instead of raw values.
  • Because retirement is the downstream product of ECC errors, having SBE rate, DBE events, and retired page counts on the same dashboard lets you watch a degradation story develop: corrected error rate accelerates, first SBE retirement lands, pending flag sets, retirement applies after reset.
  • Per-second sampling catches the pending flag as soon as it sets, so the window where a known-bad page is still allocatable is measured in the time it takes you to schedule a drain, not the time it takes someone to notice.
  • Long retention on these slow-moving counters is what makes runway extrapolation possible: a retirement every six weeks is invisible on a 24-hour graph and obvious on a 6-month one.
  • Correlating XID events from system logs with counter transitions closes the loop on the persistence-mode race (XID 48 without XID 63) that pure counter polling would miss.