NVMe volatile memory backup failed: critical warning bit 4 and a dead PLP capacitor

Your monitoring shows critical_warning nonzero on an enterprise NVMe drive. Decoding the bitmask, it is bit 4: volatile memory backup failed. The drive itself is behaving normally. IOPS are fine, latency is fine, media errors are zero. If you only looked at performance dashboards, nothing would look wrong.

That is exactly the problem. Bit 4 means the power-loss-protection (PLP) capacitor bank on the drive has failed or is degraded. The drive still accepts and acknowledges writes at full speed, but the moment this host loses power unexpectedly, any writes sitting in the controller’s volatile DRAM or FTL state are gone. You have lost the one feature that made every previous unsafe shutdown survivable.

This is a ticket-level signal, not a page. The drive is not failing. It is a risk state: you are running an enterprise drive with the data-loss profile of a consumer drive, and the clock is ticking until the next power event, kernel panic, or PDU fault.

What this means

Enterprise NVMe drives carry a volatile write cache: writes complete at the host when data lands in the controller’s DRAM, not when it lands on NAND. The controller also keeps in-flight FTL mapping updates in DRAM. On sudden power loss, a bank of hold-up capacitors supplies just enough energy for the controller to flush that DRAM state to NAND before everything dies. That is the volatile memory backup solution.

Bit 4 in the critical_warning byte of the NVMe SMART log (Log ID 02h) means the drive’s own self-check decided that backup path no longer works. The capacitor bank can no longer hold or deliver enough charge, or the charge circuit itself has failed. The bit is hex 0x10.

Two things follow:

• Every future unsafe shutdown now risks losing acknowledged writes. Filesystem journal entries, database WAL pages, anything the application believed was durable. The failure mode is silent: no performance impact, writes unprotected on power loss.
• The unsafe_shutdowns counter changed meaning. On a healthy PLP drive, an unsafe shutdown is a minor infrastructure note. On this drive, each one is a potential data-loss event.

Consumer drives without PLP never set bit 4. The field is only valid if the controller has a volatile memory backup solution at all. The inverse is the trap: the absence of PLP is invisible in SMART. A consumer drive reports a clean critical_warning of zero while offering zero power-loss protection. Bit 4 set on an enterprise drive tells you it has silently joined that category.

flowchart TD
  A[Host write acknowledged in DRAM] --> B{Power event?}
  B -->|None| C[Controller destages to NAND]
  B -->|Power lost| D{PLP healthy?}
  D -->|Yes| E[Capacitor energy flushes DRAM and FTL to NAND]
  D -->|No - bit 4 set| F[DRAM contents lost: acknowledged writes and FTL updates gone]
  C --> G[Data durable]
  E --> G
  F --> H[Filesystem or application layer corruption, often silent]

Common causes

The most common case is the first one. Capacitors age, especially in hot enclosures, and a drive that has spent years at elevated composite temperatures wears its hold-up capacitors faster. The same thermal history that ages the NAND stresses the capacitor bank.

Quick checks

All of these are read-only and safe on a production drive.

# Decode the critical warning bitmask with per-bit labels
sudo nvme smart-log /dev/nvme0 -H | grep -i "volatile"

# Confirm the raw value: bit 4 is 0x10
sudo nvme smart-log /dev/nvme0 | grep critical_warning

# How exposed are you right now? Every unsafe shutdown is now a data-loss risk
sudo nvme smart-log /dev/nvme0 | grep -E "unsafe_shutdowns|power_cycles|power_on_hours"

# Confirm no corroborating failure is hiding behind the same byte
sudo nvme smart-log /dev/nvme0 | grep -E "media_errors|available_spare|percentage_used|num_err_log_entries"

# Confirm which physical drive this is before touching anything
sudo nvme list

A few notes on what you are looking at. nvme smart-log -H prints the per-bit breakdown of the critical warning field (e.g. Volatile mem. backup failed[4] : 1), so you can read the volatile memory backup bit directly instead of decoding hex by hand. The raw critical_warning value for bit 4 alone is 16 (hex 0x10). If it is 17, 18, or higher, other bits are also set and you have more than one problem. See NVMe critical_warning is nonzero: decoding the SMART critical warning bitmask.

Also verify you are looking at an enterprise drive that actually has PLP. nvme list gives you the model. If the model is a consumer SKU, see the causes table: you never had PLP and the bit is likely firmware noise, but the underlying exposure was always there.

How to diagnose it

1. Confirm the bit and isolate it. Read critical_warning and verify only bit 4 is set. If bit 2 (reliability degraded) or bit 0 (spare below threshold) is also set, escalate: you have an actively degrading drive, not just a protection gap. Priorities shift to the replacement paths in NVMe available spare below threshold.

2. Check the drive’s error log for charge circuit evidence. nvme error-log /dev/nvme0 shows internal error entries. A cluster of internal errors near when the bit first set points at a charge circuit fault rather than gradual capacitor wear.

3. Establish when it set. SMART is polled, so find when your monitoring first observed critical_warning transition from zero. If it coincided with a firmware update, check the vendor advisory before condemning the hardware. If it coincided with a thermal incident, capacitor wear is the likely mechanism.

4. Run the drive’s own validation if the maintenance window allows. The NVMe Device Self-Test includes a segment that validates the volatile memory backup solution by measuring backup power source charge and discharge behavior. A failed PLP capacitor will fail that segment. Start an extended self-test with nvme device-self-test /dev/nvme0 -n 1 -s 2 and read the result with nvme self-test-log /dev/nvme0. Self-test competes with host I/O for controller resources, so schedule it rather than running it mid-peak.

5. Inventory what sits on the drive. This is the step that determines urgency. List the mount points, and for each one answer: does this workload depend on acknowledged writes being durable? A database with fsync semantics, a filesystem journal, a WAL, a queue’s persistence layer: all assume PLP-class durability on enterprise hardware. A rebuildable cache tier or replicated ephemeral volume does not.

6. Count the exposure. Note the current unsafe_shutdowns value and set up tracking so any increment after this point gets investigated as a potential corruption event, not a routine reboot note.

Metrics and signals to monitor

Fixes

Replace the drive (the real fix)

A failed PLP capacitor is not field-serviceable. There is no command that re-enables the backup path, no firmware toggle, no reset that recharges a dead capacitor bank. The fix is hardware replacement. Because the drive is healthy in every other respect, you can usually schedule this rather than doing it as an emergency swap.

Sequencing matters more than speed:

1. Procure the replacement. Enterprise drives with PLP, verified in the vendor datasheet, not assumed.
2. If the drive is in a RAID set, mirror, or replicated pool, fail it out cleanly and rebuild onto the new drive.
3. If it is a standalone device, migrate the workloads off, then decommission.
4. Before the old drive leaves service, record its final SMART state for your fleet records. The wear and thermal history is useful data.

Reduce exposure until replacement

If replacement takes days or weeks, lower the probability of an unsafe shutdown landing in that window:

• Verify clean shutdown paths. Make sure the OS actually issues the NVMe shutdown notification on halt and reboot. Some hypervisors and container runtimes skip clean NVMe shutdown during stop operations. A clean shutdown does not need PLP; the exposure only exists on unexpected power loss.
• Fix the power path. If this host has a history of unsafe shutdowns (check the counter), bit 4 plus a flaky PDU or PSU is a bad combination. Prioritize the power infrastructure fix.
• Reduce the durability gap in software where possible. Moving the most durability-sensitive write paths (database WAL, queue journals) to a healthy PLP drive in the same host is a cheap mitigation if one exists.

What not to do

Do not just acknowledge the alert and move on. Nothing degrades visibly until the one power event you cannot predict. Do not assume your infrastructure layer will save you either: vSAN, for example, logs this warning (VOB vob.vsan.lsom.backupfailednvmediskhealthcriticalwarning, “The disk’s backup device has failed”) but does not automatically evacuate or fail the drive on bit 4 alone, so the risk persists silently if you rely on automated handling.

Prevention

• Baseline PLP capability at provisioning. Record, per drive model, whether it has a volatile memory backup solution. The absence of PLP is invisible in SMART, so the only way to know a consumer drive is unprotected is to have looked it up when you installed it. Not knowing which drives have PLP is a common fleet-wide gap.
• Alert per bit, not on the byte. A single critical_warning != 0 alert collapses five different conditions with five different severities. Bit 3 (read-only) is a page. Bit 4 is a scheduled-replacement ticket. Per-bit alerting gets each to the right responder.
• Alert on unsafe shutdown increments. Every unsafe shutdown is a data-loss risk on any drive with bit 4 set or no PLP. Teams ignore this counter because “the server came back fine,” but the damage can be silent filesystem or application corruption that surfaces weeks later.
• Control drive temperature. Capacitor banks and NAND both age faster hot. Keeping composite temperature comfortably below WCTEMP extends the life of the PLP hardware, not just the flash.
• Track drive age for capacitor wear. High power_on_hours enterprise drives are the bit 4 population. Include PLP health in your end-of-life replacement planning alongside endurance.

How Netdata helps

Netdata collects the NVMe SMART log continuously and decodes the critical warning byte into individual dimensions, so the correlation this specific failure needs is already on one dashboard:

• nvme.device_critical_warnings_state exposes volatile_mem_backup_failed as its own dimension, so bit 4 gets its own alert instead of being buried in a generic nonzero check.
• nvme.device_unsafe_shutdowns_count sits next to it, making the actual risk exposure visible: every increment of this counter while bit 4 is set is a potential corruption event worth investigating.
• nvme.device_media_errors_rate lets you watch for delayed fallout, since lost flushes can surface later as read-time integrity errors.
• Temperature and thermal-time charts show the thermal history that likely aged the capacitor, and whether the replacement drive is headed for the same fate.
• Because collection is per-second and long-retention, you can go back and find exactly when the bit first set, which is what you need to distinguish capacitor wear from a firmware-update artifact.

Related guides

• NVMe available spare below threshold: critical warning bit 0 and end-of-life wear
• blk_update_request: I/O error, dev nvme0n1: reading NVMe I/O errors in the kernel log
• NVMe controller reset loop: repeated resets from a firmware hang
• nvme nvme0: I/O timeout, Resetting controller: what an NVMe controller reset means
• NVMe controller state not live: reading resetting, deleting, and dead from sysfs
• NVMe critical_warning is nonzero: decoding the SMART critical warning bitmask
• NVMe device disappeared: nvme0: Removing and a drive that fell off the PCIe bus
• How NVMe actually works in production: a mental model for operators
• NVMe monitoring checklist: the signals every production SSD needs
• NVMe monitoring maturity model: from survival to expert
• NVMe drive in read-only mode: critical warning bit 3 and rejected writes
• NVMe temperature threshold exceeded: critical warning bit 1, WCTEMP, and CCTEMP