NVMe drive in read-only mode: critical warning bit 3 and rejected writes

Your monitoring fires a PAGE: the NVMe device has set critical_warning bit 3 (0x08), meaning the drive has autonomously placed its media in read-only mode. Every write command the host sends is now rejected by the drive’s firmware. Reads still work. Writes do not.

This is not transient, not load-dependent, and not cleared by a reboot or controller reset. The drive has decided, based on its own internal assessment, that it can no longer safely accept writes, typically because it has run out of spare NAND blocks to remap failing cells into. It has switched itself into data-preservation mode so you can get your data off before it dies completely.

Treat this as a binary hardware fault at end of life. The correct response is: protect the data, recover what you need, replace the drive. There is no tuning fix, no kernel parameter, and no retry loop that makes the drive writable again.

What this means

The NVMe controller continuously retires bad NAND blocks and remaps them into a finite pool of spare blocks. As that pool drains, the drive first warns you through declining available_spare and critical warning bit 0 (spare below threshold), and often through rising media_errors. When the controller concludes it can no longer guarantee safe writes, it sets bit 3 and starts rejecting write commands at the device level.

flowchart TD
  A[NAND cells wear out] --> B[Controller remaps to spare blocks]
  B --> C[available_spare declines]
  C --> D[Bit 0: spare below threshold]
  C --> E[media_errors rising]
  D --> F[Spares exhausted]
  E --> F
  F --> G[Bit 3 set: media read-only]
  G --> H[All write commands rejected]
  H --> I[Recover data, replace drive]

From the host’s side, the failure looks like a software problem: applications log write errors, databases fail transactions, and filesystems on the device typically remount themselves read-only after their error handler trips (ext4’s default errors=remount-ro behavior, for example). The kernel NVMe driver does not force the remount; it passes the drive’s write rejections up the stack. The device stays visible, stays readable, and reports healthy-looking temperatures and link states, which is why teams often waste time restarting services before anyone runs nvme smart-log.

This alert is PAGE, unconditional. It cannot fire on an idle server, cannot be transient, and cannot resolve on its own. If bit 3 is set, the drive is telling you it is done accepting writes.

Common causes

The fourth row is real but less common: operator reports exist of drives setting 0x08 with available spare still at 100%, which means the controller placed media read-only for reasons other than spare exhaustion (internal media errors, firmware behavior, or electrical events such as repeated unsafe power loss). The operational response is identical either way, so do not let an unexpected spare reading talk you out of replacing the drive.

Quick checks

All of these are read-only and safe to run during an incident.

# 1. Confirm bit 3 is actually set (0x08). Values like 0x09 mean bits 0 and 3.
sudo nvme smart-log /dev/nvme0 | grep -i critical_warning

# 2. Pull the full health picture in one shot.
sudo nvme smart-log /dev/nvme0

# 3. Check spare capacity and endurance context.
sudo nvme smart-log /dev/nvme0 | grep -iE "available_spare|percentage_used|media_errors"

# 4. See what the drive logged around the transition.
sudo nvme error-log /dev/nvme0

# 5. Confirm the controller is otherwise alive (not a reset loop or dead device).
cat /sys/class/nvme/nvme0/state

# 6. See how the kernel and filesystem reacted to the rejected writes.
sudo dmesg | grep -iE "nvme|I/O error|remount" | tail -50

# 7. Check mount state of affected filesystems.
findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS | grep nvme

Reading notes:

• critical_warning: 0x08 is bit 3 alone. 0x09 adds bit 0 (spare below threshold), the expected companion when spare exhaustion is the cause.
• The error log is a circular buffer, so pull it early. High error rates overwrite the entries that explain the root cause.
• state should still read live. If it reads resetting or dead, you have a different or additional problem; see the related controller-state guides.

How to diagnose it

Diagnosis is short because the signal is binary. The real work is confirming scope and starting recovery.

1. Confirm the bit. Run nvme smart-log and verify critical_warning has 0x08 set. This is authoritative; it comes from the drive, not the OS.

2. Map the blast radius. Identify every namespace, partition, LVM volume, and mount on the device (lsblk, findmnt). Confirm which mounts are now read-only and which applications are failing writes. Anything holding deleted-but-open files or dirty page cache on that device is on borrowed time.

3. Check the trajectory. Compare available_spare against spare_thresh, and look at percent_used and media_errors. Spare at or below threshold with rising media errors is classic end-of-life spare exhaustion. If spare looks fine, the controller still made the call; treat it the same but capture the full SMART output for the vendor RMA conversation.

4. Verify your redundancy. Before touching anything else, confirm RAID partner health, replica status, or backup recency for everything on this device. The drive is one bad day away from refusing reads too.

5. Stop writing deliberately. Quiesce applications pointed at the device rather than letting them spin in retry loops. Failed-write retry storms from a database or queue flood your logs and obscure the recovery work.

6. Copy data off now. Reads still work. Copy what you need to other storage, starting with whatever is not covered by RAID, replication, or a fresh backup. Do not postpone this; the wear that ended write capability is still progressing on the read path.

7. Plan replacement. The drive is end of life. Provision a replacement, schedule the swap, and open a vendor case if the drive is in warranty. The full nvme smart-log and nvme error-log output is what the vendor will ask for.

Metrics and signals to monitor

The pattern that almost always precedes bit 3 is declining available_spare with bit 0 set, plus rising media_errors. If your monitoring only alerts on bit 3 itself, you learn about end-of-life drives at the moment they stop accepting writes instead of weeks earlier.

Fixes

No fix makes the drive writable again through the OS. dd, mkfs, partition table rewrites, and block-layer resets all fail because the drive rejects the underlying write commands. The realistic options, in order:

Recover and replace (the fix). Copy data off, verify redundancy, replace the drive, and rebuild from RAID, replication, or backup. The drive is at end of life.

Vendor tooling and RMA. Some vendors provide device-specific utilities that can reset or recondition a drive that entered read-only mode, and a secure erase (nvme format --ses=1) is sometimes suggested as a last resort. Treat this as unverified for your model and expect it to fail, because a format is itself a write operation. Never run it before your data is safe elsewhere: nvme format is destructive and erases all data on the namespace.

What not to do. Do not reboot hoping it clears; the state is in the drive’s firmware, not the host. Do not remount read-write with mount -o remount,rw; the next write still fails at the device. Do not redeploy the drive in a less critical role; a drive that has self-protected its media is not a candidate for reuse.

Prevention

You cannot prevent NAND from wearing out, but you can make bit 3 a scheduled maintenance event instead of a page.

• Alert per bit, not on critical_warning != 0. Bit 3 is an immediate outage, bit 0 is a replacement signal, bit 1 can be transient thermal. One blanket alert trains people to ignore the field that tells you the drive is dying.
• Track spare consumption rate, not just the current value. Spare depletion accelerates non-linearly; the last 10% goes faster than the first. A downward-trending available_spare is one of the strongest leading indicators of a future read-only event.
• Act on bit 0. Spare below threshold is your replacement window. Swap the drive when bit 0 fires and you will never see bit 3 in production.
• Match drive class to workload. Consumer drives in write-intensive production burn endurance fast and lack power-loss protection. Compare data_units_written-derived write rates against rated endurance before deployment.
• Keep headroom. Running drives near full raises garbage-collection pressure and write amplification, consuming endurance faster. Maintaining logical free space slows the march toward spare exhaustion.
• Protect against power loss. Unsafe shutdowns on drives without PLP risk exactly the kind of internal state damage that ends with a controller refusing writes. Fix power infrastructure when unsafe_shutdowns increments.

How Netdata helps

Netdata’s NVMe collector polls the SMART/health log and exposes the signals that bracket this failure:

• The nvme.device_critical_warnings_state chart splits critical_warning into per-bit dimensions, including read_only (bit 3), available_spare (bit 0), and nvm_subsystem_reliability (bit 2), so a read-only event is distinguishable from a thermal warning instead of hiding behind one composite alert.
• nvme.device_available_spare_perc gives you the spare trajectory over time, the leading indicator that predicts a read-only transition weeks in advance.
• nvme.device_media_errors_rate converts the lifetime media_errors counter into a rate, so the rising-error pattern that precedes end of life is visible as it develops rather than as a post-mortem number.
• nvme.device_estimated_endurance_perc tracks percent_used for procurement planning before wear becomes an incident.
• nvme.device_unsafe_shutdowns_count surfaces the power-loss history that correlates with drives self-protecting after unsafe power events.

Correlating the bit-3 alert with spare trend and media error rate in one place shortens the incident from “why are writes failing” to “the drive is end of life, start recovery” in a single look.

Related guides

• blk_update_request: I/O error, dev nvme0n1: reading NVMe I/O errors in the kernel log
• NVMe controller reset loop: repeated resets from a firmware hang
• nvme nvme0: I/O timeout, Resetting controller: what an NVMe controller reset means
• NVMe controller state not live: reading resetting, deleting, and dead from sysfs
• NVMe device disappeared: nvme0: Removing and a drive that fell off the PCIe bus
• How NVMe actually works in production: a mental model for operators
• NVMe monitoring checklist: the signals every production SSD needs
• NVMe monitoring maturity model: from survival to expert