NVMe device disappeared: nvme0: Removing and a drive that fell off the PCIe bus

You are looking at a host where an NVMe drive was present and is now gone. The kernel log shows some version of this sequence: a pcieport: AER: Uncorrectable (Fatal) line (kernels before 6.8 print Uncorrected (Fatal)), possibly followed by nvme nvme0: controller is down; will reset: CSTS=0xffffffff, then nvme nvme0: Removing (or nvme nvme0: Removing after probe failure status: -19), and often nvme0n1: detected capacity change from X to 0. After that, /dev/nvme0n1 no longer exists and /sys/class/nvme/nvme0 is empty.

This is not a slow drive or a resetting controller. The device has been removed from the PCIe bus entirely. Every outstanding and future I/O fails with EIO, and any filesystem that was mounted on it has lost an unknown amount of write data at its tail. The first job is containment: stop anything still trying to write, assess what data was on the device, and figure out whether the drive, the slot, the power, or a human pulled it.

One trap before you start: in hot-swap chassis (U.2, U.3, EDSFF), an intentional drive pull by a colleague or an automated maintenance workflow produces exactly this log signature. Check the maintenance calendar and change log before treating it as a failure.

What this means

NVMe devices are PCIe endpoints. The NVMe driver sits on top of the PCI layer, and the PCI layer can independently decide the device is gone. There are two broad paths to nvme nvme0: Removing:

1. Fatal PCIe error. The root port or endpoint reports an uncorrectable, fatal AER error. PCIe error handling isolates the device to protect the rest of the bus, tearing down the NVMe controller from underneath the driver. PCI config space reads then return all ones (0xffffffff), which is why you see CSTS=0xffffffff: the driver is literally reading a dead device.
2. Physical loss. The drive was pulled, lost power, lost link due to seating or connector problems, or the controller hardware died outright. To the kernel these look nearly identical to a fatal AER event, sometimes without the AER line at all.

The removal is terminal from the driver’s point of view. The kernel unregisters the controller, deletes the block device, and fails all queued I/O with EIO. If the device was a RAID or LVM member, the array just lost a leg.

flowchart TD
  A["pcieport: AER: Uncorrectable (Fatal)"] --> B["nvme nvme0: Removing"]
  B --> C["/dev/nvme0n1 and /sys/class/nvme/nvme0 gone"]
  C --> D{"Hot-swap or maintenance window active?"}
  D -->|"Yes"| E["Intentional removal: confirm against change log"]
  D -->|"No"| F{"Fatal AER errors logged before removal?"}
  F -->|"Yes"| G["PCIe transport fault: slot, riser, connector, retimer"]
  F -->|"No"| H["Sudden hardware or power delivery failure"]
  G --> I["Power down, reseat or replace, retest"]
  H --> I

Common causes

Quick checks

All read-only and safe to run during an incident.

# Ground truth: which NVMe controllers does the kernel currently see?
ls /sys/class/nvme/

# Namespaces (may show stale entries briefly after removal; trust sysfs over this)
lsblk | grep nvme

# The removal sequence and any preceding AER events
dmesg | grep -iE "nvme.*(removing|removed|surprise|failed)"
dmesg | grep -iE "aer|pcie.*error|bus error"

# Full kernel log around the event, for timing and order
journalctl -k --no-pager | grep -iE "nvme0|pcieport|aer"

# AER counters for the device, if it somehow still exists in sysfs
cat /sys/class/nvme/nvme0/device/aer_dev_correctable
cat /sys/class/nvme/nvme0/device/aer_dev_fatal

# Prior reset activity: a drive that resets repeatedly before vanishing
# points at firmware or link problems, not sudden death
journalctl -k --no-pager | grep "nvme0" | grep -c "reset controller"

Two gotchas on ground truth. First, lsblk and /proc/partitions can show stale entries for a short window after removal; /sys/class/nvme/ is authoritative for controller presence. Second, if the controller directory is gone, all the per-device sysfs paths above are gone with it, so pull the AER counters for the parent root port under /sys/bus/pci/devices/ instead.

How to diagnose it

1. Confirm the device is actually gone. ls /sys/class/nvme/. If the controller number is absent, the kernel has fully detached it. Do not trust lsblk alone.

2. Reconstruct the log sequence. The order matters: AER: Uncorrectable (Fatal) then Removing means PCIe error handling isolated the device. Removing with no AER and no prior resets means the device stopped answering abruptly, which points at power or instant hardware failure.

3. Rule out the human cause. On U.2, U.3, or EDSFF chassis, check the change calendar, ticket queue, and BMC/IPMI system event log for a physical pull. Intentional removal is log-identical to failure. Chasing a “failed” drive that a colleague pulled for replacement is a common way to waste an hour at 3 a.m.

4. Look at what happened before the removal. Query monitoring history for this device: correctable AER error rate, controller reset count, temperature trend, media error rate. A drive that accumulated PCIe correctable errors for weeks and then disappeared is a connector or slot problem. A drive that was clean until the second it vanished is a component or power failure.

5. Check the neighbors. If other devices on the same backplane, riser, or power rail also logged errors or disappeared, suspect the shared infrastructure, not the drive. If the device sat behind a PCIe switch, check the switch port’s AER counters too.

6. Attempt re-enumeration only after data is safe. If the filesystems on the device are unmounted and you have verified redundancy, you can try echo 1 > /sys/bus/pci/rescan to force the bus to re-probe. If the removal was a power-state glitch rather than dead hardware, the device may re-enumerate, possibly under a new controller number. The /sys/bus/pci/rescan interface is part of the kernel’s PCI ABI and has existed for many kernel versions; whether the device actually comes back depends on whether the link itself recovers. If it does not come back, the recovery path is physical: power down, reseat, and if that fails, replace.

Metrics and signals to monitor

The pattern that matters most for prediction is the correlation between correctable AER errors and link retrains. A fatal AER event rarely comes from nowhere; the physical layer usually degrades measurably first. That is the window where you can act before the drive falls off the bus.

Fixes

If it was an intentional removal

Confirm against the change record, verify the replacement or reinserted drive enumerates cleanly, and move on. After reinsertion the device may come back under a different controller number (for example nvme1 instead of nvme0), which can break anything referencing the old name. LVM and MD RAID member status can also be unreliable immediately after a surprise removal and reinsertion; verify with the array’s own tooling before declaring it healthy.

If the PCIe transport faulted (fatal AER, history of correctable errors)

1. Fail workloads off anything still depending on the device.
2. Schedule downtime. Reseat the drive, inspect the connector and backplane for damage or debris, and reseat any risers and retimers in the path.
3. After the drive is back and enumerated, watch the correctable AER counter under load. If errors resume, replace the cable or riser, or move the drive to a different slot, to isolate drive versus slot.

If the drive died (no AER, no prior signals)

Replace the drive. There is no software recovery for a controller that stopped answering the bus. If the device was a RAID or replication member, trigger the rebuild onto the replacement. If it was a single point of failure, restore from backup and treat this as a redundancy design review.

Filesystem and data recovery

Regardless of cause, if a filesystem was mounted when the device vanished, it needs a consistency check before going back into service. Unmount it if it is still wedged. Run the filesystem’s check tool once the device is re-enumerated or moved to a known-good host. Be deliberate: fsck variants that repair in place can destroy data that a recovery tool could have saved. If the data is irreplaceable, image the device first with ddrescue or equivalent before any repair pass.

If it was a power management glitch

On platforms where PCIe Active State Power Management or NVMe autonomous power state transitions are implicated (removal preceded by D3cold-to-D0 transition failures), the common workarounds are disabling ASPM via pcie_aspm=off and disabling APST via nvme_core.default_ps_max_latency_us=0 on the kernel command line (both are documented kernel parameters; reports of D3cold removal patterns are mostly from consumer and workstation chipsets, so validate the trade-off on your server platform before applying). Both are boot parameters requiring a reboot, so this is a planned maintenance action, not an in-incident fix.

Prevention

• Alert on AER, not just SMART. Correctable AER errors are the leading indicator for the physical-layer faults that end in device removal, and they are invisible to SMART polling. Monitor the sysfs counters directly.
• Alert on controller state transitions. A controller in deleting or dead for more than seconds is a removal in progress. Catching deleting buys you time to drain workloads before the block device disappears.
• Treat repeated controller resets as a pre-failure signal. A drive that resets twice in an hour gets replaced at the next window, not after it falls off the bus mid-write.
• Run redundancy on anything you care about. A single NVMe drive with no RAID, replication, or tested backup is a data loss event waiting for a connector to work loose.
• Track firmware versions across the fleet. Some removal patterns are firmware bugs triggered by specific power state transitions or command sequences, fixed in later releases.
• Document hot-swap procedures. Require that intentional pulls land in the change log. It converts a 30-minute false incident into a 30-second lookup.

How Netdata helps

• Netdata polls NVMe SMART health per device, so you get the pre-removal history: media errors rate, critical warning bits (reliability degraded, read-only), available spare, and temperature leading up to the event. That history is what separates “drive was dying for weeks” from “slot fault.”
• The nvme.device_critical_warnings_state chart gives per-bit alerting, so reliability degradation pages before the drive reaches the removal stage.
• Unsafe shutdowns and power cycle counters are tracked per device, which helps distinguish a chassis-wide power event from a single-drive failure when correlating across hosts.
• Netdata’s NVMe collector does not read the PCIe AER sysfs counters, so pair it with kernel log monitoring for AER:, Removing, and reset controller patterns; those messages are the only source for the transport-layer half of this failure.
• Per-second I/O metrics on surviving devices let you confirm whether the removal cascaded: latency spikes, queue buildup, and error retries on RAID partners or on the applications that lost their backing store.

Related guides

• nvme nvme0: I/O timeout, Resetting controller: what an NVMe controller reset means
• How NVMe actually works in production: a mental model for operators
• NVMe monitoring checklist: the signals every production SSD needs
• NVMe monitoring maturity model: from survival to expert