$ guides / smartctl-disk-monitoring / smartctl-drive-disappeared-from-bus ▌

Operations Guides

Drive disappeared from the bus: sudden controller or electronics death

A device node that was present minutes or hours ago is now gone. lsblk no longer shows it. smartctl returns “No such device.” The drive did not warn you through SMART because the component that failed is the one that would have reported the problem. This is one of the few storage failure modes that produces zero SMART telemetry before it happens.

This is a PAGE-level signal with no false positive scenario for a device that was present and is now absent. The question is not whether something is wrong, but what failed: the drive’s controller, the cable, the backplane, the HBA, or a power management transition that made a healthy device disappear temporarily.

The distinguishing feature is clean SMART history followed by sudden disappearance. If the drive’s last SMART collection showed no warnings, the failure bypassed SMART entirely. The kernel logs are your only forensic source.

What this means

This is failure archetype six in the playbook: controller or electronics failure. The severity is PAGE because data on that device is now unavailable. If the drive is part of a redundant array, the array is degraded. If there is no redundancy, the data is inaccessible until the drive returns or is recovered professionally.

The key diagnostic distinction is between permanent hardware death and recoverable disconnection. A reseated cable or a cold power cycle that brings the drive back points to a connection or firmware issue. A drive that does not return after reseating and power cycling is dead.

Common causes

Cause	What it looks like	First thing to check
Controller or PCB failure (drive-side)	Device completely gone; no response to any command; NVMe may still appear in lspci but namespace is gone	dmesg for catastrophic error before disappearance
Cable or backplane disconnection (SATA/SAS)	SATA link errors and CRC errors in dmesg before disappearance; may be intermittent	UDMA CRC count history; reseat cable
HBA or port failure	Multiple drives on the same HBA or expander port disappear simultaneously	Whether other drives on the same controller are also gone
Firmware crash (controller wedge)	Drive vanishes under heavy I/O or specific workload; may return after a full power cycle	Whether a cold boot restores the device
Power management false trigger	Drive disappears during idle or after a power state transition; may reappear spontaneously	Kernel power management parameters; USB autosuspend for USB devices
Hot-swap event (intentional)	Drive disappears during a maintenance window	Maintenance records and change log

Quick checks

Run these read-only commands to confirm the device is truly gone and build the timeline.

# Confirm the device node is absent (replace sdX/nvmeXnY with your device)
lsblk
ls /dev/sdX /dev/nvmeXnY 2>&1

# Check PCI enumeration for NVMe controllers
lspci | grep -i nvme

# Check SCSI/SATA device list
lsscsi

# Attempt SMART query (expected to fail with No such device)
smartctl -i /dev/sdX 2>&1

# Check kernel ring buffer for the removal sequence
dmesg | grep -iE "nvme|ata|I/O error|controller is down|reset|removed|disconnect" | tail -40

# Check the full kernel log timeline
journalctl -k --since "1 hour ago" | grep -iE "nvme|ata|sdX|I/O error|link"

# Check if other drives on the same controller are affected
lsblk -S

How to diagnose it

The diagnostic goal is to determine what failed and whether the drive can be recovered. Work through these steps in order.

flowchart TD
    A["Device node missing"] --> B{"Visible in lspci?"}
    B -->|"Yes (NVMe)"| C["PCIe device present but controller not initializing"]
    B -->|"No"| D{"Visible in lsblk or lsscsi?"}
    D -->|"No"| E["Device fully gone from bus"]
    D -->|"Yes"| F["Node exists but SMART fails: check RAID passthrough"]
    C --> G["Read dmesg for controller reset or CSTS errors"]
    E --> G
    G --> H{"I/O error storm before disappearance?"}
    H -->|"Yes"| I["Hardware or connection failure"]
    H -->|"No"| J{"Power state transition?"}
    J -->|"Yes"| K["Likely power management false trigger"]
    J -->|"No"| L["Sudden death: controller or PCB failure"]
    I --> M["Reseat cable or drive; try different port"]
    K --> N["Disable APST, ASPM, or USB autosuspend"]
    L --> O["Cold power cycle; if no return, replace drive"]
    M --> P{"Drive returns?"}
    P -->|"Yes"| Q["Monitor closely; check CRC errors"]
    P -->|"No"| O

Confirm the device is gone from all enumeration points. Run lsblk, lspci, and lsscsi. For SATA/SAS drives, the device node (/dev/sdX) should be absent from lsblk and lsscsi. For NVMe drives, check whether the PCI device still appears in lspci. If the PCI device is gone, the failure is electrical (the PCIe link dropped). If the PCI device appears but the namespace (/dev/nvmeXnY) is gone, the controller failed to initialize after a reset.
Read the kernel log timeline. For SATA/SAS, look for ATA exception messages such as ataX.00: exception Emask 0x10, link reset attempts (ataX: hard resetting link), and failed resume messages (ataX.00: failed to resume link). An I/O error storm often precedes the final disconnection. For NVMe, look for controller reset messages (nvme nvme0: controller is down; will reset), CSTS register values (CSTS=0xffffffff means the PCIe read returned all-ones, indicating the device stopped responding), and capacity change notifications (nvme0n1: detected capacity change from X to 0).
Check whether this is isolated or systemic. If multiple drives disappeared simultaneously, the problem is infrastructure: power supply, HBA, backplane, or SAS expander. A single drive disappearing points to that drive or its specific connection.
Distinguish hardware death from power management. NVMe drives with broken APST (Autonomous Power State Transition) can vanish under idle or load transitions. SATA link power management can cause similar symptoms. USB-attached drives can disappear when USB autosuspend activates. Check whether the disappearance correlates with idle periods or power state changes.
Try physical recovery. Reseat the drive and its cables. Try a different SATA port, SAS lane, or PCIe slot. Try a different power cable. If the drive returns after reseating, check UDMA CRC error count (SMART ID 199) for evidence of connection degradation. If the drive returns only after a full cold power cycle (not a warm reboot), suspect a firmware crash or controller wedge.
Check RAID status. If the drive is behind a hardware RAID controller, the controller may still present the virtual device even after a physical drive has failed. Use vendor-specific tools (e.g., megacli, storcli, arcconf) to check for degraded array state.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Device node presence	Binary indicator of whether the drive is on the bus	Device was present in the last poll, now absent
Kernel I/O errors (dmesg)	Catches failures that SMART cannot see, often preceding disappearance	New I/O errors, task aborts, or bus reset messages for a specific device
UDMA CRC error count (ID 199)	Transport layer degradation that can precede link failure	Count increasing over time before the drive vanishes
Unsafe shutdown count (NVMe)	High ratio of unsafe shutdowns to power-on hours correlates with controller instability	Rapidly increasing count, or abnormally high count relative to power-on hours
RAID controller status	Physical drive failure may be masked behind virtual device presentation	Array state changed to degraded, rebuild, or failed
Cross-drive correlation	Multiple drives failing simultaneously indicates infrastructure, not drive failure	Two or more drives on the same HBA, backplane, or power rail showing problems

Fixes

Hardware death: replace the drive

If the drive does not return after reseating cables, trying different ports, and performing a cold power cycle, the controller or PCB is dead. Replace the drive. There is no software fix for a dead controller.

If the drive is under warranty, start the RMA process. If the data is critical and there is no redundancy, consult a professional data recovery service before attempting anything further. Do not open the drive enclosure.

Cable or backplane failure (SATA/SAS)

If UDMA CRC errors were climbing before the disappearance, or if reseating the cable brings the drive back, replace the cable. If the drive is in a hot-swap backplane, try a different slot. Backplane connectors degrade over time, especially in high-vibration environments.

If a replacement drive develops the same CRC errors in the same slot, the backplane port or HBA port is the problem, not the drive.

Firmware crash or controller wedge

If the drive returns after a cold power cycle but not after a warm reboot, the controller firmware crashed. This can be triggered by specific I/O patterns, power state transitions, or firmware bugs. Check the manufacturer’s website for firmware updates.

For NVMe drives, if APST is implicated, disable it with the kernel parameter nvme_core.default_ps_max_latency_us=0. If PCIe ASPM is implicated, try pcie_aspm=off. These workarounds do not always prevent controller death on drives with fundamentally broken firmware or hardware.

For SATA drives exhibiting NCQ-related crashes, you can disable NCQ on the affected port.

Power management false triggers

For USB-attached drives, disable USB autosuspend for the device by writing on to /sys/bus/usb/devices/*/power/control for the relevant device, or create a udev rule to persist the setting.

For NVMe drives that disappear during idle periods, APST is the likely culprit. The kernel parameter nvme_core.default_ps_max_latency_us=0 disables APST globally.

RAID array recovery

If the drive is part of a redundant array, check the array status immediately. A degraded RAID array is one failure away from data loss. Replace the failed drive and initiate a rebuild. Monitor the rebuild closely: the sustained write load of a rebuild stresses remaining drives and can surface latent defects.

Prevention

Maintain an inventory baseline of expected devices. Without a baseline, a disappeared drive may go unnoticed until an application fails.
Monitor device presence as a first-class signal. Device node existence is Level 1 survival monitoring. A drive that vanishes from the bus should page immediately, not when an application eventually tries to read from it.
Monitor kernel I/O errors alongside SMART. SMART cannot detect controller-level failures. Kernel logs often show the I/O error storm that precedes disappearance.
Use redundant arrays for any data you cannot lose. Controller death gives no warning. Redundancy is the only protection against sudden disappearance.
Track unsafe shutdown count. A high ratio of unsafe shutdowns to power-on hours is a red flag that can precede controller instability, even when other SMART attributes look clean.
Schedule periodic extended self-tests. Self-tests cannot predict controller death, but they surface media problems that compound the impact of a sudden disappearance.

How Netdata helps

Device presence monitoring detects the exact second a device node disappears, with per-second resolution. The alert fires immediately rather than waiting for the next application-level failure.
Kernel log correlation ties the disappearance to the preceding I/O error storm. Netdata correlates the device presence signal with disk I/O errors, latency spikes, and bus reset events in the same time window.
Cross-device correlation helps distinguish single-drive failure from infrastructure problems. If multiple drives on the same HBA show errors simultaneously, the pattern points to the controller or backplane, not individual drives.
SMART attribute trends provide the baseline that confirms sudden death with clean history rather than gradual degradation. The last successful SMART collection becomes the forensic record.