$ guides / smartctl-disk-monitoring / smartctl-host-io-errors-clean-smart ▌

Operations Guides

I/O errors in dmesg with clean SMART: the failure the drive can't see

I/O error, dev sda scrolling through dmesg. The application is timing out on disk reads. You run smartctl -H /dev/sda and it returns PASSED. Reallocated sectors, pending sectors, offline uncorrectable: all zero. The ATA error log is empty.

SMART reports what the drive firmware can observe about itself: media integrity, mechanical health, thermal state, NAND endurance. It cannot see failures in the transport layer between the host and the drive. When the failure lives in the SATA cable, the backplane connector, the HBA firmware, a SCSI error recovery loop, or a PCIe link, the drive firmware has no way to observe it. The host kernel sees every failed transaction. The drive does not.

smartctl is not the source of truth for this class of failure. A monitoring stack that watches only SMART attributes will miss these failures entirely, and you will discover them when an application breaks.

What this means

The kernel’s SCSI mid-layer responds to transport and controller failures through error recovery (EH). When a command times out or returns an error sense, the kernel initiates EH, which can reset the device, the bus, or the host adapter. During EH, all I/O to the affected device freezes, and often all devices sharing the same controller freeze as well. This produces multi-second latency spikes that look like total system hangs to applications.

flowchart TD
    A["dmesg: I/O errors
smartctl: PASSED"] --> B{"UDMA CRC 199
increasing?"}
    B -->|Yes| C["Cable or backplane
transport fault"]
    B -->|No| D{"Multiple drives
on same HBA?"}
    D -->|Yes| E["HBA or controller
level fault"]
    D -->|No| F{"dmesg shows
reset or timeout?"}
    F -->|Yes| G["Controller firmware hang
or drive unresponsive"]
    F -->|No| H["Bus-level or
protocol error"]

Common causes

Cause	What it looks like	First thing to check
SATA/SAS cable or backplane degradation	UDMA CRC Error Count (ID 199) increasing. dmesg shows `ataX: hard resetting link`, link speed downshifts. SMART media attributes stay at zero.	`smartctl -A /dev/sdX \| grep UDMA_CRC` for increasing count
HBA or controller firmware hang	`DID_TIME_OUT` in kernel sense data. `task abort` and device reset messages. May affect all drives on the same controller simultaneously.	dmesg for `task abort`, `reset`, `EH complete` sequences
Drive controller firmware bug	Commands time out intermittently. Drive may disappear and reappear. SMART stays clean because the diagnostic controller is the component malfunctioning.	dmesg for device removal and re-adding events
SCSI error recovery loop	Repeated EH cycles. I/O freezes for seconds at a time across all devices on a controller. Log fills with reset and recovery messages.	dmesg for repeated `reset` or recovery sequences
Bus-level error (PCIe for NVMe)	NVMe Error Information log is clean. Kernel logs PCIe AER errors or controller reset messages. SMART/Health log shows no Critical Warning bits.	dmesg for `AER` or `pcieport` messages

Quick checks

# Check kernel ring buffer for I/O errors and recovery actions
dmesg | grep -iE "I/O error|medium error|reset|timeout|offline|task abort|device not responding" | tail -40

# Verify SMART health assessment and media attributes are actually clean
smartctl -H /dev/sdX
smartctl -A /dev/sdX | grep -iE "Reallocated|Current_Pending|Offline_Uncorrectable|UDMA_CRC"

# Check drive-side ATA error log for abort, CRC, or timeout entries
smartctl -l error /dev/sdX
# smartctl prints CCTO only when the drive logs a command-timeout error entry

# Check current vs negotiated SATA link speed
smartctl -a /dev/sdX | grep -i "SATA Version"

# For NVMe: check Error Information log and Critical Warning
smartctl -l error /dev/nvme0n1
smartctl -A /dev/nvme0n1 | grep "Critical Warning"

# Check per-device I/O latency for the suspect drive
iostat -x 1 | grep sdX

# Check if the device is still present
lsblk | grep sdX

How to diagnose it

Confirm the pattern. Verify that smartctl -H returns PASSED and that media attributes (IDs 5, 197, 198) are zero or stable. Then confirm the kernel is logging I/O errors for the same device. If both are true, you have a host-side failure that SMART cannot see.
Check UDMA CRC Error Count (ID 199). If it is increasing, the failure is in the physical transport layer: cable, connector, or backplane. This is the most common cause and the most commonly misdiagnosed as a drive failure.
Determine scope. Are errors affecting one drive or multiple drives on the same controller? Multiple drives points to HBA firmware, backplane, or power. A single drive points to that drive’s controller, its cable, or its port.
Read the SCSI sense data. The kernel logs structured sense information after failed commands. Look for hostbyte values: DID_TIME_OUT means the HBA gave up waiting for the drive to respond. DID_ERROR means a bus-level or host adapter error occurred. driverbyte DRIVER_SENSE means the drive returned an error sense, which narrows the problem toward the drive rather than the transport.
Check for SATA link resets and speed downshifts. Messages like ataX: hard resetting link followed by SATA link up X Gbps indicate the kernel is recovering from a bus error. If the negotiated speed is lower than the drive’s maximum, the link has downshifted due to accumulated errors. This is a strong indicator of cable or connector degradation.
Check whether the device has been marked dead. The kernel logs rejecting I/O to dead device after repeated EH failures. At this point, the block layer stops accepting new I/O. The drive may still respond to SMART pass-through commands because SMART commands take a different kernel path than block I/O.
For NVMe, check PCIe errors separately. The NVMe Error Information log may show zero entries while the kernel logs PCIe AER corrections or controller resets. These are PCIe link or controller issues, not NAND issues. Check dmesg for pcieport or AER messages.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Kernel I/O error rate (dmesg)	The primary signal for this failure class. SMART cannot see these errors.	Any `I/O error`, `medium error`, `task abort`, or `reset` message for a specific device
UDMA CRC Error Count (ID 199)	The one SMART attribute that directly reflects transport health rather than media health	Increasing count, especially after a maintenance event
Device presence (`/dev/sdX` exists)	Detects complete drive disappearance from the bus	Device node that was previously present is now absent
iostat await per device	SCSI EH freezes manifest as latency spikes before any error counter moves	`await` spiking to seconds on a specific device while CPU is idle
SATA link speed	Downshift from 6 Gbps to 3 Gbps or lower indicates accumulated bus errors	Negotiated speed below drive capability
SCSI EH frequency	Each EH cycle freezes I/O for the affected controller	Repeated reset or recovery messages within a short window

Fixes

Cable or backplane transport fault

Reseat the SATA or SAS cable at both ends. If CRC errors continue incrementing, swap with a known-good cable. Move the drive to a different port or bay to isolate whether the fault is the cable, the backplane port, or the drive’s connector. CRC errors are cumulative and never reset, so judge by whether the count is still increasing after the cable change, not by the absolute value.

HBA or controller firmware hang

Check the HBA vendor’s firmware changelog for known timeout or reset bugs. Update HBA firmware if a relevant fix exists. If the issue affects all drives on one controller, move drives to a different HBA to confirm the controller is the fault. For RAID controllers, check the controller’s own event log, which may show more detail than the kernel exposes.

Drive controller firmware bug

Check the drive manufacturer’s support site for firmware updates. Some firmware versions cause intermittent command timeouts without setting SMART failure bits. If a firmware update is available, apply it during a maintenance window. If the drive disappears and reappears intermittently, replace it: this pattern tends to worsen without warning, and the drive’s own diagnostics cannot self-assess a controller that is malfunctioning.

SCSI error recovery loop

If EH cycles are frequent but the drive eventually recovers each time, the issue may be a marginal link or a drive that takes too long on certain commands. Increasing the kernel’s SCSI command timeout can reduce false EH triggers, but this masks the underlying problem. The real fix is to address the root cause: cable, firmware, or failing controller.

PCIe link issue (NVMe)

Check PCIe slot seating. Reseat the NVMe drive. Check for PCIe AER errors in dmesg. If the motherboard or slot is faulty, move the drive to a different slot. Check for BIOS or firmware updates that address PCIe link stability for your platform.

Prevention

Baseline UDMA CRC Error Count at deployment. CRC errors are cumulative and never reset. Without a baseline, you cannot distinguish a historical count from an active problem. Capture the value at deployment and alert on growth, not on absolute value.

Track SATA link speed over time. A link that downshifts from 6 Gbps to 3 Gbps has accumulated transport errors. This is a leading indicator of cable or connector degradation that precedes outright failure.

Schedule regular extended self-tests. Self-tests force the drive to read every sector, surfacing latent media defects before production I/O hits them. A drive that passes extended self-tests but still generates kernel I/O errors is exhibiting a transport or controller problem, not a media problem.

Correlate across drives in the same chassis. When multiple drives on the same controller show kernel I/O errors simultaneously, the root cause is shared infrastructure: HBA, backplane, or power. Individual drive monitoring misses this pattern. Fleet-level correlation surfaces it.

How Netdata helps

Netdata collects kernel log events and disk-level metrics per second, so you can correlate a burst of dmesg I/O errors with the exact moment await spikes on the affected device, even for transient events that resolve before a human sees them.
SMART attribute collection runs alongside disk performance metrics in the same timeline, so you can confirm in seconds whether the drive’s firmware agrees with the kernel about what is happening.
The disk metrics collector tracks per-device I/O latency, queue depth, and error rates from /proc/diskstats, giving you the host-side view that SMART cannot provide.
Cross-device correlation makes it visible when multiple drives on the same controller start showing errors simultaneously, pointing to shared infrastructure rather than individual drive failure.