$ guides / smartctl-disk-monitoring / smartctl-command-timeout ▌

Operations Guides

Command_Timeout climbing: the drive is taking too long to respond

SMART attribute ID 188 (Command_Timeout) is climbing. Two interpretation traps cause false alarms here, and the symptom overlaps with at least four distinct failure modes ranging from benign background maintenance to imminent controller death.

The first trap is vendor-specific raw value encoding. On Seagate drives, the raw value packs three 16-bit counters into a single 48-bit field, producing numbers in the billions that decode to single-event counts. Monitoring tools that read the raw value as one integer will page you for a single timeout during power-on sequencing.

The second trap: Command_Timeout is not universally implemented. Some drives do not report it at all, and its absence does not mean timeouts are not happening. The host kernel often sees I/O timeouts that never reach SMART data. Check dmesg and /sys/block/sdX/stat alongside the SMART attribute.

What this means

ATA attribute ID 188 counts commands that did not complete within the expected time window. The drive firmware tracks how many operations exceeded its internal completion threshold.

For most non-Seagate vendors, the raw value is a straightforward count of timed-out commands. Any increase warrants investigation. For Seagate drives, the raw 48-bit value encodes three separate 16-bit counters:

Bytes [1:0]: total command timeouts
Bytes [3:2]: commands that took more than 5 seconds to complete, including those over 7.5 seconds
Bytes [5:4]: commands that took more than 7.5 seconds to complete

A raw value of 4295032833 (hex 0x100010001) looks catastrophic at first glance. Decoded, it means 1 total timeout, 1 command exceeding 5 seconds, and 1 command exceeding 7.5 seconds. That is a single event, not a crisis.

smartctl can split the packed field for you:

# Decode Seagate raw value into three separate 16-bit counters
smartctl -A -v 188,raw16 /dev/sdX

The -v 188,raw16 directive tells smartctl to display the raw value as three separate 16-bit values instead of one large integer. If you parse smartctl JSON output (-j), the raw.value field contains the undecoded integer and the raw.string field contains the decoded representation (for example, “1 1 1”).

Severity guidance: TICKET for any nonzero value or growth from baseline. Investigate whether timeouts are impacting application I/O. PAGE only when Command_Timeout growth is combined with I/O errors visible to the operating system. The corroboration requirement filters false pages from encoding artifacts and transient events.

flowchart TD
    A["Command_Timeout ID 188 climbing"] --> B{"Seagate drive?"}
    B -->|Yes| C["Decode raw value
smartctl -v 188,raw16"]
    C --> D{"Decoded count
actually increasing?"}
    B -->|No| D
    D -->|No| E["False alarm:
encoding artifact"]
    D -->|Yes| F{"UDMA CRC 199
also increasing?"}
    F -->|Yes| G["Transport issue:
cable, backplane, port"]
    F -->|No| H{"dmesg shows
I/O errors or timeouts?"}
    H -->|Yes| I["Host-visible failure:
escalate"]
    H -->|No| J{"Temperature
elevated?"}
    J -->|Yes| K["Thermal throttling
causing delays"]
    J -->|No| L{"SSD near full
or high wear?"}
    L -->|Yes| M["Write cliff / GC storm:
reduce write rate"]
    L -->|No| N["Controller degradation:
plan replacement"]

Common causes

Cause	What it looks like	First thing to check
Seagate encoding artifact	Raw value in billions on a Seagate drive, normalized value at 100, no dmesg errors	Decode with `-v 188,raw16`; if decoded counts are trivial, this is a monitoring false positive
Interface or cable problem	UDMA CRC Error Count (ID 199) climbing alongside Command_Timeout, possibly with SATA link downshift	Check CRC count and SATA link speed in `smartctl -a`
Thermal throttling	Temperature elevated above throttle threshold, I/O throughput dropping without load change	Check ID 194 for HDD or NVMe Composite Temperature
SSD write cliff or GC storm	Sustained write workload on a nearly full SSD, write latency spiking to seconds	Check free space, TRIM status, and SSD wear indicators
Controller degradation	Timeouts increasing without other explanation, possible drive disappearance in dmesg	Check dmesg for device resets and ATA error log entries
Background maintenance	Transient timeouts during HDD background media scan or SSD garbage collection, no sustained pattern	Verify timeouts are not accumulating; check if they correlate with idle periods

Quick checks

# Check Command_Timeout raw and normalized values
smartctl -A /dev/sdX | grep -i "command_timeout\|timeout"

# For Seagate drives: decode the packed raw value into three counters
smartctl -A -v 188,raw16 /dev/sdX

# Check ATA error log for timeout and abort entries
smartctl -l error /dev/sdX | grep -iE "ABRT|UNC|timeout|CCTO"

# Check kernel logs for I/O errors, resets, and timeout messages
# On systemd hosts, journalctl -k is more reliable than dmesg (buffer rotation)
dmesg | grep -iE "I/O error|medium error|reset|timeout|offline" | tail -20
journalctl -k --since "1 hour ago" | grep -iE "I/O error|reset|timeout|offline"

# Check UDMA CRC error count for interface correlation
smartctl -A /dev/sdX | grep -i "UDMA_CRC_Error"

# Check drive temperature
smartctl -A /dev/sdX | grep -i "Temperature"

# Check host-side disk statistics
# Field 9: I/Os currently in progress (instantaneous queue depth)
# Field 11: weighted time spent doing I/Os (ms)
cat /sys/block/sdX/stat

# Check per-device I/O latency for correlation with timeouts
iostat -x 1

# Check SMART overall health for corroboration
smartctl -H /dev/sdX

How to diagnose it

Decode the raw value. If the drive is Seagate, run smartctl -A -v 188,raw16 /dev/sdX to split the packed field. If the decoded counts are trivial (1 or 2 per field) and are not increasing across polls, the “climbing” value may be a monitoring artifact. Fix your monitoring to decode Seagate raw values before alerting.
Correlate with UDMA CRC Error Count (ID 199). If CRC errors are increasing alongside Command_Timeout, the problem is the transport layer, not the drive media. A bad cable, loose connector, or failing backplane port causes retransmission delays that the drive reports as command timeouts. Replacing the drive will not fix this. See I/O errors in dmesg with clean SMART for the full interface failure pattern.
Check dmesg for host-visible errors. The kernel sees timeouts the drive may not log in SMART. Look for “I/O error”, “medium error”, “link reset”, and “timeout” messages. If dmesg shows active I/O errors correlated with Command_Timeout growth, this meets the PAGE condition. The host perspective is often more reliable than the drive’s own accounting.
Check temperature. Thermal throttling reduces drive performance, and commands take longer to complete as a result. If ID 194 (HDD) or the NVMe Composite Temperature is elevated and I/O throughput is dropping without a workload change, thermal throttling is the likely cause. For NVMe, check Warning Composite Temperature Time for cumulative thermal exposure.
Check SSD fill level and wear state. If the drive is an SSD at or near capacity, the controller may be running garbage collection synchronously because it has no pre-erased blocks available. This is the SSD write cliff. Check Percentage Used (NVMe) or Wear Leveling Count (ATA SSD) for wear context. See Data Units Written vs rated TBW for endurance tracking methodology.
Check the ATA error log. Run smartctl -l error /dev/sdX and look for ABRT (Command Aborted) entries, which indicate the drive rejected commands due to internal state. Multiple error entries at different LBAs indicate a pattern, not a transient event.
Run a short self-test. If the drive is an HDD, a short self-test (smartctl -t short /dev/sdX) runs in the background and checks electrical and mechanical basics. A self-test that completes without error alongside elevated Command_Timeout suggests the timeouts are environmental (thermal, power, interface) rather than media-related. A self-test failure with a specific LBA confirms a media defect.
Assess the rate of change. A Command_Timeout count that has been stable for months is far less concerning than one that gained 50 in the last week. Trend the value across multiple collection intervals. Acceleration is the signal that distinguishes historical noise from active degradation.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Command_Timeout (ID 188) decoded raw value	Direct measure of commands exceeding completion threshold	Any sustained increase from baseline, not just nonzero
UDMA CRC Error Count (ID 199)	Distinguishes transport problems from drive problems	Increasing alongside Command_Timeout = cable or backplane issue
dmesg I/O errors and timeouts	Host-side view that catches failures SMART misses	Any “I/O error”, “timeout”, or “reset” messages for the device
Drive temperature (ID 194 / NVMe Composite)	Thermal throttling causes command delays	Temperature above 55C (HDD) or above vendor WCTEMP (NVMe)
iostat await and queue depth	Confirms whether timeouts are impacting application I/O	await spiking to seconds with low CPU utilization
SSD Percentage Used / Wear Leveling Count	Heavily worn SSDs have more GC overhead	High wear combined with sustained write workload
ATA Error Log entries	Drive-side record of command failures	New entries since last check
Current Pending Sector (ID 197)	Pending sectors cause retry loops that look like timeouts	Nonzero alongside Command_Timeout growth

Fixes

Seagate encoding false alarm

If the decoded raw value shows trivial counts and no growth trend, the fix is in your monitoring system, not the drive. Ensure your SMART collection decodes Seagate raw values using -v 188,raw16 or equivalent parsing logic. If your monitoring tool reads the raw.value field from smartctl JSON output rather than the raw.string field, it will see the undecoded integer and generate false alerts.

Interface or cable problems

Reseat the SATA or SAS cable at both ends. If CRC errors continue to increment after reseating, swap with a known-good cable. Move the drive to a different port or bay to isolate whether the problem is the cable, the backplane port, or the HBA. Check smartctl -a for SATA Version versus current link speed: a mismatch (for example, SATA Version is 6.0 Gb/s but current is 3.0 Gb/s) indicates the link downshifted due to errors. The drive media is healthy in this scenario. See I/O errors in dmesg with clean SMART for the full diagnostic flow.

Thermal throttling

Check chassis fans and airflow. For M.2 NVMe drives, verify the heatsink is properly mounted. If multiple drives in the same chassis show temperature spikes simultaneously, the problem is environmental, not drive-specific. Sustained high temperature accelerates NAND wear and reduces data retention.

SSD write cliff

Verify TRIM is functioning: run fstrim -v /mountpoint and confirm the bytes trimmed are nonzero. If the SSD is above 80% capacity, garbage collection inefficiency compounds. Reduce the write rate if possible, or increase over-provisioning. Some RAID controllers strip TRIM commands, so verify TRIM is actually reaching the drive if you are behind a controller. The write cliff is a performance emergency, not data loss, but it indicates the SSD is under-provisioned for its workload.

Controller degradation

If Command_Timeout is increasing without any of the above explanations, and dmesg shows device resets or the drive intermittently disappears from the bus, the drive controller may be failing. This is the failure mode SMART cannot predict because the controller that reports SMART is the component that is degrading. Run an extended self-test to check media integrity. If the self-test passes but timeouts continue, schedule replacement. Controller degradation can progress to total unavailability rapidly. See SMART says PASSED but the drive is failing for why the health check will not warn you.

Prevention

Baseline every drive at deployment. Capture a full SMART snapshot including ID 188 when the drive is first installed. This distinguishes factory-shipped counts from production growth.
Decode Seagate raw values in monitoring logic. If your fleet includes Seagate drives (IronWolf, Exos, Barracuda), apply the three-field decoding before evaluating thresholds. Alerting on the undecoded raw integer will produce constant false positives.
Track rate of change, not absolute values. A stable count of 10 accumulated over years is noise. A count that gained 10 this week is a signal. Alerting on absolute values misses the acceleration that distinguishes historical damage from active failure.
Monitor host-side I/O alongside SMART. The most dangerous failures are the ones the drive does not know about. Kernel I/O errors, device resets, and SCSI/NVMe error recovery events in dmesg catch firmware bugs and controller issues that SMART cannot see. See Current_Pending_Sector non-zero for how pending sectors cause I/O latency spikes that look like command timeouts.
Schedule periodic self-tests. Weekly short tests and monthly extended tests surface latent defects before production I/O hits them. A drive that passes an extended self-test while showing elevated Command_Timeout is more likely experiencing environmental or interface issues than media failure.

How Netdata helps

Per-second disk I/O metrics correlate latency spikes (await, queue depth) with Command_Timeout growth in the same time window. You can see whether timeouts are actually impacting application I/O or just incrementing a SMART counter in the background.
SMART attribute collection captures ID 188 alongside IDs 5, 197, 198, and 199 at each poll interval, so you can watch the rate of change across correlated attributes without manual polling.
Anomaly detection on disk latency and I/O patterns can flag changes before SMART counters update, since the drive’s firmware accounting lags behind host-observable behavior.
Kernel log correlation through systemd journal or syslog integration surfaces dmesg I/O errors and device resets alongside SMART attributes. This is the correlation that matters most for this signal.
Temperature tracking alongside disk I/O metrics distinguishes thermal throttling from media degradation without separate diagnostic commands.
Fleet-level views show whether Command_Timeout is climbing on a single drive (drive-specific problem) or across multiple drives in the same chassis (infrastructure problem: power, cooling, or backplane).