S.M.A.R.T. Disk Health Operations Guides on Netdata

Command_Timeout climbing: the drive is taking too long to respond

Mon, 01 Jan 0001 00:00:00 +0000

SMART attribute ID 188 (Command_Timeout) is climbing. Two interpretation traps cause false alarms here, and the symptom overlaps with at least four distinct failure modes ranging from benign background maintenance to imminent controller death.

The first trap is vendor-specific raw value encoding. On Seagate drives, the raw value packs three 16-bit counters into a single 48-bit field, producing numbers in the billions that decode to single-event counts. Monitoring tools that read the raw value as one integer will page you for a single timeout during power-on sequencing.

Current_Pending_Sector non-zero: unreadable sectors and I/O latency spikes

Mon, 01 Jan 0001 00:00:00 +0000

A non-zero Current_Pending_Sector (SMART attribute ID 197) means the drive has sectors it cannot reliably read. These sectors failed a read and are waiting for either a write to trigger reallocation or an offline scan to confirm the defect. The drive has not yet remapped them to its spare pool.

The operational impact is often misdiagnosed. Every time the filesystem touches one of these sectors, the drive firmware enters a multi-second retry loop. This shows up in iostat as extremely high await with low throughput and idle CPU. Teams chase this as a software freeze, a kernel bug, or a filesystem problem for hours before checking SMART data.

Data Units Written vs rated TBW: computing SSD endurance runway

Mon, 01 Jan 0001 00:00:00 +0000

SSD endurance is finite and specified in the datasheet as a TBW (Total Bytes Written) rating. The odometer is a SMART counter, but it uses non-obvious units and counts host writes, not NAND writes. This guide covers how to read the write-volume counter from smartctl, convert it to bytes correctly, compare it against rated TBW, and project when the drive reaches its endurance limit.

The counter has two names depending on the bus. NVMe drives report “Data Units Written” in units of 1000 x 512 bytes. ATA/SATA SSDs report attribute ID 241 (Total_LBAs_Written), typically in 512-byte sectors. Both measure host writes only. The drive’s NAND sees more writes due to write amplification, so this calculation produces an upper bound on remaining life, not a precise prediction.

Drive disappeared from the bus: sudden controller or electronics death

Mon, 01 Jan 0001 00:00:00 +0000

A device node that was present minutes or hours ago is now gone. lsblk no longer shows it. smartctl returns “No such device.” The drive did not warn you through SMART because the component that failed is the one that would have reported the problem. This is one of the few storage failure modes that produces zero SMART telemetry before it happens.

This is a PAGE-level signal with no false positive scenario for a device that was present and is now absent. The question is not whether something is wrong, but what failed: the drive’s controller, the cable, the backplane, the HBA, or a power management transition that made a healthy device disappear temporarily.

Drive temperature too high: HDD, SATA SSD, and NVMe thresholds

Mon, 01 Jan 0001 00:00:00 +0000

A temperature reading of 68C means very different things depending on what is in the slot. For an enterprise HDD, it is past the danger threshold and the drive is likely sustaining damage. For an NVMe SSD, it may be within normal operating range and the drive is not even throttling yet. Alerting on a single global threshold across mixed drive types produces two failure modes simultaneously: false pages on normal NVMe temperatures and missed alerts on dangerously hot HDDs.

First-observation baselining: don't page on lifetime counters at rollout

Mon, 01 Jan 0001 00:00:00 +0000

When you deploy SMART monitoring on an existing fleet for the first time, every in-service drive carries accumulated history in its lifetime counters. Offline Uncorrectable sectors, NVMe Media and Data Integrity Errors, Power-On Hours, Unsafe Shutdowns, Reallocated Sector Count, UDMA CRC Error Count. These counters started incrementing the moment the drive left the factory and never reset.

If your alerting fires on any non-zero absolute value, every drive with any history pages within minutes of enabling monitoring. A drive with 3 reallocated sectors from factory QA, a drive with 200 uncorrectable errors from a past thermal event three years ago, a drive with 5 unsafe shutdowns from a UPS failure. All produce identical alerts to a naive greater-than-zero rule.

G-Sense_Error_Rate rising: shock and vibration reaching the drive

Mon, 01 Jan 0001 00:00:00 +0000

G-Sense_Error_Rate (SMART attribute ID 221 on most drives, ID 191 on some) is a cumulative counter of shock and vibration events that exceeded the drive’s internal threshold, as detected by its built-in accelerometer. The attribute is HDD-only. SSDs and NVMe drives do not report it. Some enterprise HDD models do not expose the attribute at all, which does not mean vibration is absent, only that the drive does not instrument it.

How S.M.A.R.T. actually works: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

S.M.A.R.T. (Self-Monitoring, Analysis, and Reporting Technology) is not a tool, an agent, or a daemon. It is firmware-level self-instrumentation embedded in every modern HDD, SSD, and NVMe drive. The drive itself continuously monitors its internal health and reports what it finds. The smartctl utility from smartmontools is simply a reader: it queries the drive and prints what the firmware already knows. It has no independent intelligence about drive health.

Every SMART-based monitoring decision, every threshold, and every false alarm traces back to a single question: what does the drive firmware actually track, and how does it choose to expose it? The firmware tracks error correction outcomes, sector integrity, NAND wear, mechanical behavior, temperature, and power events. But the way it surfaces that data depends entirely on the drive interface, and the three interfaces in production today (ATA/SATA, SAS/SCSI, and NVMe) use three fundamentally different data models that are not interchangeable.

I/O errors in dmesg with clean SMART: the failure the drive can't see

Mon, 01 Jan 0001 00:00:00 +0000

I/O error, dev sda scrolling through dmesg. The application is timing out on disk reads. You run smartctl -H /dev/sda and it returns PASSED. Reallocated sectors, pending sectors, offline uncorrectable: all zero. The ATA error log is empty.

SMART reports what the drive firmware can observe about itself: media integrity, mechanical health, thermal state, NAND endurance. It cannot see failures in the transport layer between the host and the drive. When the failure lives in the SATA cable, the backplane connector, the HBA firmware, a SCSI error recovery loop, or a PCIe link, the drive firmware has no way to observe it. The host kernel sees every failed transaction. The drive does not.

Multiple drives failing at once: it's the power supply, not the drives

Mon, 01 Jan 0001 00:00:00 +0000

Three, five, or twelve drives light up at once. Spin_Retry_Count has jumped on every HDD in a shelf. Unsafe Shutdown counts are climbing across every NVMe device. A few drives may have dropped off the bus and come back. The instinct is to start filing RMAs.

Stop. When the same symptom appears on many drives at the same time, the probability that every drive independently decided to fail in the same hour approaches zero. The root cause is almost certainly external: a degrading PSU, an overloaded PDU, a failing UPS battery, or aggregate spin-up inrush sagging the 12V rail because staggered spin-up is disabled.

NVMe Available Spare below threshold: the spare block pool is running out

Mon, 01 Jan 0001 00:00:00 +0000

When smartctl -A /dev/nvme0n1 reports Available Spare below the Available Spare Threshold, the drive’s internal spare block pool is running low. The controller is reporting less reserved NAND capacity than it considers safe for continued reliable operation.

Available Spare is the percentage of reserved NAND blocks remaining for replacing worn or failed blocks. It starts at 100% and decreases monotonically as the drive consumes spares. When it reaches 0%, the next bad block causes permanent data loss for that block’s data: the drive has no spare to remap to.

NVMe Critical Warning bits: decoding the health-log bitmask

Mon, 01 Jan 0001 00:00:00 +0000

The NVMe Critical Warning byte is the single most important health signal on an NVMe drive. Unlike ATA SMART’s scattered attribute IDs, this one byte consolidates five distinct critical conditions into a compact bitmask. Most operators who find this article have just seen a non-zero value from smartctl and need to know which bit is set and what to do about it.

The field is byte 0 of NVMe Log Page 0x02 (SMART/Health Information Log). Bits 0 through 4 are defined by the NVMe specification; bits 5 through 7 are reserved. Each bit represents an independent condition, so evaluate them individually. 0x00 means no active warnings. 0x09 means two conditions are active simultaneously (bit 0 plus bit 3).

NVMe Media and Data Integrity Errors incrementing: confirmed NAND corruption

Mon, 01 Jan 0001 00:00:00 +0000

The NVMe SMART/Health Information log exposes a field called “Media and Data Integrity Errors.” When this value is zero, the controller has never returned data that failed integrity verification. When it increments, the controller detected an unrecovered data integrity error on data retrieved from NAND flash. These are errors such as uncorrectable ECC failures, CRC checksum failures, or LBA tag mismatches that exceeded the controller’s internal correction layer.

This counter is the NVMe equivalent of Offline Uncorrectable (SMART ID 198) on ATA drives. It is cumulative and monotonically increasing; it never decrements. Some enterprise NVMe drives have strong internal ECC that corrects many errors before they escalate to media errors. By the time this counter increments past that correction layer, the underlying media is failing.

NVMe media placed in read-only mode: Critical Warning bit 3

Mon, 01 Jan 0001 00:00:00 +0000

When smartctl reports Critical Warning: 0x08 on an NVMe drive, the controller has locked the media into read-only mode. The drive is refusing all writes to protect existing data. This is a hardware-enforced decision made by the drive firmware. Reboots, firmware updates, and format commands will not clear it.

The most common cause is endurance exhaustion: the spare block pool is depleted and the controller locks writes rather than risk corruption. Read-only transitions can also result from firmware bugs, thermal events, or electrical issues. An unexpected transition on a drive well below its rated endurance warrants forensic review.

NVMe NVM subsystem reliability degraded: Critical Warning bit 2

Mon, 01 Jan 0001 00:00:00 +0000

smartctl -H reports FAILED on an NVMe drive that otherwise seems fine. I/O is normal, Available Spare is healthy, and there are no media errors. The Critical Warning byte is 0x04.

Bit 2 (0x04) in the NVMe Critical Warning byte means “NVM subsystem reliability has been degraded.” Unlike bit 0 (spare below threshold), bit 1 (temperature), or bit 3 (read-only mode), bit 2 has no single metric that directly explains why firmware set it. The NVMe specification defines it as triggered by “significant media related errors or any internal error that degrades NVM subsystem reliability,” but leaves the exact conditions vendor-defined.

NVMe Percentage Used at or above 100%: rated endurance consumed

Mon, 01 Jan 0001 00:00:00 +0000

The NVMe SMART/Health Information Log (Log Page 02h) includes a field called Percentage Used that estimates how much of a drive’s rated write endurance has been consumed. At 100%, the drive has used up its warranted Total Bytes Written (TBW). Above 100%, it is operating beyond its manufacturer endurance rating.

This is not a failure signal. The NVMe specification explicitly allows values up to 255%, and drives routinely continue working past their rated endurance. A drive reporting 150% Percentage Used with 90% Available Spare and zero Media Errors is likely fine. Percentage Used is a planning signal: it tells you where the drive is in its endurance lifecycle so you can sequence replacement before the actual failure indicators (Available Spare exhaustion and Media Errors) force an emergency.

NVMe thermal throttling: throughput dropping with no errors

Mon, 01 Jan 0001 00:00:00 +0000

NVMe throughput drops during sustained I/O. No I/O errors in dmesg, no command timeouts, SMART health says PASSED, Media and Data Integrity Errors at zero. The workload has not changed, but latency is up and throughput is down by 30, 50, or more percent. Hours later, performance recovers on its own. The pattern repeats: degradation during peak load, recovery during quiet periods or overnight.

This is NVMe thermal throttling. The controller reduces performance to stay below vendor-defined temperature thresholds. Nothing in the standard error path surfaces it. The drive does not log I/O errors, the kernel does not see transport errors, and the SMART health assessment stays PASSED because the drive is operating within its protective envelope.

NVMe volatile memory backup device failed: CriticalWarning bit 4 and lost power-loss protection

Mon, 01 Jan 0001 00:00:00 +0000

You run smartctl -H /dev/nvme0n1 and get FAILED. The detail line reads - volatile memory backup device has failed. The Critical Warning byte shows 0x10. The drive is still serving reads and writes at full speed, latency is normal, Available Spare is fine, and Percentage Used is well within spec.

This is NVMe Critical Warning bit 4. It does not mean the NAND is failing or the controller is dying. It means the drive’s power-loss protection (PLP) hardware, typically supercapacitors on enterprise NVMe SSDs, has failed. Under stable power, the drive operates normally. But if power drops unexpectedly, data sitting in the drive’s volatile write buffer will be lost because the capacitor bank can no longer flush it to NAND.

Offline_Uncorrectable climbing: permanent data loss at the media level

Mon, 01 Jan 0001 00:00:00 +0000

Offline_Uncorrectable (SMART attribute 198) is the most unambiguous media-failure signal in the ATA SMART attribute set. When this counter increments, a sector was read during an offline scan or self-test and the drive firmware could not recover the data even after ECC correction and multiple retries. The data at that LBA is gone. If the filesystem or application layer did not provide redundancy (RAID, checksums, backups), the loss is permanent.

Power_On_Hours and fleet age: the context every other attribute needs

Mon, 01 Jan 0001 00:00:00 +0000

Power_On_Hours (ATA attribute ID 9, or the NVMe SMART/Health “Power On Hours” field) does not tell you anything is wrong. It tells you how long the drive has been powered on. That is the only attribute that gives every other attribute its meaning.

Ten reallocated sectors on a drive with 100 power-on hours is a manufacturing defect. Ten reallocated sectors on a drive with 50,000 power-on hours is graceful aging. The raw count is identical; the operational response is completely different. Without the age axis, you cannot distinguish “the drive is burning out” from “the drive is wearing out.”

Raw_Read_Error_Rate looks enormous: the Seagate false alarm explained

Mon, 01 Jan 0001 00:00:00 +0000

You see SMART attribute ID 1 with a raw value of 200,450,784. Or 60,000,000,000. Your monitoring system pages you at 3 a.m. because “raw read error rate is enormous.” You check the drive: smartctl -H says PASSED. Reallocated sectors: zero. Pending sectors: zero. Offline uncorrectable: zero. The raw value is enormous by design.

Seagate packs error counts and total operation counts together into the 48-bit raw value field for attribute ID 1 (Raw_Read_Error_Rate). The raw value will always be large on a healthy Seagate drive because it includes every read operation the drive has ever performed, not just errors. Alerting on raw > 0 for this attribute generates constant noise for every Seagate drive in your fleet.

Reading the ATA error log: UNC, ICRC, ABRT, CCTO, IDNF, AMNF

Mon, 01 Jan 0001 00:00:00 +0000

When smartctl -l error returns entries instead of “No Errors Logged”, the error type field in each entry tells you what failed. The ATA Summary Error Log records individual I/O failure events with the error type, the LBA where the error occurred, the command that triggered it, and a timestamp relative to the current power cycle.

The six error type codes are UNC, ICRC, ABRT, CCTO, IDNF, and AMNF. They are not equally serious. UNC means confirmed data loss. ICRC means a cable problem. ABRT may mean nothing at all. Knowing which one you are looking at determines whether you are evacuating data or reseating a cable.

Reallocated_Event_Count vs Reallocated_Sector_Ct: reading both together

Mon, 01 Jan 0001 00:00:00 +0000

Two SMART attributes track the same underlying process, sector reallocation, but count different things. Attribute ID 5 (Reallocated_Sector_Ct) counts sectors the drive has permanently retired and replaced with spares. Attribute ID 196 (Reallocated_Event_Count) counts remap operations the firmware initiated. When each bad sector fails individually and is remapped one at a time, the two counters move in lockstep. When a single physical event damages multiple sectors at once (head slap, thermal hotspot, manufacturing defect), the counters diverge: ID 196 increments once for the event while ID 5 jumps by the number of sectors involved.

Reallocated_Sector_Ct rising: the drive is burning through its spare pool

Mon, 01 Jan 0001 00:00:00 +0000

When smartctl -A shows Reallocated_Sector_Ct (SMART attribute ID 5) climbing, the drive firmware is silently remapping sectors that have become unreliable. Each reallocation consumes a sector or block from a finite spare pool reserved at the factory. Once that pool is exhausted, the next bad sector becomes an uncorrectable read error. Data loss follows.

The absolute count tells you less than the rate of change. A drive with 10 reallocated sectors that has been stable for 3 years is operating within its tolerances. A drive that gained 10 reallocated sectors in the last week is actively failing.

SAS PHY error counters: transport health for SAS drives

Mon, 01 Jan 0001 00:00:00 +0000

You are running SAS drives and something is off. I/O latency has spiked on a specific drive, or the kernel log is showing SAS link resets and error recovery messages. You reach for smartctl to check SMART health, but there is no UDMA_CRC_Error_Count attribute. SAS drives do not use ATA SMART attributes. They use SCSI log pages, and their transport health lives in a different set of counters: the SAS PHY error counters.

SATA link downshifted (6 to 3 to 1.5 Gbps): CRC errors forcing a slower link

Mon, 01 Jan 0001 00:00:00 +0000

smartctl -a shows SATA Version is: SATA 3.2, 6.0 Gb/s (current: 3.0 Gb/s). The advertised maximum and the current link speed do not match. Throughput is halved, but SMART health says PASSED and there are no reallocated, pending, or uncorrectable sectors.

This is a SATA link downshift. The kernel’s libata driver detected persistent CRC errors on the interface and automatically negotiated a lower link speed to maintain data integrity. At 3.0 Gbps the error rate drops enough for transfers to complete, but you have lost half your bandwidth. If the physical layer degrades further, the link may drop again to 1.5 Gbps.

SATA SSD wear-out: Wear_Leveling_Count and Media_Wearout_Indicator declining

Mon, 01 Jan 0001 00:00:00 +0000

SATA SSDs report remaining NAND endurance through vendor-specific SMART attributes, not the standardized NVMe Percentage Used field. Samsung exposes Wear_Leveling_Count at attribute ID 177. Intel and Solidigm expose Media_Wearout_Indicator at ID 233. Some vendors use ID 173 (also labeled Wear_Leveling_Count) or ID 231 (SSD_Life_Left). All express the same concept: firmware estimating how much rated program/erase cycle budget remains.

The problem for operators: these attributes are not standardized across vendors. The same attribute ID can carry different semantics on different drives. The THRESH column is almost always zero, so smartctl will never flag these attributes as failing regardless of how low the normalized value drops. The raw value encoding is vendor-defined and frequently misleading. A fleet with mixed SSD vendors cannot be monitored with a single threshold rule.

Scheduling SMART self-tests: short weekly, extended monthly

Mon, 01 Jan 0001 00:00:00 +0000

Drives do not run self-tests automatically. SMART firmware monitors passively, recording what it sees during normal I/O. If a sector is never read by production traffic, its degradation stays invisible until a backup job, a scrub, or a user request hits it. At that point you get an I/O error in production instead of a warning from the drive.

Self-tests are the active probing mechanism. The short test (1-2 minutes) exercises electrical and mechanical basics plus a small media sample. The extended test (hours, proportional to drive size) scans the entire surface and is the only routine mechanism that discovers latent bad sectors before production I/O reaches them. The conveyance test (~5 minutes, HDD only) checks for shipping damage and is typically run once at deployment, not on a schedule.

Seek_Error_Rate: head-positioning wear and the Seagate raw-value trap

Mon, 01 Jan 0001 00:00:00 +0000

A monitoring dashboard shows Seek_Error_Rate with a raw value of 200,009,354,607 on a Seagate drive. The on-call engineer pages the storage team. The replacement drive goes into the same bay and shows the same number. This cycle repeats across fleets because most monitoring tools and most operators do not know how Seagate encodes this attribute.

Seek_Error_Rate (ATA attribute ID 7) tracks the accuracy of the HDD actuator arm as it positions read/write heads over target tracks. On most non-Seagate drives, the raw value is a straightforward error count or rate. On Seagate drives, the raw value is a packed composite encoding both total seek operations and seek errors, producing numbers in the billions on perfectly healthy hardware. Alerting on the raw value guarantees false positives on every Seagate drive in the fleet.

Self-test completed: read failure - a bad sector found by proactive scanning

Mon, 01 Jan 0001 00:00:00 +0000

When smartctl -l selftest /dev/sdX shows “Completed: read failure”, the drive’s firmware found a sector it cannot read during an active surface scan. The LBA_of_first_error column gives you the exact logical block address of the defect. The firmware tried multiple times, exhausted its error correction, and could not recover the data at that location.

The extended self-test scans the entire media surface. Without it, a bad sector remains hidden until production I/O hits that exact LBA, producing an application-visible I/O error, a hung process, or a kernel timeout instead of a diagnostic warning.

SMART blind spots: VMs, USB bridges, and drives you think you're watching

Mon, 01 Jan 0001 00:00:00 +0000

SMART monitoring is deployed, the dashboard is green, but some drives are invisible to smartctl. No alert fired. The pattern: expected physical drive count exceeds drives returning SMART data. The drives may be healthy or failing. You cannot tell because no telemetry is collected, and the monitoring system treats “smartctl returned no data” as “no problem.”

Four configurations create this gap: virtual machines, USB-attached drives, cloud block devices, and hardware RAID controllers. In each case, smartctl cannot reach the physical drive’s firmware. The fix is to move monitoring to where SMART data is accessible and use the correct device-type flags.

SMART monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

smartctl -H returns PASSED or FAILED. Teams that stop there learn the hard way: a drive reports PASSED one day and drops off the bus the next, a “healthy” drive starts corrupting data, or a RAID rebuild fails because nobody noticed the spare sector pool degrading for months.

SMART monitoring is a spectrum of signal depth. Each level in this model closes specific diagnostic blind spots that the previous level could not answer. Use this to identify what failure modes your current monitoring is blind to and what to add next.

SMART not accessible behind a RAID controller: -d megaraid and cciss passthrough

Mon, 01 Jan 0001 00:00:00 +0000

When you run smartctl -a /dev/sda on a server with a hardware RAID controller, the query returns information about the controller’s virtual device, not the physical drive. Or it fails entirely. The RAID controller presents only virtual devices to the OS, so standard SMART queries never reach the physical hardware.

Without the correct controller-specific passthrough flag, you have no visibility into individual drive health. Monitoring may report success because it queried a device node and got a response, but the response contains no real drive data. Drives can fail silently while monitoring appears healthy.

SMART overall-health self-assessment: FAILED is the drive's own death notice

Mon, 01 Jan 0001 00:00:00 +0000

When smartctl -H reports SMART overall-health self-assessment test result: FAILED, the drive’s own firmware has concluded it is failing. At least one pre-fail attribute has crossed its vendor-defined threshold. This is not smartctl’s interpretation or a monitoring heuristic. Treat it as an unconditional page. Unlike individual attributes that require trend analysis and corroboration, FAILED is the drive saying it is done. False positives are rare in the normal case because the drive itself is making the call, not your monitoring system.

SMART says PASSED but the drive is failing: why the health check lies

Mon, 01 Jan 0001 00:00:00 +0000

The PASSED verdict from smartctl -H tells you one narrow thing: no pre-fail SMART attribute has crossed its vendor-defined threshold. It does not mean the drive is healthy, that data is safe, or that the drive will survive the week.

A drive can report PASSED while hundreds of sectors are pending reallocation, dozens have already been remapped, I/O latency is spiking to seconds, and the spare pool is burning down. Google’s 2007 study of over 100,000 drives found that 36% of failed drives had zero prior SMART warnings. Backblaze’s fleet analysis showed that 23.3% of failed drives had no non-zero values across the five attributes they consider most predictive (IDs 5, 187, 188, 197, 198). The health check is a last-resort binary, not a health indicator.

SMART self-tests keep aborting: heavy I/O interrupting the extended scan

Mon, 01 Jan 0001 00:00:00 +0000

The self-test log tells the same story every time. You started an extended self-test with smartctl -t long /dev/sdX, checked back hours later, and found another entry reading “Aborted by host” or “Interrupted (host reset)”. The drive reports no read failures, no servo errors, no electrical faults. But the full surface scan never actually completed.

This is a scheduling problem, not a drive health problem. ATA extended self-tests run in background mode by default, meaning the test has low priority and yields to host I/O. On a busy production drive, the test keeps getting deferred and eventually aborted. Latent bad sectors remain undiscovered until production I/O hits them.

smartctl disk monitoring checklist: the SMART signals every server needs

Mon, 01 Jan 0001 00:00:00 +0000

S.M.A.R.T. is firmware-level instrumentation built into every modern HDD, SSD, and NVMe drive. The drive reports on its own internal state. The smartctl tool from smartmontools reads what the firmware already knows. SMART monitoring is necessary but not sufficient: it catches gradual media degradation and endurance wear-out, but cannot predict sudden controller failures, firmware bugs, or silent data corruption. You still need redundancy, backups, and checksumming filesystems.

The single most important principle in this checklist: track rate of change, not absolute values. A drive with 50 reallocated sectors accumulated over five years is stable. A drive that gained 5 reallocated sectors this week is actively dying. Most monitoring failures come from alerting on absolute thresholds while missing the acceleration signal that distinguishes historical damage from active failure.

Spin_Retry_Count non-zero: the spindle motor is failing to spin up

Mon, 01 Jan 0001 00:00:00 +0000

Spin_Retry_Count (SMART attribute ID 10) counts how many times a hard drive’s spindle motor failed to reach operating RPM on the first attempt and retried. A healthy HDD spins up cleanly every time. Any non-zero value means the motor may not spin up on the next power cycle.

This attribute is HDD-only. SSDs have no spindle motor and do not report ID 10. If you see this attribute on an SSD, it is either a synthetic placeholder or a vendor repurposing of the ID for unrelated data.

Spin_Up_Time climbing: bearing wear and lubricant degradation

Mon, 01 Jan 0001 00:00:00 +0000

Spin_Up_Time (SMART attribute ID 3) measures how long the spindle motor takes to bring the platters from zero to rated RPM. The raw value is vendor-specific and noisy. A rising Spin_Up_Time alone is ambiguous; it becomes actionable only when you correlate it with Spin_Retry_Count (ID 10), drive temperature, and the scope of the problem across the chassis.

This attribute applies only to mechanical HDDs. SSDs have no spindle motor and report Spin_Up_Time as zero or a synthetic placeholder.

The SSD write cliff: latency spikes when the pre-erased block pool empties

Mon, 01 Jan 0001 00:00:00 +0000

Write latency on an SSD jumps from sub-millisecond to hundreds of milliseconds or seconds. Applications time out. The kernel log may show command timeouts. But smartctl -H says PASSED, every SMART attribute looks clean, and the drive has plenty of endurance left.

The write cliff happens when an SSD exhausts its pool of pre-erased NAND blocks. Under normal conditions, the controller performs garbage collection (GC) in the background: it reads valid pages from partially invalidated blocks, writes them elsewhere, and erases the now-empty block to replenish the free pool. When the write rate outpaces background GC, the free pool empties. Every host write now requires a synchronous erase cycle before it can complete, and latency explodes by 100x to 1000x.

The zombie drive: bad sectors, read retries, and high iowait with idle CPU

Mon, 01 Jan 0001 00:00:00 +0000

The system is grinding to a halt. SSH sessions lag, commands take seconds to return, and applications time out. You check top or htop and CPU is nearly idle, yet load average is climbing. iostat -x 1 tells the real story: await on one disk is spiking into the hundreds of milliseconds, sometimes multiple seconds, while every other disk looks fine. The process stuck on that I/O is in D state (uninterruptible sleep), unkillable, waiting for a read that the drive cannot complete.

UDMA_CRC_Error_Count rising: it's the cable, not the drive

Mon, 01 Jan 0001 00:00:00 +0000

Rising UDMA CRC Error Count (SMART attribute 199) is one of the most misdiagnosed signals in production. Operators see the counter climbing, swap the drive, and the replacement develops the same errors because the cable, backplane port, or HBA was the actual cause. The original drive goes back as an RMA, the replacement fails the same way, and the real fault is still in the chassis.

UDMA CRC errors indicate data corruption on the wire between the host and the drive, caught by CRC and retransmitted. They are almost never caused by the drive itself. Data integrity is preserved: each error triggers a retransmission. This is usually a TICKET-level problem, not a PAGE. The drive is not dying and your data is not being corrupted. But the retries reduce effective throughput, and in severe cases the SATA link downshifts speed to compensate.

Unexpected drive firmware or identity change: counter resets and identity drift

Mon, 01 Jan 0001 00:00:00 +0000

A cumulative SMART counter decreased between polls. Power-On Hours went backward. The drive’s model, serial number, or firmware version changed since the last snapshot. SMART counters are designed to be monotonically increasing. Identity fields are set at the factory. When either changes without a documented reason, determine whether this is routine maintenance, a firmware update, or something requiring forensic review.

The severity depends on context. A counter reset correlated with a ticketed drive replacement is informational. A firmware version change with no change management record is a TICKET. A serial number change on a drive that was never touched is a security event.

Unsafe Shutdowns / unexpected power-loss count rising

Mon, 01 Jan 0001 00:00:00 +0000

The unsafe shutdown counter is a cumulative lifetime metric tracking how many times a drive lost power without receiving a clean shutdown notification. On ATA drives, this is attribute 174 (Unexpect_Power_Loss_Ct) on some SSDs or attribute 192 (Power-Off_Retract_Count) on HDDs. On NVMe drives, it is the “Unsafe Shutdowns” field in the SMART/Health Information Log (Log Page 02h).

A non-zero value is not inherently alarming. Drives accumulate unsafe shutdowns over their lifetime from kernel panics, hard resets, UPS failures, and factory burn-in. The operational question is never “is the count non-zero?” but “is it growing, and how fast?”

Warning and Critical Composite Temperature Time: past overheating that already did damage

Mon, 01 Jan 0001 00:00:00 +0000

Two fields in the NVMe SMART/Health Information Log (Log Page 02h) record cumulative thermal exposure that a point-in-time temperature reading cannot show: Warning Composite Temperature Time and Critical Composite Temperature Time. They count the minutes the drive has spent above vendor-defined warning and critical temperature thresholds over its entire lifetime.

A non-zero value means the controller firmware accumulated a count. Even if Composite Temperature currently reads 40 degrees Celsius, these fields reveal thermal events that happened between polls, hours, days, or months ago. Thermal damage to NAND is cumulative and irreversible: sustained high temperature accelerates cell wear and degrades data retention. A drive that spent 200 minutes above its critical threshold may have aged more in those minutes than in months of normal operation. The question is not whether you can undo it (you cannot) but whether you can detect it and account for it in replacement planning.