SMART monitoring maturity model: from survival to expert

smartctl -H returns PASSED or FAILED. Teams that stop there learn the hard way: a drive reports PASSED one day and drops off the bus the next, a “healthy” drive starts corrupting data, or a RAID rebuild fails because nobody noticed the spare sector pool degrading for months.

SMART monitoring is a spectrum of signal depth. Each level in this model closes specific diagnostic blind spots that the previous level could not answer. Use this to identify what failure modes your current monitoring is blind to and what to add next.

flowchart TD
    L1["Level 1: Survival
Device presence, overall health, critical attributes"]
    L2["Level 2: Operational
Full attribute coverage, blind-spot detection, self-tests"]
    L3["Level 3: Mature
Rate-of-change tracking, error logs, endurance runway"]
    L4["Level 4: Expert
Mechanical trends, cross-drive correlation, risk scoring"]
    L1 --> L2 --> L3 --> L4

Level 1: survival

Level 1 answers one question: is the drive dead or actively dying? Without these signals, you will not know a drive has failed until an application breaks.

Device presence. The device node exists (/dev/sdX or /dev/nvmeXnY) and responds to SMART queries. If the node disappears, the drive has dropped off the bus from controller failure, cable disconnection, or catastrophic electronics failure.
SMART overall health self-assessment. The drive firmware compares its internal attribute values against vendor-defined thresholds and reports PASSED or FAILED. A FAILED result is the drive’s own death notice. Treat it as immediate data evacuation.
Reallocated sector count (ATA ID 5). Sectors or NAND blocks the firmware permanently remapped to the spare area because the original location went bad. Any non-zero value means the drive is consuming finite spare capacity.
Current pending sector count (ATA ID 197). Sectors flagged as unstable or unreadable during a read. These cause I/O latency spikes that look like system freezes. Any non-zero value in production requires investigation.
Drive temperature (ATA ID 194, NVMe Composite Temperature). Sustained high temperature exponentially reduces component lifespan. HDDs above 60 degrees C and SSDs above 70 degrees C are in the danger zone.
NVMe Critical Warning byte. A bitmask in the NVMe SMART/Health log flagging spare exhaustion, thermal threshold breach, reliability degradation, read-only mode, or power-loss protection failure. Any non-zero value requires investigation.

What this catches: drives that have died, drives the manufacturer considers failed, and drives with obvious media degradation.

What this misses: the PASSED assessment is notoriously optimistic. A drive can report PASSED while having hundreds of pending sectors and degrading performance. Vendors set generous thresholds to minimize warranty claims. Google’s 2007 study found that 36% of failed drives had zero SMART attribute warnings beforehand. Level 1 will not tell you how fast a drive is degrading, whether a cable is bad, or how much endurance remains.

Level 2: operational

Level 2 answers: what is degrading, and is it the drive or the infrastructure around it? This is the minimum a professional operation should monitor.

Offline uncorrectable sector count (ATA ID 198). Sectors that could not be recovered even after retries. Confirmed data loss at the hardware level. Alert on growth from baseline, not absolute value, to avoid paging on historical counters when you first deploy monitoring.
UDMA CRC error count (ATA ID 199). Data transfer corruption between host and drive over the SATA interface. This is almost never the drive’s fault. It is a cable, backplane, or connector problem. The most commonly misdiagnosed SMART attribute: operators replace the drive, the new drive develops the same errors because the cable was bad all along.
NVMe Available Spare. The percentage of spare NAND capacity remaining for bad-block replacement. Below the vendor threshold (typically 10%), the drive’s ability to maintain data integrity is compromised.
NVMe Percentage Used. A vendor-calculated estimate of rated endurance consumed. Above 90%, begin replacement planning. Above 100%, the drive operates beyond warranty parameters but may continue functioning.
NVMe Media and Data Integrity Errors. Confirmed data integrity failures where the controller could not recover data from NAND. Alert on growth from baseline.
SMART telemetry blind-spot detection. Compare expected physical drives against drives with successful SMART data collection. If physical drives exist but SMART data is not being collected, drives can fail silently with no alert. Common after adding a RAID controller, moving to a VM, or changing drive enclosures.
Scheduled self-tests. Drives do not run self-tests automatically. Without scheduled extended self-tests, latent bad sectors hide until production I/O hits them. A weekly short test and monthly extended test is standard practice. Schedule these in smartd.conf using the -s directive.

What this catches: cable and backplane problems distinguished from media problems, SSD endurance exhaustion before the wear-out cliff, NVMe drives entering degraded states, and RAID controller passthrough gaps.

What this misses: Level 2 checks absolute values, not trends. A drive with 5 reallocated sectors gained over three years is stable. A drive with 5 reallocated sectors gained this week is actively dying. Without rate-of-change tracking, you cannot distinguish historical damage from active failure.

Level 3: mature

Level 3 answers: how fast is it failing, and when will it reach a critical state? This is where monitoring shifts from reactive alerting to predictive planning.

Rate-of-change tracking for all counters. The single most important maturity leap. Monitor growth rate, not just current values. Accelerating reallocated sector counts, declining available spare, increasing CRC errors: the velocity of change is the actionable signal. A stable count of 10 over three years is far less concerning than a count of 10 gained in one week.
ATA error log analysis (smartctl -l error). The drive’s internal log of specific I/O errors with error type, LBA, and timestamp within the power cycle. Error types are diagnostic: UNC (uncorrectable, data loss), IDNF (ID not found, sector mapping corruption), ICRC (interface CRC, cable problem), CCTO (command completion timeout, mechanical or firmware issue). The ATA summary log holds only 5 entries. The extended log (smartctl -l xerror) provides more.
Self-test results. Check smartctl -l selftest after each scheduled test runs. A result of “Completed: read failure” with a specific LBA confirms a media defect that normal I/O had not yet hit. A passing extended self-test is strong evidence of media integrity.
Write volume for TBW runway. Track Data Units Written (NVMe) or Total LBAs Written (ATA ID 241) to calculate endurance runway: (rated TBW - current TBW) / daily write rate = days remaining. This lets you predict when an SSD will reach its rated endurance limit and schedule replacement before the wear-out cliff.
Power-on hours (ATA ID 9, NVMe). The drive’s age in operational hours. Critical for normalizing other metrics: 10 reallocated sectors over 50,000 hours means something different from 10 over 100 hours. Also essential for warranty tracking and fleet age management.
NVMe Warning and Critical Composite Temperature Time. Cumulative minutes the drive spent above its warning and critical temperature thresholds. Non-zero values even when current temperature is normal indicate past overheating events that may have caused cumulative damage.
Unsafe shutdown count. ATA: look for “Unexpect_Power_Loss_Ct” or “Power-Off_Retract_Count”. NVMe: “Unsafe Shutdowns” field. Rapid increase indicates a power stability problem that could cause data loss, especially on SSDs without power-loss protection.
Host I/O error monitoring. Kernel logs (dmesg) capture I/O failures and command timeouts that SMART attributes may not reflect. The pattern of “clean SMART, dirty kernel logs” catches firmware bugs and intermittent controller hangs that drive-side monitoring misses entirely. Key phrases to watch: “I/O error, dev sdX”, “medium error”, “Device not responding”, “task abort”.

What this catches: active degradation velocity, endurance consumption rate, cumulative thermal damage, power infrastructure instability, and failure modes that bypass SMART entirely (firmware bugs, controller hangs).

What this misses: Level 3 is thorough for individual drive health but operates in isolation. It does not correlate symptoms across drives in the same chassis, does not track mechanical early-warning signals on HDDs, and does not normalize risk across a mixed fleet.

Level 4: expert

Level 4 answers: which drives will fail next, and is the root cause the drives or the environment they live in? This is the monitoring that experienced operators add after incidents where Level 3 was not enough.

Seek error rate (ATA ID 7) normalized trend. The rate at which the HDD head positioning system fails to land accurately on the target track. Monitor the normalized value trend, not the raw value. Seagate drives report raw values in the billions as normal because the raw field encodes total seek operations, not just errors. Naive thresholding on the raw value produces constant false alarms on every Seagate drive.
Spin retry count (ATA ID 10). The count of times the spindle motor failed to reach operational speed on the first attempt. Any non-zero value on an enterprise drive is abnormal. If multiple drives show spin retries simultaneously, the root cause is the power supply, not individual drives.
G-Sense error rate (ATA ID 221). Cumulative count of shock or vibration events detected by the drive’s accelerometer. Growth indicates physical environment problems: improper mounting, dense disk shelf resonance, or nearby equipment vibration. Cross-reference with seek error rate: vibration directly causes seek errors.
Cross-drive correlation. If multiple drives in the same chassis show temperature increases, CRC errors, or spin retries simultaneously, the root cause is environmental (cooling failure, power instability, backplane degradation), not drive-specific. This pattern is invisible when monitoring each drive independently.
SMART value normalization by power-on hours and drive model. The same attribute value is more or less concerning depending on drive age and model. Normalize reallocated sector growth against power-on hours. Compare drives of the same model to detect batch defects: when multiple drives from the same manufacturing batch show similar symptoms, that is a recall risk.
Composite risk scoring. Combine multiple declining attributes into a per-drive risk score rather than alerting on individual thresholds. A drive with slowly rising reallocated sectors, declining available spare, and increasing error log entries is higher risk than one with only one signal moving. Composite scoring reduces false positives and prioritizes replacement scheduling.
Drive identity tracking. Track firmware version and serial number across the fleet. Alert on unexpected changes. Known firmware bugs have bricked drives from every major manufacturer. Without firmware version tracking, you cannot proactively identify drives running vulnerable firmware before a bug triggers.
First-observation baseline management. When deploying SMART monitoring on an existing fleet for the first time, cumulative counters (offline uncorrectable, NVMe media errors, power-on hours, unsafe shutdowns) will have non-zero historical values. The monitoring system must establish a baseline on first scrape and alert only on growth from that baseline. Paging on absolute lifetime values at first deployment floods operators with false alarms from every drive with any history.

What each level catches and misses

Level	Primary question	Catches	Blind to
Survival	Is the drive dead?	Complete failures, vendor-declared failures, gross media degradation	Degradation velocity, cable problems, endurance status
Operational	What is degrading?	Cable vs. media diagnosis, SSD wear tracking, NVMe health, monitoring gaps	Rate of failure, predictive runway, host-level errors
Mature	How fast is it failing?	Degradation velocity, endurance projection, thermal history, power events	Fleet-wide patterns, mechanical early warning, composite risk
Expert	What fails next?	Mechanical trends, batch defects, firmware vulnerabilities, composite risk	Sudden catastrophic failure (controller death, firmware crashes)

No level of SMART monitoring catches everything. Sudden mechanical failure, controller electronics death, and firmware crashes that make the drive vanish from the bus bypass SMART entirely because the component that reports SMART is the component that failed. Redundancy (RAID, replication) and backups remain the last line of defense.

How Netdata helps

Per-second collection makes rate-of-change tracking (the Level 3 leap) implicit. Reallocated sector growth is visible at hourly or weekly resolution without building a separate trend pipeline.
SMART attributes alongside disk I/O metrics (await, svctm, queue depth) in a single dashboard surfaces the “zombie drive” pattern: high I/O latency, low CPU, non-zero pending sectors. Without correlation, I/O latency spikes from bad sectors get misdiagnosed as application problems.
Blind-spot detection is automatic: if SMART collection stops returning data for a drive that should be present, that itself becomes a monitoring signal rather than a silent gap.
Anomaly detection on SMART counters flags unexpected changes without manual threshold tuning for every attribute on every drive model. Useful across mixed fleets where vendor encoding differs.
Cross-drive correlation in a single view makes environmental problems visible: when multiple drives in the same chassis show temperature or error changes simultaneously, the dashboard makes the pattern obvious rather than requiring manual comparison.
Host I/O metrics alongside SMART data close the “clean SMART, dirty kernel logs” gap that catches operators by surprise when firmware bugs or controller hangs cause production I/O errors.