$ guides / smartctl-disk-monitoring / smartctl-monitoring-checklist ▌

Operations Guides

smartctl disk monitoring checklist: the SMART signals every server needs

S.M.A.R.T. is firmware-level instrumentation built into every modern HDD, SSD, and NVMe drive. The drive reports on its own internal state. The smartctl tool from smartmontools reads what the firmware already knows. SMART monitoring is necessary but not sufficient: it catches gradual media degradation and endurance wear-out, but cannot predict sudden controller failures, firmware bugs, or silent data corruption. You still need redundancy, backups, and checksumming filesystems.

The single most important principle in this checklist: track rate of change, not absolute values. A drive with 50 reallocated sectors accumulated over five years is stable. A drive that gained 5 reallocated sectors this week is actively dying. Most monitoring failures come from alerting on absolute thresholds while missing the acceleration signal that distinguishes historical damage from active failure.

The four monitoring maturity levels

Each level builds on the previous one. A team at Level 2 already collects everything in Level 1 plus the additional signals listed. Ensure solid coverage at your current level before adding sophistication.

flowchart TD
    L1["Level 1: Survival
Device presence + SMART PASSED/FAILED"]
    L2["Level 2: Operational
+ Media, transport, NVMe health"]
    L3["Level 3: Mature
+ Rate-of-change, self-tests, endurance"]
    L4["Level 4: Expert
+ Cross-drive correlation, prediction"]
    L1 --> L2 --> L3 --> L4

Level 1: survival

Without these two signals, you will not know when a drive has died or when the drive firmware itself has declared failure.

Signal	How to check	What it detects
Device presence	`ls /dev/sdX` or `ls /dev/nvme0n1`	Complete drive failure, bus disconnection, controller death
SMART overall health	`smartctl -H /dev/sdX`	Drive firmware declaring itself PASSED or FAILED

Device presence is binary and unconditional. If the device node was present yesterday and is gone today, the drive has dropped off the bus. Track which devices are expected to exist (inventory baseline) and alert when any expected device is absent.

SMART overall health (smartctl -H) returns PASSED or FAILED. A FAILED result means at least one pre-fail attribute has crossed its vendor-defined threshold. Treat FAILED as a page-worthy event requiring immediate data evacuation.

The critical limitation: PASSED is notoriously optimistic. A drive can report PASSED while having hundreds of pending sectors, dozens of reallocated sectors, and degraded performance. Google’s 2007 study found that 36% of failed drives had zero SMART attribute warnings beforehand . Never stop at Level 1.

Level 2: operational

Missing any of these signals means you are blind to common, predictable failure modes.

Signal	Source	How to check	What it detects
Reallocated Sector Count	ATA ID 5	`smartctl -A /dev/sdX \| grep -i Reallocated`	Media degradation, spare pool consumption
Current Pending Sector	ATA ID 197	`smartctl -A /dev/sdX \| grep -i Pending`	Unstable sectors causing I/O latency spikes
Offline Uncorrectable	ATA ID 198	`smartctl -A /dev/sdX \| grep -i Uncorrectable`	Confirmed data loss at media level
Drive Temperature	ATA ID 194 / NVMe Composite	`smartctl -A /dev/sdX \| grep -i Temperature`	Thermal damage risk, throttling
UDMA CRC Error Count	ATA ID 199	`smartctl -A /dev/sdX \| grep -i CRC`	Cable, backplane, or transport failure
NVMe Available Spare	NVMe Health Log	`smartctl -A /dev/nvme0n1 \| grep -i Spare`	SSD spare pool exhaustion
NVMe Media and Data Integrity Errors	NVMe Health Log	`smartctl -A /dev/nvme0n1 \| grep -i Media`	NVMe data integrity failures
NVMe Critical Warning	NVMe Health Log byte 0	`smartctl -A /dev/nvme0n1 \| grep -i Critical`	Drive-flagged critical conditions
Telemetry blind spot	Expected vs actual drive count	`smartctl -i /dev/sdX` on all drives	Monitoring coverage gaps from RAID masking or USB bridges

The first three ATA attributes (IDs 5, 197, 198) form the core media integrity triad. Reallocated sectors are permanently remapped to spares from a finite pool. Pending sectors are unstable and waiting to be either recovered or reallocated on next write. Offline uncorrectable sectors are confirmed data loss: the drive tried and failed to recover the data. When all three are rising together, the drive is in terminal failure.

UDMA CRC Error Count (ID 199) is the most commonly misdiagnosed SMART attribute. CRC errors indicate data corruption on the SATA wire between the host and the drive. This is almost never the drive’s fault: it is a bad cable, loose connector, failing backplane port, or electromagnetic interference. If CRC errors are increasing but reallocated, pending, and uncorrectable counts are all zero, the problem is the cable, not the drive. Reseat or replace the cable first.

For NVMe drives, the monitoring center of gravity shifts. NVMe uses a standardized health log (Log Page 02h) instead of vendor-specific ATA attributes. The three signals that matter most: Available Spare (percentage of spare NAND remaining for bad-block replacement), Percentage Used (estimated endurance consumed), and the Critical Warning byte (a bitmask of five critical conditions). Teams that only look for NVMe media errors (a lagging indicator) miss the early warning that Available Spare and Percentage Used provide.

Telemetry blind spot detection is a monitoring infrastructure check, not a drive health check. Compare the number of physical drives your system should have against the number returning valid SMART data. Behind hardware RAID controllers, you need passthrough flags like -d megaraid,N or -d cciss,N to reach individual drives. Without them, SMART queries return virtual device data or fail silently.

Level 3: mature

Level 3 adds trend analysis, proactive scanning, and endurance planning. This is where you stop reacting to failures and start predicting them.

Signal	Source	Why it matters
Rate-of-change tracking	All cumulative counters	Distinguishes stable history from active failure
ATA Error Log	`smartctl -l error /dev/sdX`	Error type detail: UNC (data loss), ICRC (transport), CCTO (timeout)
Self-test results	`smartctl -l selftest /dev/sdX`	Proactive surface scan catches latent bad sectors before production I/O
NVMe Percentage Used	NVMe Health Log	Endurance consumption tracking, runway to rated TBW
Write Volume	ATA ID 241 / NVMe Data Units Written	Raw input for endurance runway calculation
Power-On Hours	ATA ID 9 / NVMe Health Log	Fleet age management, warranty tracking, attribute normalization
NVMe Composite Temperature Time	NVMe Health Log	Cumulative thermal exposure history
Unsafe Shutdown Count	NVMe Unsafe Shutdowns / ATA vendor-specific	Power stability monitoring
Host I/O errors	`dmesg`, kernel logs	Failures that SMART does not record

Rate-of-change tracking is the defining upgrade at this level. Every cumulative counter (reallocated sectors, pending sectors, CRC errors, NVMe media errors, power-on hours) should be tracked as a rate, not just a snapshot. The monitoring system must store historical values and compute deltas. A single snapshot tells you nothing about whether the drive is stable or deteriorating.

Scheduled self-tests are the active probing mechanism for SMART. Without them, SMART is passive: it only reports issues that surface during normal I/O. Sectors that are rarely read can silently degrade for months until a scrub or restore operation discovers the data is lost. Run a weekly short test and a monthly extended test. The extended test is the only reliable way to proactively discover bad sectors across the entire drive surface before the filesystem hits them.

# Schedule self-tests via smartd.conf
# Short test nightly at 2am
/dev/sda -a -s (S/../.././02)
# Extended test on the 1st of each month at 1am
/dev/sda -a -s (L/../01/../01)

NVMe Percentage Used can exceed 100%. The NVMe specification explicitly allows values up to 255%. A drive at 150% Percentage Used but with 90% Available Spare and zero media errors is operating fine, just beyond its warranty endurance rating. The failure indicator to watch is Available Spare declining toward zero, not Percentage Used alone.

Host I/O error monitoring catches failure modes that SMART misses entirely. A drive with clean SMART attributes but I/O errors and command timeouts in kernel logs is a classic pattern. Check dmesg for messages like “I/O error, dev sdX”, “medium error”, “task abort”, or SCSI error recovery loops. These indicate intermittent controller hangs, firmware bugs, or bus-level errors that never get recorded in SMART.

Level 4: expert

Level 4 adds cross-drive correlation, predictive modeling, and fleet-level analysis. This is what experienced operators build after incidents where earlier levels failed them.

Signal	Source	Why it matters
Seek Error Rate (normalized)	ATA ID 7	HDD mechanical wear, actuator degradation
Spin Retry Count	ATA ID 10	HDD spindle motor or power supply issues
NVMe Critical Warning individual bits	Bit-level tracking	Specific failure category: spare, temperature, read-only, PLP failure
Cross-drive correlation	Multiple drives in same chassis	Infrastructure problem vs individual drive failure
Drive identity tracking	Model, serial, firmware version	Change detection, known-buggy firmware versions
First-observation baselines	Fleet-wide initial SMART snapshot	Prevents alert storms when deploying monitoring on existing fleets

Cross-drive correlation is the most valuable expert technique. If one drive shows spin retries, that drive’s motor is failing. If multiple drives in the same chassis show spin retries simultaneously, the power supply is sagging. If multiple drives of the same model start showing similar SMART degradation around the same time, you have a batch defect. Individual drive monitoring cannot detect these patterns.

First-observation baseline management solves the deployment problem. When you first deploy SMART monitoring on an existing fleet, every drive with any historical errors will trigger alerts if your logic checks absolute values. Offline Uncorrectable counts, NVMe Media Errors, Power-On Hours, and Unsafe Shutdowns are all cumulative lifetime counters. The monitoring system must establish a baseline on first scrape and alert only on growth from that baseline.

Drive identity tracking (model, serial, firmware version via smartctl -i) catches unauthorized drive swaps, unexpected firmware changes, and known-buggy firmware versions. Known firmware bugs have bricked drives from every major manufacturer. Without firmware version tracking across your fleet, you cannot proactively identify drives running vulnerable firmware before the bug triggers.

Rate of change: the signal that matters most

Absolute values are context. Rate of change is the signal.

Consider a drive with 10 reallocated sectors. Is that concerning?

If those 10 appeared over 5 years and have been stable for the last 12 months: the drive contained a manufacturing defect that was contained. Monitor for growth.
If those 10 appeared in the last week: the drive is actively deteriorating. Schedule replacement now.
If the count went from 10 to 50 overnight: the drive suffered a physical event (head slap, shock, power surge). Evacuate data immediately.

Every cumulative SMART counter should be tracked as a rate. The monitoring system needs to store historical values, compute deltas over meaningful windows (hours, days, weeks), and alert on acceleration. A threshold like “reallocated sectors > 10” is nearly useless. A threshold like “reallocated sectors increased by >= 5 in 24 hours, AND current pending sectors is non-zero sustained across >= 2 polls, AND at least one corroborating signal (offline uncorrectable also increased, or self-test failed, or SMART health is FAILED)” catches active multi-signal media failure without false alarms.

This also applies to smartctl exit codes. The exit status is a bitmask: exit code 0 means all checks passed. Bit 3 (value 8) indicates the SMART health check returned FAILED. Bit 6 (value 64) indicates the device error log contains records, which may be historical. Bit 7 (value 128) indicates the self-test log contains records. A monitoring script that treats any non-zero exit as failure will generate constant false positives on drives with historical error log entries.

Pitfalls and false alarms

Pitfall	What happens	How to avoid it
Monitoring only PASSED/FAILED	36% of failed drives had zero SMART warnings (Google 2007). By the time SMART says FAILED, the drive has been dying for weeks.	Track individual attributes (IDs 5, 197, 198) alongside the health check.
Seagate raw value panic	IDs 1 and 7 show raw values in the billions on healthy Seagate drives. The raw field encodes total operation counts plus error counts, not just errors.	Use the normalized value (VALUE column), not the raw value, for Seagate drives.
Replacing drives for cable problems	CRC errors (ID 199) spike, team replaces the drive. The new drive develops the same errors because the cable was the problem.	CRC errors increasing with zero media errors = cable problem. Reseat or replace the cable first.
No scheduled self-tests	Latent bad sectors hide until production I/O hits them, producing an I/O error instead of a self-test warning.	Schedule weekly short tests and monthly extended tests via smartd.
Treating NVMe like SATA	Teams only look for NVMe media errors (lagging indicator) and miss Available Spare and Percentage Used (leading indicators).	Center NVMe monitoring on Available Spare, Percentage Used, and Critical Warning bits.
RAID controller blind spot	Standard SMART queries return virtual device data or fail. Monitoring appears to work but sees nothing.	Use controller-specific passthrough: `-d megaraid,N`, `-d cciss,N`, `-d sat+megaraid,N`.
Alerting on lifetime counters at first deployment	Every drive with any history triggers an alert. Operators learn to ignore all SMART alerts.	Baseline on first scrape. Alert on growth from baseline, not absolute values.
Single-metric alerting without drive-type context	Temperature threshold of 60 C pages for both an HDD (critical) and an enterprise NVMe (well within spec).	Make thresholds drive-type-aware. Check datasheets for rated maximums.

Netdata integration

Per-second SMART attribute collection matters operationally for two reasons:

Rate-of-change detection on short timescales. Daily smartctl cron jobs miss acceleration that happens over hours. Per-second collection catches a reallocated sector count jumping from 0 to 15 in minutes.
Cross-signal correlation. A latency spike in disk await can be confirmed against non-zero pending sectors on the same drive. A temperature rise on an NVMe drive can be correlated with throughput dropping due to thermal throttling.

For NVMe, Available Spare, Percentage Used, Media and Data Integrity Errors, and Critical Warning bits are collected as first-class metrics continuously, not just on manual smartctl runs. Device presence monitoring detects drives dropping off the bus within seconds, and kernel log correlation helps distinguish sudden disappearance (controller failure) from gradual degradation.