$ guides / smartctl ▌

SMARTCTL · DISK HEALTH PLAYBOOK

The drive reports on its own health — but PASSED lies, raw values mislead, and some failures never show up at all

smartctl reads the firmware's own instrumentation: remapped sectors, NAND wear, interface errors, temperature, mechanical wear. We trace what each signal actually means, why the overall health check misses roughly a third of failures, and what to do when one specific attribute or drive state starts to move.

> Start with the monitoring checklist → # Jump to the full guide list

SMART gets you a health check in a single command, then hands you a set of numbers that are easy to read and just as easy to misread.

The one-liner is a trap. smartctl -H returns PASSED right up until a drive carrying hundreds of Current_Pending_Sector and a climbing Reallocated_Sector_Ct finally trips a generous vendor threshold — and Google's 2007 fleet study found 36% of failed drives showed no SMART warning at all. UDMA_CRC_Error_Count spikes and a team swaps a perfectly healthy drive when the real fault was the cable. Raw_Read_Error_Rate reads in the billions on a Seagate and looks catastrophic when it is normal by design. An NVMe drive quietly flips to read-only (Critical Warning bit 3) with Available Spare at zero. A controller dies and the device simply vanishes from the bus, leaving a clean SMART history behind it.

These guides are written for engineers who already run disks in production, not for people learning what a sector is. The goal is the mental model of what the firmware is actually measuring, which signals lead a failure and which only confirm it, why rate-of-change beats absolute thresholds every time, and the runbook for the specific attribute or state you just watched move.

How disk health actually reaches you

S.M.A.R.T. is not a monitoring agent — it is the drive's firmware reporting on itself, and smartctl only reads what the firmware already knows. Health flows up a stack from the physical media to the operating system, and most surprises live between the layers: a signal that leads failure on one layer, a summary verdict that lags it on another, and whole failure modes that bypass SMART entirely.

host + kernel I/O

What the operating system experiences: <code>iostat</code> await, <code>/proc/diskstats</code>, and kernel messages like <code>I/O error, dev sdX</code>. The host sees failures — controller hangs, command timeouts, SCSI error recovery — that never appear in SMART. smartctl reads the drive; the kernel reads the experience.

HOST

transport / interface

The physical path between host and drive: SATA/SAS cable, connector, backplane, HBA, PCIe lanes. Corruption here shows as <code>UDMA_CRC_Error_Count</code> (ATA) or SAS PHY counters, and can downshift the link to a slower speed. The media is fine; the wire is not.

LINK

drive controller + firmware

The drive's brain: ECC on every read, the SSD flash-translation layer, garbage collection, wear leveling — and SMART itself. When this dies the drive vanishes from the bus, and SMART can give no warning, because the component that reports it is the one that failed.

CONTROLLER

SMART attributes + health log

The firmware's self-report. ATA exposes vendor-specific attribute IDs with normalized VALUE, RAW, and THRESH columns; NVMe exposes a standardized health log; the binary PASSED/FAILED verdict summarizes both — optimistically, against thresholds tuned to limit warranty claims.

SMART

spare pool + remapping

The finite reserve that keeps a degrading drive alive. <code>Current_Pending_Sector</code> awaits a verdict, <code>Reallocated_Sector_Ct</code> records remaps already done, <code>Offline_Uncorrectable</code> counts the ones that failed; on SSD/NVMe it is <code>Available Spare</code>. Exhaust it and the next bad block is data loss.

SPARE

NAND / platter media

Where the bits physically live. SSD NAND has finite program/erase cycles tracked as <code>Percentage Used</code> and wear-leveling count; HDD magnetic surface degrades into bad sectors and rising read-error rates. This layer wears out predictably on an SSD and erratically on an HDD.

MEDIA

mechanical assembly (HDD)

Spindle motor, actuator arm, and heads. <code>Spin_Retry_Count</code>, <code>Spin_Up_Time</code>, <code>Seek_Error_Rate</code>, and <code>G-Sense_Error_Rate</code> report wear and shock. None of it exists on an SSD — and a single failure here can end the drive instantly rather than gradually.

MECHANICAL

environment

Temperature, power, and vibration act on every layer above, often exponentially. Heat degrades NAND retention and HDD bearings; unstable power drives unsafe shutdowns and spin retries across many drives at once; vibration causes seek errors. The context that decides how fast everything else fails.

ENVIRONMENT

Why this matters: 'the disk is slow' or 'I'm seeing disk errors' can be a bad cable retransmitting, pending sectors forcing read retries, an SSD stalling on garbage collection, an NVMe drive thermally throttling, a spindle struggling to spin up, or a controller about to disappear. The symptom rhymes but each layer has a different signal — and a different fix. And the overall health check can read PASSED through most of them.

The failures you'll actually see

Most disk incidents fall into a small set of recurring shapes. Recognise the shape and triage — replace the drive, or the cable, or the power supply — gets dramatically faster.

CRITICAL

The slow-dying drive

Bad sectors accumulate. Current_Pending_Sector appears, Reallocated_Sector_Ct climbs as the spare pool is consumed, and every read that lands on a suspect sector triggers a multi-second retry. The system looks frozen — high iostat await with an idle CPU — while the drive dies in slow motion over days.

Current_Pending_Sector (ID 197) non-zero and rising
Reallocated_Sector_Ct (ID 5) climbing week over week
iostat await spiking to seconds with the CPU idle (high iowait)
ATA error log showing UNC errors at specific LBAs

Investigate →

CRITICAL

The SSD wear-out cliff

SSDs degrade gradually, then fail abruptly. Available Spare falls toward its threshold, Percentage Used passes 100%, and media errors begin — then the controller flips the drive to read-only, or dead, within hours. Unlike an HDD, the step from slightly worn to completely gone can be a single day.

Available Spare at or below the vendor threshold (typically 10%)
Percentage Used at or above 100%
Media and Data Integrity Errors starting to increment
Wear_Leveling_Count / Media_Wearout_Indicator near end of scale (SATA SSD)

Investigate →

ACTIVE

The bad cable, not the drive

UDMA_CRC_Error_Count climbs and throughput drops, but every media attribute is zero. The corruption is on the wire — a loose or failing cable, backplane port, or HBA — and CRC catches it, so data integrity holds while the link may downshift to a slower speed. Replace the drive and the new one develops the same errors.

UDMA_CRC_Error_Count (ID 199) increasing, often just after maintenance
SATA link at current: below its maximum negotiated speed
Reallocated (5), Pending (197), and Uncorrectable (198) all zero
SAS: rising invalid-dword and loss-of-sync PHY counters

Investigate →

CRITICAL

The drive that vanished

The device node was present and is now gone. A controller, PCB regulator, connector, or backplane failed — and because the component that reports SMART is the one that died, there was no warning. The SMART history is clean right up to the disappearance.

/dev/sdX or /dev/nvmeXnY present before, absent now
dmesg showing device removal, bus reset, or error-recovery failure
Clean SMART on the last successful collection
RAID array reporting a degraded or missing member

Investigate →

IMMINENT

PASSED, but dying

smartctl -H returns PASSED while the drive carries hundreds of pending sectors, dozens of reallocations, and real latency. PASSED only means no pre-fail attribute has crossed a generous vendor threshold — and 36% of failed drives (Google 2007) showed no warning at all. Trusting the summary is how surprise failures happen.

Overall health PASSED with non-zero pending / reallocated / uncorrectable
An individual attribute VALUE approaching its THRESH
Counters rising that never yet trip the binary verdict
Kernel I/O errors on a drive SMART calls healthy

Investigate →

CRITICAL

The spindle that won't spin up

Spin_Retry_Count goes non-zero — the motor failed to reach operating speed on the first attempt. A drive that struggles to spin up may not come back after the next power cycle. One drive means a failing motor; many drives at once means the power supply or the 12V rail, not the disks.

Spin_Retry_Count (ID 10) non-zero on a drive in service
Spin_Up_Time (ID 3) rising well above its own baseline
Multiple drives showing retries together (a power problem)
Unsafe-shutdown count rising alongside

Investigate →

Choosing a tool

Best Disk Health & S.M.A.R.T. Monitoring Tools

A ranked review of the tools teams actually shortlist here, what each one is genuinely good at, and how the pricing behaves as you scale.

Read the buyer's guide →

SMART monitoring maturity levels

Disk-health monitoring works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much the data on these drives matters. Anything holding real data should land at the second level.

Level 1: Survival

Know when a drive has died or declared itself dying

Survival monitoring is the floor: is the drive present, and does either its own firmware or its most basic attributes say it is failing? You will not predict failure, but you will not be blind to a drive that has already failed or called itself failed. Enough for a workstation or a disposable node; not enough for anything holding data you care about.

Device present Does /dev/sdX or /dev/nvmeXnY still exist? A vanished node is the drive gone.
SMART overall health (PASSED/FAILED) A FAILED verdict is the drive's own death notice — page on it.
Reallocated_Sector_Ct (ID 5) Any growth means the finite spare pool is being consumed.
Current_Pending_Sector (ID 197) Unreadable sectors awaiting remap; the active-degradation signal.
Drive temperature Sustained operation above the drive-type's rated maximum.
NVMe Critical Warning Any non-zero bit is a firmware-declared active problem.

Level 2: Operational

Catch drives on their way out, not just already out

Operational monitoring is what any team storing real data should run. Survival catches drives that already failed; operational catches the ones on their way. It watches the leading media and endurance signals, separates cable faults from drive faults, and confirms you can actually see every physical drive — the coverage that turns a 3am surprise into a planned replacement.

Offline_Uncorrectable (ID 198) Confirmed permanent data loss; alert on growth, key on the attribute name.
UDMA_CRC_Error_Count (ID 199) Rising CRC with zero media errors is a cable, not a drive.
NVMe Available Spare The remaining spare pool; below the vendor threshold is urgent.
NVMe Percentage Used Rated endurance consumed; watch the trend toward 100%.
NVMe Media and Data Integrity Errors Growth here is confirmed NAND-level corruption.
SMART blind-spot check Every expected physical drive actually returning SMART data.
Scheduled self-tests Short weekly, extended monthly — the drive will not scan itself.

Level 3: Mature

Read trends and logs before the summary ever changes

Mature monitoring is trend-based, not threshold-based. It tracks how fast each counter is moving, reads the error and self-test logs, projects SSD endurance runway from write volume, and watches the kernel's view alongside the drive's. This is where you catch the drive that is technically PASSED but visibly accelerating toward failure.

Rate-of-change on every counter 5 sectors gained this week beats 50 stable over five years.
ATA / NVMe error log New entries and their type (UNC, ICRC, CCTO), not just the count.
Self-test results Read/servo/electrical failures, and tests that never complete.
Write volume vs rated TBW Data Units Written / Total_LBAs_Written for endurance runway.
Power_On_Hours Fleet age; the denominator that normalizes every other attribute.
NVMe Composite Temperature Time Minutes spent overheating, even when the drive is cool now.
Host I/O errors (kernel logs) The failures the drive itself cannot see.
Unsafe shutdown count Rising counts flag power instability before it corrupts data.

Level 4: Expert

Reactive depth added after real incidents

Expert signals get added after a specific incident proved you needed them: a batch of drives that failed together, a firmware bug that bricked a model, an SSD burned out by a misconfigured workload. Most fleets never need all of these. Add the ones your own failure history points to — and always baseline on growth, not lifetime absolutes.

Seek_Error_Rate / Spin_Up_Time trends HDD mechanical wear, read via the normalized value only.
G-Sense_Error_Rate Shock and vibration actually reaching the drive.
Cross-drive correlation Same symptom on many drives at once = infrastructure, not disk.
Firmware + identity tracking Unexpected version changes, counter resets, known-buggy firmware.
First-observation baselining Alert on growth from a captured baseline, never lifetime absolutes.
Composite risk scoring Combine several declining attributes into one per-drive verdict.
SMART collection latency A drive slow to answer smartctl is an early controller-trouble sign.

Operating mistakes worth avoiding

The traps disk-monitoring teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after a surprise failure or a needlessly replaced drive.

Trusting SMART PASSED as a clean bill of health

<code>smartctl -H</code> returns PASSED until a pre-fail attribute crosses a vendor threshold set to minimize warranty claims, not to protect your data. A drive can report PASSED with hundreds of pending sectors and seconds-long read latency, and Google's 2007 fleet study found 36% of failed drives triggered no SMART warning at all. Watch the individual attributes and the NVMe health log, never the summary alone.

Alerting on absolute values instead of rate of change

A <code>Reallocated_Sector_Ct</code> of 50 accumulated over five years is stable; a count that went from 0 to 5 this week is a drive actively dying. Most monitoring asks 'is the value above X?' and misses the acceleration that is the actionable signal. Track deltas per drive over time.

Blaming the drive for a cable problem

<code>UDMA_CRC_Error_Count</code> climbs, the team swaps the drive, and the replacement develops the same errors — because the cable, connector, or backplane port was always the fault. CRC errors with zero media errors (IDs 5/197/198) point at the transport layer. Check that relationship before condemning a disk.

Panicking at Seagate raw values

Seagate encodes <code>Raw_Read_Error_Rate</code> (ID 1) and <code>Seek_Error_Rate</code> (ID 7) as packed counters whose raw values routinely reach the billions and are completely normal. Alerting on raw ID 1 or ID 7 above zero produces constant false alarms — and operators who then learn to ignore SMART entirely. Only the normalized VALUE trending toward THRESH matters for these attributes.

Never scheduling extended self-tests

Drives do not scan themselves. Without a scheduled extended self-test, latent bad sectors in rarely-read data stay hidden until a scrub, backup, or restore finally reads them — turning a preventable warning into a production I/O error. A short weekly and an extended monthly test is the cheapest insurance there is.

Not tracking SSD write endurance

Teams deploy SSDs, never compare Data Units Written against the datasheet TBW, and are surprised when a drive dies early. It died exactly on schedule — the schedule was just never computed. Trend write volume and NVMe <code>Percentage Used</code> to know the replacement date months ahead.

The RAID controller and VM blind spot

Behind a hardware RAID controller, plain <code>smartctl</code> sees only the virtual device; inside a VM or cloud instance, SMART is usually absent or meaningless. Teams believe monitoring works while it sees nothing. Use <code>-d megaraid,N</code> or <code>-d cciss,N</code> passthrough on RAID, run SMART on the hypervisor host, and reconcile expected drives against monitored ones.

Paging on lifetime counters at first deployment

Roll SMART monitoring onto an existing fleet and every drive with any history — Offline_Uncorrectable, NVMe Media Errors, Unsafe Shutdowns — fires at once. Capture a baseline on the first scrape and alert only on growth from it. Absolute lifetime values at rollout are an alarm flood, not a signal.

SMART disk-health runbooks in this section

Each guide is a focused runbook for one attribute, state, or failure pattern. Pick one when an attribute just moved, or use the categories to learn the area.

▸

Start here

▸

Media integrity: sectors and read errors

▸

Health assessment and self-tests

▸

SSD and NVMe endurance

▸

NVMe health log and critical warnings

▸

Interface, transport, and I/O errors

▸

Temperature and thermal throttling

▸

Mechanical health (HDD)

▸

Device presence and monitoring blind spots

▸

Lifecycle, power, and fleet baselining

WHERE TO GO NEXT

Setting up SMART monitoring, or staring at an attribute that just moved?

If you're starting from scratch, the monitoring checklist is the path of least regret. If a specific attribute or drive state just changed, jump straight to the runbook that matches it.

> Start with the checklist > Back to Operations Guides