SMART overall-health self-assessment: FAILED is the drive's own death notice

When smartctl -H reports SMART overall-health self-assessment test result: FAILED, the drive’s own firmware has concluded it is failing. At least one pre-fail attribute has crossed its vendor-defined threshold. This is not smartctl’s interpretation or a monitoring heuristic. Treat it as an unconditional page. Unlike individual attributes that require trend analysis and corroboration, FAILED is the drive saying it is done. False positives are rare in the normal case because the drive itself is making the call, not your monitoring system.

The critical operational mistake is treating every non-PASSED result as FAILED. When smartctl reports UNKNOWN, UNSUPPORTED, or an error reading the status, that is an instrumentation problem, not a drive failure. A USB bridge that does not pass the SMART RETURN STATUS command, or a RAID controller without passthrough, produces UNKNOWN. Paging on UNKNOWN burns operator trust and masks real failures. Do not treat non-PASSED as equivalent to FAILED.

What this means

The overall-health self-assessment works differently for ATA and NVMe drives, but the semantics are the same: the drive firmware performs an internal check and returns a pass/fail verdict.

For SATA (ATA) drives, smartctl -H issues the ATA SMART RETURN STATUS command. The drive firmware compares each pre-fail attribute’s normalized VALUE against its vendor-defined THRESH. If any pre-fail attribute’s VALUE has dropped to or below its THRESH, the drive returns a failing status. smartctl decodes the ATA output register values and prints FAILED. For a passing result, the expected register signature is CL=0x4f, CH=0xc2. Any other combination produces FAILED.

SAS drives are not ATA. smartctl reads the SCSI Informational Exceptions (IE) log page instead. The health verdict is derived from the Additional Sense Code reported in the IE log page.

For NVMe drives, smartctl -H reads the SMART/Health Information Log (Log Page 02h). The NVMe overall-health assessment incorporates the Critical Warning byte. If any critical warning bit is set, the overall-health line shows FAILED with a human-readable description of which condition triggered it. This is more comprehensive than the ATA binary check because it includes spare exhaustion, thermal thresholds, reliability degradation, read-only mode, and power-loss protection failure.

The key distinction operators must internalize:

Only the literal FAILED status triggers a page. UNKNOWN and UNSUPPORTED are instrumentation gaps that need a ticket to repair the monitoring path, not a page to evacuate data.

NVMe Critical Warning and FAILED

On NVMe drives, the Critical Warning byte drives the overall-health verdict. Each bit maps to a specific condition:

A common operator surprise: drives past their rated endurance (Percentage Used above 100%) may set bit 2 (0x04) and report FAILED even with zero media errors and adequate Available Spare. The drive firmware has made this judgment, and the FAILED result is legitimate from the firmware’s perspective. The operational urgency may differ from a FAILED caused by read-only mode (bit 3) or active media corruption. Always investigate the individual Critical Warning bits to understand the specific condition driving the verdict.

PASSED does not mean healthy

A PASSED result only means no pre-fail attribute has crossed its vendor threshold. Vendors set conservative thresholds. A drive can report PASSED while having hundreds of pending sectors, dozens of reallocated sectors, and degraded performance. Google’s 2007 study found that a significant fraction of failed drives had zero SMART attribute warnings beforehand.

The overall-health check is a binary pre-fail detector, not a comprehensive health score. Never monitor only this signal.

flowchart TD
    A["smartctl -H /dev/sdX"] --> B{Result?}
    B -->|"PASSED"| C["No pre-fail attribute crossed threshold
Still need to check individual attributes"]
    B -->|"FAILED"| D["Drive firmware judged failure
PAGE: evacuate data"]
    B -->|"UNKNOWN"| E["Instrumentation cannot read status
TICKET: fix monitoring path"]
    B -->|"UNSUPPORTED"| F["Transport or drive
does not support health queries
TICKET: fix monitoring path"]
    D --> G["Identify which attribute crossed threshold"]
    G --> H["ATA: smartctl -A, check WHEN_FAILED column
NVMe: check Critical Warning bits"]

Common causes of a FAILED verdict

A FAILED verdict means a specific pre-fail attribute has crossed its threshold. The underlying failure mode determines the attribute and the urgency.

Quick checks

These are safe, read-only commands. Run them to confirm the FAILED verdict and identify which attribute triggered it.

# Confirm the overall-health result
smartctl -H /dev/sdX

# View all SMART attributes with values, thresholds, and raw values
smartctl -A /dev/sdX

# For NVMe: view health log including Critical Warning, Available Spare, Percentage Used
smartctl -A /dev/nvme0n1

# Check the ATA error log for recent UNC, ABRT, or CCTO errors
smartctl -l error /dev/sdX

# Check the self-test log for the most recent test results
smartctl -l selftest /dev/sdX

# Check kernel logs for I/O errors, bus resets, or device removal events
dmesg | grep -iE "I/O error|medium error|reset|timeout|offline" | tail -20

# Check smartctl exit code: bit 3 (value 8) = DISK FAILING, bit 4 (value 16) = prefail at threshold
smartctl -H /dev/sdX; echo "exit code: $?"

How to diagnose it

1. Confirm the exact result string. Run smartctl -H /dev/sdX and verify the output says exactly FAILED. If it says UNKNOWN! or SMART Status not supported, you have an instrumentation problem, not a drive failure. Skip to step 8.

2. Identify which attribute crossed threshold. Run smartctl -A /dev/sdX and look for attributes where the VALUE column is at or below the THRESH column. The WHEN_FAILED column shows FAILING_NOW for the offending attribute.

3. For NVMe drives, decode the Critical Warning byte. Run smartctl -A /dev/nvme0n1 and find the Critical Warning line. A non-zero hex value means at least one bit is set. Cross-reference each set bit against the table above.

4. Correlate with individual attributes. Check Reallocated Sector Count (ID 5), Current Pending Sector (ID 197), Offline Uncorrectable (ID 198), and the ATA error log. The FAILED verdict is a summary. The individual attributes tell you how advanced the failure is and whether data loss has already occurred.

5. Check kernel logs for host-level impact. Run dmesg filtered for I/O errors, timeouts, and bus resets. A drive reporting FAILED while also generating kernel I/O errors is actively impacting production. If kernel logs are clean, the failure may still be contained, but it will not stay that way.

6. Verify the exit code. smartctl sets exit code bit 3 (value 8) when the SMART status check returns DISK FAILING. Bit 4 (value 16) indicates a pre-fail attribute is at or below threshold. A FAILED overall-health result sets bit 3, producing an exit code with the 8 bit set (commonly 8, but possibly combined with other bits).

7. Assess data redundancy. Determine whether the data on this drive has redundancy elsewhere (RAID, replication, backups). If the drive is the sole copy of any data, prioritize evacuation immediately. If redundancy exists, verify it is healthy and current before the drive degrades further.

8. For UNKNOWN or UNSUPPORTED results: diagnose the instrumentation path. This is a monitoring fix, not a data evacuation. Common causes and fixes:

   • USB-attached drives: USB-to-SATA bridge chips may not pass the SMART RETURN STATUS command. Try smartctl -d sat /dev/sdX. Not all bridge chipsets (JMicron, ASMedia, Cypress) support SMART passthrough.
   • Hardware RAID controllers: RAID controllers abstract individual drives behind a virtual device. Use controller-specific passthrough. LSI/Broadcom MegaRAID: smartctl -d megaraid,N /dev/sda. HP Smart Array: smartctl -d cciss,N /dev/sda. The N is the physical drive index on the controller.
   • Virtual machines: Hypervisors typically do not pass SMART data through to guests. Monitoring must happen on the hypervisor host, not inside the VM.

Metrics and signals to monitor

Fixes

A drive reporting FAILED must be replaced

There is no software fix for a FAILED overall-health verdict. The drive firmware has made the determination. The only correct response is data evacuation and physical replacement.

1. Evacuate data immediately. If redundancy exists (RAID, replication), verify it is healthy and current. If the drive holds the only copy of data, copy it off now.
2. Schedule replacement. Do not wait for the drive to fail completely. A drive reporting FAILED may continue to function for hours, days, or weeks, or it may die on the next I/O. The timeline is unpredictable.
3. Monitor closely until replaced. Watch for increasing I/O errors, device disappearance, or read-only mode transitions. A drive that has declared FAILED can degrade rapidly.

Do not attempt sector repair on a FAILED drive

Running dd or badblocks to force sector reallocation on a drive that has already declared FAILED is risky. The drive is in a terminal state. Forcing writes to suspect sectors may trigger further failures or cause the drive to lock up entirely, taking your last chance to read remaining data with it.

Instrumentation problems (UNKNOWN / UNSUPPORTED)

If the result is UNKNOWN or UNSUPPORTED, the fix is in the monitoring path. This is a ticket, not a page. Identify the transport layer preventing SMART passthrough and configure the correct smartctl device type flag. The drive itself may be perfectly healthy.

Prevention

• Never monitor only the overall-health assessment. PASSED does not mean healthy. Individual attributes (IDs 5, 197, 198, NVMe Available Spare, Critical Warning) provide earlier warning, often days to weeks before the overall-health changes to FAILED. The overall-health check is the last line of defense, not the first.
• Track rate of change on all counters. A drive with 50 reallocated sectors accumulated over 5 years is stable. A drive with 5 reallocated sectors gained this week is actively dying. Most monitoring checks absolute values only, missing the critical acceleration signal.
• Schedule periodic extended self-tests. Drives do not run self-tests automatically. Without scheduled extended tests, latent bad sectors hide until production I/O hits them, at which point you get a production error instead of a proactive warning.
• Baseline every drive at deployment. Capture a full SMART snapshot when the drive is first installed. This lets you distinguish factory-shipped values from production-acquired degradation.
• Configure alerting to page only on the exact FAILED string. Route UNKNOWN, UNSUPPORTED, and error-reading-status results to a ticket queue for monitoring path repair. Do not conflate instrumentation gaps with drive failures.

How Netdata helps

• Per-second SMART collection catches rate-of-change on reallocated sectors, pending sectors, and available spare before the drive crosses its threshold and declares FAILED.
• Correlation across the stack connects a FAILED verdict with kernel I/O errors, latency spikes, and temperature trends in the same time window, confirming whether the failure is impacting production right now.
• Device presence monitoring catches the failure mode that bypasses SMART entirely: sudden controller death or bus disconnection where the drive vanishes before any attribute updates.
• Fleet-level views surface whether multiple drives of the same batch are trending toward failure simultaneously, pointing to a manufacturing defect or environmental problem rather than isolated wear.

Related guides

• Current_Pending_Sector non-zero: unreadable sectors and I/O latency spikes
• How S.M.A.R.T. actually works: a mental model for operators
• smartctl disk monitoring checklist: the SMART signals every server needs
• SMART monitoring maturity model: from survival to expert
• Offline_Uncorrectable climbing: permanent data loss at the media level
• Raw_Read_Error_Rate looks enormous: the Seagate false alarm explained
• Reallocated_Event_Count vs Reallocated_Sector_Ct: reading both together
• Reallocated_Sector_Ct rising: the drive is burning through its spare pool