Offline_Uncorrectable climbing: permanent data loss at the media level

Offline_Uncorrectable (SMART attribute 198) is the most unambiguous media-failure signal in the ATA SMART attribute set. When this counter increments, a sector was read during an offline scan or self-test and the drive firmware could not recover the data even after ECC correction and multiple retries. The data at that LBA is gone. If the filesystem or application layer did not provide redundancy (RAID, checksums, backups), the loss is permanent.

A non-zero value on first observation does not necessarily mean active failure. When SMART monitoring is first deployed on an existing fleet, drives will carry historical counts from months or years of prior operation. The critical signal is growth from baseline: a counter that was 3 yesterday and is 5 today means two more sectors have permanently failed.

What this means

A sector counted by attribute 198 has been through the drive’s full error correction chain during an offline scan or self-test, and the data could not be recovered. This is confirmed hardware-level data loss at the physical media.

The distinction between attribute 197 (Current_Pending_Sector) and 198 (Offline_Uncorrectable) is timing and detection mechanism. Both can refer to the same physical bad sectors:

• 197 increments during normal I/O when a sector fails a read but has not yet been written to. The sector is “pending” reallocation.
• 198 increments during offline data collection or self-tests, when the drive proactively scans and discovers sectors it cannot recover.

A pending sector (197) may later become uncorrectable (198) if the offline scan confirms an unrecoverable read, or it may become reallocated (ID 5) if a successful write triggers remapping to a spare sector.

flowchart TD
    A[Unstable sector detected during I/O] --> B["Current_Pending_Sector ID 197"]
    B --> C{Offline scan or self-test reads sector}
    C -->|ECC recovers data| D[Sector may clear pending list]
    C -->|ECC cannot recover| E["Offline_Uncorrectable ID 198"]
    B --> F{Write reaches the sector}
    F -->|Remap to spare succeeds| G["Reallocated_Sector_Ct ID 5"]
    F -->|Spare pool exhausted| E
    E --> H[Confirmed permanent data loss]

The relationship between these three attributes tells you where the drive is in its failure trajectory:

• 197 rising, 5 and 198 stable: Firmware has not yet processed the bad sectors. Early stage.
• 197 and 198 rising together: Recovery attempts are failing. Sectors are being confirmed as unreadable.
• All three rising: Terminal failure. The drive is consuming its spare pool faster than it can compensate. Replace immediately.

One operational detail: this counter can decrease on some drives if the affected sector is subsequently overwritten, triggering reallocation and removing the sector from the uncorrectable list. The original data is still lost. The counter moved because the sector was remapped, not because the data was restored.

The attribute name gotcha

SMART attribute IDs are not standardized by the ATA specification. ID 198 is “Offline_Uncorrectable” on most drives, but some vendors repurpose the ID for a different attribute . Smartmontools maintains a vendor-specific preset database (drivedb.h) that handles name mapping, but the raw values still require vendor-aware interpretation.

Always key alert logic on the decoded attribute name (“Offline_Uncorrectable”), not the raw attribute ID. Alerting on ID 198 across a mixed fleet without checking the decoded name will produce false positives on any drive that repurposes the ID.

Common causes

Quick checks

All commands below are read-only. None modify drive state or affect production I/O.

# Check attribute 198 by name (not by ID number alone)
smartctl -A /dev/sdX | grep -i "Offline_Uncorrectable"

# Full SMART attribute table including IDs 5, 197, 198, 199
smartctl -a /dev/sdX

# Verify the decoded attribute name matches expectations
smartctl -A /dev/sdX | grep "198"

# Check overall health assessment
smartctl -H /dev/sdX

# Review the ATA error log for UNC (uncorrectable) entries
smartctl -l error /dev/sdX | grep -i "UNC\|uncorrectable"

# Check self-test results for read failures
smartctl -l selftest /dev/sdX

# Cross-reference pending and reallocated sector counts
smartctl -A /dev/sdX | grep -iE "Current_Pending|Reallocated_Sector"

# Check kernel logs for I/O errors at the host level
dmesg | grep -iE "I/O error|medium error" | tail -20

# Verify I/O latency correlation (high await = drive retrying bad sectors)
iostat -x 1 5

How to diagnose it

1. Confirm the attribute name. Run smartctl -A /dev/sdX and verify that ID 198 is decoded as “Offline_Uncorrectable”. If it shows a different name, this signal does not apply to that drive.

2. Establish whether the count is growing. Compare the current raw value against a baseline snapshot. If no baseline exists, take one now and re-check after the next offline scan or self-test. Growth from baseline is the signal that matters. A static historical count is not an emergency.

3. Run an extended self-test. This is the definitive confirmation. Start with smartctl -t long /dev/sdX and check results with smartctl -l selftest /dev/sdX when it completes (hours for large HDDs). The test runs in background and may impact I/O latency on busy drives. A result of “Completed: read failure” with a specific LBA confirms a real media defect. “Completed without error” with a non-zero ID 198 suggests either a historical count or a vendor quirk.

4. Check corroborating signals. Look for:

   • Current_Pending_Sector (ID 197) non-zero and sustained across at least two polls
   • Reallocated_Sector_Ct (ID 5) also increasing
   • ATA error log showing UNC errors at specific LBAs
   • SMART Overall Health assessment reporting FAILED
   • Host-side I/O latency spikes in iostat or kernel log entries

5. Rule out interface problems. Check UDMA CRC Error Count (ID 199). If CRC errors are increasing alongside apparent media errors, the root cause may be a cable or backplane problem causing phantom read failures, not media degradation.

6. Assess scope. If multiple drives in the same chassis show Offline_Uncorrectable climbing simultaneously, suspect an environmental or infrastructure cause (power instability, thermal event, vibration) rather than individual drive failure.

Metrics and signals to monitor

Fixes

The drive is actively failing: replace it

When Offline_Uncorrectable is growing and corroborated by any of the signals above (pending sectors sustained, self-test read failure, or health FAILED), the drive is in active failure. There is no software fix for permanent media degradation.

1. Verify all data on the drive is backed up or redundantly stored.
2. If the drive is in a RAID array, schedule a controlled replacement during a maintenance window. Do not wait for an uncontrolled failure during peak load.
3. If the drive is standalone with no redundancy, prioritize data migration immediately. Every additional bad sector is data you cannot recover.

Historical non-zero count: reconfigure alerting, do not replace

A drive with a count of 4 that has been stable for two years is very different from a drive whose count went from 0 to 4 this week. Alert on growth from baseline, not on absolute value.

In smartd.conf, use the + modifier to alert only on increases:

# Alert only when ID 198 increases (not on historical non-zero)
-U 198+

The default smartd.conf behavior (from the -a directive) is -U 198, which alerts on any non-zero value. This is appropriate for newly deployed drives with a clean baseline but will flood operators when first deployed across an existing fleet.

Counter does not reset after reallocation: use increasing mode

Some drives do not reset attribute 198 even after the bad sector is successfully reallocated or overwritten. The drive will carry a non-zero Offline_Uncorrectable count indefinitely. The -v 198,increasing directive tells smartd to treat attribute 198 as monotonically increasing:

# Attribute 198 does not reset after reallocation on this drive
-v 198,increasing

This is the recommended setting for drives exhibiting this behavior. Without it, smartd may re-alert on the same historical count every polling cycle.

Sandforce SSD false positives: suppress or override

Sandforce-based SSDs (including some Kingston, OCZ, and Corsair models) report attribute 198 incorrectly due to non-standard firmware behavior. This is documented in smartmontools ticket #258. Operators running these drives will see persistent Offline_Uncorrectable counts that never correlate with any other failure signal and never lead to actual data loss.

Workarounds include overriding the attribute decoding or suppressing the -U directive entirely for those specific drives. Identify affected drives by controller model and apply the workaround per-drive, not fleet-wide.

Forcing sector reallocation: a bandaid, not a fix

Warning: Writing to a specific LBA destroys whatever data remains there. This is a destructive, irreversible operation.

Writing to a pending or uncorrectable sector can force the drive to attempt reallocation, moving the sector from the pending/uncorrectable list to the reallocated list. This can be done with dd targeting the specific LBA or by running a filesystem scrub (ZFS scrub, mdadm check). This does not repair the drive. It consumes a spare sector and masks the symptom. Use this only to clear I/O stalls on a specific LBA while you arrange replacement.

Prevention

Baseline every drive at deployment. Capture a full smartctl -a snapshot when a drive is first installed. This establishes which counts (if any) were present from factory provisioning and which developed during production use. Without a baseline, you cannot distinguish historical damage from active failure.

Schedule periodic extended self-tests. Drives do not run self-tests automatically. Without scheduled extended tests, latent bad sectors remain hidden until production I/O hits them. Weekly short tests and monthly long tests are standard practice.

Alert on growth from baseline with corroboration. Use multi-condition alert logic to minimize false alarms across large fleets:

• Offline_Uncorrectable count has increased since baseline, AND
• At least one corroborating signal: Current_Pending_Sector (ID 197) non-zero sustained across multiple polls, OR self-test completed with read failure, OR SMART Overall Health is FAILED

Key alert logic on the decoded attribute name, not the raw ID. Some vendors repurpose ID 198. If your monitoring system alerts on “attribute ID 198” without checking the decoded name, it will false-positive on any drive that uses that ID for a different purpose.

Monitoring this with Netdata

• SMART attribute collection means you see ID 198 growth at the monitoring interval, not waiting for the next manual smartctl poll.
• Correlating ID 198 growth with ID 197 (pending), ID 5 (reallocated), and ID 199 (CRC errors) on a single timeline lets you distinguish media failure from cable problems without cross-referencing separate tools.
• Growth-from-baseline alerting prevents the fleet-wide false alarm on initial deployment: Netdata establishes the first-observation baseline and alerts on delta, not absolute value.
• Host-side I/O latency metrics alongside SMART data let you confirm whether bad sectors are causing production-visible I/O stalls.

Related guides

• How S.M.A.R.T. actually works: a mental model for operators
• smartctl disk monitoring checklist: the SMART signals every server needs
• SMART monitoring maturity model: from survival to expert
• Reallocated_Sector_Ct rising: the drive is burning through its spare pool