Current_Pending_Sector non-zero: unreadable sectors and I/O latency spikes

A non-zero Current_Pending_Sector (SMART attribute ID 197) means the drive has sectors it cannot reliably read. These sectors failed a read and are waiting for either a write to trigger reallocation or an offline scan to confirm the defect. The drive has not yet remapped them to its spare pool.

The operational impact is often misdiagnosed. Every time the filesystem touches one of these sectors, the drive firmware enters a multi-second retry loop. This shows up in iostat as extremely high await with low throughput and idle CPU. Teams chase this as a software freeze, a kernel bug, or a filesystem problem for hours before checking SMART data.

A single pending sector can self-resolve: if the sector is overwritten successfully, the firmware clears it from the pending list or remaps it. Alerting on a single non-zero reading produces false alarms. The count must be sustained across consecutive polls and corroborated by growth in Offline Uncorrectable (ID 198) before it confirms active data loss. Conversely, the count decreasing does not mean the drive is healthy. It means the failure was masked by reallocation.

What this means

The firmware flags a sector as pending when it cannot return data on the first read attempt, or when it needs excessive ECC correction. Instead of immediately remapping (which consumes the finite spare pool), the drive parks the LBA on a pending list and waits for a write to confirm whether reallocation is needed.

Read-only workloads, rarely accessed files, and filesystem caches can all prevent that triggering write from ever happening. The sector sits on the pending list indefinitely. When something does read it, the drive retries repeatedly, consuming seconds of wall-clock time on a single I/O operation.

flowchart TD
    A["Read fails on LBA"] --> B["Sector flagged as pending
ID 197 increments"]
    B --> C{"Filesystem touches that LBA"}
    C -->|Read attempt| D["Drive enters retry loop
iostat await spikes to seconds"]
    C -->|Write to that LBA| E{"Firmware attempts reallocation"}
    E -->|Write succeeds| F["Sector remapped to spare
ID 5 increments, ID 197 decrements"]
    E -->|Write fails| G["Sector becomes uncorrectable
ID 198 increments"]
    D --> H{"Pattern sustains across polls?"}
    H -->|"Yes + ID 198 rising"| I["PAGE: confirmed active data loss"]
    H -->|"Transient, self-resolves"| J["TICKET: monitor for recurrence"]

This is the “zombie drive” pattern: the system looks frozen (high iowait, idle CPU), but the root cause is a single bad sector on a drive that is otherwise functional. The drive can operate for weeks or months in this state, with intermittent latency spikes whenever a pending sector is touched.

Common causes

Smartmontools does not publish a complete list of affected MX500 firmware revisions. Its drive database records the behavior as “MX500: May flip 0 <> 1 (ticket #1227)”; smartmontools 7.2 already applied the same attribute rename to CT250/500/1000/2000 MX500 models, and 7.5 also matches CT4000 MX500.

Quick checks

These commands are read-only and safe to run on a production system.

# Check pending sector count and related media attributes
smartctl -A /dev/sdX | grep -iE "Current_Pending|Reallocated|Offline_Uncorrectable"

# Full SMART overview including health assessment
smartctl -a /dev/sdX

# Check the drive's error log for UNC (uncorrectable) errors at specific LBAs
smartctl -l error /dev/sdX

# Check self-test results for read failures with LBA details
smartctl -l selftest /dev/sdX

# Watch per-device I/O latency in real time
iostat -x 1

# Look for kernel-level I/O errors and SCSI error recovery
dmesg | grep -iE "I/O error|medium error|reset|timeout|offline" | tail -20

# Verify the drive identity has not changed unexpectedly
smartctl -i /dev/sdX | grep -iE "Model|Serial|Firmware"

How to diagnose it

1. Confirm the pending count and check for corroboration. Run smartctl -A /dev/sdX and record the raw values for IDs 5, 197, and 198. A non-zero ID 197 alone is a TICKET. If ID 198 is also increasing, the drive is converting pending sectors to permanent data loss. If ID 5 is also increasing, the spare pool is being consumed.

2. Check whether the count is sustained. Poll the attribute again after 15-30 minutes (or across your normal monitoring interval). If ID 197 drops to zero without a corresponding increase in ID 5, the sector self-resolved on a successful re-read. This is not the same as healthy: the media is marginal and will likely recur.

3. Correlate latency spikes with the drive. Run iostat -x 1 during reported slowness. Look for high await on the suspect device while CPU utilization stays low. If await spikes to hundreds of milliseconds or seconds on one device while others are normal, the drive is retrying a bad sector.

4. Find the affected LBA. The error log (smartctl -l error /dev/sdX) shows UNC errors with the LBA where the read failed. The self-test log (smartctl -l selftest /dev/sdX) also reports the LBA of any read failure. Cross-reference the LBA with the filesystem to determine which file is affected.

5. Rule out cable problems. Check ID 199 (UDMA_CRC_Error_Count). If CRC errors are increasing alongside pending sectors, the pending sectors may be phantom: the transport layer corrupted the read, so the drive flagged the sector as suspect even though the on-disk data is intact. Fix the cable first, then re-evaluate.

6. Run an extended self-test if the situation allows. smartctl -t long /dev/sdX scans the full surface and will find latent bad sectors before production I/O hits them. This takes hours on large HDDs and competes with production I/O, so schedule accordingly. If the test reports a read failure at a specific LBA, the defect is confirmed.

Metrics and signals to monitor

Fixes

Force reallocation by overwriting the affected LBA

Writing to a pending sector triggers the firmware’s reallocation path. If the write succeeds, the sector is remapped to a spare and drops off the pending list. ID 197 decrements and ID 5 increments.

# Identify the LBA from the error log or self-test log
smartctl -l error /dev/sdX | grep -i "LBA"

# Overwrite the specific sector to force reallocation.
# WARNING: This destroys data at the target LBA. Ensure data is recoverable
# from redundancy (RAID, replication, backup) before running.
dd if=/dev/zero of=/dev/sdX bs=512 count=1 seek=<LBA> oflag=direct

If dd fails because the drive rejects the write, hdparm can force reallocation at the firmware level:

# WARNING: Destructive. Forces sector reallocation at the firmware level.
# The --yes-i-know-what-i-am-doing flag is required; hdparm refuses without it.
hdparm --write-sector <LBA> --yes-i-know-what-i-am-doing /dev/sdX

This is a bandaid, not a fix. The underlying media is degrading. The sector was remapped because it failed, and the spare pool is now smaller. Plan replacement.

Replace the cable or reseat the connection

If ID 199 (UDMA CRC errors) is increasing alongside ID 197, the pending sectors may be phantom. The transport layer corrupted reads, causing the drive to flag healthy sectors as suspect. Reseat the cable at both ends, or swap with a known-good cable. After the fix, monitor whether ID 197 clears (via scrub or rewrite) without further growth.

Replace the drive

Any growth in pending sectors combined with increasing reallocated or uncorrectable counts warrants replacement. The drive has started to fail. The question is how fast, not whether. Schedule a controlled replacement during a maintenance window rather than waiting for an uncontrolled failure during peak load.

If the drive is in a RAID array, initiate a controlled rebuild now. Do not wait for the drive to fail completely.

Suppress false positives on affected SSD models

Some Crucial/Micron consumer SSDs use attribute 197 as an ECC event counter that toggles between 0 and 1 during normal FTL operation. This is not a pending sector. Smartmontools 7.2 and later handle this through the drive database (drivedb.h), renaming the attribute to avoid false alerts. On older smartmontools, add a vendor-specific attribute override:

# Override attribute 197 interpretation for affected Crucial/Micron SSDs
# Add to smartd.conf or use with smartctl
smartctl -v 197,raw48,Current_Pending_ECC_Cnt -A /dev/sdX

The -v 197,raw48,Current_Pending_ECC_Cnt syntax is valid in smartmontools 7.5. In smartd.conf, renaming attribute 197 changes the default pending-sector check from -C 197 to -C 0; an explicit -C 197 re-enables it. By contrast, -I 197 only ignores attribute-value changes for the -p, -u, or -t tracking directives and does not turn off -C 197.

Prevention

• Schedule periodic extended self-tests. Drives do not run self-tests automatically. Without a monthly extended test, latent bad sectors hide until production I/O hits them.
• Run filesystem scrubs regularly. ZFS scrubs and mdadm checks force reads across the entire device, triggering retries on marginal sectors and forcing reallocation during controlled maintenance rather than peak load.
• Baseline every drive at deployment. Capture a full smartctl -a snapshot when the drive is first installed. Some drives ship with a small non-zero reallocated count from factory provisioning. Without a baseline, you cannot distinguish factory defects from new degradation.
• Alert on rate of change, not absolute value. A stable count of 5 pending sectors that clears on rewrite is less urgent than a count that went from 0 to 5 this week and keeps climbing.
• Require multi-signal corroboration before paging. Page only when ID 197 is sustained non-zero across two or more consecutive polls and ID 198 is also increasing. This prevents false alarms from transient single-sector events while catching confirmed active data loss.

How Netdata helps

Netdata’s disk monitoring collector captures SMART attributes at per-second resolution, which is critical for correlating intermittent latency spikes from pending sector retries with the underlying SMART data.

• Correlate iostat-level latency with SMART attributes on the same timeline. When await spikes on a device, check immediately whether ID 197 is non-zero on that same drive, without switching between tools or time ranges.
• Track rate of change for IDs 5, 197, and 198 together. The relationship between these three attributes tells you whether the drive is self-healing (pending clearing via reallocation), stagnating (pending but not yet uncorrectable), or in terminal failure (all three climbing).
• Anomaly detection flags unusual latency patterns. The characteristic spike pattern of a sector retry loop (single-device await jumping to seconds while CPU stays idle) is distinguishable from normal load-driven latency, even before SMART attributes update.
• Per-device charts let you compare a suspect drive against its neighbors. Compare drives in the same chassis to determine whether the problem is drive-specific or environmental.

Related guides

• How S.M.A.R.T. actually works: a mental model for operators
• smartctl disk monitoring checklist: the SMART signals every server needs
• SMART monitoring maturity model: from survival to expert
• Reallocated_Sector_Ct rising: the drive is burning through its spare pool