The zombie drive: bad sectors, read retries, and high iowait with idle CPU

The system is grinding to a halt. SSH sessions lag, commands take seconds to return, and applications time out. You check top or htop and CPU is nearly idle, yet load average is climbing. iostat -x 1 tells the real story: await on one disk is spiking into the hundreds of milliseconds, sometimes multiple seconds, while every other disk looks fine. The process stuck on that I/O is in D state (uninterruptible sleep), unkillable, waiting for a read that the drive cannot complete.

This is the zombie drive. It still responds on the bus, and SMART still reports PASSED. But bad sectors have accumulated on the media. Every time the filesystem reads one of those sectors, the drive firmware enters an internal retry loop that can last seconds or longer. The process blocks, the kernel waits, and the system appears frozen. Meanwhile, Current_Pending_Sector (ID 197) and Reallocated_Sector_Ct (ID 5) are both climbing as the spare pool is consumed.

The distinguishing signal is the combination: high I/O latency, low CPU utilization, and non-zero pending or reallocated sectors. If CPU is high, it is an application or compute problem. If latency is high but CPU is idle, the bottleneck is the drive itself retrying failing reads.

Your first response should be data migration, not deep diagnosis. The drive is dying. Confirm the pattern quickly, start moving data, then replace the drive.

What this means

The failure mechanism is a feedback loop between the drive firmware and the kernel’s I/O path:

1. The filesystem issues a read to an LBA that sits on a degraded region of the media.
2. The drive attempts the read. Error correction code (ECC) fails to recover the data on the first pass.
3. The drive firmware retries the read internally. On drives without Error Recovery Control (SCT ERC), the firmware may retry for tens of seconds before giving up.
4. While the drive retries, the kernel blocks the issuing process in D state (uninterruptible sleep). SIGKILL has no effect on a process in this state.
5. If the read eventually succeeds, the sector is flagged as pending (Current_Pending_Sector, ID 197). The await time for that I/O reflects the full retry duration.
6. If the read never succeeds, the drive logs a UNC (Uncorrectable) error at that LBA in the ATA error log. The sector may become Offline_Uncorrectable (ID 198).
7. On the next successful write to that LBA, the drive reallocates the sector from its spare pool (Reallocated_Sector_Ct, ID 5 increments; Current_Pending_Sector, ID 197 decrements).

The 30-second window matters. The default Linux SCSI command timeout for SATA and SAS disk I/O is 30 seconds, configurable via /sys/block/sdX/device/timeout. If the drive does not complete the read within that window, the kernel fails the I/O and initiates error recovery. During those 30 seconds, the process is stuck, the CPU shows iowait, and the system appears frozen. On consumer SATA drives that lack SCT ERC, the firmware retry can exceed even the kernel timeout, leading to extended hangs. NVMe drives use a different timeout mechanism; this pattern is primarily a SATA/SAS problem.

flowchart TD
    A["Filesystem reads LBA
on degraded media"] --> B["ECC fails to
recover data"]
    B --> C["Drive retries internally
seconds to tens of seconds"]
    C --> D["Process in D state
uninterruptible sleep"]
    D --> E["CPU idle
high iowait"]
    C --> F{"Read succeeds
after retries?"}
    F -->|Yes| G["Sector flagged pending
ID 197 increments"]
    F -->|No| H["UNC error logged
at specific LBA"]
    G --> I["Next write triggers
reallocation, ID 5 up"]
    H --> J["May become
Offline Uncorrectable ID 198"]

Common causes

Quick checks

Run these read-only commands to confirm the zombie drive pattern. None of them modify data.

# Check SMART attributes 5, 197, 198 - all should be zero on a healthy drive
smartctl -A /dev/sdX | grep -iE "Reallocated_Sector|Current_Pending|Offline_Uncorrect"

# Check the ATA error log for UNC errors at specific LBAs
smartctl -l error /dev/sdX

# Extended error log provides more entries (summary log holds only 5)
smartctl -l xerror /dev/sdX

# Confirm high await on the suspect drive while others are normal
iostat -x 1

# Check kernel logs for I/O errors, medium errors, and timeout messages
dmesg | grep -iE "I/O error|medium error|timeout|reset" | tail -20

# Verify the drive's self-assessment (will likely still say PASSED)
smartctl -H /dev/sdX

# Check whether SCT Error Recovery Control is enabled or supported
smartctl -l scterc /dev/sdX

# Check the kernel SCSI command timeout for this device
cat /sys/block/sdX/device/timeout

# Identify processes stuck in D state (uninterruptible sleep)
ps -eo stat,pid,comm,wchan | awk '$1 ~ /D/'

How to diagnose it

1. Confirm the SMART signal triad. Run smartctl -A /dev/sdX and check three attributes together:

   • Reallocated_Sector_Ct (ID 5): non-zero and increasing. The drive is consuming spares.
   • Current_Pending_Sector (ID 197): non-zero. The drive has sectors it cannot reliably read right now.
   • Offline_Uncorrectable (ID 198): non-zero or increasing. Confirmed data loss at the media level.

   All three rising together is the strongest indicator of active progressive media failure. But even one non-zero value, especially ID 197, combined with the latency pattern confirms the diagnosis.

2. Correlate latency with the suspect drive. Run iostat -x 1 and watch await on the specific device. Normal await for a healthy HDD is under 10 to 20 ms. A zombie drive shows intermittent spikes to hundreds of milliseconds or multiple seconds, often when a specific file or block range is accessed. The spikes are not continuous; they happen only when the OS reads a bad LBA.

3. Read the ATA error log for UNC entries. Run smartctl -l error /dev/sdX (or smartctl -l xerror for more entries). Look for “Error: UNC” entries. Each entry includes the LBA where the read failed. The error register value 0x40 indicates an uncorrectable error. The LBA is decoded from the sector number, cylinder low, and cylinder high registers in the log entry.

   The ATA summary error log holds only 5 entries. If the displayed error count is high but only 5 entries appear, the log has wrapped. Use smartctl -l xerror for the extended comprehensive error log, which stores more entries.

4. Check kernel logs for corroborating evidence. Run dmesg filtered for I/O errors and look for messages referencing the suspect device. Key phrases include “I/O error, dev sdX”, “medium error”, “not ready”, and “Device not responding”. These messages may correspond to the same LBAs where the ATA error log shows UNC errors.

5. Verify D-state processes are blocked on the suspect device. Run ps -eo stat,pid,comm,wchan | awk '$1 ~ /D/' to find processes in uninterruptible sleep. These processes cannot be killed with SIGKILL. They unblock only when the I/O completes: either the drive eventually returns data, or the kernel timeout fires and fails the I/O.

6. Check SCT ERC status. Run smartctl -l scterc /dev/sdX. If the response is “SCT Error Recovery Control not supported,” the drive has no firmware-level read timeout. This explains the extended hangs: the drive firmware retries indefinitely until it succeeds or the kernel times out. Enterprise and NAS drives typically support SCT ERC with a 7-second default for read and write timeouts. Consumer desktop drives frequently do not support it at all.

Metrics and signals to monitor

Fixes

Migrate data immediately

This is the first action, before any further diagnosis. The drive is failing.

• Redundant array (RAID 1/5/6/10, ZFS mirror/RAID-Z): Verify array health and initiate a controlled drive replacement. Let the array rebuild onto a spare. Do this during a maintenance window, not when the drive fails during peak hours.
• Standalone drive with no redundancy: Copy critical data to another device now. Expect the copy to be slow: every read of a bad LBA will stall for seconds. Prioritize the most critical data first. You do not know how many more bad sectors will appear before the drive becomes completely unresponsive.
• Do not run fsck or filesystem repair on the failing drive as a first step. Filesystem checks read every block, which forces reads across all the bad sectors and can make the system appear more frozen than it already is. Migrate data first.

Replace the drive

There is no software fix for bad sectors on the physical media. The drive has confirmed media degradation and the spare pool is being consumed. Schedule replacement. If the drive is under warranty, the SMART data (non-zero reallocated sector count, pending sectors, and error log entries) typically supports an RMA claim.

Force sector reallocation as a temporary measure

If you must keep the drive running and a specific LBA is causing repeated hangs, you can force the drive to reallocate that sector by overwriting it. The smartmontools BadBlockHowto documents this procedure. This clears the sector from the pending list (ID 197 decrements) and triggers reallocation (ID 5 increments). The hang at that specific LBA stops.

# WARNING: This overwrites data at the specified LBA with zeros. Data at
# this LBA is already unreadable (confirmed by UNC in error log).
# If a filesystem is mounted on this device, writing directly to the block
# device bypasses the page cache and may cause coherency problems. Prefer
# unmounting first, or write through the affected file (see BadBlockHowto
# for the ext4 debugfs procedure).
# The seek offset must be calculated from the LBA: seek = LBA / (bs / 512)
# Reference: smartmontools BadBlockHowto
dd if=/dev/zero of=/dev/sdX bs=4096 count=1 seek=<LBA_OFFSET>

This is a bandaid, not a repair. The underlying media degradation continues. New bad sectors will appear. Use this only to keep the system running while you arrange replacement.

Enable SCT ERC on future drives

On drives that support it, SCT Error Recovery Control caps the firmware’s internal retry time, preventing the extended system hangs that define the zombie drive pattern:

# Set read and write ERC timeout to 70 deciseconds (7 seconds)
# This is a common enterprise default
smartctl -l scterc,70,70 /dev/sdX

With ERC enabled, the drive abandons the retry after the timeout, returns an error to the kernel, and the kernel fails the I/O cleanly. The system does not appear frozen. Consumer SATA drives frequently do not support SCT ERC at all, which is why the zombie drive pattern is more common in environments using desktop-grade disks.

Prevention

• Schedule periodic extended self-tests. Drives do not run self-tests automatically. A monthly extended self-test (smartctl -t long /dev/sdX) scans the full surface and discovers latent bad sectors before production I/O hits them. Configure smartd to schedule tests if manual scheduling is unreliable.
• Monitor individual SMART attributes, not just PASSED/FAILED. The drive’s overall health assessment uses conservative vendor thresholds. A drive can report PASSED while accumulating hundreds of pending sectors and dozens of reallocated sectors. Alert on any non-zero value of IDs 5, 197, and 198, and on any growth from baseline.
• Track rate of change, not just absolute values. Five reallocated sectors gained over five years is stable. Five gained this week is active failure. Trend the values over time and alert on acceleration.
• Baseline drives at deployment. Capture a full SMART snapshot when a drive enters service. Some drives ship with a small number of reallocated sectors from factory. Without a baseline, you cannot distinguish factory defects from production degradation.
• Enable SCT ERC where supported. This limits the blast radius of a single bad sector from a multi-second system hang to a fast, clean I/O error that the kernel and application can handle gracefully.
• Run filesystem scrubs. ZFS scrubs and mdadm checks force reads across the entire surface, surfacing bad sectors during a controlled maintenance window rather than during peak production load when the resulting hangs cause user-visible outages.

How Netdata helps

• Per-second disk await makes retry-driven latency spikes visible the moment they happen, not minutes later. The correlation between a spike on a specific device and a SMART attribute change is immediately visible in the same dashboard.
• CPU iowait collected alongside disk metrics surfaces the zombie drive signature (high iowait, low CPU, high disk await) in one view, without cross-referencing tools.
• SMART attributes (IDs 5, 197, 198, and error log entries) are collected with rate-of-change detection, showing trends over days rather than a point-in-time snapshot.
• Anomaly detection on disk latency catches unusual patterns before they cross a static threshold, flagging early-stage degradation before the first user-visible hang.

Related guides

• Current_Pending_Sector non-zero: unreadable sectors and I/O latency spikes
• How S.M.A.R.T. actually works: a mental model for operators
• smartctl disk monitoring checklist: the SMART signals every server needs
• SMART monitoring maturity model: from survival to expert
• Offline_Uncorrectable climbing: permanent data loss at the media level
• Raw_Read_Error_Rate looks enormous: the Seagate false alarm explained
• Reallocated_Event_Count vs Reallocated_Sector_Ct: reading both together
• Reallocated_Sector_Ct rising: the drive is burning through its spare pool