$ guides / smartctl-disk-monitoring / smartctl-power-supply-multi-drive ▌

Operations Guides

Multiple drives failing at once: it's the power supply, not the drives

Three, five, or twelve drives light up at once. Spin_Retry_Count has jumped on every HDD in a shelf. Unsafe Shutdown counts are climbing across every NVMe device. A few drives may have dropped off the bus and come back. The instinct is to start filing RMAs.

Stop. When the same symptom appears on many drives at the same time, the probability that every drive independently decided to fail in the same hour approaches zero. The root cause is almost certainly external: a degrading PSU, an overloaded PDU, a failing UPS battery, or aggregate spin-up inrush sagging the 12V rail because staggered spin-up is disabled.

The diagnostic fork is simple. If all drives in a chassis show spin retries, it is a power supply problem. If one drive shows spin retries, that drive’s motor is failing. Getting this wrong means replacing drives that will develop the same symptoms on the next power cycle.

What this means

The distinguishing feature is simultaneity. Multiple drives show the same SMART symptoms at the same time, with no single-drive media degradation pattern:

Spin_Retry_Count (SMART ID 10) is non-zero on multiple HDDs simultaneously. This attribute tracks spindle motor retries: the drive failed to reach operational RPM on the first attempt. When several drives show it at once, the common factor is the power source, not individual motor health.
Unsafe Shutdown Count or Power-Off_Retract_Count is increasing across multiple drives. Every drive experienced an unexpected power loss at the same moment.
Reallocated Sector Count (ID 5), Current Pending Sector (ID 197), and Offline Uncorrectable (ID 198) remain stable or absent. There is no media degradation pattern. The platters are fine; power delivery is not.

The mechanism is straightforward. HDDs draw significant inrush current during spin-up. If the 12V rail sags under aggregate load (because staggered spin-up is disabled or the PSU is degrading), drives cannot reach rated RPM on the first attempt and log spin retries. If the PSU drops out entirely or the UPS fails to hold, every drive logs an unsafe shutdown simultaneously.

flowchart TD
    A["Multiple drives show
SMART warnings"] --> B{"Same attribute across
multiple drives?"}
    B -->|Yes| C{"Spin_Retry_Count or
Unsafe_Shutdown rising?"}
    B -->|No, different attributes| D["Individual drive failures.
Treat each separately."]
    C -->|Yes| E{"Media degradation?
IDs 5, 197, 198"}
    E -->|No or minimal| F["Power infrastructure problem"]
    E -->|Yes, widespread| G["Power event likely caused
secondary media damage"]
    C -->|No| D
    F --> H["Check PSU, UPS, PDU.
Do not replace drives first."]

Common causes

Cause	What it looks like	First thing to check
PSU degradation	Spin_Retry_Count spikes on cold start or after power cycle. 12V rail droops under load.	Measure 12V rail under load. ATX spec tolerance is 11.4V to 12.6V.
Overloaded power rail	Symptoms appear on drives sharing the same power lead or rail. Other drives on different rails are fine.	Check how many drives share each PSU lead. 15 drives on one lead is too many.
UPS battery failure	Unsafe Shutdown counts increment simultaneously across all drives. UPS logs show transfer events or battery test failures.	Review UPS self-test logs and battery age.
Staggered spin-up disabled	All drives show spin retries only after a full power cycle, not during normal operation. Symptoms do not recur until next cold start.	Check BIOS, HBA, or backplane settings for staggered spin-up (SSU) or Power-Up In Standby (PUIS).
PDU overload	Symptoms correlate with load spikes. Other equipment on the same PDU circuit shows issues.	Measure actual power draw per PDU circuit against rated capacity.

Quick checks

All read-only and safe to run during production.

# Check Spin_Retry_Count across all SATA/SAS drives
for dev in /dev/sd?; do
  echo "=== $dev ==="
  smartctl -A "$dev" | grep -i "Spin_Retry"
done

# Check unsafe shutdown / power-off retract across all drives
for dev in /dev/sd?; do
  echo "=== $dev ==="
  smartctl -A "$dev" | grep -iE "Unexpect_Power|Power.Off_Retract"
done

# Check NVMe unsafe shutdowns
for dev in /dev/nvme?n1; do
  echo "=== $dev ==="
  smartctl -A "$dev" | grep "Unsafe Shutdowns"
done

# Verify media degradation is absent or stable (IDs 5, 197, 198)
for dev in /dev/sd?; do
  echo "=== $dev ==="
  smartctl -A "$dev" | grep -iE "Reallocated|Current_Pending|Offline_Uncorrectable"
done

# Check kernel logs for power or bus events
dmesg | grep -iE "power|reset|link down|link up|I/O error" | tail -30

# Check power supply voltages if IPMI or sensors are available
ipmitool sdr | grep -iE "volt|power|psu"
# or
sensors | grep -iE "12V|5V|3.3V|vin"

# Check power cycle count context
for dev in /dev/sd?; do
  echo "=== $dev ==="
  smartctl -A "$dev" | grep -iE "Power_Cycle|Power_On"
done

How to diagnose it

Confirm simultaneity. Pull SMART data from every drive in the chassis. The pattern is multiple drives with non-zero Spin_Retry_Count or recently incremented Unsafe Shutdown counts. If only one drive shows these symptoms, that drive has a motor failure, not a power problem. See the separate guide on Spin_Retry_Count behavior.
Rule out media degradation. Check IDs 5 (Reallocated), 197 (Current Pending), and 198 (Offline Uncorrectable) on all affected drives. If these are zero or stable, the drives’ surfaces are healthy and the problem is upstream. If reallocated sectors jumped on multiple drives simultaneously, a power event may have caused secondary media damage, but the root cause is still the power event.
Check PSU voltage rails under load. Use ipmitool sdr, sensors, or the BMC web interface. ATX spec tolerance for +12V is 11.4V to 12.6V, and for +5V is 4.75V to 5.25V. Readings outside these ranges indicate PSU failure. Voltage that looks fine at idle but sags under load (during a RAID rebuild or spin-up sequence) points to a PSU that can no longer handle peak current.
Review UPS logs and battery health. Check for recent transfer events, battery test failures, or runtime capacity degradation. If the UPS fails to hold during brief power blips it previously rode through, the battery is degraded. Every transfer event that results in a power drop increments Unsafe Shutdown on every drive.
Check PDU load. If multiple servers or shelves share a PDU circuit, measure actual draw against the circuit rating. Overloaded circuits cause voltage drop or breaker trips that affect everything downstream.
Verify staggered spin-up configuration. If symptoms appear only after a full power cycle (not during normal runtime or soft reboots), aggregate spin-up inrush is the likely cause. Check BIOS settings, HBA firmware settings, and backplane jumpers for staggered spin-up (SSU) or Power-Up In Standby (PUIS). Staggered spin-up was introduced in SATA revision 2.5. Whether it is active depends on the SATA power connector wiring: pin 11 of the SATA power connector must be left floating for SSU to work. On some consumer PSUs and older connectors, pin 11 is tied to ground, which disables SSU and forces all drives to spin up simultaneously.
Correlate with timestamps. Do the SMART events cluster around the same power-on hour? The ATA error log records timestamps relative to the power cycle. If all drives show their spin retries or unsafe shutdowns at the same relative timestamp, they all experienced the same power event.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Spin_Retry_Count (ID 10) per drive	Non-zero means the spindle motor failed to reach RPM on first attempt. On multiple drives simultaneously, this is a power delivery problem.	Any non-zero value appearing on multiple drives at the same time.
Unsafe Shutdowns (NVMe) / Power-Off_Retract_Count (ID 192, ATA)	Counts unexpected power losses. Multiple drives incrementing simultaneously means a shared power event.	Count increasing across multiple drives in the same polling interval.
Power Cycle Count (ID 12)	Total power cycles. Unexpected increments on multiple drives mean unplanned power events.	Increment not correlated with a known maintenance window.
Reallocated Sector Count (ID 5)	If stable, confirms media is healthy and the problem is upstream. If jumping on multiple drives, a power event caused secondary damage.	Sudden increase across multiple drives after a power event.
PSU voltage rails (via IPMI)	Direct measurement of power delivery health. Sagging or out-of-spec rails confirm PSU degradation.	12V rail below 11.4V or above 12.6V under load.
UPS battery status and transfer events	A failing UPS battery cannot hold through power blips, causing unsafe shutdowns on all connected equipment.	Battery capacity below 80% of rated, or recent transfer events logged.

Fixes

PSU degradation or failure

If voltage rails are out of spec under load, replace the PSU. This is not a component you repair in place. Check whether the replacement is rated for the peak inrush current of all drives spinning up simultaneously (if staggered spin-up is not configured). A PSU that was adequate when new may sag as capacitors age.

Do not reuse modular cables from the old PSU. Modular PSU cables from different models can deliver wrong voltages to SATA power connectors because pinouts vary between manufacturers. Storage devices have narrow voltage tolerances and fail first when receiving wrong voltages.

Overloaded power rail

If too many drives share a single PSU power lead, the resistance of the lead and connectors causes voltage drop at the drive end. Redistribute drives across multiple PSU leads so no single lead carries excessive current. Splitting drives across multiple 12V rails (if the PSU has them) also helps.

If the PSU does not have enough wattage headroom for the number of drives installed, you need a higher-wattage PSU. Adding more drives to an already-loaded PSU will make the problem worse.

UPS battery failure

If the UPS fails to hold through power events it previously survived, replace the battery (or the entire UPS if the unit is beyond its service life). Most UPS batteries need replacement every 3 to 5 years depending on cycling frequency and ambient temperature.

After replacing the battery, perform a runtime test to verify the UPS can hold the full load long enough for a clean shutdown of all connected systems.

Staggered spin-up not configured

If symptoms appear only after cold starts, enable staggered spin-up in BIOS, HBA firmware, or backplane configuration. This serializes spin-up so drives start one at a time (or in small groups), reducing peak inrush current on the 12V rail.

If the hardware does not support staggered spin-up, consider Power-Up In Standby (PUIS), which can be enabled via hdparm on supported drives. With PUIS, drives do not spin up until the host issues a SET FEATURES command, allowing software to control the spin-up sequence.

If neither option is available, you need a PSU with higher peak current capacity, or fewer drives per machine.

PDU overload

Measure actual power draw per circuit. If a circuit is near its rated capacity, redistribute load across multiple circuits or upgrade the PDU. Overloaded circuits cause voltage drop and breaker trips that affect all equipment on the circuit, not just the storage.

Prevention

Monitor PSU voltages via IPMI. Set alerts for any rail going outside ATX spec tolerances (11.4V to 12.6V for +12V, 4.75V to 5.25V for +5V). Catching voltage drift early prevents the cascading drive damage that follows a PSU failure.
Track Spin_Retry_Count and Unsafe Shutdown Count per drive. Alert when these increment on multiple drives in the same polling window. This is the single most important alert for detecting power infrastructure problems before they cause data loss.
Verify staggered spin-up at deployment time. This is a one-time check that prevents the most common multi-drive failure mode after power cycles. Document the configuration so it is not accidentally changed.
Size PSU headroom for peak inrush. The PSU’s rated wattage must cover the sum of all drives’ spin-up current, not just steady-state power. Consult drive datasheets for spin-up current requirements.
Schedule regular UPS battery tests. Most UPS units support automated self-tests. Review results monthly and replace batteries proactively.
Do not mix modular PSU cables between units. Mismatched pinouts are a common cause of repeated drive failures on “bad” drives that are actually receiving wrong voltages from incompatible cables.

How Netdata helps

Cross-drive correlation of Spin_Retry_Count. Netdata collects SMART attributes per drive on a per-chassis view. When Spin_Retry_Count increments on multiple drives in the same window, the anomaly is immediately visible without manual for loops over device nodes.
Rate-of-change detection on Unsafe Shutdown counts. Rather than alerting on a static lifetime counter, Netdata’s ML anomaly detection flags the moment multiple drives increment simultaneously, which is the signature of a power event.
Absence-of-signal as a diagnostic. When power-related attributes are moving but media degradation attributes (IDs 5, 197, 198) are flat, Netdata’s correlated dashboards make the contrast visible. This is the pattern that says “infrastructure, not drives.”
IPMI sensor integration. If the host exposes voltage rails through IPMI, Netdata collects them alongside SMART data. A sagging 12V rail visible in the same time window as a Spin_Retry_Count spike across all drives closes the diagnostic loop without switching tools.
Kernel log correlation. Netdata surfaces I/O errors, link resets, and device removal events from kernel logs, providing the timeline of what the host saw during the power event alongside what the drives reported in SMART.