$ guides / smartctl-disk-monitoring / smartctl-unsafe-shutdowns ▌

Operations Guides

Unsafe Shutdowns / unexpected power-loss count rising

The unsafe shutdown counter is a cumulative lifetime metric tracking how many times a drive lost power without receiving a clean shutdown notification. On ATA/SATA drives it is vendor-specific: smartmontools commonly maps attribute 174 to Unexpect_Power_Loss_Ct and attribute 192 to Power-Off_Retract_Count, Unsafe_Shutdown_Count, or similar names, depending on the drive. On NVMe drives, it is the “Unsafe Shutdowns” field in the SMART/Health Information Log (Log Page 02h).

A non-zero value is not inherently alarming. Drives accumulate unsafe shutdowns over their lifetime from kernel panics, hard resets, UPS failures, and factory burn-in. The operational question is never “is the count non-zero?” but “is it growing, and how fast?”

Rapid growth changes the picture. Each unsafe shutdown on an SSD without power-loss protection (PLP) risks losing data in the volatile write buffer and may leave FTL metadata inconsistent, requiring reconstruction on next power-on. On HDDs, each event triggers an emergency head retraction instead of a graceful unload. If the count is climbing, you have an ongoing power stability problem that will eventually cause data corruption.

What this means

An unsafe shutdown occurs when power is removed from the drive without the host first sending a clean shutdown command. For ATA drives, no Standby Immediate, ATA Sleep, or Idle Immediate command was issued before power loss. For NVMe drives, the controller did not receive a Shutdown Notification (CC.SHN) before power was cut.

Consequences depend on drive type and power-loss protection:

HDDs: Emergency head retraction. The heads are yanked to the landing zone rather than gracefully unloaded. Repeated events increase mechanical wear on the head actuator and landing zone surface.

SSDs/NVMe without PLP: Data in the volatile write buffer is lost. FTL (Flash Translation Layer) metadata may be inconsistent, requiring a full rebuild on the next boot. This rebuild consumes NAND write cycles, accelerating wear.

SSDs/NVMe with PLP: Onboard capacitors flush the write buffer during an unexpected power loss. The counter still increments, but data integrity is preserved. If volatile-memory backup has failed (NVMe Critical Warning bit 4, value 0x10), a PLP-equipped drive has lost that safety net and every unsafe shutdown becomes a potential data corruption event.

Common causes

Cause	What it looks like	First thing to check
UPS failure or absence	Counter increments correlate with known power events; multiple drives affected simultaneously	UPS battery health and event logs
Kernel panics or hard reboots	All drives in the host increment together; no external power event	`journalctl -k` for panic, OOM, or watchdog messages
Aggressive power management	Counter increments during suspend/resume or idle transitions; common on systems using s2idle	Whether NVMe power state transitions occur without CC.SHN
Power supply instability	Multiple drives in the same chassis show spin retries and unsafe shutdowns together; no single-drive media degradation	PSU voltage rails, staggered spin-up configuration
Firmware bugs	Counter increments on every clean boot despite proper shutdown sequence; growth is exactly 1 per boot	Drive firmware version against manufacturer advisories

Quick checks

# NVMe unsafe shutdown count
smartctl -a /dev/nvme0n1 | grep -i "unsafe shutdown"

# ATA unexpected power loss count (SSDs, attribute 174)
smartctl -A /dev/sdX | grep -iE "Unexpect_Power|Power.Off_Retract"

# Power cycle count for context: what fraction of power cycles were unsafe?
smartctl -A /dev/sdX | grep -i "Power_Cycle"
smartctl -a /dev/nvme0n1 | grep -i "Power Cycles"

# NVMe Critical Warning byte (bit 4 = 0x10 = volatile memory backup failure)
smartctl -a /dev/nvme0n1 | grep -i "Critical Warning"

# Recent kernel panics, OOM kills, or watchdog resets
journalctl -k --since "24 hours ago" | grep -iE "panic|oops|oom|watchdog"

# Power-related events in dmesg (ACPI, suspend, resume)
dmesg | grep -iE "power|suspend|resume|shutdown|acpi" | tail -30

# Spin retry count (HDDs): nonzero on multiple drives = power delivery problem
smartctl -A /dev/sdX | grep -i "Spin_Retry"

# Snapshot current unsafe shutdown count for baseline comparison
smartctl -a /dev/nvme0n1 | grep -i "unsafe" > /tmp/unsafe_baseline.txt

How to diagnose it

The diagnostic path depends on whether a single drive or multiple drives are affected, and whether the count is growing.

flowchart TD
    A["Unsafe shutdown count rising"] --> B{"Single drive or multiple?"}
    B -->|Multiple drives| C["Power infrastructure problem"]
    B -->|Single drive| D{"Growth pattern?"}
    D -->|"Exactly 1 per boot"| E["Firmware bug - check vendor advisories"]
    D -->|"Correlates with suspend/resume"| F["Power management false positive"]
    D -->|"Random spikes"| G{"Kernel events at same time?"}
    G -->|Yes| H["Panics, OOM, watchdog - fix host stability"]
    G -->|No| I["Check UPS, PSU, backplane power delivery"]
    C --> J["Check UPS, PSU rails, staggered spin-up"]

Establish the baseline. Record the current unsafe shutdown count for every monitored drive. This is critical when deploying SMART monitoring on an existing fleet for the first time. Cumulative counters will have non-zero historical values from years of operation. Alert on growth from baseline, not absolute value.
Determine the rate of change. Compare the current count against the previous measurement. A rate of zero is expected. A rate of one or more per week warrants investigation. A rate of one or more per day is a power stability emergency.
Check scope: single drive or fleet-wide.
- Single drive only: The drive’s power management behavior, firmware, or physical connection is the likely cause. Check whether the drive sits behind a hot-swap backplane or USB bridge with aggressive power management.
- Multiple drives simultaneously: The power infrastructure is the cause. This includes PSU degradation, PDU overload, UPS battery failure, or voltage regulation issues. Multiple drives showing the same symptom at the same time is the strongest signal that the problem is external to the drives.
Correlate with host events. Check kernel logs for panics, OOM kills, watchdog timer resets, and ACPI power events. Each can cause an unsafe shutdown without a visible power outage. The host may have rebooted cleanly from the operator’s perspective, but the drive experienced a power cut.
Check for power management false positives. On systems using suspend-to-idle (s2idle) or aggressive NVMe power state transitions, the NVMe controller may be power-cycled without receiving CC.SHN, incrementing the counter even though the system was intentionally entering a low-power state. This is a known issue on some platform and NVMe drive combinations. If your platform uses s2idle and the count tracks exactly with suspend/resume cycles, document it as a known false positive for that hardware.
Check for firmware bugs. Some NVMe drives increment Unsafe Shutdowns on nearly every clean boot despite proper shutdown sequences. If the count grows by exactly one per boot and all boots were clean, suspect a firmware bug rather than a power problem. Check the drive firmware version against the manufacturer’s latest release and known issues.
Check NVMe Critical Warning bit 4. If this bit is set (value 0x10 in the Critical Warning byte), the drive reports that its volatile memory backup has failed; on PLP drives this commonly means the capacitor/supercap backup subsystem. Every unsafe shutdown now carries a real data corruption risk that the PLP hardware was designed to prevent. This bit should always be zero on PLP-equipped drives.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Unsafe Shutdowns (NVMe) / Unexpect_Power_Loss_Ct (ATA 174)	Core counter for this investigation	Any increase from baseline
Power-Off_Retract_Count (ATA 192, HDDs)	Emergency head retractions causing mechanical wear	Growth correlated with power events
Power Cycle Count (NVMe / ATA 12)	Total power cycles for context: what fraction were unsafe?	Unsafe-to-total ratio increasing
NVMe Critical Warning bit 4 (0x10)	Volatile memory backup failure makes every unsafe shutdown a data loss risk on PLP drives	Bit transitions from 0 to 1
Spin Retry Count (ATA 10)	Motor could not reach operating speed: power delivery problem	Non-zero on multiple drives simultaneously
Media and Data Integrity Errors (NVMe)	FTL corruption from unsafe shutdowns may surface as media errors	New errors appearing after unsafe shutdown events
Kernel log power events	Host-side evidence of panics, watchdog resets, ACPI transitions	New entries correlated with counter increments

Fixes

UPS failure or absence

No UPS installed: Install one. Size the UPS for the full load plus headroom.
UPS battery degraded: Most UPS batteries last 3-5 years. Check runtime test results and battery health indicators. Replace on schedule, not when they fail.
UPS overload: Verify the connected load is within the UPS rating. Drives added without recalculating load can exceed capacity.
PDU or circuit issue: If multiple chassis on the same circuit show simultaneous unsafe shutdowns, the circuit or PDU is the bottleneck.

Kernel panics and hard reboots

Each kernel panic forces a hard reboot, cutting power without shutdown notification. Address the panic:

Check for OOM killer events that kill critical processes and trigger watchdog resets.
Check for driver panics or hardware faults in journalctl -k.
Check for hardware watchdog timer resets, which indicate the system was hung long enough for the watchdog to fire.
If kernel.panic sysctl is set to auto-reboot on panic, each panic becomes an unsafe shutdown.

Aggressive power management

On systems where the counter increments during suspend/resume or idle periods:

Verify whether the NVMe driver sends CC.SHN before entering low-power states. This behavior is kernel-version-dependent.
On Linux PCI NVMe devices, the kernel supports nvme_core.default_ps_max_latency_us; setting it to 0 turns off APST, as documented in the NVMe core driver source. This can avoid deep autonomous power states, but it increases idle power and may change platform suspend behavior. Verify vendor/firmware behavior before using it as a workaround.
On systems using s2idle, the counter may increment on every suspend/resume cycle. If this is a known false positive for your hardware and firmware combination, document it and suppress alerts for that pattern.

Power supply instability

If multiple drives in the same chassis show unsafe shutdowns or spin retries:

Check PSU voltage rails (12V and 5V) under load. A sagging 12V rail during spin-up is a classic symptom of an undersized or failing PSU.
Verify staggered spin-up is enabled in the BIOS or storage controller firmware. Without it, all drives attempt to spin up simultaneously, drawing peak inrush current that can sag the power rail.
Check whether the chassis has more drives than the PSU was originally sized for.
If the problem appeared after adding drives, the PSU may be at capacity.

Firmware bugs

If the counter increments by exactly 1 on every clean boot:

Check the drive’s firmware version against the manufacturer’s latest release and advisory pages.
Some NVMe drives with specific firmware versions are known to increment Unsafe Shutdowns despite clean shutdowns. A firmware update may resolve this.
Document the false-positive pattern in your monitoring configuration so alerts are suppressed for drives exhibiting this known behavior.

SSDs without power-loss protection

Consumer SSDs and entry-level enterprise SSDs without PLP capacitors are the most vulnerable to unsafe shutdown damage. Each event risks losing data in the write buffer and may force an FTL rebuild on next power-on, consuming NAND write cycles. The cumulative effect is accelerated wear and potential latent corruption that may not surface until much later.

If unsafe shutdowns cannot be eliminated due to an unstable power environment, prioritize replacing unprotected drives with PLP-equipped enterprise SSDs. This does not stop the counter from incrementing, but it eliminates the data corruption and accelerated wear consequences.

Prevention

Deploy UPS everywhere it matters. Clean power eliminates most unsafe shutdown causes.
Replace UPS batteries proactively on a 3-5 year schedule. Do not wait for failure.
Enable staggered spin-up in dense disk shelves to prevent inrush current sag.
Baseline drives at deployment. Record the unsafe shutdown count when a drive enters service. Alert on growth from that baseline, not absolute value.
Track firmware versions fleet-wide. Known-buggy firmware that miscounts unsafe shutdowns should be documented and updated when patches are available.
Monitor NVMe Critical Warning bit 4. A failed volatile-memory backup turns each unsafe shutdown from a non-event into a data loss event. This bit should always be zero on PLP-equipped drives.
Do not alert on absolute lifetime values at first deployment. When deploying SMART monitoring on an existing fleet, every drive will have a non-zero historical count. Establish a baseline and alert on deltas only.

How Netdata helps

Netdata’s smartctl collector captures unsafe shutdown counts alongside the full SMART attribute set, enabling several correlation patterns:

Rate-of-change detection: Netdata’s ML anomaly detection flags when the unsafe shutdown counter starts growing, distinguishing a stable historical baseline from a new active power problem. This eliminates false alerts from lifetime accumulated counts.
Cross-drive correlation: When multiple drives in the same host show simultaneous unsafe shutdown increments, single-node dashboards make the pattern immediately visible, pointing to infrastructure rather than individual drive failure.
Critical Warning bit monitoring: On NVMe drives, the Critical Warning byte is collected and can be alerted on per-bit. A volatile-memory backup failure (bit 4) dramatically raises the stakes of every unsafe shutdown.
Correlation with kernel events: Netdata’s system journal and kernel log collection can be correlated with unsafe shutdown timestamps to identify whether panics, watchdog resets, or ACPI events preceded the power loss.
Spin retry correlation: If spin retries (ATA attribute 10) appear alongside unsafe shutdowns on multiple drives, Netdata shows both signals in the same time window, confirming a power delivery problem.