$ guides / nvme / nvme-unsafe-shutdowns-increasing ▌

Operations Guides

NVMe unsafe shutdowns increasing: power-loss events and silent corruption risk

The unsafe_shutdowns field in the NVMe SMART log is a lifetime counter. It increments every time the drive loses power without first receiving a shutdown notification (CC.SHN) from the host. A clean reboot increments power_cycles but not unsafe_shutdowns. A power cut, kernel panic, or someone holding the power button increments both.

The counter never goes down, so the absolute number is history. What matters is the rate of change: every new increment means something cut power to the drive unexpectedly. On enterprise drives with power-loss protection (PLP) capacitors, in-flight writes in DRAM get flushed to NAND before the power dies, and the event is a footnote. On consumer drives without PLP, every increment is a data-loss roll of the dice: writes the host believes were completed may never have reached NAND.

The damage is usually silent. The server comes back up, the journal replays, the application starts, and nobody connects the corrupted database page or the checksum mismatch three weeks later to the power event that caused it. Teams ignore this counter precisely because “the server came back up fine.”

What this means

On a clean shutdown, the NVMe driver sends a shutdown notification by setting CC.SHN in the controller configuration register. The controller then has a defined window to flush volatile caches and quiesce internal state before power is removed. When power disappears without that notification, the controller records an unsafe shutdown and increments the counter on next power-up.

The consequences depend on what was in flight and whether the drive has PLP:

Drive with PLP: Capacitors hold enough charge for the controller to flush its DRAM write cache to NAND. The counter increments, but acknowledged writes are durable.
Drive without PLP: Data sitting in the DRAM write cache is gone. The host already got the completion acknowledgment. The filesystem journal replays on mount, but the journal can only recover what actually reached the media. Anything acknowledged-but-never-persisted is silently lost.
Either way: The increment is evidence of a power-infrastructure problem: PSU fault, UPS failure, PDU flapping, kernel panic, hard reset, or a watchdog-triggered power cut. The drive is the witness, not the suspect.

There is also a growing class of false positives where the counter increments without any real power loss, usually tied to firmware quirks or aggressive platform power management. Distinguishing real events from firmware noise is the first diagnostic fork.

flowchart TD
  A[unsafe_shutdowns increment] --> B{Did power actually get cut?}
  B -->|dmesg panic, watchdog, outage| C[Real power-loss event]
  B -->|clean shutdown, suspend only| D[False positive: firmware or power-state quirk]
  C --> E{Does the drive have PLP?}
  E -->|yes, bit 4 clear| F[Writes protected: investigate power infrastructure]
  E -->|no, or bit 4 set| G[Silent corruption risk: verify filesystem, app integrity, backups]
  D --> H[Check kernel version, firmware, suspend behavior]

Common causes

Cause	What it looks like	First thing to check
PSU, UPS, or PDU failure	Counter increments coincide with unexpected host reboots; may affect multiple machines on the same feed	Host power/event logs, IPMI SEL, PDU logs
Kernel panic or hard hang	Increment lines up with a crash; host rebooted without clean shutdown	`journalctl --list-boots`, kdump/pstore remnants, dmesg from previous boot
Human action: power button held, PDU outlet cycled	Single increment, no crash evidence, often during maintenance windows	Out-of-band console logs, ticketing/maintenance records
Firmware false positive on suspend or clean shutdown	Counter increments on every s2idle sleep or every clean shutdown, with no crash evidence	Compare `power_cycles` vs `unsafe_shutdowns` growth; check kernel and firmware versions
PLP capacitor failed (bit 4)	`critical_warning` bit 4 set; each subsequent unsafe shutdown now carries full data-loss risk	`nvme smart-log` critical_warning field
Hypervisor or runtime not issuing clean NVMe shutdown	Increments tied to VM stop/start or host-managed power events	Whether the platform issues shutdown notification on stop

Controller resets are a different thing. A reset is the host driver recovering an unresponsive controller in software: power was never cut, unsafe_shutdowns does not increment, and you will see a ... timeout, reset controller line in the kernel log instead. If you are seeing resets rather than unsafe shutdowns, that is a firmware or PCIe problem, not a power problem. See NVMe controller reset timeout and NVMe controller reset loop.

Quick checks

All of these are read-only and safe on a production host.

# Read the SMART log: unsafe shutdowns, power cycles, critical warning
nvme smart-log /dev/nvme0 | grep -Ei "unsafe_shutdowns|power_cycles|critical_warning|media_errors"

# Same data via smartmontools if nvme-cli is not installed
smartctl -a /dev/nvme0 | grep -i "unsafe"

# Critical warning bitmask: bit 4 (0x10) means PLP backup has failed
nvme smart-log /dev/nvme0 | grep critical_warning

# Boot history: correlate increments with crashes or unclean reboots
journalctl --list-boots

# Kernel log evidence of panic, watchdog, or power event around the last boot boundary
journalctl -k -b -1 -n 200 --no-pager

# Rule out controller resets masquerading as power events
dmesg | grep -iE "nvme.*(reset|timeout)"

# Firmware version, for checking vendor advisories
nvme id-ctrl /dev/nvme0 | grep -i "^fr"

How to diagnose it

Establish the rate. Record the current counter value with a timestamp. Check again in 24 hours and after the next reboot. A counter that increments on every clean shutdown or every suspend is a false-positive pattern, not a power problem. A counter that increments only when the host crashed or lost power is real.
Compare against power_cycles. A clean shutdown increments power_cycles only; an unsafe one increments both. If power_cycles grew by 10 over a week and unsafe_shutdowns grew by 10 too, every power-off was ungraceful, which points at the platform or the shutdown path, not random outages. If power_cycles grew and unsafe_shutdowns did not, your shutdown path is clean and the historical count is old history.
Correlate with host evidence. For each suspected increment window, find the cause: panic traces in the previous boot’s journal, watchdog messages, IPMI/BMC event log entries, outage reports, or maintenance records. If the host evidence says “clean reboot” but the counter incremented, you are in false-positive territory.
Determine PLP status. SMART does not tell you directly whether a drive has PLP. The one SMART signal you get is negative: critical_warning bit 4 means a drive that has PLP has lost it. Otherwise, identify the drive model (nvme id-ctrl /dev/nvme0 | grep -i mn) and treat any consumer-class M.2 drive in a write-intensive role as unprotected by default. For fleet planning, confirm each model’s PLP capability against the vendor datasheet when you provision it.
If the events are real and the drive lacks PLP: verify data integrity. The corruption, if it happened, is at the filesystem and application layer, not in SMART. Run application-level consistency checks (database page verification, checksum validation), verify backup integrity, and schedule a filesystem check at the next maintenance window. Do not run fsck on a mounted filesystem.
If the events are false positives: collect the evidence (kernel version, firmware version, suspend behavior) and check vendor firmware updates and kernel changelogs. Community reports document several patterns: counters incrementing on every proper shutdown on certain drive and kernel combinations, and counters climbing during s2idle sleep on some laptop platforms, with workarounds involving NVMe power-state kernel parameters. Treat these as community-reported correlations, not guaranteed fixes, and verify against your specific hardware before applying kernel-parameter changes in production.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`unsafe_shutdowns` rate (`nvme.device_unsafe_shutdowns_count`)	Each increment is a power-loss event; the rate is the actionable signal	Any increment during normal operation
`power_cycles` (`nvme.device_power_cycles_count`)	Baseline for distinguishing graceful from ungraceful power-offs	unsafe_shutdowns growing in lockstep with power_cycles
`critical_warning` bit 4	PLP backup has failed; unsafe shutdowns now carry data-loss risk on a drive that was protected	Any assertion
`media_errors` rate (`nvme.device_media_errors_rate`)	A cluster of media errors after an unsafe shutdown can be one-time power-loss corruption rather than ongoing degradation	New errors clustered after an unsafe shutdown event
Host reboot/crash evidence	Attributes each increment to a cause	Boot with no clean shutdown marker in the journal

Fixes

Real power-loss events

Fix the power path. The drive is reporting a symptom of your infrastructure. Work the chain: PSU health and redundancy, UPS battery state and runtime tests, PDU behavior, and whether the server is on the feed you think it is. If increments correlate with kernel panics, treat the panic as the incident and the unsafe shutdown as collateral.

Protect the data path going forward. For drives without PLP in write-intensive roles, the honest options are: move the workload to enterprise drives with PLP, reduce the exposure window (mount options, application fsync behavior, disabling volatile write cache where the latency cost is acceptable), and make backups frequent enough that silent loss is recoverable. Baseline PLP capability at provisioning time: PLP absence is invisible in SMART, so it must be a procurement and inventory fact, not a discovered one.

If bit 4 is set, treat the drive as a no-PLP drive immediately regardless of its enterprise pedigree, and plan replacement. PLP capacitors degrade with age.

False positives

Update firmware first. Several reported false-positive patterns are firmware-side. Check the current revision against vendor advisories.

Check the kernel side. Some reported patterns were resolved by kernel changes; others by limiting how deep NVMe power states go during suspend (for example, power-state latency parameters floated in community reports). Verify against your hardware before applying, and stage it: a wrong power-state setting can trade a cosmetic counter for real latency or stability problems.

Do not “fix” it by ignoring the counter. If you confirm a firmware false positive, annotate the monitoring baseline for that drive model so the rate alert still means something for the rest of the fleet.

Prevention

Alert on the rate, not the value. Any increment of unsafe_shutdowns during normal operation warrants a ticket. The absolute count is archaeology.
Baseline PLP capability at provisioning. Record per-model whether drives have PLP. Consumer drives in write-heavy production roles are a standing risk decision, not a default.
Monitor bit 4 on enterprise drives. A failed PLP capacitor silently converts a protected drive into an unprotected one with zero performance symptoms.
Correlate power events with host telemetry. Every unsafe shutdown should be attributable to a named cause (panic, outage, maintenance) within one investigation cycle. Unattributed increments are either missed outages or false positives, and both matter.
Verify clean shutdown behavior in your stack. Some hypervisors and runtimes do not issue clean NVMe shutdown during stop. Test it: cleanly stop a guest or host and confirm power_cycles increments while unsafe_shutdowns does not.
Run integrity verification after events. After any real unsafe shutdown on a no-PLP drive, application-level checks and backup verification are part of the incident, not optional follow-up.

How Netdata helps

Netdata collects unsafe_shutdowns as nvme.device_unsafe_shutdowns_count alongside power_cycles, so the rate of change is visible as a trend rather than something you have to remember to poll by hand.
The nvme.device_critical_warnings_state chart decodes the critical warning bitmask per bit, so a PLP backup failure (bit 4) surfaces as its own signal instead of being buried in a nonzero aggregate.
Because media_errors is tracked as an incremental rate on the same device, you can see whether a cluster of media errors followed a specific unsafe shutdown event, which changes the diagnosis from “drive is dying” to “one-time power-loss corruption.”
Per-second host metrics around the event window let you confirm what actually happened at the host: a panic, a hard hang, or a clean reboot, which is the fork between a real power event and a firmware false positive.
Fleet-wide visibility makes patterns visible: if every host on one PDU increments together, you have a power-infrastructure incident; if one drive model increments on every clean shutdown across the fleet, you have a firmware bug.