NVMe power cycles and power-on hours: fleet age and lifecycle tracking

power_cycles and power_on_hours become operational when you need to know how old a drive really is, whether it has been losing power independently of the OS, and which fleet cohort is approaching replacement together.

They are not failure alerts by themselves. Treat them as the lifecycle ledger for local PCIe-attached NVMe devices: power_cycles counts full device power-off and power-on events, not NVMe controller resets, while power_on_hours counts electrical on-time, not active I/O time. For controller-reported I/O occupancy, use controller_busy_time.

The useful signals come from ratios and deltas: cycles versus reboots, unsafe shutdowns versus cycles, busy time versus on-time, and write volume per powered hour. Absolute values are mostly baseline.

What these counters are and why they matter

Both fields come from the NVMe SMART / Health Information log, Log ID 0x02. Read them against the controller device, for example /dev/nvme0, not a namespace such as /dev/nvme0n1.

A controller reset is software recovery and does not increment power_cycles. A device that disappears and returns after a PCIe or power event can therefore tell a different story than a clean host reboot. Compare drive counters with host boot history before blaming the drive.

Do not expect one host reboot to equal one drive power cycle. Warm reboots, platform power policy, and device power-rail behavior determine whether the device actually loses power. Treat the reboot count as an expected envelope, not an exact match.

Powered age and worked age are also different. A drive can be five years old by wall clock, low by power_on_hours, and still mostly idle by controller_busy_time.

How the counters advance

A full power removal and restore increments power_cycles. If the shutdown notification did not arrive before power went away, unsafe_shutdowns also increments. If shutdown was clean, cycles can increase while unsafe shutdowns stay flat.

power_on_hours advances while the controller has power, including idle time. It is the wrong field for “how hard has this drive worked.” controller_busy_time is the better numerator, although its raw resolution is only minutes.

flowchart LR
  A[Power removed and restored] --> C[power_cycles +1]
  B[No shutdown notification before power loss] --> U[unsafe_shutdowns +1]
  P[Controller powered, even idle] --> H[power_on_hours accrues]
  Q[At least one I/O command outstanding] --> T[controller_busy_time accrues]
  C --> F[Fleet baseline]
  H --> F
  U --> F
  T --> F

For a healthy server, cycles usually track planned power work and some reboots. Hours per cycle is high, and unsafe shutdowns are zero or close to zero. With a flapping power rail, repeated suspend or resume pathology, or a reboot loop, cycles climb faster than maintenance history explains and hours per cycle collapses.

Reading them and building a fleet baseline

Use read-only commands. Replace nvme0 with the controller you are inspecting.

# Read lifecycle counters from one NVMe controller
nvme smart-log /dev/nvme0 | grep -iE "power_cycles|power_on_hours|unsafe_shutdowns|controller_busy_time|percentage_used|data units"

# Identify model, serial, and firmware
nvme list

# Snapshot all local controllers without querying namespaces
for d in /dev/nvme[0-9]*; do
  [ -e "$d" ] || continue
  case "$d" in *n[0-9]*) continue ;; esac
  echo "== $d =="
  nvme smart-log "$d" | grep -iE "power_cycles|power_on_hours|unsafe_shutdowns|controller_busy_time|percentage_used|data units"
done

Compare the cycle deltas with independent boot and kernel records:

# Boot history known to journald
journalctl --list-boots

# Reboot and shutdown records from wtmp
last -x shutdown reboot

# Current-boot storage, PCIe, and power messages
journalctl -k -b --no-pager | grep -Ei "nvme|pcie|aer|power"

A useful baseline row per drive is: timestamp, host, controller path, model, serial, firmware, power_cycles, power_on_hours, unsafe_shutdowns, controller_busy_time, data_units_written, percentage_used, and last known host boot time. Keep raw counters and compute derived values downstream.

Derived checks that pay off:

• Hours per cycle: power_on_hours / power_cycles. A sudden drop means frequent power events or a very recent deployment. Guard against a zero cycle count on new devices.
• Cycle rate: Delta cycles over 7 or 30 days. Compare it with host boots and scheduled power work over the same window.
• Unsafe ratio: unsafe_shutdowns / power_cycles. Near 1.0 means nearly every power-off was ungraceful. A lower value means at least some shutdowns were clean.
• Worked fraction: controller_busy_time / (power_on_hours * 60), using the raw SMART units because busy time is minutes and on-time is hours. High busy time with low host throughput can indicate small commands, latency-bound queue occupancy, internal overhead, or throttling. Low busy time confirms the drive is powered but mostly idle.
• Approximate DWPD: (data_units_written * 512000) / (drive_capacity_bytes * power_on_hours / 24), using raw SMART values and usable drive capacity. This is approximate because host-visible writes are not NAND writes and nominal capacity is not the physical NAND pool.

Do not alert on the counters alone. Page on the corroborated condition: cycles rising without expected boots or maintenance, unsafe shutdowns rising, media errors rising, critical warning bits set, or endurance consumption accelerating.

Where they show up in production

Steady server fleet. Cycles roughly track reboots and rare power work, subject to platform power behavior. power_on_hours becomes the denominator for wear rate, error rate, and DWPD. Track drives from the same batch, firmware, and age as a cohort when failures cluster.

Reboot loop or power instability. Cycles increase while operators did not reboot or service the host. Check PDU history, PSU and slot power, boot records, and kernel logs. A rising cycle count with rising unsafe shutdowns is a power-path problem until proven otherwise.

Edge, client, or suspend-heavy systems. Aggressive platform power management can produce cycle counts that look extreme by server standards. Do not import server thresholds into laptops or edge systems. Establish a separate baseline against actual suspend, resume, and boot behavior.

Consumer drives in write-heavy roles. Lack of power-loss protection is not directly visible in SMART. A high unsafe-shutdown count on a consumer drive is more dangerous than the same count on an enterprise drive with power-loss protection because acknowledged writes may not have reached NAND.

End-of-life planning. power_on_hours identifies drives that are old together. Pair it with percentage_used, available spare, media errors, and write rate. A high-hour drive with low wear is different from a low-hour drive consuming endurance quickly.

Tradeoffs and common misreads

• Powered age is not worked age. Use controller_busy_time and host I/O metrics before calling a drive busy or oversized.
• Cycles are not resets. NVMe controller resets are recovery events and do not increment power_cycles.
• Cycles are not reboots. A host reboot only increments the counter if the device loses and regains power.
• Cycles are not wear. Frequent power events increase risk and operational noise, but endurance wear is tracked through percentage_used, available spare, media errors, and write volume.
• SMART is self-reported. Firmware bugs, stale updates, vendor unit quirks, and low-quality controllers can make values wrong or frozen. Cross-check with host boot records, kernel logs, and block-layer I/O counters.
• SMART polling is coarse. Polling faster than the controller updates does not create resolution. Treat short bursts with suspicion and look for sustained deltas.
• Scope is local PCIe NVMe. NVMe-oF, SPDK userspace devices, RAID-hidden drives, virtual devices, and cloud volumes can expose partial, synthetic, or no SMART data.
• Multi-namespace devices share controller counters. power_cycles and power_on_hours describe the controller, not one namespace.

Signals to watch in production

How Netdata helps

• Netdata trends nvme.device_power_cycles_count and nvme.device_power_on_time, so you can inspect cycle rate and powered age without collecting one-off SMART dumps.
• Correlate cycle deltas with nvme.device_unsafe_shutdowns_count to separate clean power events from power loss.
• Compare nvme.device_controller_busy_time with host throughput to distinguish a powered-but-idle drive from a busy or latency-bound device.
• Use power_on_hours as the denominator for wear and error rates: endurance consumed per powered hour, media errors per powered hour, and approximate DWPD.
• Put lifecycle signals next to nvme.device_estimated_endurance_perc, available spare, media errors, and critical warning bits to distinguish normal aging from active degradation.
• Retain enough history to group exported data by model, firmware, powered age, and cycle history in your inventory or dashboarding layer. That is where batch and cohort risk becomes visible.
• Keep severity restrained: these counters are mostly informational baseline signals, with escalation only when deltas conflict with boot history or align with unsafe shutdowns, errors, or endurance acceleration.

Related guides

• NVMe ASPM latency spikes: PCIe power states adding first-request latency
• NVMe available spare below threshold: critical warning bit 0 and end-of-life wear
• NVMe available spare declining: watching the wear trajectory before the threshold
• blk_update_request: I/O error, dev nvme0n1: reading NVMe I/O errors in the kernel log
• NVMe controller reset loop: repeated resets from a firmware hang
• nvme nvme0: I/O timeout, Resetting controller: what an NVMe controller reset means
• NVMe controller state not live: reading resetting, deleting, and dead from sysfs
• NVMe critical_warning is nonzero: decoding the SMART critical warning bitmask
• NVMe device disappeared: nvme0: Removing and a drive that fell off the PCIe bus
• NVMe endurance runway: projecting time-to-replacement from wear signals
• NVMe error log entries growing: num_err_log_entries beyond media errors
• NVMe write cliff: SLC cache exhaustion and garbage-collection stalls