NVMe firmware version tracking: catching the bugs vendors do not advertise

The NVMe controller is an embedded computer running proprietary firmware that owns your data path: the flash translation layer, wear leveling, garbage collection, error correction, and the queue interface to the host. When that firmware has a bug, the symptoms show up as reset loops, erratic latency, premature wear, or in the worst cases silent data loss. Controller firmware bugs are more common than vendors admit, and they are usually triggered by specific command sequences or power-state transitions, so two identical drives on different firmware revisions can behave like different hardware.

Most teams do not track firmware revisions at all. The vendor advisory lands in someone’s inbox, and vulnerable drives keep serving production traffic because nobody knows which devices run the affected version. Firmware version tracking is operational hygiene: an unexpected change, or a drive sitting on a known-bad revision, is a TICKET.

This article covers how to read firmware state from a drive, how to build a fleet inventory you can alert on, what the failure modes look like when a bad revision bites, and how to run updates without being surprised by the reset behavior.

Why firmware version is an operational signal

Firmware version is not a performance metric or a wear metric. It is a state variable that changes the meaning of every other signal you collect from the drive:

A known-bad revision reinterprets your SMART data. Elevated media errors on a Samsung 980 Pro running firmware 3B2QGXA7 were not normal wear. They were a firmware defect that drove drives into read-only mode prematurely, fixed in 5B2QGXA7. Operators tracking revisions could sweep the fleet; operators who were not learned about it from failures.
A bug explains otherwise-unexplained incidents. The classic case is a controller reset loop with normal temperature, no PCIe AER errors, and no media errors. Nothing in the hardware telemetry points anywhere, because the fault is in the firmware’s handling of a specific command sequence or an autonomous power-state transition (APST). The Kingston A2000 on firmware S5Z42105 has a documented APST-triggered failure of exactly this shape.
An unexpected change is a security and change-control finding. Firmware should only move during planned maintenance. If it moved and nobody planned it, you have either unauthorized activity or an update mechanism (OS vendor tooling, a management controller) doing things you did not account for.
Version skew breaks cohort reasoning. If half your drives from one batch are on revision A and half on revision B, and only the B drives are resetting, you have a strong lead. If you are not recording versions, you cannot see the cohort at all.

Fleet-wide firmware tracking is cheap to collect and expensive to not have.

Reading the current version and slot state

Three read-only commands give you the complete firmware state of a device.

# Active firmware revision for every NVMe device (the FW Rev column)
nvme list

# Firmware Slot Information log: active slot and revision per slot
nvme fw-log /dev/nvme0

# Controller firmware capabilities: slot count, read-only slot 1,
# activation-without-reset support, activation notices
nvme id-ctrl -H /dev/nvme0 | grep -i firmware

What each one tells you:

nvme list is your inventory primitive. The FW Rev column is the string you record, trend, and compare. Collect it per host, per device, on a schedule.
nvme fw-log shows the slot table: the afi field is the active slot, and frs1, frs2, and so on hold the revision strings in each slot. This matters for two reasons. First, a newly downloaded image sitting in an inactive slot is a pending change you should know about before someone activates it. Second, unexpected content in a non-active slot is itself a finding: an attacker or a confused automation run can stage firmware without activating it.
nvme id-ctrl -H tells you what an update will do before you run it: how many slots exist, whether slot 1 is read-only (common; it holds factory firmware you cannot overwrite), and whether the controller supports “Firmware Activate Without Reset”. That last bit determines whether activation requires a disruptive controller reset, covered below.

These are safe, read-only queries. Run them as often as you like.

Building a fleet inventory that alerts

The inventory is only useful if it produces findings. Three comparisons turn a list of version strings into alerts:

Unexpected change. The active revision on any device differs from the last recorded value and no maintenance window covers it. This is a TICKET.
Known-bad match. The revision matches an entry in your internal known-bad list (built from vendor advisories, kernel quirks, and your own incident history). Also a TICKET, with severity driven by what the bug does. A data-loss or read-only bug on a drive holding primary data deserves faster handling than a cosmetic SMART reporting bug.
Unplanned skew. Drives in the same role and batch run different revisions and you did not intend it. Skew is not automatically bad, but it should be a deliberate state, not an accident you discover during an incident.

flowchart TD
    A["Scheduled collection:
nvme list + nvme fw-log"] --> B["Recorded fleet baseline
per device, per host"]
    B --> C{Revision changed?}
    C -->|"yes, no maintenance window"| D["TICKET: unexpected change
also check inactive slots"]
    C -->|"no"| E{On known-bad list?}
    E -->|"yes"| F["TICKET: plan update
assess exposure first"]
    E -->|"no"| G{Skew within cohort?}
    G -->|"yes, unintended"| H["Review: align or document"]
    G -->|"no"| I["Clean"]

The known-bad list is the part teams skip, and it is where the value is. Sources for entries:

Vendor advisories and release notes, however you receive them.
The kernel’s own record of known-bad firmware. The Linux NVMe driver maintains quirk tables in drivers/nvme/host/core.c that match on vendor ID, model, and in some cases firmware revision string, applying workarounds for documented defects. If the kernel is carrying a quirk for your drive and revision, that is a defect you should know about. git log on that file is a reasonable way to watch for new entries affecting hardware you run.
Your own incident history. A firmware revision that caused a reset loop on one host belongs on the list even if the vendor never acknowledges it.

How firmware bugs actually show up in production

When a bad revision bites, it rarely announces itself as a firmware problem. The patterns that should trigger a “check the firmware” step in triage:

Controller reset loop. Repeated Resetting controller messages in the kernel log, I/O stalls of 5-30 seconds per cycle, brief recovery, then another hang. Distinguishing feature: temperature normal, PCIe AER counters clean, media errors flat. Nothing environmental explains it. First response: read the firmware revision, search for known issues, and if a power-state trigger is suspected, disabling APST is a known workaround for several documented cases (for example nvme_core.default_ps_max_latency_us=0 as a kernel parameter). See NVMe controller reset loop for the full triage.
Erratic latency with no thermal or GC signature. Spikes that do not correlate with temperature, drive fill level, or workload pattern. Firmware timing bugs and bad power-state transitions produce exactly this shape.
Wrong or frozen SMART data. SMART is self-reported by the firmware: bugs produce incorrect values, and controllers in internal error states can stop updating counters entirely. If SMART has not moved on an active drive, suspect the reporter, not the disk.
Premature failure signatures. Media errors climbing on young drives, or drives dropping into read-only mode (critical warning bit 3) far short of rated endurance, as in the 980 Pro case. If the failure mode does not match the drive’s age and wear, the revision is a suspect.
APST-related hangs. Autonomous power-state transitions are a common trigger for firmware defects. If hangs cluster around idle-to-active transitions rather than load, see NVMe ASPM latency spikes.

The diagnostic move in every case is the same: capture the active revision and the slot table from the affected drive, compare against the fleet baseline and the known-bad list, and only then decide whether you are looking at hardware, firmware, or workload.

Updating firmware without surprises

Firmware updates are disruptive by design, and the surprise is usually the reset behavior, not the download.

The update path with nvme-cli is nvme fw-download to stage the image into a slot, then nvme fw-commit (aliased as nvme fw-activate) to control activation. The --action values define the timing:

Action	Behavior
0	Replace the image in the slot, do not activate
1	Replace the image and activate at the next controller reset
2	Activate an existing image at the next controller reset
3	Activate immediately, without reset (only if the controller supports it)

Operational points that matter:

No automatic reset happens after commit. With actions 1 and 2, activation waits for a controller reset you must trigger explicitly: nvme reset /dev/nvme0 or echo 1 > /sys/class/nvme/nvme0/reset_controller. Until that reset, the old firmware keeps running.
Expect a brief, expected controller reset at activation. All I/O to the device stalls for the reset window, typically seconds, and in-flight commands are replayed. This shows up in your kernel log and your monitoring as a controller reset. It is not a fault, but a reset is a multi-second I/O stall that cascades into application timeouts, so run it inside a maintenance window with the workload drained or replicated. Action 3 avoids the reset entirely, but only use it if nvme id-ctrl -H confirms the controller supports activation without reset.
Some devices need a real power cycle. There are two reset paths and they are not interchangeable: /sys/class/nvme/nvme0/reset_controller triggers an NVMe-level reset, while /sys/class/nvme/nvme0/device/reset triggers a PCI function-level reset. Some controllers (the Samsung PM961 on certain platforms is a documented example ) respond to neither, and the new firmware only activates after a full power cycle. Verify the new revision with nvme list afterward; do not assume activation succeeded just because the command did.
Re-baseline after the update. Record the new revision, the new slot table, and the date. If you skip this, your “unexpected change” alert fires on your own maintenance.

If a firmware file comes from a vendor in an encrypted or vendor-tool-wrapped format, that is a vendor-specific distribution problem and outside what nvme-cli handles directly. Treat any decryption or repackaging steps with the same change-control discipline as the update itself.

Signals to correlate

Firmware version is a state variable, not a time series, so monitoring it means pairing the version record with the symptom signals a bad revision produces:

Signal	Why it matters	Warning sign
Active revision vs fleet baseline	Catches unexpected changes and skew	Any diff outside a maintenance window
Inactive slot contents (`nvme fw-log`)	Staged but unactivated firmware is a pending change	Non-empty slot nobody staged
Controller resets in kernel log	The most common symptom of a firmware hang	Any reset; two or more per hour is a page
Error log entries rate	Firmware and command errors land here before media errors	Rising count with flat media errors
Media errors rate	Some firmware bugs present as phantom media degradation	Climbing errors on a young, low-wear drive
Critical warning bit 3 (read-only)	Firmware-triggered read-only transitions exist in the wild	Any assertion on a drive short of endurance
Latency without thermal or GC correlation	Timing and power-state bugs	Spikes with temperature and fill level flat

How Netdata helps

Netdata charts the symptom signals a bad firmware revision produces: media error rate, error log entry rate, per-bit critical warnings including read-only mode, and unsafe shutdown counts, so a revision-driven failure is visible even before you suspect the firmware.
Correlating rising error log entries against flat media errors on one dashboard is exactly the “firmware, not NAND” distinction you need, and it is hard to see when the counters live in separate tools.
During a firmware activation, the expected controller reset appears as a brief availability and latency blip. Per-second I/O and SMART telemetry around the maintenance window lets you confirm the reset was the expected one and that the drive came back healthy on the new revision.
Trend history on wear signals (available spare, percentage used, media errors) lets you answer “did this degradation start before or after the firmware changed”, which is often the decisive question in a vendor escalation.