NVMe error log entries growing: num_err_log_entries beyond media errors

Your NVMe alert fired because num_err_log_entries is climbing. You pull the SMART log, and media_errors is zero. The drive looks healthy, but something is writing error entries at a steady, sometimes alarming, rate.

num_err_log_entries is a superset counter: it counts every error the controller records, including admin command errors, I/O command errors, internal controller errors, and thermal events. media_errors counts only data integrity failures against the NAND. A rising num_err_log_entries with flat media_errors points away from failing flash and toward firmware, driver, or command-level issues. In a large share of cases, it points at your own monitoring stack.

One time-critical detail: the error log is a circular buffer. At high entry rates, the oldest entries, including the ones that explain the root cause, are overwritten. If this counter is moving, read the log now, not after standup tomorrow.

What this means

The NVMe controller maintains an Error Information Log (Log ID 0x01). Every time a command completes with an error status, the controller appends a 64-byte entry containing the status code, the command that failed, the namespace, and the LBA where relevant. num_err_log_entries in the SMART log (Log ID 0x02) is the lifetime count of those entries. It never decreases and it is not reset by reading the log.

The buffer itself is shallow. The maximum number of retained entries is ELPE + 1, where ELPE comes from the Identify Controller data. Most drives keep 64 or 128 entries. Once full, each new entry evicts the oldest. If a tool generates two entries per second, the entire visible window turns over in about a minute.

The practical interpretation:

• A spike in num_err_log_entries without matching media_errors means firmware, driver, or command issues, not failing NAND.
• Operator reports across firmware vendors describe the dominant status in these cases as 0x2002, typically logged when software probes optional commands the firmware does not implement. The value is the 16-bit status field nvme-cli prints for the entry: bits 7:0 hold the status code, bits 10:8 the status code type, and the 0x2000 bit is the More bit. 0x2002 therefore decodes to SC 0x02 (Invalid Field in Command, Generic status type) with the More bit set because the error also has an Error Information log entry. nvme error-log prints the decoded string alongside the hex, so you do not need to decode the bits yourself.
• media_errors rising alongside the log count is a different incident. That is active media degradation; see the related guides below.

One kernel behavior worth knowing: Linux commit d7ac8dc (merged in kernel 6.1) silenced dmesg spam from user-space passthrough commands by marking them quiet. The kernel messages stop, but the drive still logs the error internally. On a modern kernel you can see the SMART counter climbing with a completely silent kernel log.

Common causes

The monitoring-tool causes deserve emphasis because they are self-referential: the tool that alerts you about error log entries is the tool generating them. Reported examples include node_exporter 1.5.0 (the nvme and hwmon collectors each added entries per scrape on some drives), the collectd smart plugin, smartmontools 7.4 reading the self-test log against a namespace block device on single-namespace drives, and nvme-cli 2.x via libnvme issuing optional Identify commands that pre-NVMe-1.2 firmware rejects. Samsung consumer drives (950 Pro, 960 EVO, 970 EVO Plus, 980) are disproportionately reported, but Kingston and Corsair models appear too.

Quick checks

All read-only and safe to run in production.

# 1. Current counters: error log entries vs media errors
nvme smart-log /dev/nvme0 | grep -E "num_err_log_entries|media_errors"

# 2. Read the actual error entries (most recent first)
nvme error-log /dev/nvme0

# 3. Request more entries if the rate is high (default is 64)
nvme error-log /dev/nvme0 -e 128

# 4. Critical warning state: is the drive itself complaining?
nvme smart-log /dev/nvme0 | grep critical_warning

# 5. Kernel log: I/O errors or resets accompanying the entries?
dmesg | grep -iE "nvme.*(error|reset|timeout)" | tail -30

# 6. Reproduce a suspected probe: does the counter move?
nvme smart-log /dev/nvme0 | grep num_err_log_entries
nvme list >/dev/null
nvme smart-log /dev/nvme0 | grep num_err_log_entries

# 7. Temperature context for thermal-type entries
nvme smart-log /dev/nvme0 | grep -iE "^temperature|thm_temp"

On step 6: if nvme list itself increments the counter, you have found your culprit. This was the exact signature of the libnvme issue fixed in libnvme v1.7 with kernel 6.8, where tools stopped issuing unsupported Identify commands and read attributes from sysfs instead.

How to diagnose it

flowchart TD
  A[num_err_log_entries rising] --> B{media_errors rising too?}
  B -- Yes --> C[Media degradation: verify redundancy, replace drive]
  B -- No --> D[Read nvme error-log NOW - circular buffer]
  D --> E{Status code pattern}
  E -- "INVALID_FIELD probing signature" --> F[Software probing optional commands: check monitoring tools, nvme-cli, smartmontools]
  E -- "namespace not ready at boot only" --> G[Benign: ignore]
  E -- "unrecovered read/write error" --> C
  E -- "thermal or internal error" --> H[Check temperature, TMT counters, firmware version]
  F --> I{Rate matches scrape interval?}
  I -- Yes --> J[Fix or pin the tool version]
  I -- No --> K[Reproduce with nvme list / smartctl to isolate]

1. Capture the log before it rotates. Run nvme error-log /dev/nvme0 immediately. The most recent error is entry 0. Save the output off-box; at high rates the window is minutes.

2. Split the incident by media_errors. nvme smart-log /dev/nvme0 | grep media_errors. Non-zero and rising means you are in the wrong article: go to the media failure path, verify RAID or replication health, and plan replacement. Zero and flat means continue here.

3. Classify the status codes. Each entry has a Status Code and Status Code Type. The INVALID_FIELD signature (reported as 0x2002 in the probing cases above) is the classic mark of software trying optional commands the firmware does not implement. There is no spec-defined way to know an optional command is unsupported without trying it, and some firmware logs the attempt as an error. These entries are harmless but they pollute the counter and the buffer. The spec defines four status code types (Generic, Command Specific, Media and Data Integrity, and Path Related; NVM Express base spec section 4.6.1), and nvme error-log decodes each entry’s type and code to a human-readable string. A “namespace not ready” status recorded only during early boot is benign initialization noise. An “unrecovered read error” in steady state is critical regardless of what media_errors says.

4. Measure the rate and match it to a clock. Sample the counter a few times at 60-second intervals. If the increment is exactly 2 per minute and your scrape interval is 30 seconds, you have a per-scrape probe. Compare against your metrics agent, smartd, and any cron-driven smartctl or nvme invocations.

5. Isolate by reproduction. With monitoring quiet, run nvme list, then smartctl -a /dev/nvme0n1, then smartctl -a /dev/nvme0, checking the counter after each. The command that increments it is your source. For smartmontools 7.4 on single-namespace drives, the namespace block device path triggers it while the character device does not.

6. Rule out thermal and firmware causes. If status codes indicate thermal events or internal errors, correlate with composite temperature, thm_temp1_trans_count, and warning_temp_time from the SMART log, and check the firmware revision (nvme id-ctrl /dev/nvme0 | grep -i "^fr") against vendor advisories.

Metrics and signals to monitor

Fixes

Monitoring tool is generating the entries

This is the most common fix and the least invasive.

• Identify the specific tool, then pin, upgrade, or reconfigure it. For node_exporter 1.5.0, disabling the offending collector (--no-collector.hwmon or --no-collector.nvme) stops the growth at the cost of those metrics. For collectd, disable NVMe monitoring in the smart plugin. For nvme-cli 2.x against old firmware, upgrading libnvme to v1.7 or later on kernel 6.8+ resolves the nvme list behavior; where that is not possible, pinning nvme-cli 1.x avoids the unsupported Identify commands.
• For smartmontools 7.4, query the character device (smartctl -a /dev/nvme0) or use -d nvme,0xffffffff instead of the namespace block device. Note also that smartmontools 7.4 changed smartd to no longer raise LOG_CRIT for new error-log entries that do not indicate device problems, which reduces alert noise from exactly this pattern.
• Tradeoff: disabling a collector or pinning an old tool version is a workaround, not a cure. The entries are harmless, but the counter and the shallow circular buffer are now polluted, which degrades your ability to spot a real error storm later. Prefer fixes that stop the probe over fixes that stop the alert.

Driver or application command errors

If the failing commands come from your workload or a kernel path rather than a monitoring probe, the entries are telling you something real about compatibility. Update the kernel and nvme-cli together, since the passthrough and probing behavior lives in both. If the entries correlate with application I/O and dmesg shows blk_update_request: I/O error, treat it as an I/O error investigation, not a monitoring artifact.

Firmware-level issues

If entries carry internal-error status codes, or the pattern persists with all user-space probing eliminated, check the firmware revision against vendor advisories and update firmware during a maintenance window. Firmware updates on NVMe are disruptive (controller reset, possible brief I/O stall) and must be staged with redundancy verified first. There is no standard command to clear the error log, and num_err_log_entries is a lifetime counter: it does not reset on reboot or power cycle. Do not chase “resetting the counter.” Fix the cause, note the current value as your new baseline, and alert on rate-of-change from there.

Genuine media errors

If media_errors is rising or you see unrecovered read/write errors in steady state, stop treating this as noise. Verify redundancy, force rewrite of at-risk data where your stack supports it, and schedule replacement.

Prevention

• Baseline after remediation. Record the current num_err_log_entries value per drive after fixing the source. Alert on rate of increase from that baseline, not on the absolute lifetime count, which is polluted history you cannot erase.
• Read the error log on alert, automatically. When the rate alert fires, have the response capture nvme error-log output immediately. The circular buffer means the diagnostic window is short during a real error storm.
• Test monitoring changes against one host first. Before rolling a new metrics agent or smartmontools version across the fleet, run it on a single host and watch the counter for an hour.
• Track firmware versions fleet-wide. Both the false-positive probing behavior and real internal-error patterns are firmware-version-specific. Knowing which drives run which revision turns a fleet-wide mystery into a targeted fix.
• Keep kernel and nvme-cli reasonably current. The kernel passthrough quieting (6.1) and the libnvme sysfs fallback (v1.7, kernel 6.8) together eliminated the most common self-inflicted source.

How Netdata helps

• Netdata charts num_err_log_entries as a rate (nvme.device_error_log_entries_rate), which is the only actionable form of this counter. A flat line at a high value is history; a rising line is now.
• The media-errors chart (nvme.device_media_errors_rate) sits next to it, so the first diagnostic split, “failing NAND vs noisy software,” is a two-second comparison instead of two SSH sessions.
• Critical warning bits are charted individually (nvme.device_critical_warnings_state), so you can immediately see whether the drive agrees something is wrong, and which condition it is reporting.
• Temperature and thermal-management charts let you confirm or eliminate the thermal class of log entries without manual SMART parsing.
• Because Netdata collects at per-second granularity, you can see whether the error-log rate steps in sync with a scrape or cron cycle, which is the signature of a monitoring-induced false positive.

Related guides

• NVMe available spare below threshold: critical warning bit 0 and end-of-life wear
• blk_update_request: I/O error, dev nvme0n1: reading NVMe I/O errors in the kernel log
• NVMe controller reset loop: repeated resets from a firmware hang
• nvme nvme0: I/O timeout, Resetting controller: what an NVMe controller reset means
• NVMe controller state not live: reading resetting, deleting, and dead from sysfs
• NVMe critical_warning is nonzero: decoding the SMART critical warning bitmask
• NVMe device disappeared: nvme0: Removing and a drive that fell off the PCIe bus
• How NVMe actually works in production: a mental model for operators
• NVMe monitoring checklist: the signals every production SSD needs
• NVMe monitoring maturity model: from survival to expert
• NVMe NVM subsystem reliability degraded: critical warning bit 2
• NVMe percentage used at 100%: reading the endurance-consumed estimate