$ guides / smartctl-disk-monitoring / smartctl-first-observation-baseline ▌

Operations Guides

First-observation baselining: don't page on lifetime counters at rollout

When you deploy SMART monitoring on an existing fleet for the first time, every in-service drive carries accumulated history in its lifetime counters. Offline Uncorrectable sectors, NVMe Media and Data Integrity Errors, Power-On Hours, Unsafe Shutdowns, Reallocated Sector Count, UDMA CRC Error Count. These counters started incrementing the moment the drive left the factory and never reset.

If your alerting fires on any non-zero absolute value, every drive with any history pages within minutes of enabling monitoring. A drive with 3 reallocated sectors from factory QA, a drive with 200 uncorrectable errors from a past thermal event three years ago, a drive with 5 unsafe shutdowns from a UPS failure. All produce identical alerts to a naive greater-than-zero rule.

The fix is straightforward: capture a baseline at first observation and alert on growth from that baseline, not on the absolute value. This is the difference between “something happened at some point in this drive’s life” and “something happened since you started watching.”

Why absolute values are useless at rollout

Cumulative SMART counters are monotonically increasing by design. They tell you that an event occurred at some point in the drive’s lifetime, not when. A drive that accumulated 50 Offline Uncorrectable sectors during shipping damage in 2021 and has been stable since is in a fundamentally different state than one that gained 50 sectors last week. Without a baseline, both drives produce the same alert.

New drives are not exempt. Factory burn-in routinely leaves non-zero counts on several attributes:

Power-On Hours: brand-new drives may show 1-2 hours from factory testing
Total Data Written: factory testing, firmware installation, and QA write data to the drive
Reallocated Sector Count: some SSDs ship with a handful of remapped blocks from manufacturing QA
NVMe Unsafe Shutdowns: the drive went through testing and burn-in at the factory. A single-digit count is expected
UDMA CRC Error Count: may show a small count from initial cable seating during integration

None of these indicate problems in your environment. Without a deployment-time snapshot, you cannot distinguish “the factory did this” from “production did this.”

How baselining works

The mechanism has three stages: capture, compare, and alert on delta.

flowchart TD
    A["Monitoring sees drive"] --> B{"Baseline recorded?"}
    B -->|"No"| C["Record current values as baseline"]
    C --> D["No alert this cycle"]
    B -->|"Yes"| E["Compare current vs baseline"]
    E --> F{"Counter changed?"}
    F -->|"No change"| G["Normal operation"]
    F -->|"Increased"| H["Alert: new event since baseline"]
    F -->|"Decreased"| I["Anomalous: drive swap or reset"]

Capture the baseline

At first scrape, record the full SMART attribute set for every drive. This snapshot is both the reference point for growth alerts and a forensic record of the drive’s state at deployment.

# Capture full SMART snapshot at deployment for archival
mkdir -p /var/lib/smart-baselines
smartctl -x -j /dev/sdX > /var/lib/smart-baselines/sdX-$(date +%Y%m%d).json

The -x flag enables all SMART features and logs. The -j flag produces structured JSON output (available since smartmontools 7.0) for programmatic parsing. For fleet-wide capture, run this against every device before enabling alerting. Store the output somewhere that survives host rebuilds and monitoring system reinstallation.

Distinguish two moments:

First-observation baseline: what the monitoring system records when it first scrapes a drive. If monitoring comes online after drives have been in production, this baseline includes production history.
Deployment-time snapshot: what the operator captures when physically installing a drive. This is the true zero-time reference, useful for later triage.

When monitoring is deployed onto an existing fleet, the first-observation baseline is your starting point. It absorbs all prior history into a single reference. From that moment forward, growth is the signal.

Compare against baseline

Each subsequent scrape compares the current value to the baseline value:

Current equals baseline: no change, no alert
Current greater than baseline: growth detected. This is the signal that matters
Current less than baseline: anomalous. Counters are cumulative and should never decrease. Investigate drive swap, firmware update, or counter reset

For rate-of-change analysis, track the timestamp of each observation alongside the value. The growth rate (sectors per day, media errors per week) is more diagnostic than the absolute delta. A drive that gains 1 reallocated sector per month is in a different category than one that gains 10 per day.

Alert on growth, not absolute value

The alerting logic transforms from “is this counter above zero?” to “did this counter increase since the last observation?” This single change eliminates the rollout flood while preserving sensitivity to new events. A drive that had 17 Offline Uncorrectable sectors at baseline and now has 18 triggers an alert. A drive that had 17 at baseline and still has 17 does not.

The recommendation for severity: page only when the count has increased since baseline. Historical non-zero values get a ticket (investigate, but do not assume it just happened).

Which counters need baselining

Not all SMART signals need baseline treatment. Some are threshold-based or represent current state rather than accumulated history. The counters that require first-observation baselining share one property: they are monotonically increasing lifetime accumulators.

Signals that represent current state do not need this treatment:

Signal	Why no baseline needed
Current Pending Sector (ID 197)	Can decrease (sectors resolve on rewrite). Alert on sustained non-zero, not growth from baseline
Drive Temperature	Instantaneous reading, not cumulative
NVMe Available Spare	Current percentage. Alert on threshold crossing
NVMe Percentage Used	Current estimate, not a counter. Alert on threshold
NVMe Critical Warning bits	Current bitmask state, not cumulative

Implementation in common tools

smartd configuration

For operators using smartd, directive syntax controls whether alerts fire on absolute value or growth:

-U 198 reports when attribute 198 changes from its previously recorded value
-U 198+ reports only when the value increases (growth-based)

The + suffix is the built-in mechanism for growth-based alerting. For an existing fleet with historical values, use the + variant to avoid re-alerting on pre-existing counts.

smartd maintains state across daemon restarts using savestates. The state file stores previous attribute values, temperature min/max, and error log entry counts. Without state persistence, smartd loses track of previous values on restart and may re-alert on already-known values.

Prometheus and smartctl_exporter

The Prometheus community smartctl_exporter exposes raw SMART counter values as metrics. It does not perform baselining or delta calculation. You must implement growth-based alerting in PromQL.

A naive rule like smartctl_device_smart_attributes_raw_value > 0 fires on every drive with any history at first scrape. Use a time-windowed comparison instead:

# Alert when Offline Uncorrectable increased in the last hour
delta(smartctl_device_smart_attributes_raw_value{attribute_id="198"}[1h]) > 0

This naturally implements baselining: the delta over the time window only fires on growth within that window, not on historical accumulated values. At first scrape, there is no previous data point in the window, so no alert fires. The window length defines your detection latency: a 1-hour window catches growth within the last hour.

For slower-moving counters, use a longer window or combine with changes() to detect any modification. The principle is the same: alert on the rate of change, never on the absolute value.

Deployment-time snapshots

The baseline serves double duty. Beyond alerting, it is a forensic record. When a drive develops 3 reallocated sectors, the deployment snapshot confirms whether those sectors were present at install time or appeared in production. Without it, you cannot distinguish factory reallocations from production ones.

For new drive installations, capture a full SMART snapshot at physical install time. This gives you a true zero-time reference even if the monitoring system was not running yet.

Common mistakes

Alerting on absolute non-zero at first deployment: Every drive with any history pages. This floods operators at the exact moment they are validating the monitoring system, training them to ignore SMART alerts before they have seen a real one.
Losing the baseline on monitoring system rebuild: If the monitoring system is rebuilt or state files are lost, the new “first observation” becomes the baseline. Growth that occurred between the old baseline and the rebuild is silently absorbed. Store baselines in persistent storage that survives host rebuilds.
Assuming new drives have zero counters: Factory testing writes data, spins up drives, and may cause unsafe shutdowns during burn-in. Brand-new drives routinely show small non-zero values for Power-On Hours, Total LBAs Written, Reallocated Sector Count, and Unsafe Shutdowns.
Ignoring counter decreases: A cumulative counter that decreases is anomalous. It may indicate a drive swap (new drive with lower counters), a firmware update that reset counters, or manufacturer diagnostics that forced sector reallocation.
Treating first-observation baseline as deployment-time truth: If monitoring was deployed after drives entered production, the first observation already includes production history. Growth from that baseline is valid, but the baseline itself is not a clean install-time record. Capture deployment-time snapshots separately for forensic purposes.
Using the same alert logic for all counters: Current Pending Sector (ID 197) can decrease when sectors resolve on rewrite. Alerting on growth from baseline for ID 197 is less useful than alerting on sustained non-zero across multiple polls. Match the alerting strategy to the counter’s behavior.

Correlating SMART growth with Netdata

SMART counter growth rarely tells the full story in isolation. Netdata’s value here is correlation, not just collection.

Per-second collection means growth is detected on the next collection cycle, not on the next 5-minute scrape. For a counter like Offline Uncorrectable that may jump in bursts, tighter polling reduces the window between event and detection.
Correlated timelines let you confirm whether SMART counter growth represents active failure. View Offline Uncorrectable growth alongside I/O latency (await, svctm, iowait), kernel I/O errors from dmesg, and drive temperature in a single view. Growth that coincides with await spikes and UNC errors in the kernel log is a confirmed active failure. Growth without corroborating signals may warrant investigation but not an immediate page.
ML-based anomaly detection flags unexpected changes in SMART counter values without explicit threshold rules. The anomaly model learns each drive’s normal behavior rather than checking against a fixed threshold, so drives with non-zero baselines from factory or pre-monitoring history do not generate false positives.
Historical retention lets you scroll back to when monitoring started and verify whether specific sectors or errors were present at first observation or appeared later. This provides the deployment-time snapshot automatically, as long as monitoring was active from install.