Ceph health detail: mapping ceph_health_detail checks to a cause

ceph_health_detail is the bridge between the umbrella status an alert fires on (ceph_health_status: 0=HEALTH_OK, 1=HEALTH_WARN, 2=HEALTH_ERR) and the specific fault you have to fix. Each health check is a separate gauge with name, severity, and message labels; value 1 means active, 0 means inactive. The name label is the check code (OSD_FULL, PG_AVAILABILITY, MON_CLOCK_SKEW, etc.) and is what you build alert routing, dashboards, and post-incident timelines around.

This article is a lookup table for converting an active check into the first thing to inspect. It assumes you have read the umbrella status page and the production mental model; it does not re-explain CRUSH, PG peering, or BlueStore internals. Use it in two directions: when an alert names a check code, jump to the matching row to find the likely fault domain and first correlating signal; when CLI output shows a (MUTED) prefix or the JSON mutes array is populated, remember that muted checks still report value 1 in the ceph_health_detail metric. Alert rules that fire on ceph_health_detail == 1 keep firing for muted checks. Either alert on the umbrella ceph_health_status instead, or include a mute-awareness matcher.

Reading the metric correctly

The metric is per-check, not per-incident. A single code like PG_DEGRADED aggregates many PGs and many OSDs; the count of affected items lives in the message label and in dedicated metrics like ceph_pg_degraded, not in the gauge value. Treat the gauge as a boolean “this fault class is active” signal, then drill into the dedicated metrics for magnitude.

Two consumption patterns work well in production:

• Alert on the umbrella, route on the detail. Page or ticket on ceph_health_status leaving 0, then use the ceph_health_detail{name="..."} labels in the alert payload to route to the right runbook.
• Track check-code activity over time. Stacking ceph_health_detail by name over a week reveals which checks oscillate (commonly OSD_FLAPPING, MON_CLOCK_SKEW, SLOW_OPS) versus which are sticky when they fire (commonly OSD_FULL, MDS_ALL_DOWN, PG_AVAILABILITY with incomplete PGs).

ceph health history complements the metric with first-seen and last-seen timestamps per check, useful for correlating check onset with deployments, topology changes, or capacity events.

Check-code reference

The table maps the check codes you will see most often in production to the underlying fault domain, the metrics that quantify magnitude, and the first CLI command to run. Severity reflects triage posture: PAGE conditions are reserved for checks where data is at risk or unavailable, even when the umbrella stays at HEALTH_WARN.

Codes not listed here usually surface during version-specific transitions: BlueStore legacy-format checks, deprecated Filestore warnings, or DAEMON_OLD_VERSION when mixed daemon versions persist beyond mon_warn_older_version_delay. When you encounter an unfamiliar code, the message label almost always contains the affected item list and the configured threshold, which is enough to start the investigation without the table.

Mapping checks to the right fault domain

The fastest path from a check code to a fix is to identify which subsystem owns the fault. A capacity check (OSD_FULL, OSD_NEARFULL) needs immediate free space or CRUSH reweighting and is independent of PG logic. A PG-availability check (PG_AVAILABILITY with incomplete PGs) almost always traces back to specific OSDs in the acting set being down, or to unfound objects blocking recovery, not to capacity. A performance check (SLOW_OPS, OSD_FLAPPING) needs per-OSD latency and OS-level disk metrics, not PG state.

flowchart TD
  A["ceph_health_detail
name=..."] --> B{"Which subsystem?"}
  B -->|Capacity| C["OSD_FULL / OSD_NEARFULL"]
  B -->|PG state| D["PG_AVAILABILITY / PG_DEGRADED"]
  B -->|OSD performance| E["SLOW_OPS / OSD_FLAPPING"]
  B -->|MON health| F["MON_CLOCK_SKEW"]
  B -->|Subsystem| G["LARGE_OMAP / MDS_*"]
  C --> C1["ceph osd df tree
+ recovery flags"]
  D --> D1["ceph pg dump_stuck
+ unfound objects"]
  E --> E1["per-OSD latency
+ iostat + dmesg"]
  F --> F1["chronyc on MONs
+ election epoch rate"]
  G --> G1["bucket reshard list
+ mds cache status"]

Muting: when and how to use it without lying to yourself

ceph health mute <CODE> [duration] [--sticky] suppresses a check from the umbrella status without removing it from ceph_health_detail. Two behaviours matter operationally.

A non-sticky mute clears automatically when the underlying condition resolves. This is the right tool for transient checks during planned maintenance, for example OSD_DOWN while a host reboots under a noout flag. A sticky mute persists for the full duration even if the condition oscillates, useful for noisy checks that flap during a known network remediation.

Threshold-based mutes disappear if the condition worsens. If you mute OSD_DOWN with one OSD down and a second OSD goes down, the mute is removed automatically. This protects you from muting a check that then escalates, but it also means a mute is not a durable suppression for a check that can grow in scope.

Because muted checks still report value 1 in the Prometheus metric, build mute-awareness into alerting. Either alert on the umbrella ceph_health_status and rely on the ceph_health_detail labels for routing context, or maintain an explicit allow-list of muted codes in the alert rule. The risk of relying on the raw metric alone is that you page on a check you intentionally suppressed.

Cross-cutting gotchas

One check can mask another. OSD_NEARFULL often coexists with PG_DEGRADED after an OSD loss: degraded PGs need to backfill, target OSDs are at backfillfull, and recovery stalls. Treat a cluster with both codes as a single incident with two symptoms, not two unrelated alerts.

PG_DEGRADED during recovery is expected. Alerting on PG_DEGRADED alone produces false tickets during every OSD event. Correlate with ceph_pool_recovering_bytes_per_sec: degraded with non-zero recovery is healing; degraded with zero recovery is stalled and dangerous.

Per-OSD matters more than cluster average. OSD_NEARFULL can fire on a single OSD while the cluster average is at 60%. The cluster-wide capacity metric will not show the problem; only ceph osd df tree and per-OSD alerting will. The same applies to OSD latency: SLOW_OPS on one OSD with cluster-average latency in single digits points at a localised disk failure, not a systemic issue.

MDS checks are deployment-conditional. If your cluster does not run CephFS, any MDS_* check firing indicates either a misconfigured alert or an unexpected CephFS deployment. Do not invest in MDS runbooks for an RBD-only cluster.

Crash checks deserve attention even when muted. RECENT_CRASH with archived crashes hidden still represents real daemon instability. Investigate each crash ID with ceph crash info <id>; a single OSD segfault may be transient, but recurring crashes on the same daemon indicate a real bug or hardware fault.

How Netdata helps

• The Ceph collector scrapes ceph_health_detail per check per second, so onset and clear of each check code is timestamped to the second. When you correlate a check firing with a host-level signal (CPU steal, NIC error counter, disk latency spike), per-second resolution matters.
• Correlating OSD_DOWN with per-OSD ceph_osd_up and host dmesg (for OOM kills) narrows the cause from “an OSD went down” to “OSD 47 was OOM-killed on host ceph-node-12 at 03:14:07”.
• Correlating SLOW_OPS with per-OSD commit/apply latency and per-device iostat distinguishes a network stall from a local disk stall.
• Correlating PG_DEGRADED with ceph_pool_recovering_bytes_per_sec separates expected healing (degraded count falling, recovery non-zero) from stalled recovery (degraded flat, recovery near zero, possibly with norecover or backfill_toofull set).
• Correlating MON_CLOCK_SKEW with host-level NTP and chrony offset metrics identifies which MON is drifting, which the umbrella check does not surface.

Related guides

• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• How Ceph actually works in production: a mental model for operators
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert