Ceph HEALTH_ERR: reading the umbrella status and finding the real fault

ceph_health_status == 2 is the most severe top-level status Ceph reports, and one of the most over-paged signals in production. The reason is structural: HEALTH_ERR is an umbrella aggregation. It tells you that one or more child health checks crossed into ERR severity. It does not tell you which subsystem failed, whether I/O is actually blocked, or whether the condition will self-resolve when PGs finish peering.

Treating the umbrella as the page to action creates two failure modes. First, it duplicates the more specific child signals (OSD_FULL, PG down, MON quorum loss) that already have their own PAGE conditions with tighter gating, producing parallel escalations for the same incident. Second, it fires on transient cold-start states. A cluster-wide OSD restart will briefly push PGs through peering and incomplete, which trips HEALTH_ERR for tens of seconds before clearing. Operators who page on HEALTH_ERR without context spend their nights chasing ghosts.

The correct posture is to treat HEALTH_ERR as a TICKET that points you at ceph health detail. The child health checks you find there carry the tighter PAGE-worthy conditions. This article walks through how to read the umbrella, how to localise the actual fault via the name label, and how to filter cold-start false positives from genuine incidents.

What this means

HEALTH_ERR is computed by the MON cluster from a set of internal health checks. Each check has a severity (HEALTH_WARN or HEALTH_ERR) and a short code (the name label). When any active check has ERR severity, the umbrella reports HEALTH_ERR. When any check has WARN severity, the umbrella reports HEALTH_WARN.

The child checks that can push the umbrella to ERR are the ones that represent data risk or unavailability: capacity exhaustion, PG availability, MON quorum, MDS damage, and a small number of module and configuration errors. The full set of ERR-severity codes is version-dependent, so verify the list against the Ceph version in your cluster.

The critical operator insight is that HEALTH_ERR and its child checks fire simultaneously. When OSD_FULL goes active, the umbrella reports ERR and ceph_health_detail{name="OSD_FULL"} reports active at the same instant. They are not staged. The child check is the diagnosis; the umbrella is the announcement.

flowchart TD
  ERR["HEALTH_ERR
ceph_health_status = 2"]
  ERR --> Q["MON quorum lost"]
  ERR --> F["OSD_FULL / POOL_FULL"]
  ERR --> PG["PG_AVAILABILITY
(down / incomplete)"]
  ERR --> MDS["MDS_DAMAGED"]
  ERR --> MOD["MGR_MODULE_ERROR"]
  Q -.-> Q1["Child PAGE:
ceph_mon_quorum_status"]
  F -.-> F1["Child PAGE:
OSD_FULL detail"]
  PG -.-> PG1["Child PAGE:
ceph_pg_down /
ceph_pg_incomplete"]
  MDS -.-> MDS1["TICKET - CephFS only"]
  MOD -.-> MOD1["TICKET - check MGR"]

Because the child checks have their own per-signal PAGE conditions with longer sustain windows and proper cold-start gating, HEALTH_ERR itself is best treated as a TICKET (sustained > 120 seconds), not a PAGE. The umbrella is redundant with its children for the genuinely catastrophic cases and noisy for the transient ones.

Common causes

Quick checks

These are read-only and safe to run on a production cluster.

# Inspect the active health checks - this is the primary diagnostic
ceph health detail

# Same data, machine-readable, for filtering by code or severity
ceph health detail --format json-pretty | jq '.checks'

# Top-level cluster status and quorum summary
ceph status

# Per-OSD capacity - look for the fullest OSD, not the average
ceph osd df tree

# Stuck PGs by category - pass a meaningful duration, not the default
ceph pg dump_stuck inactive 60
ceph pg dump_stuck stale 60
ceph pg dump_stuck undersized 300

# MON quorum state
ceph mon stat
ceph quorum_status

# CephFS state, if applicable
ceph fs status

# Confirm whether any checks have been muted
<!-- TODO: verify the exact subcommand for listing muted health checks in the deployed Ceph version; ceph health detail --format=json-pretty exposes mute state via .mutes -->
ceph health detail --format json-pretty | jq '.mutes'

The single most important command in this list is ceph health detail. The umbrella does not localise the fault. The detail output does.

How to diagnose it

1. Read ceph health detail and extract every active check code. Each line names a subsystem. The name label is the localisation signal.
2. Bucket each code by category: capacity (OSD_FULL, POOL_FULL), availability (PG_AVAILABILITY), control plane (MON_*), CephFS (MDS_*, FS_*), or module (MGR_MODULE_ERROR).
3. Check cold-start context. If a cluster-wide OSD restart, MON restart, or host reboot happened in the last few minutes, transient peering and incomplete PGs may be driving the ERR. Typical peering completes within 60 to 120 seconds; large clusters may take longer. Wait for the child signal’s sustain window (300 seconds for ceph_pg_incomplete and ceph_pg_down) before escalating.
4. For each remaining code, drill into the corresponding subsystem signal:
   • OSD_FULL: cross-check ceph osd df tree. A single full OSD blocks writes to every PG that maps to it.
   • PG_AVAILABILITY: identify the affected PGs with ceph pg dump_stuck, then ceph pg <pgid> query for the specific blocked PG.
   • MON-related codes: check ceph quorum_status and the per-MON ceph_mon_quorum_status metric.
   • MDS_DAMAGED or MDS_ALL_DOWN: check ceph fs status and the MDS daemon log.
   • MGR_MODULE_ERROR: identify the failing module with ceph mgr module ls and the MGR log.
5. Correlate with the child signal’s own PAGE condition. The child signal is what tells you whether I/O is actually blocked, data is actually at risk, or whether the cluster is degraded but still functional.

Metrics and signals to monitor

The umbrella is the entry point. The child signals are the alerting surface.

Fixes

Cluster full (OSD_FULL, POOL_FULL)

When the cluster or an individual OSD crosses the full ratio (default 0.95), Ceph refuses all writes. Reads still succeed. This is a hard stop.

• Free space: delete snapshots, expired RGW multipart uploads, or non-critical data.
• Add capacity: new OSDs, or reweight under-used OSDs higher.
• Force RGW garbage collection: radosgw-admin gc process.
• Last resort (risky): temporarily raise mon_osd_full_ratio to buy time while capacity is added. This is dangerous because it removes the safety margin during the window before capacity lands.

PG down or incomplete (PG_AVAILABILITY)

These PGs cannot serve I/O. The condition does not self-resolve without intervention.

• Identify the failed OSDs in the acting set with ceph pg <pgid> query.
• Check whether recovery is blocked by capacity (backfill_toofull), unfound objects (recovery_unfound), or operator flags (norecover, nobackfill).
• If objects are genuinely unfound and the acting set is permanently gone, ceph pg <pgid> mark_unfound_lost revert|delete is the last-resort escape hatch. This command causes data loss. The action argument is required. Choose revert to pick a known-good copy or delete to accept data loss.
• For incomplete PGs after a cold start, wait for peering before any destructive action.

MON quorum loss

Without a majority quorum, the cluster cannot accept map updates. Existing clients with cached maps continue briefly, then stall.

• Check clock synchronisation on MON hosts: chronyc tracking or ntpq -p. Clock skew above mon_clock_drift_allowed (default 0.05 s) destabilises elections.
• Check network connectivity between MON hosts.
• If one MON is destabilising quorum, stopping its daemon temporarily can let the others form a stable majority. This is a control-plane action that further reduces quorum headroom until the remaining MONs reform; only do this when you are sure which MON is the outlier.
• Inspect MON store size on each MON host; a bloated store slows elections and crash recovery.

MDS damaged (CephFS only)

MDS_DAMAGED indicates metadata corruption in the MDS journal or cache. Client I/O is not data-path for RADOS, but CephFS itself may be partially or fully unavailable.

• Read the MDS daemon log for the damage reason before running anything.
• Recovery typically involves cephfs-journal-tool.

• Do not run destructive repair commands without first understanding which journal segment is damaged.

MGR module failure (MGR_MODULE_ERROR)

The MGR is still running, but a Python module has failed. Monitoring, dashboard, balancer, or PG autoscaler behaviour may be impacted, but client I/O is unaffected.

• Identify the failing module from ceph health detail and ceph mgr module ls.
• Try disabling and re-enabling the failing module.
• Restart the MGR daemon as a last resort; failover to the standby should be quick.

Transient cold-start ERR

If ceph health detail shows PGs in peering or incomplete immediately after a coordinated restart, wait. Typical peering completes within 60 to 120 seconds; very large clusters may take longer. Escalate only if the condition persists past the child signal’s sustain window (300 seconds for ceph_pg_incomplete and ceph_pg_down).

Prevention

• Alert on child signals, not the umbrella. ceph_health_status == 2 is a TICKET. The PAGE conditions belong to the specific subsystem signals (ceph_pg_down, ceph_pg_incomplete, MON quorum loss, OSD_FULL).
• Gate HEALTH_ERR with a sustain window. 120 seconds filters most cold-start transients without delaying response to real incidents.
• Maintain capacity headroom. Stay comfortably below the backfillfull ratio (default 0.90), not near it. A cluster that loses an OSD near the backfillfull threshold has no room to recover.

• Watch the noout trap. ceph_osd_flag_noout set for > 24 hours while OSDs are down is a common preventable cause of cascading data-loss risk.
• Keep MON clocks synchronised. Monitor NTP/chrony on MON hosts; do not disable the clock skew check as a workaround.
• Run deep scrubs on schedule. noscrub and nodeep-scrub are maintenance flags, not a permanent configuration.

How Netdata helps

• The Ceph collector exposes ceph_health_status per second, so the transition into HEALTH_ERR is visible immediately and the sustain duration is precise.
• ceph_health_detail{name="..."} is collected as a labelled metric, so each child check is its own time series. The dashboard lets you correlate the umbrella going red with the specific code that drove it.
• MON quorum (ceph_mon_quorum_status), PG states (ceph_pg_down, ceph_pg_incomplete), capacity (ceph_cluster_total_used_raw_bytes, ceph_osd_*_ratio), and OSD flags (ceph_osd_flag_noout) are in the same view, so the diagnostic flow from umbrella to subsystem is one click.
• Anomaly detection on the umbrella and on the child signals catches the moment of transition without requiring static thresholds for every check code.
• Cold-start transients show up as short-lived spikes in peering PG counts alongside the umbrella blip, which makes them easy to distinguish from sustained incidents.

Related guides

• Ceph HEALTH_WARN: which warnings are noise and which are structural
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph monitoring checklist: the signals every production cluster needs
• How Ceph actually works in production: a mental model for operators
• Ceph monitoring maturity model: from survival to expert
• Ceph OSD_FULL: all writes stopped at the 95% full ratio
• Ceph nearfull: the 85% warning that decides whether the cluster can heal
• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph OSD fullness imbalance: one OSD full while the cluster average looks fine
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph OSD down: telling a dead disk apart from a network blip
• Ceph OSD flapping: OSDs cycling up and down and the peering storm that follows