Ceph PG inconsistent (OSD_SCRUB_ERRORS): scrub found replica divergence

A scrub or deep-scrub finished comparing replicas for a placement group and found they disagree. The cluster surfaces this as OSD_SCRUB_ERRORS (often paired with PG_DAMAGED) and the affected PG sits in active+clean+inconsistent. Client reads still succeed because Ceph serves them from a consistent replica, but at least one copy in the acting set is corrupt.

The danger is not the symptom. Reads work, and Ceph did what it was designed to do: detect silent divergence. The danger is that the corruption was found, not fixed, and the window during which an uncorrupted replica survives is your margin of safety. If the OSD holding the good copy fails before you repair, the object becomes unreadable or unwritable.

The second danger is the repair command itself. ceph pg repair {pgid} resolves the inconsistency by copying data from the primary to the replicas. If the primary is the corrupt copy, repair propagates the corruption to the surviving replicas and destroys the good data. The Ceph documentation explicitly warns against running pg repair before you have identified which replica is authoritative.

The safe path: identify the PG, enumerate the inconsistent objects, determine which OSD holds the corrupt copy, then repair (or manually remove the bad object) once you know the source of truth.

What this means

The inconsistent PG state is set when a scrub finds that replicas disagree on object data, size, metadata, omap (extended attributes or key-value data), or snapset information. A light scrub compares metadata. A deep-scrub additionally checksums object data, which is what catches bit rot that left all metadata intact.

When Ceph detects an inconsistency, it does not silently fix it. The PG remains active+clean+inconsistent. Reads are served from the primary or another consistent replica. Writes continue, but the inconsistency record is preserved until a repair resolves it.

Health checks you will see:

Relevant Prometheus metrics from the MGR module:

• ceph_pg_inconsistent (per pool_id): count of PGs in the inconsistent state. Alert on this.
• ceph_pg_failed_repair (per pool_id): count of PGs where automatic repair was attempted and failed. Escalation signal, not a normal state.
• ceph_pool_objects_repaired (counter, per pool_id): cumulative count of objects repaired.

Severity is TICKET for sum(ceph_pg_inconsistent) > 0 of any duration. Reads still work, but this is a “before the good replica’s OSD dies” ticket, not a “next week” ticket. If automatic repair has been tried and failed (ceph_pg_failed_repair > 0), it becomes urgent manual intervention.

flowchart TD
    A[Scrub finds inconsistency] --> B[PG: active+clean+inconsistent]
    B --> C[Health: OSD_SCRUB_ERRORS / PG_DAMAGED]
    C --> D{Run ceph pg repair?}
    D -- BLIND --> E[Primary may be corrupt]
    E --> F[Corruption propagates to good replicas]
    D -- DIAGNOSE FIRST --> G[rados list-inconsistent-obj]
    G --> H[Identify bad replica via SMART + shard errors]
    H --> I[Safe repair or manual object removal]

Common causes

Quick checks

These commands are read-only and safe to run.

# Confirm which PGs are inconsistent and from which health check
ceph health detail | grep -E 'OSD_SCRUB_ERRORS|PG_DAMAGED|inconsistent'

# List inconsistent PGs cluster-wide
ceph pg ls inconsistent

# Per-pool: list PGs flagged inconsistent
rados list-inconsistent-pg {pool}

# Acting set and primary for a specific PG
ceph pg {pgid} query | jq '.acting, .up, .info'

# Object-level inconsistencies inside a PG
rados list-inconsistent-obj {pgid} | jq .

# Snapset inconsistencies (when object list is empty)
rados list-inconsistent-snapset {pgid} | jq .

# Recent scrub results for the PG
ceph pg {pgid} query | jq '.info.stats.scrub_stats'

# SMART health on every OSD in the acting set
ceph device ls-by-daemon osd.{id}
ceph device get-health-metrics {devid}

If rados list-inconsistent-obj returns “Operation not permitted”, the client key lacks caps. It needs allow r on the pool. Re-run as client.admin or grant the missing cap.

If rados list-inconsistent-obj returns an empty object list for a PG the cluster still marks inconsistent, the inconsistency is at the PG metadata level. Check rados list-inconsistent-snapset {pgid} and the inconsistency payload in ceph pg {pgid} query.

How to diagnose it

The diagnosis has one goal: figure out which OSD holds the corrupt copy before issuing any write.

1. Identify the inconsistent PGs and their acting sets.

   ceph pg ls inconsistent -f json | jq '.pg_stats[] | {pgid, state, acting}'
   

2. For each inconsistent PG, list the inconsistent objects and read the error codes.

   rados list-inconsistent-obj {pgid} | jq '.inconsistencies[0]'
   

   The errors array on each entry tells you the failure mode:

   Error	Meaning
   data_digest_mismatch	Object content differs between replicas.
   size_mismatch	Object sizes differ.
   omap_digest_mismatch	OMAP (extended attributes / key-value) differs.
   read_error	One replica returned a read error, often a bad sector.

3. Cross-reference each shard in the shards array against the acting set from step 1. Each shard entry names the OSD (osd) and whether its copy was considered authoritative. The OSD that is not authoritative, or that reports read_error, is the suspect.

4. Pull SMART for every OSD in the acting set, not just the suspect. The goal is independent evidence about which device is unhealthy.

   ceph device ls-by-daemon osd.{id}
   ceph device get-health-metrics {devid}
   smartctl -A /dev/{device}
   

   Rising Reallocated_Sector_Ct, Current_Pending_Sector, or Offline_Uncorrectable is strong evidence that this OSD’s copy is the corrupt one.

5. Check host-level memory health if SMART is clean and the inconsistency pattern matches recent writes.

   dmesg -T | grep -iE 'edac|memory|hardware error'
   

   Non-ECC RAM on an OSD host, or a failing DIMM with EDAC reporting correctable errors that occasionally become uncorrectable, can produce data_digest_mismatch with no disk fault.

6. Decide on the authoritative copy. If two replicas agree and one disagrees, the two agreeing copies are authoritative. If the primary is the outlier, blind ceph pg repair will propagate the corruption.

7. Only after you know the source of truth, proceed to repair.

Metrics and signals to monitor

Fixes

The right fix depends on which replica is corrupt. The single rule: never run ceph pg repair until you have evidence about which copy is authoritative.

Standard case: primary is the good copy, one replica is corrupt

This is the safe path the repair command was designed for.

# Re-verify the inconsistency
rados list-inconsistent-obj {pgid} | jq .

# Run repair
ceph pg repair {pgid}

# Watch progress: ceph_pool_objects_repaired should increase
# and the PG should return to active+clean
ceph pg {pgid} query | jq '.info.stats'

Repair triggers a recovery-style operation that overwrites the inconsistent copy with the primary’s data. On BlueStore, internal checksums make authoritative copy selection more reliable than on legacy FileStore, but the primary bias still exists when checksums are unavailable, which is why the diagnosis step is non-negotiable.

Dangerous case: primary is the corrupt copy

This is the scenario called out in the official docs and vendor knowledge bases. The primary OSD has the wrong digest and the two replicas agree with each other. Running ceph pg repair here overwrites the good replicas with corrupt data from the primary.

The fix is to remove the bad object from the primary OSD using ceph-objectstore-tool, then trigger repair so the primary pulls the object back from a good replica.

WARNING: this procedure is destructive and requires stopping the OSD. The exact ceph-objectstore-tool invocation is version-dependent. On Reef and later (FileStore removed, BlueStore only), refer to the current ceph-objectstore-tool documentation for the remove and list syntax. Test on a non-production OSD first.

High-level steps:

1. Stop the primary OSD: systemctl stop ceph-osd@{id}.
2. Use ceph-objectstore-tool with --op remove against the OSD’s data path, the PG ID, and the object ID to remove the bad object from the primary’s local store.
3. Mark the object as missing on that OSD so the next read pulls it from a peer.
4. Start the OSD: systemctl start ceph-osd@{id}.
5. Trigger repair: ceph pg repair {pgid}.

Repair does not resolve: failed_repair

If ceph_pg_failed_repair is non-zero for the PG, automatic repair was attempted and failed. Common reasons:

• The inconsistency is at the snapset or PG metadata level, not the object level, so the object-copy repair path does not apply.
• The OSD that holds the authoritative copy is down or unfound.
• An underlying device error prevents the read.

Run rados list-inconsistent-snapset {pgid} for metadata-level inconsistencies, and check whether the authoritative OSD is up and reachable. If the authoritative copy is on a down OSD, recovering that OSD (or declaring the object lost via ceph pg mark_unfound_lost revert|delete) may be required.

Erasure-coded pools

Repair semantics differ for EC pools. The rados list-inconsistent-obj output includes shard-level information, and repair uses the EC plugin’s reconstruction logic. The primary bias concern still applies: if the OSDs holding the corrupt shard are also the ones the primary would consult as authoritative, blind repair can propagate bad data. Walk the same diagnosis path: enumerate shards, find the disagreeing one, verify SMART, then repair.

Automatic repair

osd_scrub_auto_repair (default false) enables automatic repair when scrub finds errors, up to osd_scrub_auto_repair_num_errors (default 5). It applies to BlueStore and EC pools. If you have this enabled, your monitoring of ceph_pg_inconsistent becomes more important, not less: auto repair quietly fixes single-object divergences, which means the underlying cause (bad RAM, marginal disk, firmware bug) can keep producing new inconsistencies without you ever seeing a ticket. Track ceph_pool_objects_repaired as a rate. Any non-zero rate is a sign to investigate.

Prevention

• Keep deep scrubs running. Suppressing noscrub and nodeep-scrub is the single most common way operators miss corruption until it spreads. If you must suppress for performance, set a duration limit and alert if the flags are set for more than 24 hours.
• Watch scrub recency. ceph_health_detail{name="PG_NOT_DEEP_SCRUBBED"} active for more than 24 hours means verification debt is accumulating and corruption can grow undetected.
• Use ECC RAM on OSD hosts. Non-ECC RAM produces data_digest_mismatch with no disk fault and is one of the hardest causes to diagnose.
• Monitor SMART aggressively. Reallocated_Sector_Ct, Current_Pending_Sector, and Offline_Uncorrectable are leading indicators of the bit rot that scrub eventually catches.
• Do not enable osd_scrub_auto_repair without also alerting on ceph_pool_objects_repaired rate. Silent auto repair masks the underlying hardware cause.
• Make the operator rule explicit: never run ceph pg repair without first running rados list-inconsistent-obj. Bake it into your runbook so the 3 a.m. operator does not have to remember.
• Track Squid-specific scrub scheduling issues. Squid 19.2.x introduced scrub scheduling changes that can leave deep scrubs taking far longer than expected, extending the window for undetected corruption. If scrubs stall after an upgrade, check whether osd_scrub_disable_reservation_queuing needs to be set to true.

How Netdata helps

• The Ceph collector surfaces ceph_pg_inconsistent, ceph_pg_failed_repair, and ceph_pool_objects_repaired per pool, so divergence is visible the moment scrub reports it rather than the next time someone runs ceph health detail.
• ceph_health_detail with labels for name (OSD_SCRUB_ERRORS, PG_DAMAGED, PG_NOT_DEEP_SCRUBBED) lets you alert on the specific integrity check, not just the umbrella HEALTH_WARN.
• Per-OSD commit and apply latency correlate with the bad replica. When scrub flags a PG, latency outliers on the OSDs in its acting set point at the failing device.
• Per-second collection shortens the window between “scrub found something” and “operator saw it”, which matters when the only good replica is one OSD failure away from disappearing.
• Correlating scrub health with host-level SMART and memory signals on the same view turns “PG inconsistent” into “OSD 47 has new reallocated sectors and elevated commit latency” in one place, rather than across three dashboards.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph blocked ops: client I/O stuck behind a single slow OSD
• Ceph BlueStore RocksDB compaction stalls: periodic latency spikes
• Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device
• Ceph BlueStore allocator fragmentation: rising latency at moderate fullness
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph client latency vs OSD latency: fast disks, slow clients
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph MON_CLOCK_SKEW: clock drift between monitors and election churn