Ceph failed_repair: when ceph pg repair cannot fix the inconsistency

A failed_repair PG state means Ceph attempted to repair a data inconsistency that scrub or deep-scrub detected, and could not. The PG is now active+clean+inconsistent+failed_repair, the cluster is at HEALTH_WARN (typically via the OSD_SCRUB_ERRORS health check), and the inconsistency will not heal on its own. Reads still succeed from consistent replicas, but the cluster has stopped trying to fix the divergence until you intervene.

This is the escalation of the inconsistent state, not a separate problem. A scrub found that at least two replicas of the same object differ in content, size, or checksum. You or the auto-repair logic ran ceph pg repair, and the repair could not determine which copy is authoritative, or the authoritative copy could not be propagated. The failed_repair flag is sticky: re-running ceph pg repair immediately will usually fail the same way.

The path forward is destructive. You must decide manually which replica is correct, then either repair from it, force the cluster to accept loss, or replace failing hardware first. This article walks through that decision tree.

What this means

The failed_repair flag is set by Ceph’s PG state machine when an automatic repair attempt fails. The operational meaning is unambiguous: Ceph has given up trying to fix this PG without operator intervention.

Three operational realities follow:

1. Reads continue to succeed. The PG is still active. failed_repair is a data-integrity state, not an availability state. Clients will not see errors unless they read from the corrupt copy and the read path surfaces a checksum mismatch.

2. The PG is at risk. If the OSD holding the authoritative copy fails before you repair, you lose the ability to recover the correct data. There is no timer, but the window is real.

3. ceph pg repair again, unattended, will not work. It will either be a no-op or fail the same way. Gather evidence first, then choose a targeted repair path.

Check your defaults. osd_scrub_auto_repair is false on many clusters. If set to true, Ceph automatically attempts repair when scrub finds up to osd_scrub_auto_repair_num_errors (historically 5) errors. A failed_repair in that case means auto-repair fired and failed before you ran any command.

flowchart TD
    A[Scrub finds divergence] --> B[active+clean+inconsistent]
    B --> C{ceph pg repair attempted}
    C -->|succeeds| D[objects repaired, returns to clean]
    C -->|fails| E[active+clean+inconsistent+failed_repair]
    E --> F[rados list-inconsistent-obj + list-inconsistent-snapset]
    F -->|inconsistent objects found| G[compare digests across shards]
    F -->|snapset inconsistency only| G2[handle snapset repair]
    F -->|no usable result| H[no authoritative candidate]
    G --> I{primary shares majority digest?}
    I -->|yes| J[re-run ceph pg repair]
    I -->|no| K[ceph-objectstore-tool remove or export/import]
    G2 --> K
    H --> K
    K --> L{loss acceptable?}
    L -->|no| M[deeper manual recovery or restore from backup]
    L -->|yes| N[mark_unfound_lost revert or delete]

Common causes

The “all replicas disagree” case is the one most operators hit first and find most confusing. Ceph cannot pick an authoritative copy when no two replicas share the same digest. The objects are still listed by rados list-inconsistent-obj, but the shards section shows no majority. This is by design.

Quick checks

These commands are read-only. Run them before touching any data.

# Identify the affected PGs and the health checks firing
ceph health detail | grep -E 'inconsistent|failed_repair|SCRUB|DAMAGED'

# Show the full PG state, acting set, and up set
ceph pg <pgid> query

# List inconsistent objects in the PG (replicated and most EC pools)
rados list-inconsistent-obj <pgid> --format=json-pretty

# List inconsistent snapsets (run this even if the obj list is empty)
rados list-inconsistent-snapset <pgid> --format=json-pretty

# Confirm whether auto-repair is enabled
ceph config get osd osd_scrub_auto_repair
ceph config get osd osd_scrub_auto_repair_num_errors

# Show OSD tree up/down/in/out state for the acting set
ceph osd tree

# Check SMART for the device backing each OSD in the acting set
smartctl -a /dev/<device>

# Look for kernel-level media or link errors
dmesg -T | grep -iE 'error|ata|nvme|medium' | tail -100

An empty inconsistents array from rados list-inconsistent-obj does not mean the PG is clean. It means no object-level inconsistency was found at the level this command reports. Always also run rados list-inconsistent-snapset. If both return empty but the PG stays inconsistent+failed_repair, examine the full ceph pg <pgid> query output for state details.

How to diagnose it

1. Confirm the PG and its state string. ceph pg <pgid> query should show active+clean+inconsistent+failed_repair. Note the acting set: these are the OSDs you need to investigate.

2. Pull the inconsistency report. Run both rados list-inconsistent-obj and rados list-inconsistent-snapset. Save the output. This is your evidence.

3. Inspect each object listed. For each inconsistent object, the report includes a shards section showing each replica’s errors and hash. Errors like read_error, checksum_error, size_error, or digest_error tell you what kind of divergence you have.

4. If you see read_error on any shard, treat that as physical storage failure first. Run dmesg and smartctl -a on the host of the OSD reporting read_error. Do not attempt software repair before ruling out (or accepting) hardware failure. A failing disk will keep producing new inconsistencies.

5. Cross-check digests. For each shard in the report, the data_digest (or omap_digest) field tells you which replicas agree. If two of three replicas agree, the third is the corrupt one. If all three disagree, you have no authoritative candidate, and Ceph will refuse to pick.

6. Map objects to OSDs. The shard number in the report corresponds to the position in the acting set. Shard 0 is the primary, shard 1 is the first replica, and so on. This tells you which OSD’s data to trust or distrust.

7. Verify the primary. The primary coordinates repair and is the default source when no majority digest exists. If the primary’s disk has SMART errors or read_error, do not trust it as authoritative.

8. Determine whether the inconsistency is data-only or includes the snapset. Snapset inconsistencies can require different handling and may block snapshot operations until resolved.

Metrics and signals to monitor

Fixes

There is no safe one-size-fits-all repair. Every path below assumes you have already identified the authoritative replica through the diagnosis above.

Repair when the primary is correct and one replica is corrupt

This is the easy case. If two of three replicas agree and the primary is among them, re-running ceph pg repair <pgid> should succeed.

# Re-attempt repair after confirming the primary is correct
ceph pg repair <pgid>

Watch ceph_pool_objects_repaired increment, then verify the PG returns to active+clean. If it does not, the primary was probably the corrupt copy (next case).

Repair when the primary is the corrupt copy

pg repair uses the primary as the default source. If the primary is corrupt, you must remove the corrupt object from the primary OSD so that recovery pulls it from a known-good replica. This requires ceph-objectstore-tool and the OSD must be stopped. The procedure is destructive and irreversible.

Warning: Stopping the primary OSD triggers repeering and temporarily marks the PG degraded. On size-2 or EC pools, this reduces redundancy further. Verify the acting set has other healthy replicas before proceeding.

# Get the object ID from your earlier diagnosis output
# (rados list-inconsistent-obj <pgid>)

# Stop the OSD whose copy is corrupt (the primary in this case).
# Unit name varies by deployment: ceph-osd@<id> (legacy) or ceph-<fsid>@<id> (cephadm).
systemctl stop ceph-osd@<id>

# Confirm the OSD is down
ceph osd tree | grep osd.<id>

# Use ceph-objectstore-tool to remove the specific object.
# Read man ceph-objectstore-tool first. Syntax varies by Ceph release.
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-<id> --op remove --pgid <pgid> <object-id>

# Restart the OSD. Recovery will pull the missing object from a good replica.
systemctl start ceph-osd@<id>

The tool refuses to operate on a running OSD, returning Mount failed with '(11) Resource temporarily unavailable'. That error means the OSD is still up: stop it and retry.

Repair when all replicas disagree (no authoritative copy)

This is the hard case. Ceph cannot pick, and neither can the cluster. Your options, in increasing order of pain:

1. Look outside the acting set. If the PG was recently remapped, a previous acting-set member may still hold an older, uncorrupted copy. Check ceph pg <pgid> query for the history and probe those OSDs.

2. Restore from a higher-level backup (RBD image-level snapshot, RGW versioning, application backup). This is the cleanest path if it exists.

3. Accept loss. ceph pg <pgid> mark_unfound_lost revert reverts to the older version of the object (useful when an older version is known good). ceph pg <pgid> mark_unfound_lost delete removes the object entirely. Either choice is a data-loss decision. Document it.

The action argument to mark_unfound_lost is required. Running it without revert or delete will fail.

Replace the failing hardware

If SMART, dmesg, or read_error shards point at a specific disk, replace it before further repair work. A failing disk will keep generating new inconsistencies, and any “successful” repair on top of failing media is provisional.

The standard replacement path: mark the OSD out, wait for backfill to complete, mark it down, destroy it, replace the device, redeploy. Do not skip the backfill wait.

FileStore-specific notes (legacy only)

If you are still on FileStore, the manual repair procedure involves locating and removing the object file from the filesystem directly. This does not work on BlueStore. The FileStore repair walkthrough in the official Ceph docs is explicitly marked as not applicable to BlueStore OSDs. BlueStore has its own internal checksums and requires ceph-objectstore-tool for any object-level manipulation.

Prevention

• Run deep-scrub on schedule. failed_repair only surfaces because scrub found the inconsistency. Suppressing scrub indefinitely (via noscrub/nodeep-scrub) means corruption accumulates undetected. Track deep-scrub recency as an operational metric.

• Replace disks proactively on SMART signals. Reallocated sectors, current pending sectors, and offline uncorrectable are leading indicators. A disk that produces one inconsistency will produce more.

• Use ECC RAM. Non-ECC memory is a documented cause of mystery inconsistencies that look like disk corruption but move around the cluster.

• Decide deliberately on osd_scrub_auto_repair. On a healthy, well-monitored cluster, auto-repair can fix single-replica inconsistencies before you see them. Without good SMART monitoring, leave it off: auto-repair on top of failing hardware can cascade.

• Avoid the noout trap. noout left set after maintenance means degraded PGs are not recovered, which extends the window in which a second failure turns inconsistent into incomplete or down.

• Version-pin your repair playbooks. The exact ceph-objectstore-tool syntax and the JSON shape of ceph pg dump change across releases. Any scripts you write for repair triage need version awareness.

How Netdata helps

During a failed_repair incident, correlate these per-second, per-pool metrics to understand the escalation and confirm your repair is working.

• ceph_pg_failed_repair (per pool): Any nonzero value means Ceph has given up on repair for that PG. Alert on any nonzero value.

• ceph_pg_inconsistent (per pool): The precursor state. Correlate the timeline with failed_repair to confirm the escalation path (inconsistent, then failed repair after a repair attempt).

• ceph_health_detail{name="OSD_SCRUB_ERRORS"} and PG_DAMAGED: Exposed as health-check labels. Alert on these specific checks rather than the umbrella HEALTH_WARN so the right responder gets paged.

• Per-OSD apply and commit latency: A degrading disk often shows rising latency before it starts producing checksum errors. Cross-reference latency outliers with SMART attribute changes on the same host to identify the failing disk before the next scrub run.

• ceph_pool_objects_repaired (counter): Confirms successful repairs elsewhere in the cluster. A flat counter alongside nonzero failed_repair means normal healing is broken for that specific PG.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph blocked ops: client I/O stuck behind a single slow OSD
• Ceph BlueStore RocksDB compaction stalls: periodic latency spikes
• Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device
• Ceph BlueStore allocator fragmentation: rising latency at moderate fullness
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph client latency vs OSD latency: fast disks, slow clients
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph MON_CLOCK_SKEW: clock drift between monitors and election churn