Ceph unfound objects: the cluster cannot locate a surviving copy

When a placement group enters recovery_unfound or backfill_unfound, the cluster has objects it knows should exist but cannot locate on any OSD that is currently up and in. ceph health detail surfaces this as the OBJECT_UNFOUND check, and the cluster-wide gauge ceph_num_objects_unfound rises above zero. Every known replica or erasure-coded chunk is on an OSD that is down, destroyed, or has not yet been probed.

Recovery of the affected PGs is blocked at the unfound object. Client reads and writes to those objects stall indefinitely rather than erroring out. The only paths forward are to bring back the missing data or to explicitly accept the loss.

The 300-second sustain on ceph_num_objects_unfound is deliberate. After a cluster-wide restart, PGs can briefly appear unfound while OSDs rejoin and peering completes. Unfound persisting past that window is real, and the rest of this article assumes you are past the cold-start window.

What this means

An object becomes “unfound” when the PG’s recovery process has exhausted the set of OSDs it believes might hold a copy (might_have_unfound) and none of them has the object. The PG cannot complete recovery, so it stays in recovery_unfound or backfill_unfound. Client I/O to the specific unfound objects blocks indefinitely. The rest of the PG and the rest of the cluster continue to serve I/O normally.

Unfound is not the same as inconsistent (replicas differ but a copy exists) or degraded (fewer copies than configured but at least one is available). Unfound means the cluster has no readable copy it can reach right now.

Common causes

Quick checks

Safe, read-only commands. Run them in order.

# Confirm the unfound condition and get the affected PG IDs
ceph health detail

# List the unfound objects for a specific PG, with need/have versions and locations
ceph pg <pgid> list_unfound

# Full PG query; inspect might_have_unfound and the acting set
ceph pg <pgid> query

# Which OSDs are down right now
ceph osd tree | grep down

# Recovery flags that could be silently blocking progress
ceph osd dump | grep flags

# Capacity on target OSDs (backfill_toofull can look similar from the dashboard)
ceph osd df tree

# Stuck PGs across all categories, for blast-radius context
ceph pg dump_stuck unclean

How to diagnose it

1. Confirm the PG is genuinely stuck on unfound, not transiently peering. If the cluster restarted in the last 5 to 10 minutes, wait. Unfound during cold-start peering usually clears as OSDs rejoin.

2. Get the list of affected PGs from ceph health detail or ceph pg dump_stuck unclean. Cross-reference with PG state metrics: ceph_pg_recovery_unfound and ceph_pg_backfill_unfound tell you which pools have PGs blocked at the unfound stage.

3. For each affected PG, run ceph pg <pgid> list_unfound. The output shows num_missing, num_unfound, the object IDs (oid), the need and have versions, and any locations the cluster has recorded.

4. Run ceph pg <pgid> query and inspect might_have_unfound. This is the list of OSDs the cluster thinks might still hold a copy. Each entry has a probe status. If any entry shows the OSD has not been queried yet, or the OSD is down, the cluster has not finished probing and mark_unfound_lost will refuse to proceed.

5. Cross-reference the PG acting set and the might_have_unfound list with ceph osd tree. Determine which OSDs are down and whether they are recoverable (host reboot, daemon restart) or destroyed (disk failure, host decommissioned).

6. Check whether any down OSD can be brought back. If it can, do that first. Bringing back the OSD that holds the only surviving copy is the only resolution that does not lose data.

7. If the unfound objects have oid values starting with .ceph-internal::hit_set_, you are looking at the deprecated cache tiering hit_set archive case. This is a known source of spurious unfound objects across many Ceph versions, and the standard fix is different (see Fixes below).

flowchart TD
    A[ceph_num_objects_unfound > 0] --> B{Cold start in last 10 min?}
    B -- yes --> C[Wait, recheck]
    B -- no --> D[query PG: list_unfound + might_have_unfound]
    D --> E{All sources probed?}
    E -- no --> F[Restart down OSDs, wait for probe]
    E -- yes --> G{Down OSD recoverable?}
    G -- yes --> H[Bring it back, let recovery run]
    G -- no --> I{hit_set archive oid?}
    I -- yes --> J[mark_unfound_lost delete]
    I -- no --> K[mark_unfound_lost revert/delete, accept loss]
    F --> D

Metrics and signals to monitor

Severity guidance: ceph_num_objects_unfound > 0 is a TICKET at 300 seconds. Escalate to PAGE only if you have restart-awareness context (no OSD restarted in the last 600 seconds), because cold-start peering can produce transient unfound. If unfound co-occurs with ceph_pg_incomplete or ceph_pg_down, those signals already carry their own PAGE conditions at 300 seconds.

Fixes

Recover the OSD holding the copy (preferred)

If any OSD in might_have_unfound is down but its data is intact, bring it back. Restart the daemon, reboot the host, remount the disk. When the OSD rejoins and completes peering, the unfound object reappears and recovery proceeds normally.

This is the only resolution that preserves the data. Everything below this point accepts some loss.

Clear the probe backlog first

If mark_unfound_lost fails with a message like “pg has N unfound objects but we haven’t probed all sources, not marking lost”, the cluster has not finished probing every OSD in might_have_unfound. Either wait for probing to complete, or restart the OSDs listed as not-yet-queried or down so the probe can run. You cannot mark objects lost until probing is done.

Mark the unfound objects as lost

When probing is complete and no copy is recoverable, the explicit loss path is:

# Destructive: accepts data loss. Read the rest of this section first.
ceph pg <pgid> mark_unfound_lost revert|delete

The argument is required.

• delete forgets the object entirely. Subsequent reads return ENOENT.
• revert rolls the object back to a previous known version, or forgets it if it was new. revert is not available for erasure-coded pools.

Known gotcha (Tracker #19377): if num_missing == num_unfound for the PG, mark_unfound_lost revert can add log entries without actually recovering the objects, because the recovery path needs at least one found object to trigger. The documented workaround is to create a missing object that is not unfound (for example, take down the primary, write a single new object, bring the primary back) so the recovery loop has something to iterate over.

ceph-objectstore-tool fix-lost (low level)

For cases where you want to repair lost objects on a specific OSD’s store directly:

# Destructive and requires the OSD to be stopped first.
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-<id> --op fix-lost

You can scope it with --pgid <pgid> or a specific object ID. This is a low-level tool and requires the OSD to be stopped, which means taking it out of the acting set temporarily.

Cache tier hit_set archive objects

If the unfound objects are .ceph-internal::hit_set_*_archive objects from a deprecated cache tier, the established workaround is ceph pg <pgid> mark_unfound_lost delete on those specific objects. Field reports indicate this is safe because these are internal bookkeeping objects, not user data. Do not deploy new cache tiers: cache tiering was deprecated in the Reef release and is being removed.

ceph osd lost is a different and dangerous action

Do not confuse mark_unfound_lost with ceph osd lost <id> --yes-i-really-mean-it. The latter tells the cluster to proceed without a known-good copy of everything that was on that OSD. The official docs warn that the cluster cannot guarantee the remaining copies are consistent and up to date. Use it only when you have already accepted the loss and have no other path.

Prevention

• Do not run at capacity levels that prevent recovery. At backfillfull (default 0.90), target OSDs refuse backfill data and recovery stalls. A cluster that cannot heal is one failed OSD away from unfound objects. See Ceph capacity death spiral.
• Do not leave noout set indefinitely. noout keeps dead OSDs in CRUSH, blocks auto-out, and extends the window in which a second failure creates unfound objects.
• Replace down OSDs promptly. The faster a down OSD is replaced or its data redistributed, the smaller the window for a cascading failure to create unfound objects.
• Retire cache tiering. If you still have cache tiers, plan their removal. Hit_set archive objects are a known source of spurious unfound across many Ceph versions.
• Keep deep scrub running. Unfound is about availability, but inconsistent replicas are about integrity. A PG that is both inconsistent and unfound has no good resolution path. See Ceph health detail.

How Netdata helps

• Per-second ceph_num_objects_unfound: collected every second, so you see the exact moment unfound objects appear and whether the count is flat, rising, or clearing.
• PG state correlation: ceph_pg_recovery_unfound and ceph_pg_backfill_unfound are collected alongside ceph_pg_incomplete and ceph_pg_down, so you can tell whether unfound has escalated to a PG-level availability failure without a separate query.
• OSD transition context: ceph_osd_up and ceph_osd_in transitions are timestamped, giving you the restart-awareness context the playbook recommends for deciding whether to escalate unfound from TICKET to PAGE.
• Flag monitoring: ceph_osd_flag_noout, ceph_osd_flag_norecover, and ceph_osd_flag_nobackfill are tracked as gauges, so a forgotten noout or a stalled recovery flag is visible next to the unfound count.
• Anomaly context: unfound is normally zero, so any non-zero value triggers anomaly scoring. Correlation with co-occurring OSD restart and flag state helps distinguish a cold-start blip from a genuine loss condition.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph blocked ops: client I/O stuck behind a single slow OSD
• Ceph BlueStore RocksDB compaction stalls: periodic latency spikes
• Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device
• Ceph BlueStore allocator fragmentation: rising latency at moderate fullness
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph client latency vs OSD latency: fast disks, slow clients
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph MON_CLOCK_SKEW: clock drift between monitors and election churn