Ceph PG down: no surviving replica for reads or writes

A placement group in the down state has no replica that can serve I/O. Reads and writes to objects in that PG fail. Clients see EIO or ENODEV, and ceph health detail reports PG_AVAILABILITY with one or more PGs flagged down. Data is unavailable, not just degraded.

A PG can briefly pass through down during peering after an OSD failure, before activating on a surviving replica. The 300-second sustain on sum(ceph_pg_down) > 0 filters that transient window. Once a PG has been down past the sustain, no OSD in the acting set can serve it, and the cluster will not heal it without operator action.

This article covers how to distinguish a real down PG from adjacent states, how to read peering_blocked_by from ceph pg <pgid> query, and the recovery paths available.

What this means

The down PG state means a replica carrying the data needed to serve the PG is offline. The PG cannot peer to a usable acting set, so neither reads nor writes succeed. Contrast this with adjacent states operators often confuse:

• peered: the PG has enough OSDs to form a peer relationship, but not enough to reach the pool’s min_size. Recovery can still run and may heal the PG to min_size automatically. I/O is blocked, but the cluster knows what to do.
• incomplete: the PG is missing write history or has no healthy copy. This is a stronger failure than down and usually indicates data loss is already in play.
• stale: the monitor has not heard from the PG’s primary. The PG may still be serving I/O from the OSD’s perspective; the monitor just does not know.

down is the state where the cluster has a concrete blocked peer dependency: a specific OSD (or set of OSDs) holds the interval the PG needs, and that OSD is not reachable. The recovery question is always: can you bring that OSD back, and if not, what do you accept losing?

flowchart TD
    A[PG fails to peer] --> B{Replica with needed data reachable?}
    B -- yes --> C[activating then active]
    B -- no, OSD down --> D[PG state: down]
    B -- no, history missing --> E[PG state: incomplete]
    D --> F{Bring OSD back?}
    F -- yes, restart daemon --> G[peering resolves, PG active]
    F -- no, disk gone --> H[Accept loss: ceph osd lost]
    H --> I[PG peers past missing history or becomes incomplete]

Common causes

Quick checks

Run these read-only. None modify cluster state.

# Triage: list stuck inactive PGs (includes down PGs)
ceph pg dump_stuck inactive

<!-- TODO: verify whether 'down' is a valid dump_stuck filter keyword; 'inactive' is the documented catch-all -->

# Get the precise PG IDs from health
ceph health detail | grep -A3 PG_AVAILABILITY

# Inspect one PG's peering detail
ceph pg <pgid> query | jq '.recovery_state.peering_blocked_by'

# See the acting set vs up set for the PG
ceph pg <pgid> query | jq '.acting, .up'

# Check which OSDs are down
ceph osd tree | grep down

# Verify the OSD daemon state from the cluster's view
ceph osd dump | jq '.osds[] | select(.up == 0) | {osd, up, in, weight}'

# Confirm no global flags are blocking recovery
ceph osd dump | grep flags

# Confirm MON quorum and clock health (peering needs a stable MON)
ceph quorum_status
ceph time-sync-status

# Confirm whether the pool is erasure coded (changes failure semantics)
ceph osd pool ls detail | grep <poolname>

The peering_blocked_by output from ceph pg <pgid> query is the single most important diagnostic. It names the OSD IDs whose absence is blocking peering. If it lists OSDs that are genuinely down in ceph osd tree, the path is to bring them back. If it lists OSDs that are up, you are looking at a peering stall (see below).

How to diagnose it

1. Identify every down PG and how long it has been stuck. Use ceph pg dump_stuck inactive and filter for down. Anything under 300 seconds may still be transient peering; confirm with a second sample a minute later.
2. For each stuck PG, pull peering_blocked_by. ceph pg <pgid> query | jq '.recovery_state.peering_blocked_by' tells you which OSD IDs are blocking. This is the root of the diagnosis.
3. Cross-reference against OSD state. For each OSD listed, check ceph osd tree and the daemon’s systemd unit on its host. Determine whether the OSD is down because the daemon stopped, the host is down, or the disk failed.
4. Check the acting set vs up set. ceph pg <pgid> query | jq '.acting, .up'. If the acting set contains NONE positions, the PG cannot find a valid peer. Common on erasure coded pools after multiple failures.
5. Verify no flags are blocking recovery. ceph osd dump | grep flags. norecover and nobackfill do not cause down directly, but they prevent healing once the blocked OSD returns. noout can mask a permanently failed disk by keeping it down+in rather than down+out.
6. Check MON and clock health. ceph quorum_status and ceph time-sync-status. Peering requires MON agreement on the OSD map. Clock skew beyond mon_clock_drift_allowed (default 0.05s) can destabilize elections and stall peering broadly, not just on one PG.
7. If peering_blocked_by is empty but the PG is still down, look at requested_info_from in the same query output. On Reef 18.2.x there is a known pattern where this references an OSD that is up, but peering never completes. A targeted restart of that OSD typically unsticks it.
8. For erasure coded pools, verify the EC profile matches the failure count. EC recovery requires k chunks to reconstruct. If the acting set is below k, the PG cannot serve reads.

Metrics and signals to monitor

Fixes

Bring the blocked OSD back

If peering_blocked_by names OSDs that are down in ceph osd tree, the safest option is to restart those OSD daemons. This works when the disk is intact and the daemon simply stopped, OOM-killed, or the host rebooted.

# On the OSD host
systemctl start ceph-osd@<id>

# Verify it rejoined
ceph osd tree | grep osd.<id>

Once the OSD is up, the PG should transition out of down through activating to active within seconds to minutes, depending on PG log replay. Watch ceph pg <pgid> query and ceph_pool_recovering_bytes_per_sec to confirm progress.

If the OSD comes back but the PG stays down, the OSD may not actually hold the interval the PG needs (the log was truncated, or an earlier failure lost it). At that point you are looking at incomplete rather than down, and the recovery path is harder.

Clear a peering stall by restarting the referenced OSD

On Reef 18.2.x there is a documented pattern where peering_blocked_by is empty but requested_info_from names an OSD that is already up. The peering exchange is stuck on that OSD. A targeted restart of the referenced OSD daemon typically clears it.

This is non-destructive: the OSD’s data is intact, you are only restarting the daemon to reset its peering state. Confirm after restart that the PG transitions to activating.

Unset noout if it is masking a dead OSD

If noout has been set for an extended period and an OSD has failed permanently, the OSD stays down+in and the cluster never marks it out to trigger remapping.

# Confirm the flag
ceph osd dump | grep noout

# Clear it once maintenance is actually complete
ceph osd unset noout

After unsetting, Ceph waits mon_osd_down_out_interval (default 600s) before marking the OSD out, which triggers remapping. If the data was already recovered elsewhere, the PG can peer past the dead OSD.

Mark an OSD as lost

Destructive. Last resort. Ceph’s own documentation warns that ceph osd lost “cannot guarantee consistency.” Only use this when the OSD’s disk is permanently gone and no other path remains.

# Confirm the OSD is down and out
ceph osd tree | grep osd.<id>

# Mark it lost (irreversible)
ceph osd lost <id> --yes-i-really-mean-it

This tells the cluster to peer past the missing history. Outcomes vary:

• If the surviving replicas had enough information, the PG peers and recovers.
• If the lost OSD held the only authoritative copy of some interval, affected PGs transition to incomplete or surface unfound objects.

Snapshot the current state (OSD map, ceph pg dump, ceph health detail) before running it, so you can investigate any objects that later appear unfound. Mark-lost should be the last action after every other recovery option is exhausted.

Handle unfound objects after a loss

If marking an OSD lost surfaces unfound objects, recovery is blocked until you decide what to do with them.

# Review the unfound objects per PG
ceph health detail | grep unfound

# Decide per PG: revert to a known older version, or delete
ceph pg <pgid> mark_unfound_lost revert
# or
ceph pg <pgid> mark_unfound_lost delete

revert rolls back to the last known consistent version of the object. delete removes the object entirely. Both represent data loss; the choice depends on whether stale data is preferable to no data for the workload.

Force-create a PG (size=1 pools only)

For pools with size=1 and no redundancy, Ceph documents ceph osd force-create-pg <pgid> as a last-resort recovery. Not applicable to replicated pools with size >= 2 or erasure coded pools, and the docs note the PG may revert to incomplete if the underlying OSD data is actually missing.

Prevention

• Do not leave noout set. This is the single most common preventable cause of PGs that cannot recover. Set noout for maintenance, unset it when maintenance ends. Alert on ceph_osd_flag_noout == 1 for more than 24 hours.
• Keep capacity below backfillfull on the worst OSD, not the cluster average. Recovery needs somewhere to go. A cluster whose fullest OSD is past backfillfull cannot heal from the next failure.
• Monitor per-OSD, not cluster averages. A down PG almost always traces back to specific OSDs. Cluster-level OSD counts hide the one OSD whose failure dropped a PG below min_size.
• Verify EC pool sizing against failure domains. Erasure coded pools fail down when the acting set drops below k. Ensure CRUSH failure domains match the redundancy the EC profile assumes.
• Keep MON clocks synchronized. mon_clock_drift_allowed is 0.05s by default. Peering stalls cascade when MON elections flap.
• Track recovery rate, not just degraded state. A down PG that clears but leaves degraded PGs with zero recovery rate is still a problem. Correlate ceph_pg_down with ceph_pool_recovering_bytes_per_sec.

How Netdata helps

• Per-second ceph_pg_down per pool shows the exact second a PG went down and when it cleared, instead of sampling against ceph pg dump on a slow interval.
• Correlate ceph_pg_down with ceph_osd_up transitions in the same chart view. When a PG goes down, the OSD that took it down is visible in the same timeframe, removing the cross-referencing step from triage.
• ceph_osd_flag_noout as a first-class metric surfaces the noout trap directly, rather than requiring an operator to remember to check ceph osd dump.
• ML anomaly detection on ceph_healthcheck_slow_ops and ceph_pool_recovering_bytes_per_sec flags peering stalls and stalled recovery before they trip a static threshold.
• ceph_health_detail labels per check let you alert on PG_AVAILABILITY specifically, rather than the umbrella HEALTH_ERR, with the check name and severity attached.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert
• Ceph OSD down: telling a dead disk apart from a network blip
• Ceph OSD flapping: OSDs cycling up and down and the peering storm that follows
• Ceph OSD_FULL: all writes stopped at the 95% full ratio
• Ceph OSD fullness imbalance: one OSD full while the cluster average looks fine