Ceph backfill_toofull: recovery blocked because target OSDs are full

A PG in backfill_toofull is not making progress toward active+clean. The source OSD has the data, CRUSH has chosen the target, but the target refuses the reservation because it is above backfillfull_ratio. Client reads and writes still succeed; the redundancy gap is not closing.

The cluster is usually not at the hard full ratio (0.95 default). It is at backfillfull (0.90 default) on one or more individual OSDs, and that is enough to halt recovery for every PG whose up set includes them. Symptoms: HEALTH_WARN (or HEALTH_ERR on older releases), a flat degraded-object count, and recovery bytes/sec near zero.

The fix is always capacity headroom: delete data, add OSDs, or reweight the fullest OSDs down so CRUSH stops placing new replicas on them. Below: how to confirm the state, identify the blocking OSDs, and pick a remediation that matches the urgency.

What this means

Ceph enforces three ascending per-OSD capacity thresholds:

They must remain ascending: nearfull < backfillfull < full < failsafe_full. Violating the order trips OSD_OUT_OF_ORDER_FULL.

When a PG needs backfill (OSD failure, reweight, CRUSH change, autoscaler activity), the primary requests a backfill reservation on each target OSD in the new up set. If any target is at or above backfillfull_ratio, the reservation is rejected. The primary drops its local reservation, waits osd_backfill_retry_interval (default 30s), and retries. The PG stays in backfill_toofull until the target frees space or the threshold moves.

The sibling state recovery_toofull is similar but stricter.

Seeing both together means the cluster is essentially at the write-blocking cliff, not just the recovery-blocking one.

backfill_toofull was demoted from HEALTH_ERR to HEALTH_WARN in Ceph v15.2.0 (PR #28204). Nautilus and earlier surface it as ERR.

flowchart TD
    A[OSD fails or rebalance starts] --> B[Primary requests backfill on target OSDs]
    B --> C{Target OSD utilization}
    C -->|Below backfillfull 0.90| D[Backfill proceeds, PG heals]
    C -->|At or above backfillfull 0.90| E[Reservation rejected]
    E --> F[PG enters backfill_toofull]
    F --> G[Waits osd_backfill_retry_interval 30s]
    G --> B
    F --> H[Degraded PG count flat, recovery bytes per sec near zero]

Common causes

The transient case is by design. While PGs are being moved off an OSD, the monitor cannot count those outgoing PGs as freed space because backfill order is unpredictable and something could fail mid-stream. The OSD may appear too full for incoming backfills until outgoing backfills complete.

Quick checks

All read-only. Safe on a production MON at any time.

# Confirm the active health checks and message text
ceph health detail | grep -E 'BACKFILL|TOOFULL|OSD_'

# Per-OSD utilization. Sort interactively; UTIL column index varies by release.
ceph osd df tree

# PGs stuck in non-clean states, with their current state string
ceph pg dump_stuck unclean

# Identify which OSDs each stuck PG is trying to backfill onto
ceph pg <pgid> query | jq '[.peering, .recovery_state]'

# Confirm no recovery-disabling flags are set
ceph osd dump | grep flags

# Recovery rate and degraded count
ceph pg stat

# Cluster-wide capacity and pool breakdown
ceph df detail

If ceph pg dump_stuck unclean shows PGs in active+remapped+backfill_toofull or active+degraded+backfill_toofull, the diagnosis is confirmed. Cross-reference those PG IDs against ceph osd df to find the OSDs above 90%.

How to diagnose it

1. Confirm the state is backfill_toofull, not full. Run ceph health detail and check whether OSD_FULL is active. If it is, writes are failing, not just recovery. Switch to the OSD_FULL playbook.

2. Identify the blocking OSDs. Cross-reference the stuck PGs from ceph pg dump_stuck unclean with per-OSD utilization in ceph osd df tree. The OSDs above backfillfull_ratio (default 0.90, but check the configured value) are the ones rejecting reservations.

3. Verify the configured threshold. ceph osd dump exposes full_ratio, backfillfull_ratio, and nearfull_ratio from the OSDMap. The authoritative command to change it is ceph osd set-backfillfull-ratio <value>. The mon_osd_backfillfull_ratio config option only applies at cluster creation; afterwards the OSDMap value wins.

4. Rule out false positives. If every target OSD is clearly below the configured backfillfull ratio and the PG is still flagged, check the Ceph version. Older releases had a known false-positive (Bug #39555, fixed in v15.2.0) where monitors counted all PGs remapped to an OSD, projected their total size, and flagged backfill_toofull even on a 35%-full OSD. A separate report (Bug #61839) describes the same symptom on Pacific 16.2.7 with OSDs well below a 0.97 ratio, attributed to too few PGs per OSD on large drives.

5. Check whether the situation is transient. If an OSD is actively being evacuated, wait one or two osd_backfill_retry_interval cycles (default 30s each) and recheck. Outgoing backfills freeing space will eventually unblock incoming ones.

6. Quantify the gap. Compare current per-OSD utilization against the threshold. If the worst OSD is at 91% and the threshold is 90%, a small reweight or a small deletion will clear it. If the worst OSD is at 94%, the cluster needs real capacity added.

Metrics and signals to monitor

Fixes

Free space on the blocking OSDs

Lowest-risk remediation. Delete snapshots, run RGW garbage collection (radosgw-admin gc process if RGW is deployed), delete non-critical data, or flatten old RBD snapshots. The goal is to push the blocking OSDs below backfillfull_ratio so the existing retry loop picks the reservation back up.

Only works if there is genuinely deletable data on the affected OSDs. Cluster-wide deletion may not move the needle on a specific OSD if CRUSH places new writes back onto it.

Reweight the fullest OSDs down

Two distinct commands. Know which one you are running:

# Temporary reweight override (0..1). CRUSH placement changes; does not persist
# across restart. Use this for the short-term "stop placing new data here" case.
ceph osd reweight <osd.id> 0.8

# Permanent CRUSH weight change. Persists across restarts.
ceph osd crush reweight osd.<id> 0.8

ceph osd reweight tells CRUSH to place fewer new objects on that OSD. Existing data migrates off through normal recovery. As utilization drops below backfillfull, the stuck PGs unblock. ceph osd crush reweight changes the underlying CRUSH weight.

Both trigger additional data movement, which competes with the recovery you are trying to unblock. Step the weight down gradually (0.9, then 0.8, then lower if needed) and watch recovery rate and client latency. Once the OSD is below threshold and recovery completes, restore the original weight if the underlying capacity is unchanged.

Add OSDs

The structural fix. New OSDs shift CRUSH placement, giving backfill new targets that are well below backfillfull. The only durable remediation when the cluster is genuinely out of capacity.

Provisioning lead time is the constraint. Adding OSDs also triggers a large rebalance, which consumes I/O and network. Consider tuning osd_max_backfills (default 1) and osd_recovery_max_active (default 3) if client impact is a concern.

On Quincy and later with mClock active, osd_max_backfills may be overridden unless osd_mclock_override_recovery_settings = true.

Tune backfillfull_ratio (cautiously)

# Raise the threshold (example: from 0.90 to 0.92)
ceph osd set-backfillfull-ratio 0.92

Buys time when the cluster is genuinely close to the threshold and a small reweight or deletion is imminent. It does not add capacity; it just lets backfill place data on OSDs that were previously off-limits.

Riskiest fix. Pushing backfillfull closer to full narrows the window between recovery-blocked and writes-blocked. Operators report the new threshold takes 10 to 15 minutes to fully propagate, and in some cases requires restarting the relevant OSDs. Never raise backfillfull above full, and never raise full itself unless you are in an actual write-stopped emergency with a deletion or OSD-add already in flight.

Manual upmap (last resort)

# Force a specific PG off the blocking OSD onto another
ceph osd pg-upmap-items <pgid> <old_osd> <new_osd>
# Remove the upmap entry once backfill completes

Reported as a successful workaround for stuck false-positive cases. It manually moves a single PG, bypassing the reservation logic.

Sharp-edged. You are overriding CRUSH placement, which can defeat failure-domain isolation if the target OSD shares a host or rack with another replica. Use only for individual stuck PGs after the capacity path is exhausted, and remove the upmap entry as soon as the PG is clean.

Prevention

• Plan capacity against failure scenarios, not averages. Model “if I lose the largest host, do the surviving OSDs stay below backfillfull?” The cluster needs enough headroom to absorb a full failure domain of recovery. Stay at least 20% below backfillfull under normal operation.
• Monitor per-OSD utilization, not cluster averages. A cluster at 75% average can have OSDs at 91%. CRUSH does not guarantee even distribution. Alert on the worst OSD, not the mean.
• Track recovery rate alongside degraded counts. A flat ceph_num_objects_degraded with ceph_pool_recovering_bytes_per_sec near zero is the leading indicator of a stalled recovery, regardless of cause. Capacity is only one of several stall modes.
• Keep the three thresholds ascending and sensible. OSD_OUT_OF_ORDER_FULL is a self-inflicted wound from careless ratio edits. Defaults (0.85 / 0.90 / 0.95) exist for a reason; deviate only with a documented capacity plan.
• Watch the noout trap. A long-forgotten noout combined with an OSD failure produces exactly the conditions (degraded PGs, stalled recovery) that surface backfill_toofull when capacity is tight. ceph_osd_flag_noout set for more than 24 hours without a maintenance ticket is itself an alert.

How Netdata helps

• ceph_pg_backfill_toofull and ceph_pg_recovery_toofull, collected per pool, are the direct signal that recovery is blocked by target OSD capacity rather than by I/O contention or a forgotten flag.
• ceph_health_detail rows for OSD_BACKFILLFULL and PG_BACKFILL_FULL correlate the health-check view with the raw PG-state view, helping distinguish a real capacity stall from a transient retry cycle.
• Per-second ceph_pool_recovering_bytes_per_sec next to ceph_num_objects_degraded makes a flat recovery curve obvious within minutes, before anyone has to run ceph pg dump_stuck manually.
• ceph_osd_full_ratio and ceph_osd_nearfull_ratio expose the configured thresholds from the OSDMap, so alerts fire against what the cluster actually enforces rather than hardcoded defaults.
• ML anomaly detection on per-OSD utilization surfaces the OSD drifting toward backfillfull faster than its peers, often the first warning before any PG state flips.

Related guides

• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert
• Ceph OSD_FULL: all writes stopped at the 95% full ratio