Ceph OSD_FULL: all writes stopped at the 95% full ratio

The cluster suddenly stops accepting writes. Client applications report ENOSPC errors. ceph status returns HEALTH_ERR. ceph health detail shows OSD_FULL active. Reads still succeed, but every write, update, and delete fails cluster-wide.

This is a hard stop, not a throttle. Ceph refuses all write operations once any OSD crosses the configured full_ratio (default 0.95). CRUSH spreads every PG across multiple OSDs, so a single full OSD can block writes to hundreds of PGs even when cluster-average utilization looks moderate.

The condition cannot self-resolve. Capacity must be freed or added before writes resume. Recovery itself requires spare space, so a cluster at full cannot heal from an additional OSD failure. Treat this as an active incident.

What this means

Ceph applies capacity thresholds to every OSD. The thresholds are stored in the OSDMap and are configurable at runtime.

Once any OSD’s utilization reaches full_ratio, the cluster stops accepting writes for every PG whose acting set includes that OSD. Because CRUSH distributes PGs across OSDs, one full OSD typically blocks writes to a large fraction of PGs. Writes stop cluster-wide even if cluster-average utilization is well below 95%.

Reads continue from surviving replicas, which makes the failure deceptive: read-heavy workloads keep working while write-dependent applications error out.

flowchart TD
    A[Client write] --> B[CRUSH hashes object to PG]
    B --> C{Any OSD in PG acting set at full_ratio?}
    C -- yes --> D[Write rejected ENOSPC]
    C -- no --> E[Write accepted]
    F[Single OSD crosses 0.95] --> G[PGs with that OSD in acting set blocked]
    G --> D

A separate per-OSD failsafe at osd_failsafe_full_ratio (default 0.97) is enforced inside the OSD daemon. Even if you raise the cluster-wide full_ratio above 0.97, individual OSDs still refuse writes at their failsafe. Raising full_ratio above 0.97 buys little because the failsafe remains.

Common causes

Quick checks

Run these read-only checks first. None of them modify cluster state.

# Confirm the OSD_FULL health check is active and see which OSDs are full
ceph health detail | grep -A3 OSD_FULL

# Per-OSD utilization, hierarchical view. Sort by the UTIL column.
ceph osd df tree

# View the currently configured thresholds from the OSDMap
ceph osd dump | grep -E "full_ratio|nearfull_ratio|backfillfull_ratio"

# Cluster-wide raw usage and per-pool breakdown
ceph df detail

# Check whether recovery is blocked by capacity on target OSDs
ceph health detail | grep -E "TOOFULL|BACKFILL"

# Identify PGs stuck because target OSDs are too full
ceph pg dump_stuck unclean

# If RGW is deployed, inspect the garbage collection queue depth
radosgw-admin gc list --include-all | head -50

How to diagnose it

Run the quick checks above, then make one decision: is this imbalance or genuine capacity shortage?

• Imbalance: only a few OSDs are at full_ratio while the rest have headroom. Fix with CRUSH reweighting (below), not by adding capacity.
• Genuine shortage: all OSDs are near full. Fix by adding capacity or deleting data.

Three traps change effective capacity without obvious symptoms:

• OUT OSDs shrink the denominator. Marked-OUT OSDs subtract their capacity from the cluster total. A cluster can cross full_ratio without new writes if enough OSDs go OUT. Check ceph osd tree for out entries.
• Snapshots retain data invisibly. RBD and CephFS snapshots hold changed blocks. ceph df detail includes this in pool usage, but live object counts do not reflect it.
• RGW GC stalls. Deleted objects consume space until garbage collection reclaims them. If raw usage climbs despite deletions, check radosgw-admin gc list --include-all.

Metrics and signals to monitor

Fixes

Work through these in order of safety. The goal is to get below the full ratio on every OSD so writes resume, then address the root cause.

Free capacity by deleting data

The fastest unblock. Delete snapshots, old RBD images, orphaned RGW objects, or non-critical pool data. Confirm space is actually reclaimed with ceph osd df after deletion. If RGW is the source, deleting objects queues them for GC, which must then run before space is freed (see below).

Add OSDs or expand devices

Adding OSDs increases the denominator and lets CRUSH rebalance away from the full OSDs. This is the durable fix for genuine capacity shortage. Plan for the rebalance itself to consume I/O; on a cluster already at full, recovery may be slow.

Force RGW garbage collection

If RGW deleted objects but GC has not reclaimed the space, force it:

# Force-process the entire GC queue. Safe but I/O-intensive.
radosgw-admin gc process --include-all

This is read/write heavy on the OSDs hosting the .rgw.buckets data pool. Run it during a low-traffic window if possible, but on a cluster that is already full and blocking writes, reclaiming space is the priority.

Reweight unbalanced OSDs

If a small number of OSDs are full while the rest have headroom, lower the weight of the full OSDs to shift data away:

# Temporary reweight (0.0-1.0 utilization override).
# May be overwritten by automatic reweight-by-util.
ceph osd reweight <osd-id> 0.8

# Permanent CRUSH weight change.
# Weight is device-size-based (e.g., 3.0 for 3 TB), not a 0-1 ratio.
<!-- TODO: verify exact argument format and weight scale for this Ceph version -->
ceph osd crush reweight osd.<id> <weight>

ceph osd reweight is a temporary override stored in the OSDMap. ceph osd crush reweight changes the CRUSH weight permanently. Either triggers data movement off the reweighted OSD, which competes with client I/O. Reweighting one OSD can cascade: the recipients may approach their own limits. Watch ceph osd df as rebalancing progresses.

Last resort: raise the full ratio (dangerous)

If no data can be deleted and no capacity can be added immediately, you can raise the full ratio temporarily. This is a stopgap, not a fix, and it carries real risk:

# Change the runtime full ratio stored in the OSDMap
ceph osd set-full-ratio 0.96

Caveats:

• The failsafe at 0.97 still applies per-OSD. Raising the cluster ratio above 0.97 has no effect because the OSD daemon refuses writes at its own failsafe.
• Recovery becomes harder and may become impossible as you approach the failsafe.
• This buys hours, not days. The underlying capacity shortage remains.

mon_osd_full_ratio in ceph.conf is a bootstrap-only parameter. Changing it on a running cluster has no effect. The runtime threshold lives in the OSDMap and is modified with ceph osd set-full-ratio, ceph osd set-nearfull-ratio, and ceph osd set-backfillfull-ratio.

If the ratios are misordered (nearfull must be less than backfillfull, which must be less than full), Ceph raises OSD_OUT_OF_ORDER_FULL. Verify ordering after any change.

Prevention

• Alert on the most-full OSD, not the cluster average. A single OSD at 95% stops writes even when the average is 75%. Page on any OSD crossing nearfull (0.85).
• Keep cluster-average utilization at least 20% below backfillfull. Recovery after an OSD failure needs spare space. A cluster at 85% that loses one host can jump past backfillfull and lose the ability to heal.
• Monitor RGW GC rate. If your cluster runs RGW, track ceph_rgw_gc_retire_object. A near-full cluster with GC near zero is heading for OSD_FULL.
• Track snapshot growth. RBD and CephFS snapshots retain changed data invisibly. Audit snapshot age and count regularly.
• Model failure scenarios. Ask: if the largest host fails, do the surviving OSDs have room for recovery? Capacity planning that only looks at current utilization misses this.
• Watch for OUT OSDs shrinking usable capacity. A cluster can cross full without any new writes if OSDs are marked OUT and their capacity is subtracted.

How Netdata helps

• Per-second ceph_health_detail metrics surface OSD_FULL alongside OSD_NEARFULL and OSD_BACKFILLFULL, showing the capacity cascade as it develops.
• Per-OSD utilization is collected individually, not just as a cluster average. Alerting on the most-full OSD catches the single-OSD-at-95% case that cluster averages hide.
• ML anomaly detection on per-OSD fill rate flags accelerating growth before the threshold is crossed.
• Correlation between ceph_pg_backfill_toofull, recovery rate, and OSD utilization shows whether recovery is already capacity-blocked, the precursor state that precedes a full stop.
• For RGW deployments, the GC retire rate metric shows stalled garbage collection before it becomes an OSD_FULL incident.
• Configured threshold metrics (ceph_osd_full_ratio, ceph_osd_nearfull_ratio) are tracked alongside utilization, so alerts use the actual configured ratios rather than hardcoded values.

Related guides

• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• How Ceph actually works in production: a mental model for operators
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert