Ceph nearfull: the 85% warning that decides whether the cluster can heal

OSD_NEARFULL or POOL_NEAR_FULL in ceph health detail means HEALTH_WARN. Clients are still reading and writing, and nothing looks broken yet. The nearfull ratio (default 0.85) is not a polite reminder to plan storage. It is the point where Ceph warns it is running out of the spare space it needs to heal itself.

One OSD failure at 85% can push surviving OSDs past backfillfull (0.90), at which point backfills refuse to start and recovery stalls. If another OSD fails in that window, you are left with degraded PGs that cannot be recovered, and the next stop is OSD_FULL at 95% where all client writes return ENOSPC.

What this means

Ceph has three capacity thresholds, all stored in the OSDMap after cluster creation:

Ceph enforces nearfull < backfillfull < full, and the OSD_OUT_OF_ORDER_FULL health check fires if you set them in the wrong order.

The non-obvious part: backfillfull does not produce a HEALTH_ERR. It produces backfill_toofull PG states and silently stalls recovery. A cluster sitting at 88% looks like it is functioning, but if an OSD fails, the PGs assigned to it cannot be redistributed because no surviving OSD will accept the backfill. The warning you ignored becomes the recovery that never starts.

flowchart TD
    A["Cluster at 85%+ (nearfull)"] --> B["OSD fails"]
    B --> C["PGs need to move to surviving OSDs"]
    C --> D{"Surviving OSDs under 90%?"}
    D -->|"Yes"| E["Backfill proceeds, cluster self-heals"]
    D -->|"No"| F["backfill_toofull blocks recovery"]
    F --> G["Degraded PGs accumulate"]
    G --> H["Second failure risks data loss"]

This cascade is why the headroom rule exists. To survive losing one failure domain (host, rack) worth of OSDs and still have recovery proceed, you need enough spare capacity that the redistributed data does not push surviving OSDs past backfillfull. The operator rule of thumb: stay at least 20% below backfillfull. With the default backfillfull at 0.90, that means keeping raw utilization below roughly 70-72%.

Common causes

Quick checks

These are read-only and safe to run during production.

# Which OSDs and pools triggered nearfull
ceph health detail | grep -iE 'nearfull|near_full'

# Per-OSD utilization with failure domain hierarchy
ceph osd df tree

# Actual configured thresholds from the OSDMap (not ceph.conf)
ceph osd dump | grep -E 'full_ratio|nearfull_ratio|backfillfull_ratio'

# Pool-level usage and max_avail
ceph df detail

# PGs already blocked by capacity
ceph pg dump_stuck unclean | grep -iE 'toofull'

# Failure domains and any down/out OSDs reducing capacity
ceph osd tree

# Recovery flags that may be silently stopping healing
ceph osd dump | grep -E 'norecover|nobackfill|noout'

How to diagnose it

1. Confirm the real thresholds. The defaults are 0.85 / 0.90 / 0.95, but your cluster may have been tuned. Run ceph osd dump | grep -E 'full_ratio|nearfull_ratio|backfillfull_ratio'. These values live in the OSDMap, not ceph.conf, so editing the config file has no effect on a running cluster. Use ceph osd set-nearfull-ratio, ceph osd set-backfillfull-ratio, and ceph osd set-full-ratio to change them.

2. Check whether recovery is already blocked. Look for backfill_toofull or recovery_toofull PG states. If you see them alongside OSD_NEARFULL, the trap has already sprung: the cluster cannot heal because target OSDs are too full. This is more urgent than the nearfull warning itself.

3. Find the fullest OSDs, not the cluster average. ceph osd df tree shows per-OSD utilization. The OSD that triggers nearfull is the most-full one, not the average. A cluster at 70% average with one OSD at 86% is already in nearfull. CRUSH does not guarantee even distribution, and variance of 10-20% between OSDs is common.

4. Estimate runway. Track daily growth rate from ceph df samples over time: (total_bytes - used_raw_bytes) / daily_growth_rate. Use the worst-case growth, not the average. Account for replication: a 3x replicated pool uses 3x raw space per usable byte.

5. Model losing your largest failure domain. If you lose the host with the most OSDs, can the remaining OSDs absorb that data without crossing backfillfull? This is the question the nearfull warning is actually asking. If the answer is no, you are already in the danger zone regardless of the current cluster-wide percentage.

Metrics and signals to monitor

The most common monitoring mistake is alerting on a hardcoded 0.85. If an operator tuned mon_osd_nearfull_ratio to 0.80 for a small cluster, your 0.85 alert fires too late. Always alert against the actual configured ratio exposed as ceph_osd_nearfull_ratio.

Fixes

Reweight oversized OSDs (fast, temporary)

If a small number of OSDs are disproportionately full, reweight them down so CRUSH moves data elsewhere:

# Caution: triggers data movement. Do one OSD at a time and watch recovery impact.
ceph osd reweight <osd-id> 0.9

This only helps if other OSDs have room. If the whole cluster is tight, reweighting just moves the problem.

Add capacity (the real fix)

New OSDs are the only durable fix for genuine capacity exhaustion. Adding OSDs gives CRUSH more targets and lowers per-OSD utilization. Plan for the rebalance I/O impact during the addition, especially on clusters where client I/O and recovery share the same network or disks.

Delete data or snapshots

Check for snapshot accumulation, which consumes space invisibly:

ceph df detail

RBD and CephFS snapshots from weeks ago on heavily-written volumes can be enormous. Removing stale snapshots frees space, but the deletion itself generates I/O as Ceph flattens the deltas.

Force RGW garbage collection

If you run RGW and recently deleted large volumes of objects, the data may still be queued for GC rather than removed:

# Processes the GC queue once and exits
radosgw-admin gc process

Deleted objects and aborted multipart uploads are not removed immediately. A GC backlog can consume significant space that ceph df attributes to the pool without any obvious live-data cause.

Temporarily raise backfillfull_ratio (emergency only)

If recovery is already blocked by backfill_toofull and you need it to proceed while you add capacity, you can raise backfillfull_ratio slightly with ceph osd set-backfillfull-ratio. This is risky: it lets OSDs accept more data and pushes them closer to full_ratio, where writes stop entirely. Restore the default as soon as capacity is added. The same caution applies to raising full_ratio to unblock writes: it buys time but does not solve the underlying shortage.

Prevention

• Alert against the configured ratio, not 0.85. Use ceph_osd_nearfull_ratio as the threshold in your alert, not a hardcoded constant.
• Track per-OSD max utilization. The cluster average hides the OSD that crosses first. Alert when any OSD exceeds 80% of its configured nearfull ratio.
• Keep raw utilization below roughly 70%. This is the 20% headroom rule below backfillfull. It is the difference between surviving an OSD failure and deadlocking recovery.
• Model failure scenarios in capacity planning. Project what happens if you lose your largest failure domain. If the answer is crossing backfillfull, you need more capacity before the next failure, not after.
• Watch for small clusters. The default thresholds assume enough OSDs that losing one is a small percentage of capacity. On a 3-5 node cluster, losing one node is a large fraction of total space, and the defaults are too aggressive. Consider lowering nearfull for small clusters so the warning fires while there is still time to act.
• Track daily growth rate. Runway estimation only works if you know your growth rate. Sample ceph df regularly and trend it, then compute days-to-nearfull against the worst-case growth, not the average.

How Netdata helps

• Per-second per-OSD utilization shows the OSD that crosses nearfull first, not just the cluster average that hides it.
• Correlate nearfull with PG state. When OSD_NEARFULL fires, Netdata shows ceph_pg_backfill_toofull and ceph_pg_recovery_toofull on the same timeline, so you immediately know whether recovery is already blocked.
• Recovery rate trending. ceph_pool_recovering_bytes_per_sec next to degraded PG counts tells you whether healing is progressing or stalled, which is the question that actually matters once capacity gets tight.
• Configured-ratio awareness. Netdata surfaces ceph_osd_nearfull_ratio and ceph_osd_full_ratio as metrics, so alerts track the real threshold even if operators tune it away from the defaults.
• Per-device-class breakdown. ceph_cluster_by_class_total_bytes separates HDD, SSD, and NVMe capacity, which matters when one device class fills faster than another.

Related guides

• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• How Ceph actually works in production: a mental model for operators
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert