Ceph RGW garbage-collection backlog: deleted data still consuming space

You deleted several terabytes of S3 objects, but ceph df shows raw usage barely moving. The cluster is approaching nearfull and the write freeze is coming. The deletes returned 204 to clients, so they succeeded from the S3 layer’s perspective, but the underlying RADOS objects are still on disk, queued behind the RADOS Gateway garbage collector.

This is one of the quietest contributors to “the cluster is full but we deleted everything”. RGW does not free object data inline on delete. It marks the head object as deleted, enqueues the data objects (tail segments, multipart parts) for asynchronous garbage collection, and relies on a background GC thread on each gateway to drain the queue. When that thread stops making progress, deleted data keeps consuming capacity indefinitely.

The signal that matters is ceph_rgw_gc_retire_object per RGW instance. When the cluster is nearfull and that counter stops moving, you are in this failure mode. This guide walks the diagnosis, the safe ways to force GC forward, and the dangerous shortcuts to avoid.

What this means

GC in RGW is a background worker that reads the per-instance GC queue (sharded across rgw_gc_max_objs shards, default 32), selects entries whose rgw_gc_obj_min_wait (default 7200 seconds) has elapsed, and issues RADOS deletes for the data objects. The counter ceph_rgw_gc_retire_object, exported per RGW instance, increments on each retire.

If rate(ceph_rgw_gc_retire_object[1h]) is approximately zero while the cluster is nearfull, deleted data is queuing without being reclaimed. Combine this with capacity signals (ceph_cluster_total_used_raw_bytes / ceph_cluster_total_bytes and the OSD_NEARFULL health check) and you have the diagnosis.

The operator playbook treats this as a TICKET when both conditions hold for more than an hour: cluster nearfull AND GC retire rate near zero. It is not a hard PAGE because the underlying data is still safe and reads succeed. The danger is the capacity trajectory toward a hard write stop.

flowchart TD
    A["S3 DELETE returns 204"] --> B["Tail and multipart objects\nenqueued for GC"]
    B --> C{"GC thread\ndraining?"}
    C -->|yes| D["ceph_rgw_gc_retire_object\nincrements"]
    D --> E["Capacity freed"]
    C -->|no| F["Queue grows"]
    F --> G["Raw usage flat or rising\ndespite deletes"]
    G --> H{"Cluster nearfull?"}
    H -->|yes| I["TICKET:\nstalled GC plus capacity"]
    H -->|no| J["Latent risk only"]

Common causes

Quick checks

# Check daemon responsiveness and capacity state
ceph health detail | grep -E 'NEARFULL|FULL|OSD_FULL|SLOW_OPS'

# Per-OSD utilization. The worst OSD matters more than the average.
ceph osd df tree

# Pool-level and raw bytes view
ceph df detail

# GC queue depth, including not-yet-eligible entries.
# Can be slow on a multi-million-entry queue; pipe to wc -l first if needed.
radosgw-admin gc list --include-all | head -50

# Per-instance GC retire counter. The daemon ID does not always match hostname;
# list daemons via `ceph orch ps` or systemctl to find the correct id.
ceph daemon rgw.$(hostname -s) perf dump | jq '.rgw | {gc_retire_object: .gc_retire_object}'

# GC-related tunables
ceph config get client.rgw rgw_enable_gc_threads
ceph config get client.rgw rgw_gc_obj_min_wait
ceph config get client.rgw rgw_gc_max_concurrent_io

All commands above are read-only.

How to diagnose it

1. Confirm the cluster is actually nearfull. Look at ceph osd df tree, not just ceph df. A single OSD at backfillfull can block writes for every PG mapped to it, even when cluster average utilization is moderate.
2. Confirm the GC counter is flat. rate(ceph_rgw_gc_retire_object[1h]) should be greater than zero on at least one RGW instance during any window where deletes are happening. If it is zero across every instance for an hour while deletes are happening, GC is stalled.
3. Inspect the queue. radosgw-admin gc list --include-all shows entries the GC has not yet processed. Compare entry timestamps against rgw_gc_obj_min_wait. If every entry is younger than the min-wait, you do not have a stall, you have a recent delete burst. Wait it out.
4. Check whether GC threads are running. At least one RGW in each zone must have rgw_enable_gc_threads = true. Read-only or edge gateways sometimes have this disabled, and configuration drift can leave the entire fleet without a GC worker.
5. Look at RGW logs for lock contention. Repeated RGWGC::process() failed to acquire lock on gc.N lines indicate multiple gateways racing on the same GC shard lock. The work still gets done, just slowly and unevenly.
6. Verify the OSD layer can actually delete. If the cluster is at OSD_FULL, RADOS delete operations cannot make progress. GC will appear to spin without retiring anything. Freeing capacity anywhere breaks the deadlock.
7. Distinguish GC backlog from multipart orphans. If radosgw-admin gc list --include-all is small but raw usage keeps climbing, the growth is not the GC queue. It is orphaned __shadow* and __multipart* objects from incomplete uploads. Use the rgw-orphan-list tool to identify them. Do not delete shadow objects by hand.

Metrics and signals to monitor

Fixes

Force GC forward manually

# Manually process the GC queue, including not-yet-expired entries.
# Safe to run from any RGW host.
radosgw-admin gc process --include-all

--include-all is what makes this an effective emergency lever. Without it, the command processes only entries whose rgw_gc_obj_min_wait has elapsed, which during a stall may be a no-op if the queue is stuck on something else. With it, GC processes every entry it can acquire a lock on.

Run it once and re-check radosgw-admin gc list --include-all and the retire counter. If the queue shrinks and the counter climbs, the issue was throughput. If it does not, the issue is lock contention, full OSDs, or orphans rather than a real queue backlog.

Raise GC concurrency

If GC is making progress but too slowly to outrun a capacity cliff, raise per-thread concurrency:

# Increase concurrent RADOS deletes per GC thread (default 10).
<!-- TODO: verify default of rgw_gc_max_concurrent_io across Ceph versions -->
ceph config set client.rgw rgw_gc_max_concurrent_io 20

This trades OSD I/O headroom for faster draining. Do not raise it during a deep-scrub window or while recovery is saturating disks. Watch ceph_osd_apply_latency_ms and ceph_healthcheck_slow_ops after the change.

Ensure at least one RGW has GC threads enabled

# Global default
ceph config get client.rgw rgw_enable_gc_threads

# Per-instance. Replace <id> with the daemon ID from `ceph orch ps` or systemctl.
ceph tell rgw.<id> config get rgw_enable_gc_threads

If every RGW in the zone has GC threads disabled (a common drift on read-only or edge gateways), set it explicitly on at least one and restart that daemon:

ceph config set client.rgw.rgw1 rgw_enable_gc_threads true
# Then restart rgw1 to pick up the change.

Break the full-cluster deadlock

If GC is stalled because OSDs are at OSD_FULL, RADOS deletes themselves cannot complete. You cannot GC your way out of this from RGW alone. Options, in order of safety:

1. Add capacity (new OSDs). The cleanest fix.
2. Delete data the cluster can actually reclaim: snapshots, non-RGW pools, anything where removal translates to immediate raw space.
3. As a last resort, temporarily raise mon_osd_full_ratio. This is dangerous, buys hours not days, and must be reversed the moment capacity is freed. Treat it as a documented change.

Do not bypass GC casually

The radosgw-admin bucket rm --bypass-gc flag exists and looks attractive in a capacity emergency. Do not reach for it without understanding the data-safety implications. Manual deletion of shadow and multipart objects is explicitly unsafe: you cannot reliably distinguish shadow objects of deleted parents from shadow objects of live ones. Prefer radosgw-admin gc process --include-all and let the GC machinery do the bookkeeping.

Prevention

• Alert on the composite condition. Ticket when the cluster is nearfull AND rate(ceph_rgw_gc_retire_object[1h]) is approximately zero for more than one hour. Either signal alone is noise. Together they are the signature of this failure mode.
• Verify GC threads on every RGW deployment. Make rgw_enable_gc_threads = true on at least one gateway per zone part of the deployment checklist. Drift here is the most common silent cause.
• Do not change rgw_gc_max_objs after first deployment. Default is 32. Changing it reshards the GC queue and can strand existing entries.
• Apply lifecycle policies for incomplete multipart uploads. Use AbortIncompleteMultipartUpload to prevent __multipart* and __shadow* objects from accumulating. GC does not reliably clean these up on its own.
• Watch GC queue depth during bulk delete operations. Bucket lifecycle expirations, backup rotation, and test-data teardown can enqueue tens of millions of entries in hours. If retire rate cannot keep up, throttle the delete workload before capacity does it for you.
• Track raw utilization on the worst OSD, not the cluster average. Capacity death spirals start with one backfillfull target, not with cluster-wide fullness.

How Netdata helps

• Per-second ceph_rgw_gc_retire_object per RGW instance. A flat line on this counter is the leading indicator. Per-second resolution matters because GC retire events are bursty, and minute-level aggregation hides brief stalls.
• Capacity signals beside the GC counter on the same timeline. ceph_cluster_total_used_raw_bytes, ceph_osd_nearfull_ratio, and the OSD_NEARFULL / OSD_FULL health checks should be read alongside GC retire rate. A flat GC line that is not moving capacity is benign. A flat GC line while raw bytes climb toward nearfull is the TICKET.
• Pre-built health detail labels. The ceph_health_detail series with the name label surfaces OSD_NEARFULL, OSD_FULL, SLOW_OPS, and LARGE_OMAP_OBJECTS as individual booleans, so you can correlate GC stalls with capacity thresholds and OSD-layer contention without writing the queries yourself.
• Anomaly detection on retire rate. Netdata flags a drop in GC retire rate before the absolute zero threshold is crossed, which gives lead time before the cluster reaches nearfull.
• Per-pool and per-OSD capacity drill-down. When GC cannot make progress because target OSDs are full, the per-OSD utilization view identifies the blocker faster than ceph df alone.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph blocked ops: client I/O stuck behind a single slow OSD
• Ceph BlueStore RocksDB compaction stalls: periodic latency spikes
• Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device
• Ceph BlueStore allocator fragmentation: rising latency at moderate fullness
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph client latency vs OSD latency: fast disks, slow clients
• Ceph deep scrub performance impact: I/O saturation that mimics an incident
• Ceph degraded objects: reduced redundancy and the race against a second failure
• Ceph FS_DEGRADED: standby MDS failed to take over a rank
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault