Ceph BlueStore RocksDB compaction stalls: periodic latency spikes

Periodic latency spikes on Ceph OSDs that appear and clear on their own schedule are a signature of BlueStore RocksDB compaction. The cluster reports brief bursts of slow ops on a small set of OSDs, commit latency climbs for seconds to minutes, then returns to baseline without intervention. Health checks may briefly show SLOW_OPS before clearing.

This is not a steady-state problem. Spillover, hardware failure, and deep scrub all produce sustained latency. Compaction stalls produce periodic degradation because RocksDB compacts in level cycles: when L0 SST files exceed a trigger, compaction starts, saturates the DB device (or the shared block device), and then releases it. The cycle repeats whenever the next trigger threshold is hit.

The most common drivers are small-object pools (high object count per OSD means a large RocksDB metadata working set) and RGW bucket indexes (OMAP keys stored in RocksDB). Both increase the volume of metadata that RocksDB must compact, raising both the frequency and the duration of compaction cycles.

What this means

BlueStore stores object metadata, omap data, and allocator state in RocksDB. Writes arrive at an in-memory memtable, flush to L0 SST files on the DB device, and compact through L1 and beyond over time. Without compaction, L0 file count grows, read amplification rises, and RocksDB eventually hits stop-writes triggers that block all metadata operations.

The stall mechanism is contention. When compaction runs, it issues sequential reads and large writes against the same DB device that BlueStore uses for synchronous metadata commits. If the DB device is shared with the data device (no separate block.db), compaction I/O competes directly with client writes. If the DB device is dedicated but undersized or saturated, compaction saturates it on its own.

The visible signature is periodic:

  • ceph_osd_commit_latency_ms jumps 5 to 50x baseline for the affected OSDs
  • ceph_healthcheck_slow_ops rises above zero briefly, then clears
  • ceph_osd_apply_latency_ms stays closer to baseline (the data device is not the bottleneck)
  • the cycle repeats on a roughly regular cadence tied to write rate

Between cycles, the OSD looks healthy. That is the key distinguishing feature.

flowchart TD
  A[Heavy omap or small-object writes] --> B[RocksDB memtable fills]
  B --> C[L0 SST files accumulate on DB device]
  C --> D{L0 trigger threshold hit?}
  D -->|No| B
  D -->|Yes| E[Compaction cycle starts]
  E --> F[DB device I/O saturated]
  F --> G[Commit latency spikes]
  G --> H[Transient slow ops]
  H --> I[Compaction completes]
  I --> B

Common causes

CauseWhat it looks likeFirst thing to check
Small-object workloadHigh object count per OSD, low bytes-per-object, periodic spikes across many OSDsceph osd pool stats for object rate versus byte rate
RGW bucket index OMAPSpikes concentrated on OSDs hosting the .rgw.buckets.index pool; RGW LIST latency degrades in stepradosgw-admin bucket stats; ceph_health_detail{name="LARGE_OMAP_OBJECTS"}
DB device shared with dataAll OSDs on the host stall together during compaction; no separate block.dbceph-volume lvm list for block.db presence
DB device undersized or saturatedSpikes on OSDs whose block.db is near full; spill starts mid-cycleceph daemon osd.<id> bluefs stats for slow_used_bytes
Pre-Pacific column-family layoutOlder OSDs never resharded; compaction is monolithic and slowceph-bluestore-tool label and CF inspection on the OSD path
Debug build of Ceph packagesCompaction takes 2 to 3x longer than expected on every OSDceph --version and verify a RelWithDebInfo build

Quick checks

# Sample commit and apply latency across OSDs - look for the sawtooth
ceph osd perf

# Slow ops and BlueStore slow-op detail
ceph health detail | grep -E 'SLOW_OPS|BLUESTORE_SLOW_OP'

# BlueFS slow device usage - nonzero means spillover, not compaction
ceph tell osd.<id> bluefs stats

# RocksDB compaction queue depth - high means compaction is behind writes
ceph daemon osd.<id> perf dump | jq '.rocksdb'

# In-flight operations on a suspect OSD
ceph daemon osd.<id> dump_ops_in_flight

# Per-PG scrub state - rules out deep scrub as the cause
ceph pg dump | grep -E 'scrubbing|deep'

All of these are read-only. None modify cluster state.

How to diagnose it

  1. Establish periodicity. Capture ceph osd perf every 30 seconds for 10 to 15 minutes. A compaction stall shows a regular sawtooth on commit latency for the same set of OSDs. Random spikes across different OSDs suggest a different cause.

  2. Localize to OSDs. Identify which OSDs spike together. OSDs sharing a host and a DB device spike together. OSDs on different hosts spiking together suggests a workload pattern (a small-object pool spread across the cluster).

  3. Rule out spillover. Run ceph tell osd.<id> bluefs stats. If slow_used_bytes is nonzero, the DB has spilled to the slow device. Spillover produces persistent degradation, not periodic, and the fix is DB migration, not compaction tuning.

  4. Rule out deep scrub. Check ceph pg dump for scrubbing or deep states on PGs mapped to the spiking OSDs. Deep scrub produces sustained high apply latency and high read I/O on the data device, not just commit latency spikes.

  5. Confirm compaction is the driver. On a suspect OSD, capture ceph daemon osd.<id> perf dump during a spike and between spikes. Compare the rocksdb counters (compact queue length, bytes written during compaction, L0 file count). A compaction-driven spike correlates with active compaction work and a falling L0 count.

  6. Check the workload driver. If the pattern is cluster-wide, identify the pool. Small-object pools (CephFS metadata, RGW index, or any pool with millions of small objects) are the usual cause. Use ceph osd pool stats and ceph df detail to find pools with high object count relative to bytes stored.

  7. Verify the build type. On Pacific, Quincy, and Reef backports, a build-type bug was reported to cause RocksDB to run effectively in debug mode, multiplying compaction time. If compaction seems uniformly slow across all OSDs regardless of workload, verify the package was built with CMAKE_BUILD_TYPE=RelWithDebInfo.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
ceph_osd_commit_latency_ms per OSDReflects DB device and WAL path latencyPeriodic 5 to 50x spike on the same OSDs
ceph_osd_apply_latency_ms per OSDReflects data device latencyStays near baseline during a pure compaction stall
ceph_healthcheck_slow_opsCount of ops exceeding osd_op_complaint_time (default 30s)Brief nonzero bursts that clear without intervention
rocksdb perf counters (admin socket)Compaction queue depth and L0 file countSustained high queue depth means compaction is losing ground to writes
bluefs slow_used_bytes (admin socket)Spillover indicatorAny nonzero value means spillover, not a pure compaction stall
Per-pool object rate versus byte rateIdentifies small-object driversHigh object rate with low byte rate indicates a metadata-heavy workload

Fixes

Reduce compaction pressure

The most effective long-term fix is reducing the working set RocksDB has to compact.

  • Reshard RGW bucket indexes. Use radosgw-admin bucket reshard for older buckets, or enable dynamic resharding. Target roughly 100,000 to 200,000 objects per index shard.
  • Move small-object pools off overloaded OSDs. Reweight OSDs that host both high-churn metadata pools and high-throughput data pools.
  • Add PGs to small-object pools if PGs per OSD is low. More PGs spread the OMAP load across more OSDs and more RocksDB instances.

Trigger manual compaction

When compaction has fallen behind writes, a manual compaction can catch it up during a maintenance window.

# Online, non-blocking compaction of one OSD
ceph tell osd.<id> compact

Online compaction runs concurrently with client I/O and adds latency while it runs. Schedule it during low-traffic periods, or rotate it across OSDs.

For severe cases, offline compaction is more thorough but requires OSD downtime:

# Stop the OSD first, then run offline compaction
ceph-kvstore-tool bluestore-kv <path> compact

Offline compaction is disruptive. The OSD must be stopped and marked out. Use it only when online compaction cannot keep up.

Tune bluestore_rocksdb_options

Defaults changed significantly across releases. Verify the active configuration before tuning:

ceph config get osd bluestore_rocksdb_options

Squid (19.2.x) defaults enable LZ4 compression, max_background_jobs=4, write_buffer_size=16777216 (16MB), max_write_buffer_number=64, and compaction_readahead_size=2MB. These differ substantially from pre-Squid releases (which used kNoCompression, a 256MB write_buffer_size, max_write_buffer_number=4, and the now-deprecated max_background_compactions=2).

On pre-Squid releases, common operator adjustments include increasing max_background_jobs (the replacement for the deprecated max_background_compactions and compaction_threads) to allow more parallel compaction. The recycle_log_file_num option is reported by some operators to be ignored, but upstream RocksDB documents it; do not rely on either claim without testing on your release.

Any change to bluestore_rocksdb_options requires an OSD restart to take effect. Test on one OSD before applying cluster-wide.

Verify rocksdb_cf_compact_on_deletion

RGW usage logging and bucket index operations generate tombstones that accumulate in RocksDB. Without column-family-level compaction on deletion, iteration performance degrades over weeks until the OSD is restarted. The rocksdb_cf_compact_on_deletion feature was added in Reef and backported to Pacific 16.2.13.

If you are on a release that predates this feature and you see latency growing over weeks between OSD restarts, an upgrade is the structural fix.

Address discard contention

On SSDs whose firmware handles concurrent TRIM poorly, the default `bdev_async_discard_threads` can produce latency spikes that compound with compaction. Setting it to 1 serializes discards and removes the contention. Investigate this only if you see compaction-correlated spikes on SSD-backed OSDs with otherwise healthy DB devices.

Prevention

  • Size the DB partition correctly. Target roughly 4% of the data partition size for replicated workloads, more for EC pools or heavy OMAP usage. Undersized DB leads to spillover, which presents as persistent latency, not periodic.
  • Upgrade to a release with rocksdb_cf_compact_on_deletion. Reef and later have this enabled for all column families by default. It is the single most important fix for RGW-driven compaction pressure.
  • Verify build type on upgrades. A build-type bug affected packages for years. Verify packages were built RelWithDebInfo after upgrade.
  • Monitor per-OSD commit latency, not just cluster averages. Compaction stalls affect individual OSDs or small groups. Cluster-level latency aggregates hide them.
  • Track RGW bucket growth and reshard proactively. Do not wait for LARGE_OMAP_OBJECTS health warnings. Monitor per-bucket object counts against shard count.
  • Apply the Squid OSD crash fix if you are at 75% or fuller. The ceph_assert(cut_off == p->length) assertion hits fragmented OSDs and was reportedly fixed in v19.2.3. Existing corruption requires non-destructive repair with ceph-bluestore-tool fsck --path <path> --bluefs_replay_recovery=true.

How Netdata helps

  • Per-second per-OSD commit and apply latency reveals the sawtooth pattern of compaction stalls in real time, where 30-second or 60-second scrapes collapse the spikes into noise.
  • ML anomaly detection on commit latency flags the periodic spikes without requiring static thresholds tuned per OSD and device class.
  • Correlated health-check metrics (ceph_healthcheck_slow_ops, plus ceph_health_detail for slow-ops and BLUEFS_SPILLOVER) appear alongside the latency signal so you can confirm compaction versus spillover in one view.
  • Per-pool metrics (ceph_pool_objects, ceph_pool_percent_used) surface the small-object or RGW index pool that is driving compaction pressure.
  • Host-level disk I/O metrics on the DB device let you confirm that the saturation is on block.db and not the data device, which is the key diagnostic step.