$ guides / ceph ▌

CEPH · OPERATIONS PLAYBOOK

Ceph's cliff-edges: a cluster that throttles at 85%, stops writing at 95%, and heals only if you left it room

A distributed store where CRUSH places every object across OSDs, monitors hold the maps by Paxos majority, and the same disks and network carry client I/O, replication, recovery, and scrub all at once. We trace how that design behaves under load, where a single full OSD or a flapping daemon turns local trouble into a cluster-wide stop, and what to do when it does.

> Start with the monitoring checklist → # Jump to the full guide list

Ceph is self-healing until it isn't, and the line between the two is capacity you forgot to leave and flags you forgot to clear.

The defaults keep you alive. Until one OSD crosses the full ratio and Ceph refuses every write cluster-wide — a hard stop, not a slowdown, and reads keep working so it hides in plain sight. Until a cluster running hot loses a disk and recovery has nowhere to put the data, so backfill_toofull appears and degraded PGs stop healing. Until an OSD with a slow disk misses a heartbeat, gets marked down, triggers a peering storm across hundreds of PGs, and the storm slows its neighbours into missing heartbeats too. Until a monitor's clock drifts past 50ms and the quorum churns through elections while the whole cluster freezes. Until deep scrub finds a checksum mismatch on a weekend and the good replica's disk dies before anyone repairs it.

These guides are written for engineers who already run Ceph, not for people deciding whether to. The goal is the mental model of how RADOS, CRUSH, the OSDs, and the monitors actually behave under load, the failure patterns that keep recurring, the monitoring story that catches them before they page anyone, and the runbooks you wish someone had handed you before your last incident.

How Ceph actually runs in production

Ceph is not one thing. It is a placement algorithm, a fleet of per-disk daemons, a Paxos control plane, and a local storage engine — with client I/O, replication, recovery, and scrub all competing for the same disks and network. Most production failures live between these layers, not inside any one of them.

clients + RADOS interfaces

Block (RBD), file (CephFS), and object (RGW / S3) interfaces all sit on top of RADOS. Every client independently computes where its data lives — there is no central lookup — and talks straight to the primary OSD for each object.

CLIENT

CRUSH placement

CRUSH deterministically maps objects into placement groups and PGs onto OSDs using a failure-domain hierarchy (host, rack, row). No lookup table, so any topology change recomputes placement and moves data. A wrong CRUSH rule can leave PGs unable to reach <code>active+clean</code> even with disks to spare.

CRUSH

placement groups (PGs)

The sharding and coordination unit. Objects hash into thousands of PGs; each PG has an acting set, a primary, and replicas, and lives in exactly one state. <code>active+clean</code> is health — every other state (<code>degraded</code>, <code>undersized</code>, <code>peering</code>, <code>incomplete</code>, <code>down</code>, <code>inconsistent</code>) is work or trouble.

OSD daemons

One OSD per physical device. Each one simultaneously serves client I/O as primary, replicates to secondaries (or computes EC chunks), peers after topology changes, scrubs for integrity, runs recovery, and heartbeats with peers and monitors. Miss a heartbeat past <code>osd_heartbeat_grace</code> and peers mark it down.

OSD

BlueStore (device + DB/WAL)

The OSD's local engine, writing raw block devices with a RocksDB metadata store. Three devices may be in play: bulk data, a fast DB partition, and a WAL. When the DB device fills, metadata spills to the slow device and latency jumps by 10-100x — a cliff, invisible in capacity metrics.

STORE

monitors + cluster maps

Monitors hold the OSD, MON, CRUSH, PG, and MDS maps and agree on every update by Paxos majority. They are off the data path but critical during any topology change. Lose quorum and no map updates commit; clients coast on cached maps until those go stale, then I/O stalls.

MON

managers (MGR)

The active/standby pair running the Prometheus exporter, dashboard, balancer, and PG autoscaler. Almost every metric in these guides comes from the MGR module. If MGR fails, monitoring and some PG operations halt while client I/O keeps flowing — so a blind spot, not an outage.

MGR

capacity thresholds

The cluster-wide circuit breakers. At the nearfull ratio (0.85) Ceph warns and throttles; at backfillfull (0.90) it blocks recovery onto full OSDs; at full (0.95) it stops <em>all</em> writes. One OSD hitting full is the whole cluster's problem, and a full cluster cannot heal itself.

CAPACITY

Why this matters: 'Ceph is slow' or 'writes are failing' can come from a single full OSD, a flapping daemon triggering a peering storm, BlueStore metadata spilling to a slow disk, a recovery storm starving client I/O, a monitor quorum that keeps re-electing, or an OMAP storm on one RGW bucket index. The symptom rhymes but each layer has a different signal — and a different fix.

The failures you'll actually see

Most Ceph incidents fall into a small set of recurring patterns. Recognise the shape, and triage gets dramatically faster.

CRITICAL

The write freeze

One OSD — or the cluster average — crosses the full ratio (default 0.95) and Ceph refuses all writes cluster-wide. It is a hard stop that cannot self-resolve, and because reads keep working it hides until applications start failing on write. A cluster left running hot gets there fast when losing an OSD subtracts its capacity from the total.

ceph_health_detail{name="OSD_FULL"} active, writes returning ENOSPC
Cluster or any OSD utilization at or above ceph_osd_full_ratio
One OSD near full while the cluster average looks moderate
backfill_toofull appearing as recovery runs out of room

Investigate →

CRITICAL

Stuck and unavailable placement groups

After OSD losses, PGs cannot find enough authoritative data to serve I/O. incomplete means the write history is missing; down means no replica is available at all. Both fail reads and writes for the affected objects and will not self-resolve once peering has had time to finish.

sum(ceph_pg_incomplete) > 0 sustained past normal peering
sum(ceph_pg_down) > 0 — reads and writes to those PGs fail
PGs stuck in peering for more than ~10 minutes
Unfound objects blocking recovery of the affected PGs

Investigate →

CRITICAL

Monitor quorum loss

Fewer than a majority of monitors agree, so no cluster map update can commit. Clients keep running on cached maps for a while, then new connections fail and I/O stalls as maps go stale. Clock skew, a MON-to-MON partition, a full MON disk, or one monitor on slow storage can all trigger the election churn behind it.

sum(ceph_mon_quorum_status) below floor(count/2)+1
ceph -s itself taking seconds to respond
MON election epoch incrementing, leader changing repeatedly
MON_CLOCK_SKEW active alongside the instability

Investigate →

ACTIVE

The OSD flapping cascade

An OSD with a slow disk, network jitter, or memory pressure misses heartbeats and is marked down; its PGs peer and recover elsewhere; it returns and peering reverses; the underlying fault makes it miss again. Each flap mints an OSD map epoch every OSD must process, and the peering overhead can slow healthy OSDs into flapping too — the feedback loop that has killed more clusters than any other.

One or more OSDs alternating up/down in ceph osd tree
OSD map epoch incrementing several times per minute
SLOW_OPS appearing across many OSDs, not just one
PG states cycling through peering / recovering / active+clean

Investigate →

ACTIVE

Slow requests and blocked I/O

Operations exceed osd_op_complaint_time (default 30s) and register as slow ops — stuck, not merely slow. Because a write waits for every replica to acknowledge, one slow OSD can block client I/O far beyond itself. Disk stalls, op-queue saturation, a blocked BlueStore kv_sync thread, or RocksDB compaction are the usual causes; deep scrub and recovery cause transient versions that must be ruled out.

ceph_healthcheck_slow_ops > 0 sustained past a couple of minutes
Commit or apply latency spiking on specific OSDs
Client latency elevated cluster-wide, not localised
Blocked ops concentrated behind a single primary OSD

Investigate →

IMMINENT

Silent data corruption

Deep scrub finds a replica whose data does not match the others — bit rot, a firmware bug, non-ECC RAM, or a torn write. Ceph detected it, which is the system working as designed, but reads still succeed from a good copy so it is easy to ignore. If that copy's OSD fails before the inconsistency is repaired, the data is gone. A blind ceph pg repair can copy the corrupt primary over the good replica.

sum(ceph_pg_inconsistent) > 0 / OSD_SCRUB_ERRORS active
PG state active+clean+inconsistent, or PG_DAMAGED
ceph_pg_failed_repair > 0 after an automatic repair attempt
SMART degradation on an OSD in the affected acting set

Investigate →

Ceph monitoring maturity levels

Ceph observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your cluster matters. Most production clusters should land at the second level.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor. With these signals you can answer one question: is the cluster healthy, and can it still accept data? You will not learn what broke, but you will learn that something broke before users do. Survival is enough for lab clusters and non-critical storage.

Cluster health status ceph_health_status — OK / WARN / ERR, the umbrella over every check.
Monitor quorum A majority of MONs in quorum, or no map update can commit.
OSD up / in counts How many disks are alive and participating in placement.
Cluster capacity utilisation Raw used vs total against the nearfull and full ratios.

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production clusters should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident on its own: stuck PGs, slow OSDs, stalled recovery, capacity pressure, corruption.

Individual PG states Not just aggregates — degraded, undersized, stale, down, incomplete, inconsistent.
Per-OSD commit and apply latency Outliers localise a failing disk or a saturated DB/WAL device.
Slow ops count ceph_healthcheck_slow_ops — operations stuck past 30s.
Recovery / backfill rate Degraded PGs with zero recovery means healing has stalled.
Unfound / degraded / misplaced objects Redundancy status and the potential-data-loss signal.
Scrub inconsistency detection ceph_pg_inconsistent and OSD_SCRUB_ERRORS — corruption found.

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone up. A flag left set after maintenance, one OSD filling faster than the rest, clock drift creeping toward election churn, scrubs falling behind. None of these page you on day one. They become the incident on day thirty.

OSD cluster flags The noout trap: noout / norecover / nobackfill left set.
Per-OSD capacity outliers One OSD near full stops writes even at a moderate average.
Per-pool capacity and quotas ceph_pool_percent_used and pool quota headroom.
Clock skew MON_CLOCK_SKEW before it turns into quorum churn.
Recovery-flag correlation Degraded PGs plus norecover/nobackfill means healing is off.
Scrub recency PG_NOT_DEEP_SCRUBBED — verification debt piling up.
OSD flapping detection OSD_FLAPPING and rapid map-epoch churn.
CRUSH placement safety TOO_FEW_OSDS / POOL_NO_REDUNDANCY from bad rules.

Level 4: Expert

Subsystem-specific observability after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. BlueStore internals, RGW OMAP health, MDS cap pressure, per-device-class capacity. Most teams never need every signal here. Add the ones your incident history says you do.

BlueStore DB/WAL and compaction BLUEFS_SPILLOVER and compaction stalls behind periodic latency.
RGW latency and OMAP health GET/PUT latency, queue length, LARGE_OMAP_OBJECTS.
MDS cap pressure and recall Cache oversized, recall throttle, client eviction (CephFS).
Per-device-class capacity HDD vs SSD vs NVMe fill tracked separately.
Erasure-coding shard recovery EC pools recover differently and cost more CPU.
PG autoscaler activity Splits and merges cause brief I/O stalls.
OSD memory target vs RSS Eviction pressure and the OOM kills that read as flapping.

Operating mistakes worth avoiding

The traps Ceph teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

Dismissing nearfull warnings

The 85% warning gets waved off as premature. But recovery needs spare capacity: at 85%, losing OSDs pushes the survivors past backfillfull (90%) and blocks the very recovery meant to heal the failure. The cluster loses the ability to self-heal exactly when it needs to. Treat nearfull as a capacity deadline, not a nag, and keep headroom below backfillfull for a full OSD's worth of data.

Monitoring only cluster-wide metrics

Cluster-average capacity looks fine while one OSD sits at 96% — and a single full OSD stops writes cluster-wide. Cluster-average latency looks fine while one OSD stalls every request routed through it. Ceph's failures are lumpy; per-OSD capacity and per-OSD latency are where they show up first. Aggregate dashboards hide the outlier that pages you.

The noout trap

Setting <code>noout</code> for maintenance and forgetting to unset it is the single most common preventable Ceph outage. DOWN OSDs never get marked OUT, so recovery never starts, degraded PGs pile up silently, and the next failure lands on already-reduced redundancy. Alert whenever noout is set for more than a maintenance window, and urgently when noout is set while an OSD is down.

Treating HEALTH_WARN as background noise

WARN fires constantly during normal recovery, so teams mute it — and then miss the WARN that means nearfull, scrub errors, a set noout flag, or clock skew. Never silence the umbrella status. Split transient recovery warnings from structural ones using the name labels in ceph_health_detail, and alert on the specific checks that matter.

Ignoring scrub inconsistencies

Deep scrub finds a corrupt replica on a weekend; by Monday the cluster reads HEALTH_OK again and the ticket is closed. Those inconsistent replicas are a ticking bomb — if the good copy's OSD fails before repair, the data is lost. Investigate every ceph_pg_inconsistent promptly, and never run a blind <code>ceph pg repair</code>, which may overwrite the good replica with the corrupt primary.

Not watching recovery rate

Teams watch whether PGs are degraded but not whether recovery is actually progressing. A cluster with degraded PGs and a healthy recovery rate is healing normally; the same degraded count with zero recovery is a stalled, dangerous state one failure away from loss. Alert on degraded-and-not-recovering, not on degraded alone, or you drown in false tickets during normal healing.

Underestimating BlueStore DB spillover

When the dedicated DB device fills, RocksDB metadata spills onto the slow data device and performance collapses by 10-100x — but the OSD does not crash and capacity metrics look fine, so it is easily misread as a random slowdown. Only commit latency and BlueFS slow-device usage reveal it. Size the DB partition for the object count you actually store, and monitor for BLUEFS_SPILLOVER.

Not separating public and cluster networks

When client traffic and replication/recovery share the same NICs, a recovery storm saturates the link that clients depend on, and a routine OSD failure becomes a latency incident. A separate cluster network keeps rebuild traffic off the client path; if you cannot separate physically, at least throttle recovery so it cannot consume the whole link.

Ceph runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

Cluster health status

▸

Capacity and the write freeze

▸

OSD daemons and flapping

▸

Placement groups and peering

▸

Monitors, quorum, and clock skew

▸

Slow requests and OSD latency

▸

BlueStore backend

▸

Recovery, backfill, and cluster flags

▸

Scrub, corruption, and data loss

▸

CephFS and the metadata servers

▸

RGW / S3 object gateway

WHERE TO GO NEXT

Setting up Ceph monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.

> Start with the checklist > Back to Operations Guides