Ceph monitor quorum lost: the cluster can no longer update its maps

Ceph monitor quorum loss is a PAGE condition. Without a majority of MONs agreeing through Paxos, the cluster cannot commit any map update: no OSD up/down transitions, no PG state changes, no pool edits, no CRUSH adjustments. Existing clients keep running on cached maps for a while, but new client connections fail immediately, and as soon as those cached maps go stale, in-flight I/O stalls.

The trigger formula is sum(ceph_mon_quorum_status) < floor(count(ceph_mon_quorum_status) / 2) + 1. For 3 MONs you need 2 in quorum. For 5 MONs you need 3. The standard alert sustain is 300 seconds, long enough to ride out a normal sub-second leader election and short enough to page before clients start timing out en masse.

MONs are not on the data path. That is the only reason this incident is not immediately catastrophic. But MONs are on the metadata path: every map change, every OSD failure, every rebalance decision goes through them. A cluster without quorum cannot react to anything else going wrong. Treat this as a stop-the-line incident.

What this means

Paxos requires a majority quorum to commit any map update. When quorum is lost, each MON daemon enters one of three states: probing (looking for peers), electing (trying to choose a leader), or synchronizing (a rejoining MON catching up on map history). A healthy MON is either leader or peon. Persistent electing is the classic signature of quorum loss.

While quorum is down:

  • Existing clients continue serving I/O against their cached OSD maps. Reads and writes against already-known PGs keep working until those maps go stale.
  • New client mounts, RBD attach operations, and any client that needs a fresh map fail immediately.
  • Any concurrent failure (an OSD going down, a host reboot) cannot be recorded. The cluster cannot mark the OSD down, cannot start recovery, cannot protect itself from the next failure.
  • CLI commands that require MON agreement (ceph osd ..., ceph pg ..., pool changes) hang or error out. Read-only inspection via the admin socket may still work if the MON process is alive on the host.

The blast radius widens with time. After the OSD map cache window expires on clients, in-flight operations stall. RBD-attached VMs may hit guest I/O watchdogs. CephFS clients may see cap recalls fail. RGW requests that need new pool metadata fail. Recovery time matters.

Common causes

CauseWhat it looks likeFirst thing to check
MON daemon crashOne or more MON processes not running; systemd shows failed unitsystemctl status ceph-mon@<id> and journal on the MON host
MON-to-MON network partitionMONs running but cannot reach each other; mon_status shows probing indefinitelyICMP and TCP port check from each MON host to peers; check switch and firewall
MON disk fullMON process running but unable to write Paxos proposals; data directory at 100%du -sh /var/lib/ceph/mon/ceph-<id>/store.db and df on the MON host
Clock skewMON_CLOCK_SKEW health check active; MONs repeatedly re-enter electingceph time-sync-status and chronyc tracking on each MON host
MON store corruptionMON process crashes on startup, logs RocksDB errors, refuses to rejoinMON log file and ceph daemon mon.<id> perf dump if reachable

Quick checks

Start with read-only inspection. None of these change cluster state.

# Top-level status. If this hangs, MONs are unreachable from your admin host.
timeout 10 ceph status

# Quorum membership and leader. Slow responses indicate MON stress.
ceph quorum_status -f json | jq '{epoch: .election_epoch, quorum: .quorum_names, leader: .quorum_leader_name}'

# Per-MON state from the admin socket. Run on each MON host.
ceph daemon mon.<id> mon_status

# Health detail, focused on MON checks.
ceph health detail | grep -E 'MON_|CLOCK|quorum'

# Clock skew as the cluster sees it.
ceph time-sync-status

# Current MON map (addresses the MONs believe are correct).
ceph mon dump

# Paxos and election counters on the local MON.
ceph daemon mon.<id> perf dump | jq '.paxos, .mon'

On each MON host:

# Is the daemon actually running?
systemctl status ceph-mon@<id>

# Time sync health.
chronyc tracking    # or: ntpq -p
timedatectl status

# Disk space on the MON data dir.
df -h /var/lib/ceph/mon/ceph-<id>
du -sh /var/lib/ceph/mon/ceph-<id>/store.db

# Recent MON log lines.
journalctl -u ceph-mon@<id> --since "30 min ago" | tail -100

How to diagnose it

  1. Confirm the alert is real. Run ceph status with a short timeout. If the CLI itself hangs past a few seconds, the MONs your admin host talks to are unreachable or stuck. That is itself diagnostic.

  2. Identify which MONs are out. From a working MON host, run ceph daemon mon.<id> mon_status. The state field tells you whether that MON sees itself as leader, peon, probing, electing, or synchronizing. Cross-check with ceph quorum_status if any MON is responsive.

  3. Check the election epoch trend. A rapidly incrementing election epoch in ceph mon stat means MONs are fighting over leadership. This is typical of clock skew or a flapping peer, not of a clean host outage.

  4. Verify time sync on every MON host. chronyc tracking shows the offset. Anything beyond mon_clock_drift_allowed (default 0.05s) shows up as MON_CLOCK_SKEW in ceph health detail. The upstream troubleshooting guide is blunt on this point: persistent electing without clock skew is rare and warrants deeper investigation.

  5. Verify MON-to-MON connectivity. From each MON host, test TCP reachability to the others on port 3300 (msgr2) or 6789 (legacy msgr1), whichever the cluster uses. Check monmap addresses in ceph mon dump to be sure you are testing the addresses Ceph actually uses. A MON stuck in probing with peers reachable on ICMP but unreachable on the messenger port usually means a firewall or a stale monmap.

  6. Check MON disk space. The MON store lives in /var/lib/ceph/mon/ceph-<id>/store.db. A full filesystem prevents Paxos writes and drops the MON out of quorum silently. This is common on MONs co-located with other services.

  7. Inspect logs on any down MON. journalctl -u ceph-mon@<id> and the MON log file show assertion failures, RocksDB errors, store corruption messages, or OOM kills.

flowchart TD
  Q[Quorum-lost alert fires] --> C{ceph -s responds within 10s?}
  C -->|No, CLI hangs| H[Check MON hosts: alive? reachable on 3300/6789?]
  C -->|Yes, slowly| D[ceph daemon mon.X mon_status on each MON]
  D --> St{Persistent electing or probing?}
  St -->|Electing cycle| Sk[Check chronyc tracking on all MONs]
  St -->|Probing only| Net[Check MON-to-MON reachability vs monmap]
  Sk --> Skew{MON_CLOCK_SKEW active?}
  Skew -->|Yes| NTP[Fix time sync, wait one election cycle]
  Skew -->|No| Disk[Check MON data dir disk space]
  Disk --> Full{Any MON disk full?}
  Full -->|Yes| Free[Free space, restart MON]
  Full -->|No| Net
  Net --> Part{Partition or wrong monmap?}
  Part -->|Yes| FixNet[Resolve network or inject corrected monmap]
  Part -->|No| Crash[Investigate MON logs and store]

Metrics and signals to monitor

SignalWhy it mattersWarning sign
ceph_mon_quorum_status per MONDirect quorum membership; the alert sourceAny MON at 0; sum below floor(N/2)+1
ceph_health_detail{name="MON_CLOCK_SKEW"}Leading indicator for election stormsActive for more than 60 seconds
ceph_health_detail{name="MON_DOWN"}MON daemon not reachable by peersAny MON down for more than 60 seconds
Election epoch rateInstability even when quorum technically holdsMultiple increments per minute
Paxos commit latency (paxos.commit_latency from perf dump)MON cluster struggling to agreeSustained above 500ms
MON store size on diskBloated store slows elections and startupGrowth faster than 100MB/day, or above 10GB total
chrony offset on MON hostsUnderlying clock drift driverOffset trending toward 50ms or beyond
Disk usage on MON data dirSilent killer when MON host fills upAny MON host above 80% on the store volume

Fixes

The fix depends on which cause you found. In all cases, do not add or remove MONs mid-incident unless you understand exactly what you are doing. Changing MON membership while quorum is broken can leave the cluster with no valid monmap source and make recovery harder. The official guidance is to add new MONs first, wait for quorum to reform, and only then remove broken ones.

Clock skew

This is the most common root cause and the easiest to fix.

  1. Identify the skewed MON via ceph time-sync-status and ceph health detail | grep CLOCK.
  2. On the affected MON host, check chronyc tracking (or ntpq -p). Confirm the time source is reachable and the offset is shrinking.
  3. If chrony or ntpd is stopped or misconfigured, fix the configuration and restart the daemon: systemctl restart chronyd (or ntpd).
  4. Wait for the offset to drop below mon_clock_drift_allowed. The MON will rejoin quorum on its own once elections stabilize.
  5. Do not manually restart MONs to “force” rejoining. Let Paxos settle.

VM-based MON hosts are the usual culprit. Live migration, overloaded hypervisors, and missing tsc clock sources all cause drift. Bare-metal MON hosts with working chrony rarely skew past a few milliseconds.

MON disk full

  1. Identify which MON host has a full data volume via df -h /var/lib/ceph/mon/ceph-<id>.
  2. Free space. Common offenders: rotated but unflushed journals, core dumps left in /var/lib/ceph, log bloat, other services co-located on the MON host.
  3. If the store itself is bloated (large store.db), compact it: ceph daemon mon.<id> compact. This causes a brief latency spike but reclaims space and improves startup time.
  4. Once disk space is available, restart the MON if it did not rejoin automatically: systemctl restart ceph-mon@<id>.

Do not delete files under store.db directly. The store is a RocksDB database; manual deletion corrupts it.

MON-to-MON network partition

  1. Confirm with ceph mon dump that the addresses in the monmap are the ones MONs are actually trying to reach. A stale monmap after a host migration or IP renumbering is a common cause.
  2. From each MON host, test TCP reachability to peers on the msgr2 port (3300) or legacy msgr1 port (6789). ICMP alone is not sufficient.
  3. Resolve the underlying issue: switch config, firewall rules, routing, or a stale monmap.
  4. If the monmap itself is wrong on every surviving MON, you need to inject a corrected monmap. This is a destructive operation: it involves stopping MONs, rebuilding the monmap with monmaptool, and replacing it on each MON’s data directory. Follow the upstream “Recovery using a rebuilt monmap” procedure for your deployment type (bare metal, cephadm, or Rook). Do not improvise.

MON crash or store corruption

  1. Identify the failed MON from ceph mon stat and systemctl status ceph-mon@<id>.
  2. Inspect the journal and log file. Assertion failures, RocksDB corruption errors, and OOM kills all show up here.
  3. If the host itself is the problem (failed hardware, kernel panic), bring the host back or evacuate the MON role.
  4. If the store is corrupted, the recovery path is to destroy and rebuild the MON. For non-cephadm deployments this is documented upstream as the “Recovery using OSDs” procedure, which builds a fresh MON store from OSD maps. For cephadm and Rook deployments, follow the deployment-specific variant. The rebuild tool has known limitations: it cannot recover MDS maps, partially created pools, or non-admin keyrings.
  5. Once the rebuilt MON is back and in quorum, validate cluster state with ceph health detail before declaring the incident over.

Prevention

  • Run at least three MONs on three separate physical hosts in three separate failure domains. Five MONs is appropriate for larger or stretch clusters.
  • Put MON data on SSD-backed storage. A slow MON store slows every Paxos round and every election.
  • Dedicate the MON host or budget enough CPU and memory that the MON is never starved by a noisy neighbor.
  • Configure chrony (or ntpd) on every MON host, point at reliable upstream sources, and alert on offset drift well before the 50ms threshold.
  • Monitor MON store size over time. A growth rate above 100MB/day usually means excessive OSD map churn, often from flapping OSDs.
  • Track election epoch rate. Even when quorum holds, frequent elections indicate a fragility that will eventually become a real outage.
  • Periodically exercise the monstore-recovery procedure in a staging cluster. Operators who have never run it will struggle during a real incident.

How Netdata helps

  • The per-second ceph_mon_quorum_status series, broken out by ceph_daemon, lets you see the exact moment each MON dropped out and whether the loss was simultaneous (network or shared dependency) or staggered (host-by-host cascade).
  • Correlating quorum state with host-level CPU, memory, disk usage, and disk I/O on the MON hosts distinguishes host resource starvation from a Ceph-internal failure.
  • The ceph_health_detail{name="MON_CLOCK_SKEW"} and MON_DOWN signals surface the leading indicators before quorum actually breaks.
  • ML anomaly detection on MON host clock offset, disk latency, and network retransmits catches slow drift toward quorum loss before the alert fires.
  • Election epoch rate and Paxos commit latency from the MON admin socket, collected per second, show election storms even when the binary quorum signal has not yet flipped.