Ceph MON_DOWN: a monitor out of quorum and reduced redundancy

MON_DOWN fires when one or more monitor daemons are not part of the active quorum. The common case is a 3-monitor cluster with one monitor gone: quorum still holds with 2 of 3, but the cluster has lost redundancy and tolerates zero further monitor loss before consensus collapses. The Prometheus Ceph mixin surfaces this as CephMonDown (warning) and escalates to CephMonDownQuorumAtRisk (critical) when the number of down monitors equals the minimum quorum count.

Client I/O is typically unaffected because clients operate against cached cluster maps, but the cluster can no longer accept map updates safely if one more monitor fails, and any topology change (OSD flap, recovery, weight change) is one Paxos failure away from stalling.

The playbook treats this as a TICKET when at least one monitor is out of quorum but the majority still holds, sustained for more than 300 seconds. Quorum loss itself is the PAGE condition. This article covers the TICKET case: reduced redundancy with quorum intact.

What this means

Ceph monitors maintain cluster maps (OSD map, MON map, CRUSH map, PG map, MDS map) through Paxos consensus. A strict majority of provisioned monitors must participate for any map update to commit. With 3 monitors you need 2 in quorum; with 5 you need 3. MONs are not in the data path, but they are the source of truth that every client and OSD consults to learn where data lives.

When one monitor drops out, the surviving majority continues to operate. Reads and writes to OSDs proceed using cached maps. What changes is fault tolerance:

• 3-MON cluster, 1 down: 2 of 3 in quorum. Quorum holds. One more monitor failure loses quorum. Zero fault tolerance remaining.
• 5-MON cluster, 1 down: 4 of 5 in quorum. Two more failures would lose quorum. Reduced but not critical.
• 5-MON cluster, 2 down: 3 of 5 in quorum. One more failure loses quorum.

MON_DOWN sustained is not a data-loss signal. It is a redundancy signal. The cost of leaving it unfixed is that the next unrelated monitor problem (host reboot, network blip, disk fill) takes the whole cluster’s control plane down. Treat it as same-shift work.

flowchart TD
  A["MON_DOWN health check fires"] --> B{"Quorum still held?
sum(in_quorum) >= floor(n/2)+1"}
  B -- Yes --> C["TICKET: reduced redundancy
investigate the out-of-quorum MON"]
  B -- No --> D["PAGE: quorum lost
cluster cannot commit map updates"]
  C --> E{"MON daemon running?"}
  E -- No --> F["Check host, disk, systemd unit"]
  E -- Yes --> G["Check clock skew,
MON store, network"]
  F --> H["Restore MON
verify ceph_mon_quorum_status = 1"]
  G --> H

Common causes

Quick checks

Run these read-only. None mutate cluster state.

# Cluster status: confirm MON_DOWN and see which checks fire alongside it
ceph health detail

# Quorum membership: which MONs are in, who is leader, election epoch
ceph quorum_status -f json | jq '{epoch: .election_epoch, quorum: .quorum_names, leader: .quorum_leader_name}'

# Per-MON status from the admin socket of the suspect daemon
ceph tell mon.<id> mon_status

# MON map: provisioned vs quorum sets
ceph mon dump

# Time sync health cluster-wide
ceph time-sync-status

# Clock source on each MON host
chronyc tracking    # or: ntpq -p

On the affected MON host:

# Find the correct systemd unit (Quincy+ units include the cluster fsid)
systemctl -l | grep ceph-mon | grep active

# MON daemon runtime status (local admin socket; works even if MON is out of quorum)
ceph daemon mon.<id> mon_status

# Disk space on the MON data device
df -h /var/lib/ceph/mon/ceph-<id>

# Store size on disk
du -sh /var/lib/ceph/mon/ceph-<id>/store.db

# Recent MON log lines for crashes or election failures
journalctl -u ceph-mon@<id> --since "30 min ago" | tail -100

The quorum array in ceph quorum_status is a list of ranks. A missing rank means that monitor is out of quorum. The state reported by ceph tell mon.X mon_status for an out-of-quorum monitor is typically probing, electing, or synchronizing.

How to diagnose it

1. Confirm which monitor is out. Cross-reference ceph health detail, ceph mon dump, and ceph quorum_status. Note the election epoch: if it is climbing rapidly, you have an election storm, not a single down monitor.

2. Verify the daemon is running on its host. On Quincy and later the unit name includes the cluster fsid, so systemctl status ceph-mon@<id> alone may not resolve; use systemctl -l | grep ceph-mon. If the unit is inactive or failed, that is your root cause path.

3. Check clock sync. This is the single most common cause of mysterious monitor problems. Paxos depends on tight time synchronization, and the default mon_clock_drift_allowed is only 0.05 seconds. Run ceph time-sync-status cluster-wide and chronyc tracking (or ntpq -p) on each MON host. If MON_CLOCK_SKEW is active alongside MON_DOWN, fix time sync first.

4. Check the MON data device. The monitor writes every committed map update synchronously to its RocksDB store. A slow or full disk causes fsync stalls, election timeouts, and eventually the monitor dropping out. Look for MON_DISK_LOW (30% free) or MON_DISK_CRIT (5% free) in ceph health detail. Confirm with df and iostat -x 1 on the MON data device. A full monitor filesystem can crash the daemon with Caught signal (Bus error). Do not manually delete files from the MON data directory; use ceph-monstore-tool or the admin socket compaction command.

5. Check network reachability between MON hosts. Monitors must reach each other on the MON network. A partition where the daemon is running but isolated produces a monitor stuck in probing. Ping and a TCP check on the MON port between every pair of MON hosts will surface this.

6. Check the MON store size and Paxos latency. A bloated store slows startup and elections. ceph daemon mon.<id> perf dump exposes Paxos commit_latency and accept_latency. du -sh /var/lib/ceph/mon/ceph-<id>/store.db shows the on-disk size. Healthy clusters usually keep this under a few GB; growth faster than 100 MB/day suggests excessive OSD map churn, often from flapping OSDs.

7. Check for the freshly-added-monitor false positive. If the down monitor was just added and the cluster is on a recent Ceph release, you may be hitting a known bug where new monitors incorrectly report MON_DOWN. If this is the case, the monitor is healthy; upgrade or wait out the bootstrap.

Metrics and signals to monitor

There is no dedicated Prometheus metric for clock skew from the MGR module; rely on ceph_health_detail{name="MON_CLOCK_SKEW"} and the time-sync check, plus NTP/chrony offset metrics scraped from the MON hosts themselves. MON store size is likewise not in the standard Prometheus pipeline; monitor it with a host-level check on the MON data directory.

Fixes

MON daemon stopped or crashed

Restart the daemon on the affected host using the correct systemd unit. On Quincy+ the unit name includes the cluster fsid, so find it first:

# Identify the correct unit
systemctl -l | grep ceph-mon | grep active

# Start the unit (name varies by deployment)
sudo systemctl start <unit>

Then watch the election epoch and ceph quorum_status until the monitor rejoins. If the daemon crash-loops, read the MON log. Caught signal (Bus error) points to a full or failing MON data device. Do not delete files manually from the MON data directory; use ceph-monstore-tool to compact a corrupted or bloated store.

Clock skew

Fix the underlying time sync; do not widen mon_clock_drift_allowed. The default of 0.05 seconds is tight for good reasons. Steps:

1. Confirm NTP/chrony is running on every MON host: systemctl status chronyd (or ntpd).
2. Check that the configured time source is reachable: chronyc sources.
3. Verify the offset is small and stable: chronyc tracking.
4. On virtualized MON hosts, verify the hypervisor clock source and any guest agent. VMs are notorious for clock drift, especially after live migration.

Once clocks stabilize, MON_CLOCK_SKEW clears and the out-of-quorum monitor should rejoin on the next election.

MON host disk full or failing

Free space on the MON data device. Common sources of consumption:

• The MON store itself, if it has grown large. Trigger compaction:

  ceph daemon mon.<id> compact
  

  If the problem persists after compaction, set mon_compact_on_start (default false) to true and restart the monitor. Persistent growth may indicate a bug preventing metadata pruning, or excessive OSD map churn from flapping OSDs.
• Unrelated log files or crash dumps on the same filesystem.
• A failing device producing slow fsyncs. Check SMART and iostat.

If the device is failing, plan to migrate the MON data directory to a healthy device. SSD-backed MON stores are strongly recommended; HDD-backed stores slow elections and recovery.

Network partition

Heal the underlying connectivity. If a monitor host is reachable on the public network but isolated from peers on the MON network, it will stay in probing. Confirm port reachability between every pair of MON hosts. If the partition is sustained and you need to restore quorum urgently, you can stop the isolated monitor’s daemon so the surviving majority stops waiting on it, then bring it back once the network is healthy.

Bloated MON store

Compact the store as above. If compaction does not help and the store continues to grow, the root cause is usually upstream: OSD flapping generating rapid OSD map epochs. Address the flapping first. See the related guide on OSD flapping.

Prevention

• Run at least 3 monitors across 3 failure domains. 5 is better for larger or geographically distributed clusters. A 3-MON cluster tolerates one monitor loss; a 5-MON cluster tolerates two.
• Put MON data on SSD. HDD-backed MON stores slow elections and extend recovery after a daemon restart.
• Keep NTP/chrony healthy on MON hosts. Monitor the offset. The 0.05 second mon_clock_drift_allowed default is unforgiving on drifting VMs.
• Watch the MON data device free space. Alert on MON_DISK_LOW (30%) before it becomes MON_DISK_CRIT (5%).
• Track MON store growth. A custom host check on du -sh /var/lib/ceph/mon/ceph-<id>/store.db catches bloat before it slows elections.
• Track election epoch rate. Even when quorum holds, frequent elections indicate instability that will eventually drop a monitor.
• Upgrade to a fixed release if you are adding monitors on an affected version, to avoid the false MON_DOWN on freshly added monitors.

How Netdata helps

• The Ceph collector surfaces ceph_mon_quorum_status per ceph_daemon, so the exact monitor that dropped out is visible at per-second resolution rather than only when ceph health detail happens to run.
• ceph_health_detail exposes MON_DOWN, MON_CLOCK_SKEW, MON_DISK_LOW, and MON_DISK_CRIT as labeled gauges, letting you correlate the down monitor with the health checks firing alongside it. Clock skew and a down monitor appearing together is a different diagnosis than a crashed daemon.
• Host-level metrics on the MON nodes (disk utilization, disk latency, chrony offset, network errors) align with the per-monitor quorum status, so a slow MON data device or drifting clock is visible in the same window as the MON_DOWN event.
• Sustained-condition alerting (the playbook’s > 300 second threshold for MON_DOWN) avoids paging on brief election flaps during normal maintenance while still escalating real quorum risk.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert
• Ceph OSD down: telling a dead disk apart from a network blip
• Ceph OSD flapping: OSDs cycling up and down and the peering storm that follows
• Ceph OSD_FULL: all writes stopped at the 95% full ratio
• Ceph OSD fullness imbalance: one OSD full while the cluster average looks fine