Ceph FS_DEGRADED: standby MDS failed to take over a rank

FS_DEGRADED fires when at least one CephFS rank is failed or damaged and a standby did not promote. Clients can usually still reach the filesystem through surviving ranks, but you are running without the failover reserve the MDS cluster was sized to provide.

FS_DEGRADED is the precursor to MDS_ALL_DOWN. If the last active rank fails before you restore a healthy standby, CephFS becomes fully unavailable. The playbook classifies FS_DEGRADED active for more than 120 seconds as a TICKET. If CephFS is a primary storage interface, treat 120 seconds as the upper bound on response time, not a soft target.

For the broader mental model of active/standby MDS and recovery storms, see How Ceph actually works in production. This page stays on the FS_DEGRADED symptom.

What this means

The monitors report FS_DEGRADED when one or more MDS ranks are in a failed or damaged state and not served by an active daemon. For a multi-active MDS filesystem (max_mds > 1), a single rank can be offline while the rest serve metadata. For a single-rank filesystem, the filesystem is being served by what may now be your last daemon.

A rank with no daemon is down:failed. Once the monitor assigns it to a daemon, the rank is up. Standby daemons exist to take over down:failed ranks. The failure in front of you is one of three things: no qualifying standby exists, a qualifying standby exists but cannot complete promotion, or the rank is up:damaged and refuses any takeover until repaired.

flowchart TD
  A[Rank holder fails or stops beaconing] --> B{Healthy standby available?}
  B -- no --> C[FS_DEGRADED]
  B -- yes --> D{Rank state?}
  D -- down:failed --> E[Standby assigned]
  D -- up:damaged --> F[Standby refused]
  F --> C
  E --> G[up:replay then up:rejoin]
  G --> H{Rejoin completes?}
  H -- yes --> I[up:active]
  H -- no --> C
  C --> J{Last rank down?}
  J -- yes --> K[MDS_ALL_DOWN]
  J -- no --> L[Reduced redundancy]

The rightmost branch is the race. FS_DEGRADED gives you time to act before MDS_ALL_DOWN fires and CephFS goes dark for clients.

Common causes

Quick checks

# Filesystem-level view: ranks, standbys, rank states
ceph fs status

# Per-daemon state across all MDS
ceph mds stat

# MDS-related health checks (FS_DEGRADED, MDS_DAMAGED, MDS_ALL_DOWN)
ceph health detail | grep -E 'MDS|FS_'

# Full MDS map dump for rank-by-rank detail
ceph fs dump -f json-pretty

# Required standby count for the filesystem
ceph fs get <fs_name> | grep -i standby

# Daemon-level state for the MDS you suspect
ceph daemon mds.<id> status

# Metadata pool health: MDS journal flushes depend on it
ceph df | grep -E 'metadata|cephfs'

# Cluster-wide slow ops, which can block MDS journal flushes
ceph health detail | grep -i SLOW_OPS

# MDS memory pressure (OOM can take down both active and standby)
ps -eo pid,rss,cmd | grep ceph-mds | grep -v grep

All of the above are read-only.

How to diagnose it

1. Confirm the rank state. Run ceph fs status and identify any rank that is not active. The same output shows the standby count. For multi-active MDS, identify which rank number is degraded.
2. Check for the damaged case first. Run ceph health detail | grep -E 'MDS_DAMAGED|FS_DEGRADED'. If MDS_DAMAGED is active and the rank is up:damaged, automatic failover is intentionally blocked. Standbys will not promote to a damaged rank; you must inspect and mark the rank repaired first. This is the most serious CephFS state in the playbook’s Signal 1.21.
3. Verify the standby count and type. Run ceph mds stat. A common mistake is assuming standby-replay daemons count as general standbys. They do not. A standby-replay daemon follows one specific active rank and will not step in for a different failed rank.
4. Verify standby_count_wanted. If the count is unmet, you do not have the reserve. Cross-check the value against the actual standby daemon count from ceph mds stat.
5. Check daemon affinity. If a standby daemon exists but belongs to another filesystem, it will not promote. ceph fs dump shows per-daemon filesystem assignment. The mds_join_fs setting and refuse_standby_for_another_fs can both suppress cross-filesystem failover.
6. Check the metadata pool. MDS journal flushes go to RADOS. If the metadata pool is nearfull or OSDs hosting it are slow, the standby can stall in up:rejoin because it cannot safely reply to the journal replay. Correlate with ceph_osd_commit_latency_ms on the metadata pool OSDs and with SLOW_OPS.
7. Check MDS host resources. MDS OOM kills can take down both the active daemon and any standby colocated on the same host. Check dmesg for oom-killer events around the time FS_DEGRADED first fired. Check ceph_mds_mem_rss history if you have it.
8. On Rook-managed clusters, check the operator. Rook has historically skipped MDS deployment reconciliation when cluster health is HEALTH_ERR, creating a chicken-and-egg state where the operator will not recreate the MDS pod because the cluster is degraded because the MDS pod is missing. Check the operator logs for reconcile skips.

Metrics and signals to monitor

Fixes

Damaged rank: do not fail over, repair first

If the rank is up:damaged, a standby will not promote. This is by design. You must inspect the MDS journal, address the underlying metadata corruption, and mark the rank repaired before failover can proceed. The playbook notes that recovery from a damaged rank may require ceph-recover-journal. Treat this as the most serious path: forcing a standby onto a damaged rank can propagate corruption.

No healthy standby available

If standby_count_wanted is unmet, you have no general standby to promote. Options, in order of preference:

• Start additional MDS daemons on healthy hosts. The fastest way to restore failover reserve is to add capacity, not to lower the bar.
• Reduce the active workload’s pressure on the surviving ranks while you bring standbys online.
• As a temporary measure, lower standby_count_wanted to suppress the related insufficient-standby warning. This does not fix the underlying problem; it only quiets the secondary alert.

standby-replay daemons are bound to a specific rank. Adding standby-replay for rank 0 does nothing for a failed rank 1.

Standby exists but is stuck in rejoin or replay

A standby that enters up:replay or up:rejoin and stays there is usually blocked by the metadata pool or by a slow OSD. Check, in order:

• Is the metadata pool nearfull or at backfillfull? Recovery into the metadata pool may be blocked.
• Are OSDs serving the metadata pool slow? Look for ceph_osd_commit_latency_ms outliers and any active SLOW_OPS health check.
• Is there a network partition between the standby MDS host and the OSDs?
• Is the metadata pool’s BlueStore DB spilled to the slow device on the relevant OSDs? DB spillover on metadata pool OSDs is a plausible contributor to journal flush stalls.

If the underlying stall clears, the rejoin usually completes on its own. If it does not, and you have ruled out metadata pool problems, you can force a fresh assignment with ceph mds fail <fs_name>:<rank>. This is disruptive: in-flight metadata operations on that rank stall during journal replay. Only do this after the underlying cause is resolved, or you will reproduce the stuck state on the new standby.

Standby exists but will not promote due to affinity

If ceph fs dump shows the available standby bound to another filesystem, and you have multi-filesystem Ceph, failover is suppressed by mds_join_fs affinity or by refuse_standby_for_another_fs. The cleanest fix is to bring up a standby bound to this filesystem.

Rook-managed cluster: break the reconcile loop

Force manual failover as a last resort

ceph mds fail <fs_name>:<rank> instructs the monitor to mark the current rank holder failed and assign a standby. This is the same mechanism automatic failover uses, invoked manually. Use it when automatic failover has clearly failed and you have confirmed a healthy standby exists. Expect a brief metadata operation stall during journal replay.

Prevention

• Size standby_count_wanted for the failure you expect. For multi-active MDS, that means at least one general standby per filesystem in addition to any standby-replay daemons. standby-replay is a fast-failover optimization for one rank, not a failover reserve.
• Track ceph_mds_mem_rss against host limits. MDS OOM is a recurring root cause of FS_DEGRADED escalating to MDS_ALL_DOWN. The playbook treats MDS memory as a stability requirement, not a performance optimization.
• Keep the metadata pool on fast, well-provisioned OSDs. MDS journal flushes are latency-sensitive. BlueStore DB spillover, nearfull conditions, or slow OSDs on the metadata pool will stall failover.
• Periodically test failover in maintenance windows. Running ceph mds fail against a non-critical filesystem confirms that standbys actually promote under your configuration.
• Alert on FS_DEGRADED at 120 seconds, not on MDS_ALL_DOWN. Once MDS_ALL_DOWN fires, CephFS is already unavailable to clients. The warning is the actionable signal.
• Watch cap pressure on the surviving ranks. When a rank fails in a multi-active filesystem, the survivors absorb its load. ceph_mds_server_cap_revoke_eviction starting to increase is a sign the survivors are about to fall over too.

How Netdata helps

• ceph_health_detail{name="FS_DEGRADED"} is collected per second, so you see the transition from healthy to degraded within seconds instead of waiting on cron-driven ceph status polling. The same metric is available for MDS_DAMAGED and MDS_ALL_DOWN, letting you distinguish a repair-required state from a no-standby state without leaving the chart.
• Correlating FS_DEGRADED with ceph_mds_mem_rss, ceph_mds_slow_reply, and the cap revocation counters shows the resource pressure that preceded the rank failure. The cause is usually visible in the minute before the health check fires.
• Per-OSD ceph_osd_commit_latency_ms on the metadata pool OSDs reveals whether a stuck up:rejoin is really a CephFS problem or a RADOS latency problem underneath. This avoids the wrong fix (forcing failover) when the right fix is addressing slow OSDs.
• Anomaly detection on MDS request latency, inode cache counters, and memory can surface slow drift toward OOM or cap pressure before the rank fails.
• The recovery-rate and PG-state signals let you confirm that the underlying metadata pool is healthy enough to accept journal replay once a standby is assigned.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph blocked ops: client I/O stuck behind a single slow OSD
• Ceph BlueStore RocksDB compaction stalls: periodic latency spikes
• Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device
• Ceph BlueStore allocator fragmentation: rising latency at moderate fullness
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph client latency vs OSD latency: fast disks, slow clients
• Ceph degraded objects: reduced redundancy and the race against a second failure
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators