Ceph CephFS client eviction (EBLACKLISTED): blocklisted clients and failed I/O

CephFS clients suddenly see EBLACKLISTED errors on open file handles. Reads and writes that worked seconds ago fail; new operations against the same mount return the same error code. The MDS has forcibly evicted the client and added its address to the cluster-wide OSD blocklist. Every OSD now refuses traffic from that client address, and the mount is effectively dead until the blocklist is cleared or expires.

Eviction is the MDS’s last-resort response to a client that fails to return capabilities (“caps”) under cache pressure. The mechanism is by design, but the operator impact is severe: file descriptors that were live a moment ago now return hard errors.

The cascade variant is worse. When many clients are slow at once, typically because they sit behind overloaded NFS-Ganesha gateways that are themselves CephFS clients, the MDS evicts one, frees a little pressure, then evicts the next. The eviction counter climbs, the blocklist grows, and operators end up clearing entries one at a time while the underlying pressure has not been resolved.

This article covers how to confirm an active eviction, identify which clients are stuck, and clear the blocklist only after the underlying pressure is fixed. The pattern is CephFS-only. RBD exclusive-lock loss can produce a similar blocklist entry but a different client-side symptom.

What this means

CephFS delegates caching of file metadata and data to clients via capabilities. The MDS tracks every cap it has granted. When the MDS inode cache fills, the MDS sends cap-revoke messages asking clients to return caps so it can free inodes and stay under mds_cache_memory_limit (default 4GB on modern releases).

A client that fails to return caps within session_timeout (default 60s) is treated as unresponsive. The MDS terminates the session and adds the client address to the OSD blocklist. The blocklist propagates cluster-wide via the OSD map, so every OSD refuses operations from that client. Open file descriptors on the evicted client start returning EBLACKLISTED, which client libraries typically surface as EIO or EACCES depending on the call path.

Two thresholds govern the upstream pressure that drives eviction:

• mds_cache_memory_limit bounds the MDS inode cache. When the cache approaches the limit, the MDS begins issuing cap revocations to free inodes.
• mds_health_cache_threshold (default 1.5x the cache limit) is where the MDS reports MDS_HEALTH_CACHE_OVERSIZED. Reaching this point does not evict clients, but it indicates the MDS is losing the cap-recall race.

There is also a mds_cap_revoke_eviction_timeout option in some Ceph releases that governs automatic eviction specifically for unreturned cap revokes. The session_timeout path is the one most operators encounter in production incidents.

flowchart TD
  A[MDS cache approaches limit] --> B[MDS revokes caps from clients]
  B --> C{Client returns caps within session_timeout?}
  C -- yes --> D[MDS frees inodes, pressure clears]
  C -- no --> E[MDS evicts client session]
  E --> F[Client addr added to OSD blocklist]
  F --> G[Blocklist propagates cluster-wide via OSD map]
  G --> H[Open FDs fail with EBLACKLISTED]
  H --> I[All client OSD ops refused]

The blocklist is not a CephFS construct. It is an OSD-map-level entry, which is why an evicted CephFS client cannot simply reconnect to a different OSD or MDS and keep working. The eviction follows the client address across the entire cluster.

Common causes

Quick checks

Run these read-only commands to confirm an eviction is in progress and locate the affected client.

# Top-level health, including MDS-related checks
ceph health detail | grep -E 'MDS|CLIENT|RECALL'

# Clients with outstanding recalled caps (the precursor state)
<!-- TODO: verify JSON field name in current release; recent versions may use num_recall_caps or similar -->
ceph daemon mds.<id> session ls | jq '.[] | select(.recalled_caps > 0) | {id, addr, recalled_caps}'

# Current OSD blocklist entries
ceph osd blocklist ls
# On pre-Pacific releases the command is `ceph osd blacklist ls`

# MDS cache occupancy versus limit
ceph daemon mds.<id> cache status

# MDS rank state and failover status
ceph fs status
ceph mds stat

# MDS process RSS (cross-check against mds_cache_memory_limit)
ps -eo pid,rss,args | grep ceph-mds

# Counter of clients evicted for unreturned caps
# (Prometheus: ceph_mds_server_cap_revoke_eviction)
ceph daemon mds.<id> perf dump | jq '.mds_server.cap_revoke_eviction'

The session ls output is the single most useful artifact. Any client with recalled caps greater than zero is on the path to eviction. The size of the recall queue and how long it has been outstanding matter more than the mere presence of non-zero values.

How to diagnose it

1. Confirm the eviction. ceph osd blocklist ls lists the affected client address. ceph daemon mds.<id> session ls no longer shows that client because the session is gone. The kernel logs on the client host will show blocklisting messages.
2. Check for upstream cache pressure. Look at ceph_health_detail{name="MDS_HEALTH_CACHE_OVERSIZED"} and ceph_health_detail{name="MDS_CLIENT_RECALL"}. The first means the MDS is over its cache limit; the second means clients are slow to return caps. Both firing together is the classic precursor pattern.
3. Identify slow clients before they are evicted. In session ls output, entries with high recall counts are the clients the MDS is actively waiting on. If you can see them, eviction has not yet happened and you may have time to intervene.
4. Read the eviction counter as a rate. ceph_mds_server_cap_revoke_eviction is a counter. A single non-zero sample is expected over the cluster lifetime. What matters is increase(ceph_mds_server_cap_revoke_eviction[5m]) or the per-second rate.
5. Determine whether this is a cascade. If the eviction counter keeps climbing and the blocklist keeps growing, multiple clients are stuck. NFS-Ganesha gateway overload is the most common cascade driver because each gateway is itself a CephFS client.
6. Correlate with MDS memory. ceph_mds_mem_rss approaching mds_cache_memory_limit confirms cache pressure as the upstream cause. If RSS is well below the limit, the cause is more likely client-side: slow client, network issue, or a workload holding caps longer than the MDS can tolerate.
7. Rule out network. A client that cannot reach the MDS to acknowledge cap revokes looks identical to a stuck client. Check MDS logs and client-side kernel logs for connection resets or timeouts before assuming the client is misbehaving.

Metrics and signals to monitor

Fixes

Clear the blocklist only after the root cause is fixed

Clearing the blocklist (ceph osd blocklist rm <addr>) without resolving the underlying pressure lets the same client, or another victim, get evicted again within minutes. In an active cascade this is a losing battle.

Blocklist entries expire on their own after osd_blocklist_default_expire (default 3600s). For an isolated single-client eviction where the workload has stopped or the client has been restarted, expiry may be sufficient. Manual clearing is appropriate when the root cause is fixed and you need to restore a specific client immediately, or when an entry was added in error.

# Remove a specific entry after root cause is resolved
ceph osd blocklist rm <client_addr>
# On pre-Pacific releases the command is `ceph osd blacklist rm <client_addr>`

# WARNING: clears every blocklist entry cluster-wide. Every previously
# blocklisted address can resume traffic immediately. Only use when you
# understand why every entry exists and have confirmed root cause is fixed.
ceph osd blocklist clear

Address MDS cache pressure

If MDS_HEALTH_CACHE_OVERSIZED is firing or ceph_mds_mem_rss is consistently at the limit, the MDS is the bottleneck. Two responses, in order of preference:

• Increase mds_cache_memory_limit. Effective when the active metadata working set genuinely exceeds the current limit. Costs MDS host RAM. Pair with monitoring of ceph_mds_inodes_with_caps to confirm the new limit fits the working set.
• Add active MDS ranks. CephFS supports multiple active MDS daemons with subtree partitioning. This spreads the metadata cache and cap bookkeeping across daemons. Costs additional MDS hosts and adds operational complexity, but it is the right structural answer when a single rank cannot keep up.

Address client-side slowness

If the MDS is not under cache pressure but specific clients keep getting evicted, the problem is on the client side.

• Identify the workload. find, recursive rsync, recursive du, and similar walks across large trees pin caps faster than the MDS can recall them. Run them off-peak, restrict them to subtrees, or route them through a dedicated MDS rank.
• Investigate NFS-Ganesha load. NFS-Ganesha gateways are CephFS clients and can be evicted like any other. When a gateway is evicted, every NFS client behind it loses access to the filesystem. Distribute load across multiple gateways and monitor gateway host CPU, memory, and network. A single overloaded gateway is the most common cascade source.
• Kernel client tuning. Some kernel CephFS client versions hold caps longer than necessary. Upgrading the client kernel is often the right first step. Kernel client behavior differs significantly across kernel versions; verify before deploying mount option changes broadly.

Manual eviction of a stuck client

If a single client is blocking MDS progress with a high recall count for an extended period and has not yet been evicted, you can force eviction. This is disruptive to that client’s open file handles.

# Force-evict a specific client by ID
<!-- TODO: verify exact command syntax and behavior in current release -->
ceph tell mds.<id> client evict id=<client_id>

Use only when leaving the client in place is worse than evicting it. After manual eviction, the client follows the same blocklist path as an automatic eviction.

Prevention

• Size mds_cache_memory_limit for the working set. Track ceph_mds_inodes_with_caps over time. If it routinely approaches the limit during normal operation, raise the limit or add MDS ranks.
• Treat MDS_CLIENT_RECALL as an early warning. This health check fires before eviction. Alert on it the same way you would alert on degraded PGs: it indicates a client is not responding fast enough to MDS pressure.
• Watch the cap-revoke-eviction rate. A non-zero increase(ceph_mds_server_cap_revoke_eviction[5m]) is never benign. Even a single eviction is worth a post-incident review.
• Distribute NFS gateway load. Single-gateway deployments are the most common cascade source. Multiple gateways with shared storage reduce the blast radius of a single slow gateway.
• Restrict recursive filesystem walks. find and rsync against the CephFS root are the most common single-client trigger. Schedule them, scope them, or route them to a dedicated rank.
• Keep kernel clients current. Client-side cap-handling bugs are fixed in newer kernels, and the difference between a 4.x and a 6.x kernel client can be the difference between stable operation and recurring evictions.

How Netdata helps

• Per-second collection of ceph_mds_server_cap_revoke_eviction as a counter, so increase() over short windows surfaces evictions the moment they happen rather than at minute granularity.
• ceph_osd_blocklist_count tracked alongside the eviction counter, so you can watch evictions convert to live blocklist entries in real time.
• MDS_CLIENT_RECALL and MDS_HEALTH_CACHE_OVERSIZED exposed as ceph_health_detail series, surfacing the precursor signals that precede eviction.
• Correlation with ceph_mds_caps, ceph_mds_inodes_with_caps, and ceph_mds_mem_rss to distinguish MDS-side cache pressure from client-side slowness.
• ML anomaly detection on cap recall rates and MDS memory, flagging the slow drift toward cache exhaustion before any eviction fires.
• Per-second MDS reply latency and the ceph_mds_slow_reply counter, which often spike before eviction as the MDS struggles to make progress under cap pressure.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph blocked ops: client I/O stuck behind a single slow OSD
• Ceph BlueStore RocksDB compaction stalls: periodic latency spikes
• Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device
• Ceph BlueStore allocator fragmentation: rising latency at moderate fullness
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph client latency vs OSD latency: fast disks, slow clients
• Ceph deep scrub performance impact: I/O saturation that mimics an incident
• Ceph degraded objects: reduced redundancy and the race against a second failure
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural