Elasticsearch cluster_block_exception: blocked by, the read-only blocks explained

Elasticsearch returns cluster_block_exception when a write or metadata operation hits an active index or cluster-level block. The blocked by array contains a FORBIDDEN or TOO_MANY_REQUESTS string, a numeric code, and an API label. Writes stop. Depending on the code, deletes and metadata changes may also fail.

This article maps the four numeric codes most common in production, explains how to confirm which block is active, and gives the exact commands to clear it. Most incidents involve the /12 flood-stage disk watermark, but manual index blocks and cluster-level overrides produce the same exception and require different fixes.

What this means

Elasticsearch uses index-level and cluster-level blocks to protect data or enforce administrative states. When a block is active, the coordinating node rejects the operation and returns cluster_block_exception. The error body contains a status and a block identifier with a numeric code that identifies the active gate.

The four index-level codes in the read-only and write-block path are:

A cluster without an elected master surfaces as SERVICE_UNAVAILABLE/1/no master, which is a coordination failure rather than an index-level block. There is also a cluster-level setting, cluster.blocks.read_only, that blocks the entire cluster. It does not carry an index-level numeric code, but it produces the same cluster_block_exception on every write attempt.

In Elasticsearch 7.x and 8.x, the /12 flood-stage block is automatically removed once disk usage on the affected node drops below the high watermark (90% by default). If disk stays above that threshold, or if you need to restore writes immediately after freeing space, you must clear the block manually. The other blocks (/5, /8, /9) are persistent until explicitly removed.

flowchart TD
    A[cluster_block_exception] --> B{Read the code in blocked by}
    B -->|/12| C[Flood-stage disk watermark]
    B -->|/8| D[Index write block]
    B -->|/5| E[Index read-only block]
    B -->|/9| F[Index metadata block]
    C --> G[Free disk space then clear index.blocks.read_only_allow_delete]
    D --> H[Clear index.blocks.write]
    E --> I[Clear index.blocks.read_only]
    F --> J[Clear index.blocks.metadata]

Common causes

Quick checks

# Cluster health and node count
curl -s 'http://localhost:9200/_cluster/health?filter_path=status,number_of_nodes,unassigned_shards'

# Disk usage per node
curl -s 'http://localhost:9200/_cat/allocation?v'

# Index-level blocks on all indices
curl -s 'http://localhost:9200/_all/_settings?flat_settings=true' | grep -E 'index.blocks'

# Cluster-level read-only block
curl -s 'http://localhost:9200/_cluster/settings?flat_settings=true&filter_path=*.cluster.blocks*'

# Node resource pressure
curl -s 'http://localhost:9200/_cat/nodes?v&h=name,disk.used_percent,heap.percent,cpu'

# Write thread pool rejections (distinguishes saturation from a block)
curl -s 'http://localhost:9200/_cat/thread_pool/write?v&h=node,name,rejected,completed'

How to diagnose it

1. Read the exact code in the exception. The blocked by array contains the numeric code. This determines whether you are dealing with disk pressure (/12), a manual index block (/5, /8, /9), or a cluster-level setting.
2. Check disk allocation if the code is /12. Run GET /_cat/allocation?v. Any node above 95% triggers the flood-stage block on every index that has a shard on that node. A node at 93% can spike to 95% during a large merge. Use GET /_cat/shards?v to confirm which indices have shards on the affected node.
3. Check index settings for manual blocks. Use GET /<index>/_settings?flat_settings=true and inspect the index.blocks.* keys. If the index metadata API is blocked by /9, query settings via a wildcard scope such as GET /*/_settings?flat_settings=true.
4. Check cluster-level settings. Run GET /_cluster/settings?flat_settings=true&filter_path=*.cluster.blocks*. A cluster.blocks.read_only: true overrides every index.
5. Correlate with recent events. Look for ILM failures that stopped deletions, snapshot cleanup jobs that did not run, maintenance scripts that set blocks, or AWS OpenSearch JVM pressure alarms.

Metrics and signals to monitor

Fixes

Flood-stage disk watermark (/12)

The block is index.blocks.read_only_allow_delete. First, free disk space. The fastest way is usually to delete old indices:

# Delete an old index to reclaim space immediately
curl -X DELETE 'http://localhost:9200/<old-index>'

Deleting documents does not reclaim space until segments merge, so index deletion is preferred over bulk-deletes when you are racing to drop below the watermark.

If you cannot delete indices, reduce replica count or force-merge read-only indices. Merges require temporary disk, so do not force-merge on a nearly full node. If you must keep the data, add nodes or expand disks before clearing the block.

After freeing space:

• In 7.x and 8.x, Elasticsearch automatically removes the block once disk drops below the high watermark (90%). You do not need to do anything if you have enough headroom.
• If you need to unblock immediately, or if auto-release did not fire:

# Clear the flood-stage block on all indices
curl -X PUT 'http://localhost:9200/_all/_settings?expand_wildcards=all' -H 'Content-Type: application/json' -d '
{"index.blocks.read_only_allow_delete": null}
'

Setting the value to null resets it to the default. false also works.

Warning: If disk is still above the flood stage, Elasticsearch will reapply the block almost immediately. Fix the capacity problem first.

System indices on 8.x: If you see illegal_state_exception: Cannot override settings on system indices when clearing blocks on .fleet-*, .security-*, or similar, free disk space and wait for automatic removal rather than forcing the setting.

Index write block (/8)

Caused by index.blocks.write: true or AWS OpenSearch JVM pressure protection.

# Clear the write block
curl -X PUT 'http://localhost:9200/<index>/_settings' -H 'Content-Type: application/json' -d '
{"index.blocks.write": null}
'

If you are on AWS OpenSearch and did not set the block manually, investigate JVM memory pressure. The service may reapply the block if heap pressure remains high.

Index read-only block (/5)

This blocks all writes including deletes.

# Clear the read-only block
curl -X PUT 'http://localhost:9200/<index>/_settings' -H 'Content-Type: application/json' -d '
{"index.blocks.read_only": null}
'

Index metadata block (/9)

Prevents metadata changes.

# Clear the metadata block
curl -X PUT 'http://localhost:9200/<index>/_settings' -H 'Content-Type: application/json' -d '
{"index.blocks.metadata": null}
'

Cluster-level read-only block

If the entire cluster is blocked, check both persistent and transient settings, then clear the block:

# Clear cluster-level read-only block
curl -X PUT 'http://localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '
{"persistent": {"cluster.blocks.read_only": null}, "transient": {"cluster.blocks.read_only": null}}
'

Prevention

• Project disk runway. Track daily growth and alert when any node crosses 85% (low watermark). Merges can spike usage by 10-15% temporarily, so maintain headroom.
• Audit ILM policies. Indices that should be deleted or shrunk must actually transition. A stuck ILM policy silently fills disk.
• Avoid manual blocks without runbook cleanup. If you set index.blocks.read_only during maintenance, set a reminder to revert it.
• Do not rely on auto-release for /12 in an emergency. Automatic removal only happens below the high watermark. If you are racing to free space, be prepared to clear the block manually after deletion.
• Review disk watermark settings. Non-default cluster.routing.allocation.disk.watermark.* values change when blocks trigger. Verify they match your operational runway.
• Watch for archived settings on upgrades. Custom watermark settings from older versions can leave archived.cluster.routing.allocation.disk.watermark.* entries that interfere with block clearing. Remove archived keys if present.

How Netdata helps

• Per-node disk utilization alerts fire before the 95% flood stage, giving time to act.
• Indexing rate and write thread-pool rejection metrics flatline when a block is active, making the failure visible before users report it.
• JVM heap and GC monitoring correlates with AWS OpenSearch JVM-pressure blocks and helps distinguish /8 from disk-related /12.
• Cluster health and node count tracking detects unassigned shards and relocation storms that often precede flood-stage incidents.

Related guides

• Elasticsearch all shards failed: diagnosing search_phase_execution_exception
• Elasticsearch CircuitBreakingException: [parent] Data too large - causes and fixes
• Elasticsearch cluster health red: unassigned primaries and how to recover
• Elasticsearch cluster health yellow: unassigned replicas vs real allocation blocks
• Elasticsearch fielddata circuit breaker tripped: text-field aggregations and the keyword fix
• Elasticsearch heap pressure death spiral: GC, node removal, and the cascade
• Elasticsearch JVM heap usage high: reading the sawtooth and the post-GC floor
• Elasticsearch this action would add too many shards: max_shards_per_node limit
• Elasticsearch monitoring checklist: the signals every production cluster needs
• Elasticsearch monitoring maturity model: from survival to expert
• Elasticsearch long GC pauses: old-generation stop-the-world and node drops
• Elasticsearch node OOM-killed: heap ceiling, page cache, and container limits