Elasticsearch Operations Guides on Netdata

Elasticsearch all shards failed: diagnosing search_phase_execution_exception

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch all shards failed: diagnosing search_phase_execution_exception

You run a search and Elasticsearch returns search_phase_execution_exception with reason all shards failed. Every shard copy involved in the query returned a failure to the coordinating node. The outer error is a container; the actual root cause lives in the per-shard failure reasons inside the response body. Do not assume the cluster is down. This error fires on clusters with green health and stable nodes when a query is malformed, a mapping is incompatible, or a resource limit is breached uniformly across every target shard.

Elasticsearch ALLOCATION_FAILED after max retries: reroute and corrupt shard recovery

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch ALLOCATION_FAILED after max retries: reroute and corrupt shard recovery

A shard that repeatedly fails allocation exhausts index.allocation.max_retries (default 5) and becomes permanently UNASSIGNED. Elasticsearch stops automatic placement. Cluster health is RED if the shard is a primary, YELLOW if it is a replica. Indexing to the affected index is blocked until the primary is assigned.

This state typically follows transient node restarts, disk pressure events, or translog corruption. After the fifth failed attempt, the allocator stops retrying. The shard will not move without explicit operator action: POST /_cluster/reroute?retry_failed=true for transient failures, or forced allocation with accepted data loss when no valid copy remains.

Elasticsearch authentication failures: audit logs, brute force, and credential drift

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch authentication failures: audit logs, brute force, and credential drift

Elasticsearch does not expose authentication failure counts through _nodes/stats or _cluster/health. The security audit log is the only structured source for authentication_failed, access_denied, and run_as_denied events. Without it, brute force attempts, credential stuffing, and expiring service tokens are invisible until they cause an outage.

What this means

When xpack.security.audit.enabled is true, each node writes security events to a local audit log file. The events that matter for auth issues are:

Elasticsearch CircuitBreakingException: [parent] Data too large - causes and fixes

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch CircuitBreakingException: [parent] Data too large - causes and fixes

When a search or indexing request returns HTTP 429 with CircuitBreakingException: [parent] Data too large, data for [<http_request>] would be [X], which is larger than the limit of [Y], the parent circuit breaker has rejected the operation. This is Elasticsearch protecting the JVM from an out-of-memory kill, not a client-side rate limit.

Since version 7.0, the parent breaker tracks real memory usage by default. It can trip even when individual child breakers are within limits. The node is under genuine heap pressure. Determine quickly whether the cause is a single abusive query or structural memory exhaustion.

Elasticsearch cluster health red: unassigned primaries and how to recover

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch cluster health red: unassigned primaries and how to recover

Cluster health red means at least one primary shard is unassigned. Queries against affected indices return partial results or fail; writes are blocked. yellow only signals missing replicas, but red signals active data unavailability.

Cluster health is a lagging indicator. A red status sustained longer than two minutes after the cluster has formed is a real fault; a brief flash during startup is normal. By the time the status turns red, a node has likely departed, a disk has crossed a watermark, or a shard copy has been rejected as corrupt.

Elasticsearch cluster health yellow: unassigned replicas vs real allocation blocks

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch cluster health yellow: unassigned replicas vs real allocation blocks

GET /_cluster/health returning status: yellow means all primaries are assigned but at least one replica is not. Some clusters are yellow by design. Others are yellow because a disk cascade, stuck allocator, or failed shard is blocking recovery. Benign yellow resolves. Structural yellow persists. If your cluster has been yellow for more than thirty minutes, something is actively blocking allocation. Teams that treat yellow as cosmetic often miss the transition from transient recovery to a real incident, leaving them one node failure away from red.

Elasticsearch cluster state too large: field count, index count, and per-node heap

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch cluster state too large: field count, index count, and per-node heap

Every node holds a copy of the cluster state in heap. When it grows large, the cost is paid everywhere: 200 MB of state consumes 200 MB on every node, and the elected master burns additional CPU and heap serializing and publishing updates. Symptoms show up indirectly: the master feels sluggish, pending tasks queue for minutes, heap pressure climbs on nodes that should be idle, and master elections stall indexing and shard allocation. The usual drivers are too many indices from per-minute or per-hour time-series patterns; a mapping explosion from uncontrolled dynamic fields; excessive aliases; or churn from frequent template and setting changes. Raw size measured via /_cluster/state is a rough proxy. The indicators that matter are field-count growth, cluster state version churn, pending-task age, and master node heap pressure.

Elasticsearch cluster_block_exception: blocked by, the read-only blocks explained

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch cluster_block_exception: blocked by, the read-only blocks explained

Elasticsearch returns cluster_block_exception when a write or metadata operation hits an active index or cluster-level block. The blocked by array contains a FORBIDDEN or TOO_MANY_REQUESTS string, a numeric code, and an API label. Writes stop. Depending on the code, deletes and metadata changes may also fail.

This article maps the four numeric codes most common in production, explains how to confirm which block is active, and gives the exact commands to clear it. Most incidents involve the /12 flood-stage disk watermark, but manual index blocks and cluster-level overrides produce the same exception and require different fixes.

Elasticsearch coordinating node overload: aggregation merge, heap spikes, and 429s

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch coordinating node overload: aggregation merge, heap spikes, and 429s

HTTP 429 or 503 responses appear on search requests while data nodes look healthy. Heap spikes on one node while others stay flat. The slow log shows heavy aggregation queries. That node is the coordinator, and it is running out of heap during the reduce phase.

Every node can act as a coordinating node. For each search, the coordinator broadcasts the query to relevant shards, collects partial results, and merges them. Aggregations compute locally per shard and reduce in memory on the coordinator. High-cardinality terms aggregations, deep pagination, or large fetch sizes force the coordinator to hold massive intermediate structures in heap. If the estimate exceeds the circuit breaker limit, the request is rejected. If the breaker is too slow, the node may suffer long GC pauses or disconnect from the cluster.

Elasticsearch CPU saturation: search, merges, GC, and hot-spotting

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch CPU saturation: search, merges, GC, and hot-spotting

Search latency climbs, bulk indexing slows, and clients see EsRejectedExecutionException or HTTP 429. On data nodes, CPU is pinned above 90% and load average rises. Elasticsearch burns CPU in query execution, text analysis, segment merges, and JVM garbage collection. If saturation persists, search and write thread pool queues grow until the cluster rejects work.

The first trap is assuming all CPU belongs to Elasticsearch. os.cpu.percent includes every process on the host, co-located containers, and kernel work. In containers this diverges sharply from process.cpu.percent, which tracks only the Elasticsearch process. A node may show 89% OS CPU and 67% process CPU, while the container runtime reports a third figure. Attribute CPU correctly before tuning thread pools or adding nodes.

Elasticsearch disk full: emergency recovery and freeing space safely

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch disk full: emergency recovery and freeing space safely

Writes fail with TOO_MANY_REQUESTS/12/index read-only / allow delete (api). Cluster health is red or yellow and data nodes are pinned above 90 percent disk. Indexers buffer or drop data while the cluster attempts shard relocations onto already-full disks. Recover without corrupting metadata or amplifying pressure.

What this means

Elasticsearch uses three disk watermarks. Low (85 percent) stops new shard allocation. High (90 percent) starts relocating shards off the node. Flood stage (95 percent) forces every index with a shard on that node into index.blocks.read_only_allow_delete: true, which stops writes.

Elasticsearch disk I/O saturation: merges, fsync, and page-cache starvation

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch disk I/O saturation: merges, fsync, and page-cache starvation

When Elasticsearch data nodes show climbing I/O wait while indexing and search latency rise, but CPU is not the bottleneck, the cluster stays green while throughput falls and thread pool queues grow. This pattern usually traces to one of three disk pressures: background segment merges rewriting data faster than storage can absorb, translog fsync overhead from durability guarantees, or OS page-cache eviction forcing every search to read from disk. This guide shows how to tell them apart, confirm the bottleneck with safe read-only checks, and relieve pressure.

Elasticsearch disk watermark cascade: from low watermark to cluster-wide read-only

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch disk watermark cascade: from low watermark to cluster-wide read-only

Writes fail with cluster_block_exception or FORBIDDEN/12/index read-only / allow delete (api). Kibana becomes unreachable; Logstash and Beats buffer or drop data. The cluster did not fail at once. It crossed a sequence of thresholds that turned single-node disk pressure into a cluster-wide write outage. With homogeneous disk sizes, every data node likely hit the thresholds within minutes, leaving no relocation target and no relief valve.

Elasticsearch disk watermark tuning: thresholds, max_headroom, and multiple data paths

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch disk watermark tuning: thresholds, max_headroom, and multiple data paths

Elasticsearch uses disk watermarks to decide when a node is too full to accept shards. The defaults (85%, 90%, 95%) were designed for small disks. On modern nodes with multi-terabyte volumes, those percentages leave hundreds of gigabytes free while still blocking allocation and triggering expensive rebalancing. Elasticsearch 8.x introduced max_headroom to cap the free-space requirement on large disks, but the interaction between percentages, absolute bytes, and max_headroom is not obvious. If you run multiple data paths, watermarks apply per path rather than per node, so a partially full disk can cause node-wide allocation restrictions. This article explains how the allocator evaluates disk space, how to tune thresholds without causing relocation storms, and why multiple data paths complicate the picture.

Elasticsearch document indexing failures: index_failed, bulk item errors, and version conflicts

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch document indexing failures: index_failed, bulk item errors, and version conflicts

A bulk request can return HTTP 200 while rejecting individual documents inside it. indices.indexing.index_failed climbs, but pipelines that only check HTTP status miss the rejections and documents disappear. This guide covers three failure classes: mapper parsing and type conflicts, per-item bulk errors hidden in HTTP 200 responses, and version_conflict_engine_exception under concurrent updates. It also distinguishes these from node-level write rejections and circuit breaker trips, which produce different symptoms and require different fixes.

Elasticsearch EsRejectedExecutionException: write thread pool rejections and HTTP 429

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch EsRejectedExecutionException: write thread pool rejections and HTTP 429

HTTP 429 responses from Elasticsearch, or stack traces containing EsRejectedExecutionException, mean the write thread pool queue is full. The write thread pool (named bulk before Elasticsearch 6.3) executes indexing, bulk, update, and delete operations on each data node using a bounded queue. When all threads are busy and the queue fills, the node rejects the operation. This guide covers how to confirm the diagnosis, distinguish it from other rejection paths, and fix the root cause instead of masking the symptom.

Elasticsearch expensive queries: leading wildcards, regex, deep pagination, and scripts

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch expensive queries: leading wildcards, regex, deep pagination, and scripts

Search latency spikes and data-node CPU saturation usually trace to one of four query patterns: leading wildcards or unbounded regex, deep from+size pagination, runtime Painless scripts, and deep aggregations. Elasticsearch uses a scatter-gather read path: the coordinating node broadcasts every query to one copy of every target shard. A single expensive query fans out, and the slowest shard sets overall latency. Coordinating-node heap pressure rises when it merges large intermediate result sets during the fetch phase or aggregation reduce phase.

Elasticsearch exposed without authentication: open clusters and snapshot exfiltration

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch exposed without authentication: open clusters and snapshot exfiltration

TCP/9200 is externally reachable and responds without credentials. When Elasticsearch binds to a public interface with security disabled, anyone who can reach the HTTP port can query indices, modify cluster state, and register snapshot repositories. The fastest exfiltration path is not reading documents individually. It is registering an attacker-controlled snapshot repository and copying entire indices out in a single background operation.

Elasticsearch fielddata circuit breaker tripped: text-field aggregations and the keyword fix

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch fielddata circuit breaker tripped: text-field aggregations and the keyword fix

Queries return CircuitBreakingException: [fielddata] Data too large... and HTTP 429s while JVM heap on one or more data nodes climbs toward the breaker limit. The node rejects queries to protect itself before OOM. This almost always means a query is aggregating, sorting, or scripting against an analyzed text field that lacks a keyword sub-field, forcing Elasticsearch to load an expensive fielddata cache into heap.

Elasticsearch FORBIDDEN/12/index read-only / allow delete (api) — flood stage recovery

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch FORBIDDEN/12/index read-only / allow delete (api) — flood stage recovery

When Elasticsearch returns cluster_block_exception with FORBIDDEN/12/index read-only / allow delete (api), every write, update, and index creation fails with HTTP 403. Search continues to work.

This happens when a data node crosses the flood-stage disk watermark (95% by default). Elasticsearch auto-applies index.blocks.read_only_allow_delete to every index with a shard on that node, preventing writes to avoid Lucene segment corruption from a full disk.

Elasticsearch heap pressure death spiral: GC, node removal, and the cascade

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch heap pressure death spiral: GC, node removal, and the cascade

When a node drops from _cat/nodes and network tests pass, check its JVM GC logs. Stop-the-world pauses over 10 seconds cause the master to remove the node. Survivors then absorb recovery traffic, their heap climbs, and they begin missing fault-detection checks too. This feedback loop is the heap pressure death spiral. It masquerades as network instability because operators check connectivity while the real problem is memory saturation.

Elasticsearch high disk watermark [90%] exceeded: shard relocation and the cascade

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch high disk watermark [90%] exceeded: shard relocation and the cascade

When Elasticsearch logs high disk watermark [90%] exceeded on [node] ... shards will be relocated away, the allocator immediately begins moving shards off the affected node. Relocation generates disk I/O and network traffic on both source and target. If targets were already close to their own watermarks, incoming shards can push them past 90%. This is the disk watermark cascade: a self-reinforcing loop where one full node triggers relocations that make other nodes full, eventually leaving the cluster with no legal allocation target and, if flood stage is hit, read-only indices.

Elasticsearch ILM stuck: indices not rolling over, shrinking, or deleting

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch ILM stuck: indices not rolling over, shrinking, or deleting

Disk usage climbs steadily. Old indices that should have been deleted remain. Shard count grows, and the cluster approaches cluster.max_shards_per_node. In ILM, indices are stuck in one phase for hours or days. This is the ILM stuck pattern: silent accumulation that becomes a disk watermark crisis, heap pressure, or unassigned shard storm when the cluster runs out of room.

ILM polls every ten minutes by default. When an index cannot advance, it sits. Because the failure is gradual, it rarely pages until a secondary limit is breached. Detect the stuck state early and fix the root cause before accumulation triggers cascading failures.

Elasticsearch indexing pressure rejections: memory backpressure before heap failure

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch indexing pressure rejections: memory backpressure before heap failure

Bulk indexing clients report rejections while cluster health is green, disks are below the high watermark, and write thread pool queues are not saturated. Yet the nodes are pushing back. Pull _nodes/stats/indexing_pressure and you will see climbing coordinating_rejections, primary_rejections, or replica_rejections. This is the indexing pressure framework, introduced in Elasticsearch 7.9, enforcing memory-based backpressure. It tracks in-flight indexing bytes at the coordinating, primary, and replica stages. The default limit is 10% of the JVM heap for coordinating and primary work, and 1.5 times that limit for replica operations. It fires before write thread pool rejections, indicating that in-flight write memory is too high rather than disk or CPU.

Elasticsearch indexing rate dropped to zero: where the write path stalls

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch indexing rate dropped to zero: where the write path stalls

Your ingestion pipeline reports healthy connections, but Elasticsearch stopped accepting writes. The index_total counter is flat, upstream queues are building, and documents are erroring or disappearing. Because index_total increments for every document, update, delete, and individual bulk item, a sustained rate of zero means the write path is stalled. The cluster may still report green health, nodes may still respond to pings, and search may still work, but the pipeline is backed up.

Elasticsearch ingest pipeline bottleneck: grok, enrich, and per-processor time

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch ingest pipeline bottleneck: grok, enrich, and per-processor time

Logstash or Beats instances log HTTP 429 errors. Elasticsearch indexing rate drops while upstream volume is flat. The write thread pool queue grows, but cluster health is green and there is no disk pressure or heap pressure. The culprit is often a single slow processor inside an ingest pipeline. A grok pattern backtracking on malformed logs, an enrich processor doing synchronous lookups, or a heavy Painless script throttles every document before it reaches Lucene. Because ingest processing runs on the write path, the backlog overflows the write thread pool queue and returns bulk rejections to upstream clients.

Elasticsearch IOException: Too many open files -- file descriptors, segments, and ulimit

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch IOException: Too many open files – file descriptors, segments, and ulimit

IOException: Too many open files means the Elasticsearch process has reached its per-process RLIMIT_NOFILE. Once open_file_descriptors reaches max_file_descriptors, the kernel returns EMFILE on every new open() call. The node cannot create Lucene segments, accept transport or HTTP connections, or maintain cluster membership. Each shard is a Lucene index composed of multiple segment files, and every network connection consumes a descriptor.

Elasticsearch JVM heap usage high: reading the sawtooth and the post-GC floor

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch JVM heap usage high: reading the sawtooth and the post-GC floor

Your Elasticsearch alert fires: jvm.mem.heap_used_percent has crossed 75 percent and is holding there. You pull up the graph and see a jagged sawtooth climbing toward the ceiling. The first instinct is to add heap or restart the node. Both are usually wrong.

The sawtooth is normal. Elasticsearch runs on the JVM with a young generation that fills with short-lived objects and empties on young garbage collections. The peak of the tooth is noise. The signal that matters is the post-GC floor: the minimum heap used immediately after a collection. In a healthy node, the floor stays between roughly 30 and 50 percent of max heap, and young GC dominates. When the floor trends upward, old generation objects are accumulating. Old GC pauses stop the world, and once a pause exceeds the cluster fault detection timeout, the master removes the node and triggers shard reallocation. That reallocation places more heap pressure on the survivors, beginning a death spiral.

Elasticsearch Limit of total fields [1000] in index has been exceeded — mapping explosion

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch Limit of total fields [1000] in index has been exceeded — mapping explosion

Every write suddenly returns illegal_argument_exception: Limit of total fields [1000] in index [X] has been exceeded. Indexing stops. The temptation is to raise index.mapping.total_fields.limit and move on. Do not. This is a mapping explosion: dynamic mapping creates a new field for every unique key in your documents. The 1000-field limit is a guardrail, not the root cause. Runaway mappings bloat the cluster state, inflate heap on every node, and eventually destabilize the master. Diagnose the source, relieve pressure safely, and fix the data shape so it does not recur.

Elasticsearch long GC pauses: old-generation stop-the-world and node drops

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch long GC pauses: old-generation stop-the-world and node drops

In Elasticsearch 8.x, nodes can drop out of the cluster without logging errors. The master logs node-left with reason disconnected, while the departed node shows no ERROR entries because its JVM was frozen in an old-generation stop-the-world GC pause. A single pause longer than 10 seconds fails a fault-detection check; roughly 30 seconds of total unresponsiveness triggers removal. Once the master reallocates shards, remaining nodes face additional heap pressure and the cascade continues.

Elasticsearch mapper_parsing_exception: type conflicts and failed document indexing

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch mapper_parsing_exception: type conflicts and failed document indexing

Document counts do not match what your pipeline sent. The _bulk endpoint returns HTTP 200, yet documents are missing from queries. In the Elasticsearch logs you see mapper_parsing_exception with messages like failed to parse field [fieldname] of type [typename] in document with id [id]. The indices.indexing.index_failed counter is climbing. This is a per-document schema rejection, not a cluster outage, and it silently drops data.

Elasticsearch mapping explosion: dynamic mapping, cluster state bloat, and master pressure

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch mapping explosion: dynamic mapping, cluster state bloat, and master pressure

Intermittent master elections, climbing heap on every node, and spiking indexing latency are hallmarks of a mapping explosion. Queries slow down. Administrative operations such as index creation or snapshot management crawl. If your data sources include unstructured JSON, such as application logs with variable keys, Kubernetes labels, or user-generated metadata, check mapping growth first.

Dynamic mapping creates a new field for every unique key it encounters. Over hours, an index can accumulate tens of thousands of fields. Every mapping is part of the cluster state, which every node holds in its JVM heap. The master serializes and publishes the updated state on every mapping change. As the state grows, publication slows, pending tasks queue up, and the master becomes unstable. Unchecked, this leads to heap pressure death spirals and cascading node removals.

Elasticsearch master instability: frequent elections and metadata overload

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch master instability: frequent elections and metadata overload

Index creation requests time out. _cluster/health hangs or returns timeouts. The node listed by _cat/master changes every few minutes outside planned maintenance. Shard allocation stalls, and new indices stay red or unassigned even though all data nodes are reachable. These symptoms indicate a master node that cannot keep up with cluster state updates, triggering repeated elections and leaving the cluster without stable coordination.

Elasticsearch master_not_discovered_exception: no elected master and stalled writes

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch master_not_discovered_exception: no elected master and stalled writes

HTTP 503 and master_not_discovered_exception mean bulk indexing, index creation, mapping updates, and shard allocation checks are being rejected. Search requests that do not need fresh cluster state may return cached results briefly, but the cluster cannot process writes or administrative work. The data nodes may be healthy, but without an elected master, the cluster cannot update shard routing, publish state changes, or acknowledge document writes. The root cause is usually one of four problems inside the master-eligible cohort.

Elasticsearch merge storms: segment explosion, I/O saturation, and refresh tuning

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch merge storms: segment explosion, I/O saturation, and refresh tuning

Search latency climbs, indexing slows, and heap usage rises while cluster health stays green. The cause is often a merge storm. Background Lucene segment consolidation has fallen behind, leaving nodes with hundreds or thousands of small segments. Each extra segment adds search overhead, consumes file descriptors, and increases memory pressure. This guide covers how merge storms develop, how to confirm the diagnosis, and how to fix them without making things worse.

Elasticsearch monitoring checklist: the signals every production cluster needs

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch monitoring checklist: the signals every production cluster needs

Elasticsearch failures cascade. A long GC pause on one node causes it to miss fault detection checks; the master removes it. Shards relocate to survivors, increasing heap pressure and thread pool load. If disk is near the high watermark, relocation I/O pushes other nodes toward flood stage, which sets indices to read-only and blocks writes. By the time GET /_cluster/health returns red, the leading indicators fired minutes ago.

Elasticsearch monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch monitoring maturity model: from survival to expert

Incidents escalate when teams monitor only the survival layer and miss leading indicators that predict cascades. A rising heap floor, a growing segment count, or a master task backlog surface long before cluster health turns red.

This guide organizes Elasticsearch monitoring into four levels: survival, operational, mature, and expert. Each level adds signals that reduce mean time to detection and prevent composite failure patterns. Build level 1 before going live, level 2 before handling production traffic, and levels 3 and 4 after your first serious incident.

Elasticsearch node left the cluster: fault detection, reallocation, and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch node left the cluster: fault detection, reallocation, and recovery

Your cluster health turned yellow and number_of_nodes dropped by one. The master logs a NODE_LEFT event, shards are unassigned, and the remaining nodes absorb extra load. In the next minute, the allocator decides whether to move data. Misread the cause and a transient restart becomes an expensive reallocation storm, or a genuine hardware failure goes unaddressed while replicas rebalance.

This guide covers how Elasticsearch decides a node is gone, what happens to its shards, and how to recover without deepening the incident.

Elasticsearch node OOM-killed: heap ceiling, page cache, and container limits

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch node OOM-killed: heap ceiling, page cache, and container limits

An Elasticsearch node leaves the cluster, restarts seconds later via systemd or a supervisor, and is killed again. Kernel logs show the OOM-killer terminated the Java process. heap.percent often looks reasonable right up until the kill.

The JVM heap is only one component of resident set size. Off-heap allocations, memory-mapped Lucene segments, and co-located processes all compete for the same memory budget. In containers, the cgroup limit is the hard boundary, not the host’s physical RAM.

Elasticsearch pending cluster tasks backlog: the master can't keep up

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch pending cluster tasks backlog: the master can’t keep up

Index creation hangs, shards stop allocating, and administrative settings updates time out. GET /_cluster/pending_tasks shows a queue that grows instead of draining, with some tasks waiting for minutes. The elected master cannot keep up with cluster state updates, and every metadata-dependent operation stalls.

Elasticsearch processes cluster state changes serially on the master. Index creation, mapping updates, shard allocation decisions, and settings changes all enter a single priority-ordered queue. When tasks arrive faster than the master can compute, serialize, and publish the updated state to the rest of the cluster, the backlog grows. A healthy cluster typically holds zero to five pending tasks, each resolving in under a second. Once the queue exceeds a few hundred tasks, or any individual task ages past several minutes, the cluster is approaching an operational outage.

Elasticsearch refresh_interval tuning: search visibility vs indexing throughput

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch refresh_interval tuning: search visibility vs indexing throughput

Elasticsearch makes newly indexed documents searchable within one second by default. Every refresh creates a new Lucene segment, invalidates per-segment query caches, and adds merge debt. During bulk loads, reindexing, or high-throughput indexing, the default interval becomes a throughput bottleneck. Tuning refresh_interval requires understanding its interaction with the write path, the merge scheduler, and the query cache.

In self-managed clusters, the default index.refresh_interval is one second, though recent versions implement lazy refresh: an index refreshes every second only if it received a search request in the last 30 seconds. If idle, auto-refresh stops until the next search arrives. Explicitly setting refresh_interval overrides lazy refresh and forces refreshes on the interval regardless of search traffic. In Elastic Cloud Serverless, the default is five seconds with a hard floor of five seconds (or -1). Setting the interval to -1 disables automatic refresh.

Elasticsearch replica lag: sequence-number gaps and in-sync set removal

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch replica lag: sequence-number gaps and in-sync set removal

A replica shard is falling behind its primary. The local checkpoint on the replica is lower than the global checkpoint on the primary, and the gap is growing across consecutive samples. If the replica cannot acknowledge writes fast enough, Elasticsearch removes it from the in-sync set. At that point, wait_for_active_shards semantics change for the replication group and data redundancy is reduced. The replica must undergo peer recovery to rejoin. Read sequence-number checkpoints, find the bottleneck, and stop the lag before the master drops the replica.

Elasticsearch search latency high: query phase, fetch phase, and the slow shard

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch search latency high: query phase, fetch phase, and the slow shard

Elasticsearch P95 search latency jumped from 50 ms to multiple seconds. Cluster health is green, search thread pool queues are growing, and users report timeouts. Elasticsearch executes every search as a two-phase scatter-gather across targeted shards, so the slowest shard sets the floor for response time. The _nodes/stats API reports cumulative counters, not per-shard or per-query latencies, meaning one bad shard or one expensive query can be invisible to cluster-wide averages while destroying tail latency. Determine whether time is lost in the query phase, the fetch phase, or the coordinating node merge step, then isolate the outlier.

Elasticsearch search thread pool rejections: failed queries and small search queues

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch search thread pool rejections: failed queries and small search queues

Your application logs show HTTP 429 responses from Elasticsearch. The error is es_rejected_execution_exception and the message points to the search thread pool. User-facing queries are failing, not just slowing down.

The search thread pool uses a fixed number of threads and a bounded queue. The default queue size is 1000. Because search is a scatter-gather operation across shards, a single expensive query can hold a thread for seconds while the coordinating node waits for every shard to respond. The queue drains slowly, and under burst or pathological load it fills fast. Once full, Elasticsearch rejects new search requests immediately.

Elasticsearch shard recovery stuck: throttling, translog replay, and concurrent limits

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch shard recovery stuck: throttling, translog replay, and concurrent limits

You restart a data node during a rollout. Ten minutes later the cluster is still yellow. You check _cat/recovery and see a replica pinned at 38 percent for the last hour. The network is not saturated, the source node is healthy, and there are no obvious errors in the logs, yet the shard refuses to reach STARTED state.

Stuck recovery usually comes down to four constraints: a bandwidth throttle capping transfer speed, a large translog that must replay sequentially, a concurrent recovery limit serializing work, or a disk watermark on the target node blocking allocation entirely.

Elasticsearch slow search after restart: cold OS page cache and warmup

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch slow search after restart: cold OS page cache and warmup

Restart an Elasticsearch node and searches that normally return in tens of milliseconds now take seconds. CPU stays low, disk read throughput spikes, and iowait climbs. This is not a failing disk, a runaway query, or a JVM heap problem. It is a cold OS page cache.

Elasticsearch relies on the OS filesystem cache to serve search requests from Lucene segment files. After any restart, that cache is empty. The kernel must read segments from disk into memory on demand. Until the working set is resident, queries incur disk I/O that should have been cache hits. Depending on the ratio of dataset size to available RAM, warmup can last minutes to hours.

Elasticsearch snapshot failed or partial: backups that silently stop working

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch snapshot failed or partial: backups that silently stop working

You check backups and the last successful snapshot is three days old. Or you see PARTIAL snapshots where only some indices were captured. Cluster health is green, indexing and search are fine, but your recovery point is slipping. Elasticsearch does not fail the cluster when snapshots break, so the problem surfaces only when someone asks, “When did we last test a restore?”

Elasticsearch this action would add too many shards: max_shards_per_node limit

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch this action would add too many shards: max_shards_per_node limit

Creating an index or rolling over a data stream returns HTTP 400 validation_exception: “this action would add [N] shards, but this cluster currently has [X]/[Y] maximum normal shards open”. The cluster has hit cluster.max_shards_per_node, which defaults to 1000 open shards per non-frozen data node. Raising the limit via _cluster/settings unblocks writes but postpones the outage. The durable fix is consolidation.

Elasticsearch thread pool queue growing: the precursor to rejection

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch thread pool queue growing: the precursor to rejection

A climbing write or search queue in _cat/thread_pool means a node is receiving work faster than it can complete it. Rejections are the lagging indicator. By the time clients see EsRejectedExecutionException, the cluster is already degraded.

For the write pool, the default queue size is 10000 (ES 7.x+). For search, it is 1000. Sustained write queues above 1000 or search queues above 100 warrant investigation. The management pool is different: even small amounts of sustained queuing mean the master is falling behind on cluster state operations, which blocks allocation, mapping updates, and recovery.

Elasticsearch TLS certificate expiry: cluster fragmentation and client lockout

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch TLS certificate expiry: cluster fragmentation and client lockout

A node reboots and never rejoins the cluster. Kibana shows connection errors while your data pipeline buffers events. curl returns a TLS handshake failure even though the Elasticsearch process is still listening on port 9200. In Elasticsearch 8.x, security is enabled by default: every node and client relies on TLS certificates. When they expire, failure is abrupt and total. Transport-layer expiry fragments the cluster by rejecting inter-node handshakes. HTTP-layer expiry locks out clients while the cluster internals may still operate. There is no built-in grace period. At the expiry timestamp, connections fail immediately, often with no prior warning in the application logs.

Elasticsearch too many segments per shard: search slowdown and force-merge

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch too many segments per shard: search slowdown and force-merge

Searches that used to complete in tens of milliseconds are now taking hundreds. CPU on data nodes climbs without a traffic spike. Cluster health stays green, but query latency degrades and timeouts increase. The problem is invisible at the cluster level: shards have accumulated too many Lucene segments, and every query scans every segment in every relevant shard. When a shard holds more than roughly 100 segments, the background merge policy has fallen behind. Search slows, file descriptor usage rises, and merge I/O competes with indexing and queries. Since Elasticsearch 7.7, most segment metadata moved off-heap, so the node may not run out of JVM heap, but search performance degrades all the same.

Elasticsearch too many shards per node: overallocation and the heap tax

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch too many shards per node: overallocation and the heap tax

Searches slow from milliseconds to seconds. Nodes drop out after long GC pauses, triggering shard relocations that overload the remaining nodes. Cluster health stays green while _cat/nodes shows climbing heap and segments.memory growing steadily. The cause is usually shard overallocation: thousands of open shards with individual nodes carrying more than their heap can sustain.

Each shard is a self-contained Lucene index. Elasticsearch keeps per-segment metadata in heap memory, and that overhead is charged per field, not per segment. A node with 1,000 shards may be spending multiple gigabytes of heap on metadata alone, leaving less room for caches, aggregations, and in-flight requests.

Elasticsearch translog growing: flush problems, durability, and slow recovery

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch translog growing: flush problems, durability, and slow recovery

Shard recovery estimates climb from minutes to hours. A node restart stalls during translog replay. Disk usage creeps up on data nodes, and uncommitted translog size per shard is past the flush threshold. The translog is growing faster than Elasticsearch can flush it to Lucene commit points.

Every indexing operation appends to the per-shard write-ahead log before the document is searchable. A flush commits those operations to a Lucene segment and truncates the log. When flush cannot keep pace with the write path, uncommitted operations accumulate. During recovery, Elasticsearch replays every operation sequentially. A multi-gigabyte translog translates directly into a multi-hour recovery window, extending time-to-recovery and increasing cascade risk if another node fails during replay.

Elasticsearch unassigned shards: reading allocation explain and fixing each reason

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch unassigned shards: reading allocation explain and fixing each reason

Yellow or red cluster health with unassigned_shards > 0 means the allocator cannot place one or more shard copies on any node. Missing primaries block queries and risk data loss; missing replicas only cost redundancy. Do not guess from the cluster color. The allocator already knows why it rejected every node. Ask it.

What this means

Unassigned primaries make their data unreachable. Affected indices return partial results or fail. Unassigned replicas remove redundancy; a second failure on those primaries drops the data. The master allocator evaluates every node through a chain of deciders: disk watermarks, allocation filters, awareness attributes, the same-shard rule, and retry limits. When every node is rejected, the shard stays UNASSIGNED until the blocking condition clears or you intervene.

How Elasticsearch actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How Elasticsearch actually works in production: a mental model for operators

Elasticsearch is a distributed search and analytics engine built on Apache Lucene. Every node runs interacting subsystems that compete for the same resources. If you treat it as a black box that stores JSON, you will miss the failure modes that kill clusters at 3 a.m. The interactions between subsystems determine whether a cluster survives a traffic spike or collapses into a heap-pressure death spiral.