$ guides / elasticsearch ▌

ELASTICSEARCH · OPERATIONS PLAYBOOK

Running Elasticsearch in production, without the 3 a.m. surprises

A JVM with a hard heap ceiling, shards that are self-contained Lucene indexes, a master holding the whole cluster state, fork-free but fork-fragile persistence through refresh and merge, and disk watermarks that turn an index read-only. The mental model of how Elasticsearch actually behaves under load, where it tends to break, what to monitor as your operation matures, and the runbooks for the incidents you'll see.

> Start with the monitoring checklist → # Jump to the full guide list

Elasticsearch is famously easy to start with a single node and famously unforgiving once the shard count, the field count, the heap, or the disk grows past what the defaults assumed.

The defaults work. Until the heap floor creeps past 85% and an old-generation GC pause exceeds the fault-detection timeout, the master removes the node, and shard reallocation piles pressure onto the survivors. Until a disk crosses the flood-stage watermark and every write to the index comes back cluster_block_exception ... FORBIDDEN/12/index read-only / allow delete (api). Until dynamic mapping invents a field per unique key and the cluster state balloons on every node with Limit of total fields [1000] in index has been exceeded. Until bulk indexing outruns the merge scheduler, segments pile up, and the write thread pool starts answering with EsRejectedExecutionException. Until one high-cardinality aggregation trips the parent breaker with CircuitBreakingException: [parent] Data too large.

These guides are written for engineers who already run Elasticsearch, not for people learning what an index is. The goal is to give you the mental model of how the cluster actually behaves under load, the failure patterns that keep recurring, the monitoring story that catches problems before they page anyone, and the runbooks you wish someone had handed you before your last incident.

How Elasticsearch actually runs in production

Elasticsearch is not just a search box. It is a master node publishing a cluster state, a set of self-contained Lucene shards behind bounded thread pools, a JVM heap with a hard ceiling guarded by circuit breakers, and a disk subsystem governed by watermarks. Most production failures live between these layers, not inside any one of them.

clients / coordinating node

Every request lands on a coordinating node. For writes it routes each document to the right primary by <code>_id</code> hash; for searches it scatters the query to one copy of every relevant shard and gathers the results. The slowest shard sets search latency, and merging large aggregations here can spike heap on this one node.

COORDINATE

cluster coordination (master)

One elected master holds the cluster state — every index, shard, mapping, alias, and node. It is published synchronously on every change. A slow or overwhelmed master, or a state bloated by mapping explosion, stalls allocation, index creation, and recovery for the whole cluster.

MASTER

thread pools + circuit breakers

Dedicated bounded pools (<code>write</code>, <code>search</code>, <code>get</code>, <code>management</code>) queue work; when a queue fills the pool rejects with HTTP 429. Circuit breakers (<code>parent</code> at 95% of heap, <code>fielddata</code>, <code>request</code>) reject operations that would OOM the node before they run.

ADMIT

shard layer (Lucene)

Each shard is a self-contained Lucene index and the unit of work distribution and failure. Primaries are fixed at creation; replicas add redundancy. The allocator places shards subject to disk watermarks, awareness attributes, and filters. Too many shards taxes heap, file descriptors, and the cluster state.

SHARD

write path (translog → segment)

A write hits the <code>translog</code> on disk for durability, enters an in-memory buffer, becomes searchable on <code>refresh</code> as a new Lucene segment, and becomes self-durable on <code>flush</code> when the translog is truncated. Then it replicates to replica shards.

INDEX

merge + segments

Background merges combine small segments into larger ones and reclaim deleted-document space. When ingest outruns the merge scheduler, segments pile up — search slows, file descriptors climb, and segment metadata consumes heap.

MERGE

JVM heap + GC

The heap has a hard ceiling (keep it ≤26 GB for compressed OOPs) holding caches, breakers, segment metadata, and the cluster state. Young GC is frequent and cheap; long old-generation GC pauses stop the world and get the node removed from the cluster.

HEAP

OS page cache + disk

Lucene mmaps segment files and relies on the kernel page cache, not heap, for fast reads. Disk watermarks (low 85%, high 90%, flood 95%) drive allocation: above flood stage indices go read-only. Merges, fsyncs, recovery, and snapshots all compete for this I/O.

DISK

Why this matters: a single symptom — Elasticsearch is slow, or red, or rejecting writes — can come from heap pressure, a long GC pause, a disk watermark, a mapping explosion bloating the cluster state, a merge backlog, an expensive aggregation on the coordinating node, or a cold page cache. The symptom is shared but each layer has a different signal and a different fix.

The failures you'll actually see

Most Elasticsearch incidents fall into a small set of recurring patterns. Recognise the shape, and triage gets dramatically faster.

CRITICAL

The heap pressure death spiral

Something fills the heap — too many shards' segment metadata, a mapping explosion bloating the cluster state, fielddata on a text field, or a giant aggregation. Old GC fires more often, each pause is stop-the-world, and once a pause exceeds the fault-detection timeout the master removes the node. Its shards relocate onto the survivors, raising their heap pressure. The cascade can take down the cluster.

jvm.mem.heap_used_percent sustained above 85% with the post-GC floor rising
jvm.gc.collectors.old.collection_time climbing, individual pauses over 10s
breakers.parent.tripped incrementing and write/search queues growing
number_of_nodes dropping during GC peaks, then unassigned shards appearing

Investigate →

CRITICAL

The disk watermark cascade

A node crosses the high watermark (90%) and the allocator relocates shards off it — generating I/O that pushes the target nodes toward their own watermarks. With homogeneous storage they all approach the flood stage (95%) together, where index.blocks.read_only_allow_delete is set and every write returns FORBIDDEN/12/index read-only / allow delete (api).

high disk watermark [90%] exceeded in the logs
relocating_shards elevated while indexing rate drops
index.blocks.read_only_allow_delete appearing in index settings
writes failing cluster-wide with cluster_block_exception

Investigate →

IMMINENT

The mapping explosion

Dynamic mapping invents a new field for every unique key in incoming JSON — Kubernetes labels, user-defined tags, unbounded keys. Field counts climb into the tens of thousands, the cluster state balloons on every node (each holds a full copy in heap), the master strains to publish it, and new documents are rejected with Limit of total fields [1000] in index has been exceeded.

Limit of total fields [N] in index has been exceeded on indexing
total mapping field count climbing without bound in _cluster/stats
master node heap rising in step with cluster state changes
pending cluster tasks backing up during mapping updates

Investigate →

ACTIVE

The merge backlog and segment explosion

A high ingest rate creates many small segments. The merge scheduler can't keep up — I/O is saturated or the refresh interval is too low — so segment count per shard climbs past 100. Search slows because every segment is scanned, file descriptors grow, and segment metadata eats heap. Force-merging a live index makes it worse.

segments.count per shard above 100 and rising in actively searched indices
merges.current pinned at max_thread_count with merges.total_time climbing
search latency degrading with no change in query pattern
open file descriptors and segments.memory per node trending up

Investigate →

ACTIVE

The write rejection storm

The write thread pool queue fills and the node rejects with HTTP 429 / EsRejectedExecutionException. Clients without retry-and-backoff silently lose documents; clients that retry aggressively make it worse. The cause is usually an undersized cluster, hot-spotted shards, a slow ingest pipeline, or merge/GC stalls reducing effective throughput.

thread_pool.write.rejected incrementing (track the delta, not the total)
EsRejectedExecutionException or HTTP 429 at the indexing client
write queue oscillating near its limit before rejecting
indexing rate dropping while ingest sources keep sending

Investigate →

IMMINENT

Red cluster: unassigned primaries

Cluster health turns red when one or more primary shards are unassigned — data for those indices is unavailable and queries return partial results or fail. A node left, a disk watermark blocks allocation, a shard copy is corrupt, or allocation has exhausted its retries. Cluster health is a lagging indicator; the leading signals came first.

GET /_cluster/health status:red with nonzero unassigned_shards
unassigned.reason of NODE_LEFT or ALLOCATION_FAILED in _cat/shards
searches returning all shards failed for affected indices
allocation/explain naming the rule that blocks placement

Investigate →

Elasticsearch monitoring maturity levels

Elasticsearch observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your cluster matters. Most production clusters should land at the second level.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor. With these signals you can answer one question: is the cluster still functioning? You will not learn what broke, but you will learn that something broke before users do. Survival is enough for dev clusters and throwaway log stacks.

Node reachability (GET /) Does the node answer on 9200 — or is it down, partitioned, or stuck in GC?
Cluster health status Green, yellow, or red? Red means a primary is unavailable.
jvm.mem.heap_used_percent per node How close is each node to the heap ceiling and GC trouble?
Node count vs expected Did a node leave the cluster without your permission?
Disk used percent vs watermarks Is any node approaching 85% / 90% / 95%?
unassigned_shards Are any primaries or replicas without a home?

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production clusters should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident on its own: rejections, GC, watermarks, merges, unassigned shards, master pressure.

Thread pool rejected (write, search) Are writes or queries being refused? Track the rate, not the total.
Thread pool queue depth Queuing is the precursor to rejection — catch it early.
Old GC count and time (rate) Is old-generation GC firing more often and pausing longer?
Search and indexing latency Query phase, fetch phase, and per-document write time vs baseline.
Unassigned shard count and reason What is the allocator refusing to place, and why?
Pending cluster tasks Is the master keeping up with cluster state changes?
Circuit breaker trips Is parent / fielddata / request rejecting to avoid OOM?
Segment count and merge activity Are merges keeping up with the refresh rate?
Disk watermark proximity How many days until the high watermark at this growth rate?
Snapshot status and ILM health Are backups completing and is ILM transitioning indices?

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone up. The heap floor creeping, the cluster state growing, a translog outpacing flush, fielddata appearing where doc_values should be, a replica falling out of the in-sync set. None of these will page you on day one. They become page-out incidents on day thirty.

Heap sawtooth floor (post-GC minimum) Is the after-GC baseline rising — long-lived objects accumulating?
segments.memory per node How much heap is segment metadata consuming as shards grow?
Fielddata cache size and evictions Is a text field being aggregated without doc_values?
Cluster state size / field count Is mapping explosion bloating the state on every node?
Translog size and uncommitted ops Is flush keeping up — and how slow will recovery be?
Refresh and flush time Is the write path I/O-bound?
Shard recovery activity Are recoveries progressing, or stuck and competing with traffic?
Per-index indexing/search rates Is one node or index hot-spotting while others idle?
Replica lag (sequence-number gap) Is a replica falling behind the primary's global checkpoint?

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. Cluster state version churn, per-shard segment distribution, merge throttle time, page-cache adequacy, indexing pressure, adaptive replica selection. Most teams never need every signal here. Add the ones your incident history says you do.

Cluster state version churn rate How fast is the state changing — a master-overload leading indicator?
Per-shard segment count distribution Which individual shard is the outlier?
merges.total_throttled_time Are merges being throttled because I/O is under pressure?
OS page-cache adequacy (os.mem.free) Is there enough free RAM to cache the working segment set?
Indexing pressure stats (7.9+) Memory backpressure at coordinating / primary / replica stages.
Hot threads (GET /_nodes/hot_threads) What exactly is each node's CPU doing right now?
Adaptive replica selection stats Which nodes are being avoided for being slow?
Script compilation rate / cache evictions Are per-request dynamic scripts hammering compilation?

Operating mistakes worth avoiding

The traps Elasticsearch teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

Treating green as healthy

A green cluster can be one GC pause away from catastrophe. Green only means shards are assigned — not that the cluster has headroom. It can have a rising heap floor, a stuck ILM, failed snapshots, and a shard count climbing toward the death spiral, all while reporting green. Monitor heap, disk, and shard count independently of cluster health.

Watching heap peak instead of the heap floor

Everyone watches <code>heap_used_percent</code>, but that's the sawtooth peak, and the sawtooth is normal. The operationally critical signal is the minimum <em>after</em> old GC — the floor. If the floor is rising, no amount of GC will save you, and it's the best single leading indicator of the death spiral. Almost nobody tracks it.

Ignoring shard count as a first-class resource

Teams add indices without thinking about shard accumulation. Each shard carries fixed overhead in heap, file descriptors, and the cluster state. Shards grow by ten a day for a year, nobody graphs shards-per-node, and one day the cluster enters the death spiral with 4000 shards per node. Keep it under a few hundred per node and use ILM rollover.

Letting dynamic mapping run unbounded

Indexing unstructured JSON with dynamic mapping creates a field per unique key. Kubernetes labels and user tags can push a mapping into tens of thousands of fields, bloating the cluster state on every node and destabilising the master. Set <code>index.mapping.total_fields.limit</code>, use <code>dynamic: strict</code> or the <code>flattened</code> type, and shape the data in an ingest pipeline.

Setting heap above ~26 GB

A heap above roughly 26–30.5 GB disables compressed ordinary object pointers and effectively wastes about 30% of the heap, while starving the OS page cache that Lucene depends on. Set <code>-Xms</code> equal to <code>-Xmx</code>, leave half of RAM for the page cache, and run two 30 GB nodes rather than one 60 GB node.

Raising limits instead of fixing root causes

Increasing the thread pool queue size, raising a circuit breaker limit, or bumping <code>cluster.max_shards_per_node</code> just delays the failure and adds memory pressure. A bigger queue absorbs the overload until one more GC pause tips it over. Fix the capacity, the query, or the topology — don't tune your way out of a capacity problem.

Tolerating yellow cluster health

Yellow means replicas are unassigned. Teams run yellow for weeks because "it's just test indices," which trains everyone to ignore it — so when a critical index loses its replicas for a real reason, nobody notices until a node failure causes data loss. Investigate sustained yellow with <code>allocation/explain</code>.

Assuming a successful snapshot is a valid backup

Snapshots run on a schedule and silently fail on repository permissions, a full bucket, or a concurrent-operation conflict — and a <code>PARTIAL</code> snapshot looks almost like success. Nobody notices until disaster recovery. Alert on "no successful snapshot in more than twice the schedule interval," and test restores periodically.

Elasticsearch runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

JVM heap, GC, and circuit breakers

▸

Cluster health, shards, and allocation

▸

Disk, watermarks, and flood stage

▸

Mapping explosion and cluster state

▸

Thread pools, rejections, and queues

▸

Indexing, refresh, merges, and translog

▸

Search latency and the coordinating node

▸

Master stability and coordination

▸

Recovery, snapshots, and ILM

▸

Security and access

WHERE TO GO NEXT

Setting up Elasticsearch monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.

> Start with the checklist > Back to Operations Guides