$ guides / logstash / logstash-out-of-memory-java-heap-space ▌

Operations Guides

Logstash OutOfMemoryError: Java heap space and how to recover

java.lang.OutOfMemoryError: Java heap space in the Logstash log means the JVM could not satisfy an allocation request and GC could not reclaim enough heap to proceed. The process exits or gets kernel OOM-killed, and every pipeline it was running stops with it.

Before the hard crash, there is usually a warning period: the GC death spiral. Heap fills, garbage collection runs longer and more frequently, throughput collapses. This can last minutes or hours before the JVM finally fails to allocate. If you catch the spiral, you can intervene before data is lost. If you only catch the OOM, you are in recovery mode.

The default JVM heap in Logstash ships at 1GB (-Xms1g -Xmx1g in jvm.options). This is too small for most production workloads.

What this means

OutOfMemoryError: Java heap space means the JVM’s garbage collector could not free enough heap to satisfy an allocation request. The allocation might be a batch of event objects, a large JSON document being parsed, a filter’s internal buffer, or any other heap consumer. When the JVM exhausts heap and GC cannot reclaim enough, it throws this error and terminates.

This is distinct from two related failure modes:

GC death spiral: The process stays alive but spends increasing time in garbage collection, leaving less time for event processing. Throughput approaches zero while the process appears healthy to a liveness check. This can persist for minutes or hours and often precedes a hard OOM.
Direct buffer memory OOM: The error reads java.lang.OutOfMemoryError: Cannot reserve N bytes of direct buffer memory instead of Java heap space. This happens when Netty-based inputs (Beats, TCP, HTTP) exhaust direct memory.

The critical operational question is whether you are seeing the sudden terminal crash, or the slow spiral that predicts it. Different symptoms, different response windows.

flowchart TD
    A[Heap fills with live objects] --> B[GC frequency and duration increase]
    B --> C[Throughput drops, events accumulate in queue]
    C --> D[Heap fills faster from backlog]
    D --> B
    B --> E[Post-GC floor rises, old-gen fills]
    E --> F{JVM can still allocate?}
    F -- No --> G["OutOfMemoryError: Java heap space"]
    F -- Barely --> H[Process alive, near-zero useful work]
    H --> G
    G --> I[Process exits or kernel OOM-kills]

Common causes

Cause	What it looks like	First thing to check
Heap too small	1GB default with production traffic; OOM under normal load	`jvm.options` for `-Xms` / `-Xmx` values
Oversized events or JSON field explosion	Single events consuming hundreds of MB during parse; stack trace in JSON parsing code	Event source for unusually large payloads
In-flight events too large	OOM scales with `pipeline.workers * pipeline.batch.size`	Worker count and batch size in config
In-memory queue at capacity	Memory queue full, heap dominated by queued event objects	Queue type and `events_count`
Filter plugin memory leak	Post-GC floor rises steadily over hours or days with no config change	Old-gen pool trend after GC

Quick checks

Run these read-only commands to assess the situation. If the process has already crashed, skip to checking logs and dmesg.

# Check if the process is still alive
pgrep -f org.logstash.Logstash

# Check API responsiveness (slow response indicates GC pressure)
time curl -sS --max-time 5 http://127.0.0.1:9600/_node/stats/jvm?pretty

# Check heap usage and memory pools
curl -sS http://127.0.0.1:9600/_node/stats/jvm | python3 -c "
import sys,json
j = json.load(sys.stdin)['jvm']
m = j['mem']
print(f\"Heap: {m['heap_used_in_bytes']//1048576}MB / {m['heap_max_in_bytes']//1048576}MB ({m['heap_used_percent']}%)\")
old = m['pools']['old']
print(f\"Old-gen: {old['used_in_bytes']//1048576}MB / {old['max_in_bytes']//1048576}MB\")
"

# Check GC time and frequency
curl -sS http://127.0.0.1:9600/_node/stats/jvm | python3 -c "
import sys,json
gc = json.load(sys.stdin)['jvm']['gc']['collectors']
for gen, s in gc.items():
    print(f\"{gen}: {s['collection_count']} collections, {s['collection_time_in_millis']}ms total\")
"

# Check queue depth (growing queue under GC pressure compounds the problem)
curl -sS http://127.0.0.1:9600/_node/stats/pipelines | python3 -c "
import sys,json
p = json.load(sys.stdin)['pipelines']
for name, d in p.items():
    q = d.get('queue', {})
    print(f\"{name}: {q.get('events_count',0)} events queued\")
"

# Check for kernel OOM kill in system logs (may require root)
dmesg -T | grep -i 'out of memory\|oom.*kill\|killed process'
journalctl -u logstash --since '1 hour ago' | grep -i 'OutOfMemoryError\|heap space'

# Check current JVM heap settings
grep -E '^-Xm' /etc/logstash/jvm.options

How to diagnose it

Confirm whether the process is alive or dead. If dead, check dmesg or journalctl for OOM killer messages. If alive but unresponsive, suspect GC death spiral rather than a completed OOM.
Distinguish hard OOM from GC death spiral. A hard OOM produces OutOfMemoryError in the log and the process exits. A GC death spiral shows high GC overhead (more than 20% of wall time), rising old-gen usage, and degraded throughput, but the process stays alive. The spiral may eventually trigger a hard OOM, or the kernel may OOM-kill the process when total RSS exceeds system limits.
Compute GC overhead as a percentage of wall time. Take two samples of jvm.gc.collectors.old.collection_time_in_millis 60 seconds apart. Divide the delta by 60000 (wall time in ms). If old-gen GC alone consumes more than 10% of wall time, you are in significant memory pressure. Above 20%, throughput is severely impaired.
Check the post-GC floor, not the peak. The meaningful heap signal is the level after garbage collection, not before. A rising post-GC floor (old-gen used_in_bytes after collections) indicates live objects accumulating. A sawtooth pattern where the floor stays stable is normal JVM behavior.
Calculate the in-flight event count. Multiply pipeline.workers by pipeline.batch.size. On a 16-core machine with default settings (16 workers, batch size 125), that is 2000 events in flight. If events average 500KB after JSON expansion, the in-flight set alone needs approximately 1GB of heap. With the default 1GB heap, this is a guaranteed OOM.
Look for oversized events. If the OOM stack trace shows jruby.RubyStringConverter or JSON parsing code, a single large event may be the trigger. The Jackson string length limit defaults to 200MB (-Dlogstash.jackson.stream-read-constraints.max-string-length=200000000 in jvm.options). A single event near that limit can consume most of the heap during parsing.
Check for persistent queue overhead. Each pipeline with PQ enabled requires at least head and tail pages in native memory (default 64MB each). With 10 pipelines, PQ alone consumes approximately 1.28GB of native memory before any heap usage. This does not count toward heap, but it counts toward total process RSS and can trigger container OOM kills.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`jvm.mem.heap_used_percent`	Overall heap pressure	Sustained above 85% after GC
`jvm.mem.pools.old.used_in_bytes`	Long-lived object accumulation	Rising post-GC floor over hours
`jvm.gc.collectors.old.collection_time_in_millis`	GC overhead stealing processing time	More than 10% of wall time (compute as rate)
`flow.output_throughput`	Whether the pipeline is doing useful work	Drops to zero while input continues
`queue.events_count`	Backlog building under memory pressure	Growing while throughput drops
Process RSS	Total memory footprint vs system or container limit	RSS approaching container limit

Fixes

Increase JVM heap size

The most common fix. The default 1GB heap is insufficient for production.

In /etc/logstash/jvm.options, set:

-Xms4g
-Xmx4g

Always set -Xms equal to -Xmx. If they differ, the JVM spends cycles resizing heap at runtime, which adds latency and can trigger unnecessary GC cycles. The equal setting also means heap_max_in_bytes is fixed from startup, making heap percentage metrics stable from the first second.

Do not set heap above 50% of available system memory. The JVM needs room for off-heap allocations (Netty buffers, PQ memory-mapped files, JVM internal structures). In containers, set heap relative to the container memory limit, not the host. The bundled JDK in Logstash 8.x and 9.x has container support enabled (-XX:+UseContainerSupport), so it detects cgroup limits.

After changing jvm.options, restart Logstash. There is no hot-reload for JVM settings.

Reduce in-flight event count

If increasing heap is not possible due to memory constraints, or the OOM persists after increasing, reduce the number of events held in memory simultaneously.

The in-flight count is pipeline.workers * pipeline.batch.size. Lower pipeline.batch.size in logstash.yml or pipelines.yml:

pipeline.batch.size: 64

This trades throughput for memory safety. Each worker holds fewer events, so the total in-flight set is smaller. Test the throughput impact, as smaller batches mean more overhead per event.

Do not reduce pipeline.workers below what your CPU can handle. Fewer workers means slower processing, which grows the queue, which can increase memory pressure rather than decrease it.

Address oversized events

If the OOM stack trace points at JSON parsing or string conversion, a single large event is the likely trigger. Sources of oversized events:

Log producers embedding full stack traces, JSON payloads, or binary blobs in a single log line
Multiline codec accumulating many lines into one event without a size limit
JSON with deeply nested or exploded field structures

Mitigations:

Add size guards at the source (log producer configuration)
Use the truncate filter to cap field lengths before they consume heap
Split large events earlier in the pipeline
Review the Jackson string length limit setting and lower it if your events should never approach 200MB

Enable heap dump on OOM

-XX:+HeapDumpOnOutOfMemoryError writes a heap dump file when the OOM occurs. This file is your best tool for post-mortem analysis. It is included in the default jvm.options file in recent Logstash versions. Verify it is present:

grep HeapDumpOnOutOfMemoryError /etc/logstash/jvm.options

If not present, add it. The dump is written to the JVM working directory by default. A heap dump for a 4GB heap produces a file of approximately 4GB. Ensure the working directory has enough disk space, and clean up old dumps.

Analyze the dump with Eclipse MAT, VisualVM, or jhat to identify which objects dominate the heap.

Distinguish from direct buffer OOM

If the error message is Cannot reserve N bytes of direct buffer memory rather than Java heap space, increasing -Xmx will not fix it. Direct memory has its own limit. By default, the JVM sets MaxDirectMemorySize equal to -Xmx, so a 1GB heap also implies 1GB of direct memory.

In Logstash 8.x, Beats/TCP/HTTP inputs allocate Netty buffers from direct memory. Options:

Set pipeline.buffer.type: heap in logstash.yml to move these allocations onto the Java heap (this is reportedly the default in Logstash 9.0+)
Increase direct memory explicitly with -XX:MaxDirectMemorySize in jvm.options

Even with pipeline.buffer.type: heap, some plugins may still use direct memory. If direct buffer OOM persists, check plugin-specific buffer behavior.

Prevention

Size heap for production, not defaults. 1GB is a development default. Size to your workload, keep -Xms equal to -Xmx, and never exceed 50% of system or container memory.
Monitor the post-GC floor, not the peak. A heap alert that fires on heap_used_percent > 80% will trigger during every normal GC cycle and get silenced. Alert on the post-GC old-gen level rising over time instead.
Calculate in-flight against heap. Know your pipeline.workers * pipeline.batch.size product. Estimate average event size. Ensure the product fits comfortably in heap with room for GC overhead.
Track GC overhead as a rate. Compute delta(collection_time_in_millis) / delta(wall_time) for old-gen collections. Alert above 10% sustained. This catches the GC death spiral before it becomes a hard OOM.
Enable and verify heap dump. Confirm -XX:+HeapDumpOnOutOfMemoryError is in jvm.options. Ensure disk space for the dump file.
Watch total RSS in containers. Container OOM kills happen when RSS exceeds the limit, not when heap exceeds max. PQ pages, direct buffers, and JVM overhead all add to RSS.
Gate alerts on uptime. Cold start behavior (JIT compilation, class loading, PQ replay) can look like memory pressure. Require jvm.uptime_in_millis > 300000 before firing heap or GC alerts.

How Netdata helps

Per-second JVM heap metrics let you see the sawtooth pattern and, more importantly, the post-GC floor trend that predicts OOM before it happens. Standard polling intervals (15 to 60 seconds) often miss the GC floor between collections.
GC collection time and count are collected at per-second resolution, making it straightforward to compute GC overhead as a percentage of wall time and alert on the death spiral pattern.
Correlation between heap, GC, throughput, and queue depth in a single view lets you distinguish a GC death spiral (high GC, dropping throughput, growing queue) from an output bottleneck (low GC, dropping throughput, growing queue) or a CPU-bound filter (low GC, high CPU, growing queue).
ML-based anomaly detection on heap usage and old-gen trends can surface the slow post-GC floor climb that a fixed threshold would miss.
Container-aware memory metrics show RSS alongside heap, which is critical for diagnosing container OOM kills where heap is within limits but total process memory exceeds the cgroup cap.