Logstash ships with a 1GB JVM heap (-Xms1g -Xmx1g in config/jvm.options), unchanged through 7.x, 8.x, and 9.x. That default works for trivial pipelines. It is catastrophically small for production and is the single most common cause of GC death spirals, throughput collapse, and OOM kills.

The fix: increase the heap, set minimum and maximum to the same value, and account for off-heap memory. Set the heap too large and you starve the OS and risk longer full-GC pauses. Set it without understanding off-heap allocation and you get OOM kills with a heap that looks comfortably under capacity. Ignore container cgroup limits and the JVM sizes itself against host RAM, not the container limit.

The 1GB default and what it costs

At 1GB, the heap holds in-flight events, filter plugin state, codec buffers, and the memory queue (if used). With default pipeline.batch.size of 125 and pipeline.workers set to CPU core count, a busy pipeline exhausts 1GB in seconds.

The failure pattern is the GC death spiral: heap fills, GC runs more frequently and for longer durations, throughput drops, events accumulate faster, heap fills faster, and full-GC pauses eventually freeze the pipeline. The process stays alive (process checks pass) while throughput goes to near zero. The monitoring API itself may become unresponsive during prolonged GC pauses.

Signals that distinguish this from normal GC cycling:

  • Post-GC heap floor above 90% sustained
  • Old-gen pool above 85% of its max
  • GC overhead above 20% of wall time
  • Output rate dropping while heap stays high

A 1GB heap reaches this state quickly under any real ingestion load. The first fix is increasing heap. The long-term fix is sizing it correctly.

Why -Xms and -Xmx must be equal

Elastic guidance is explicit: set Xms and Xmx to the same value to prevent the heap from resizing at runtime, which is costly.

When -Xms is less than -Xmx, the JVM starts with a smaller heap and grows it as needed. Each growth operation allocates and zeroes new memory pages, potentially triggers a full GC, and produces an application-visible pause during the resize.

For a streaming pipeline processor, heap resizing pauses are destructive. They happen under load (when the JVM decides it needs more heap), which is exactly when you can least afford pauses. Setting -Xms equal to -Xmx allocates the full heap at startup and eliminates runtime resizing.

The trade-off: the full heap is reserved immediately, even if the pipeline is idle. On a dedicated Logstash host this does not matter. On a shared host or memory-constrained container, you must size the heap to fit within the available memory budget from the start.

How to set the heap

The canonical location is jvm.options:

  • RPM/DEB installs: /etc/logstash/jvm.options
  • Tarball installs: <logstash_home>/config/jvm.options
  • Docker: set via LS_JAVA_OPTS environment variable or a custom jvm.options mounted into the container

Set both values:

-Xms4g
-Xmx4g

Remove or comment out any conflicting -Xms/-Xmx lines. The file is read top to bottom, and later flags on the JVM command line take precedence.

LS_JAVA_OPTS alternative: Set heap via the LS_JAVA_OPTS environment variable:

export LS_JAVA_OPTS="-Xms4g -Xmx4g"

This appends to the JVM arguments constructed from jvm.options. If jvm.options already contains -Xms1g -Xmx1g, the LS_JAVA_OPTS values appear later on the command line and take precedence.

LS_HEAP_SIZE is deprecated. Use jvm.options or LS_JAVA_OPTS instead.

Bug note (fixed in 7.17/8.1): In versions before 7.17 and 8.1, LS_JAVA_OPTS was silently ignored if jvm.options was absent. If LS_JAVA_OPTS seems to have no effect on an older version, verify that jvm.options exists (even if empty).

Sizing heap for production

Elastic recommends no less than 4GB and no more than 8GB for typical ingestion. Heap should not exceed 50-75% of total physical memory, leaving room for off-heap allocation, JVM overhead, and OS page cache.

What consumes heap

ConsumerDescription
In-flight eventspipeline.batch.size (default 125) multiplied by pipeline.workers (default CPU cores). Each event includes all fields.
Filter plugin stateGrok pattern caches, translate dictionaries, GeoIP database references, Ruby filter state.
Codec buffersMultiline codec accumulates partial events in memory until pattern completion.
Memory queueIf using the default memory queue (not PQ), queued events live in heap.
pipeline.buffer.type=heap allocationsSince Logstash 9.0, input plugin buffers (Beats, TCP, HTTP, Elastic Agent) default to heap instead of direct memory.

Sizing checklist

  • Calculate peak in-flight events. batch_size * pipeline.workers * peak_event_size gives a floor. A pipeline with batch_size=125, 8 workers, and 10KB events holds approximately 10MB per batch cycle. Filter state and codec buffers add overhead.
  • Account for all pipelines. In multi-pipeline setups, all pipelines share one JVM heap. Sum the in-flight and state requirements across all pipelines.
  • Factor in pipeline.buffer.type (9.0+). If upgrading from 8.x where pipeline.buffer.type defaulted to direct, allocations that previously consumed direct memory now consume heap. Without a heap increase, this can cause OOM after upgrade.
  • Leave room for off-heap. Reserve at least the same amount as heap for off-heap allocation, JVM internal structures, and OS page cache.
  • Persistent queue overhead. Each PQ requires memory-mapped space for head and tail pages (64MB each, 128MB per pipeline). With 10 pipelines, that is 1.28GB before any heap or direct memory. This is off-heap but competes for physical memory.

When bigger is not better

An oversized heap lengthens full-GC pause times. With G1GC, a large heap can produce multi-second full-GC pauses when old-gen finally fills.

  • Larger heap: less frequent GC, but longer pauses when they happen
  • Smaller heap: more frequent GC, but shorter pauses and faster recovery

For a latency-sensitive streaming pipeline, a 4-8GB heap with frequent but short young-gen collections is preferable to a 16GB heap with rare but devastating full-GC pauses.

Off-heap memory: the hidden budget

JVM heap is not the total memory footprint. Off-heap memory includes:

ComponentWhat it is
Direct memory buffersNetty I/O buffers for network input/output plugins. Default MaxDirectMemorySize equals -Xmx.
JVM internalCompressed class space, code cache, thread stacks, GC data structures.
Memory-mapped filesPQ page files are memory-mapped. Each pipeline’s PQ uses at least 128MB of mapped space.
flowchart TD
    A["Total Physical or Container Memory"] --> B["JVM Heap
-Xms = -Xmx
4-8GB typical"]
    A --> C["Direct Memory
MaxDirectMemorySize"]
    A --> D["JVM Overhead
class space, code cache,
thread stacks, GC"]
    A --> E["PQ Memory-Mapped Files
128MB per pipeline"]
    A --> F["OS Page Cache + Other Processes"]
    C --> G{"pipeline.buffer.type"}
    G -->|"heap (9.0+ default)"| H["Input buffers on heap"]
    G -->|"direct (8.x default)"| I["Input buffers on direct memory"]

MaxDirectMemorySize

By default, the JVM sets MaxDirectMemorySize equal to -Xmx. A 4GB heap reserves up to 4GB of direct memory. On a host with 16GB RAM, a 4GB heap could consume up to 8GB (heap + direct) before JVM overhead or OS needs.

The recommendation is to set -XX:MaxDirectMemorySize to half of heap size:

-XX:MaxDirectMemorySize=2g

for a 4GB heap. MaxDirectMemorySize is not included in the default jvm.options.

pipeline.buffer.type and the 9.0 migration

In Logstash 8.x, pipeline.buffer.type defaulted to direct for Beats, TCP, HTTP, and Elastic Agent inputs. In 9.0.0, the default changed to heap.

  • 8.x to 9.0 upgrade risk. If you upgrade to 9.0 without adjusting heap size, buffer allocations that previously lived in direct memory now consume heap. This can cause OOM.
  • Migration options. Either set pipeline.buffer.type: direct to preserve old behavior, or set pipeline.buffer.type: heap and increase heap accordingly.

Setting pipeline.buffer.type: heap has a diagnostic advantage: buffer allocations are visible in heap dumps, making OOM debugging easier. Direct memory exhaustion does not appear in standard heap analysis tools.

Container and cgroup awareness

In containers (Kubernetes, Docker), the JVM must see the container memory limit, not host RAM. Without cgroup awareness, the JVM sizes itself against total host memory and can be OOM-killed when it exceeds the container limit.

JDK 10+ behavior: -XX:+UseContainerSupport is enabled by default on JDK 10+. The bundled JDK in modern Logstash supports this. However, MaxRAMPercentage defaults to 25%, which may be too conservative for Logstash.

To set heap as a percentage of the container limit:

-XX:MaxRAMPercentage=50

Or set explicit -Xms/-Xmx values that fit within the container limit minus off-heap overhead.

Practical container sizing: For a container with an 8GB memory limit:

  • Heap: 4GB (-Xms4g -Xmx4g)
  • Direct memory: 2GB (-XX:MaxDirectMemorySize=2g)
  • JVM overhead + PQ mapped files + OS: approximately 2GB

This leaves no headroom for PQ growth or unexpected off-heap allocation. In practice, a container running Logstash with PQ needs more room. Either increase the container limit or reduce heap.

cgroup v1 vs v2. On older systems with cgroup v1, container memory detection may be less reliable. Verify by checking the actual heap the JVM selects after startup.

Verifying the configuration

After changing jvm.options and restarting Logstash, verify the JVM picked up the correct values:

# Check effective heap via the monitoring API
curl -sS http://127.0.0.1:9600/_node/stats/jvm | python3 -c "
import sys, json
m = json.load(sys.stdin)['jvm']['mem']
print(f\"Heap max: {m['heap_max_in_bytes'] / 1024 / 1024:.0f} MB\")
print(f\"Heap used: {m['heap_used_in_bytes'] / 1024 / 1024:.0f} MB ({m['heap_used_percent']}%)\")
"

If jcmd is available:

# jcmd requires the same user that runs the JVM
jcmd $(pgrep -f org.logstash.Logstash | head -1) VM.flags

If the reported heap max does not match your configured -Xmx, the configuration was not applied. Common causes:

  • Multiple jvm.options files (check installation vs config directory)
  • LS_JAVA_OPTS overriding the file on older versions
  • jvm.options file missing entirely (the LS_JAVA_OPTS bug on pre-7.17/8.1)

Common pitfalls

  • Heap too large for the host. An 8GB heap on a 12GB host leaves 4GB for off-heap, OS, and other processes. Leads to swapping or OOM kills even though heap metrics look fine.
  • Ignoring off-heap after upgrading to 9.0. If pipeline.buffer.type changed from direct to heap, the same heap size now carries more allocation. Increase heap or set pipeline.buffer.type: direct explicitly.
  • Setting -Xms less than -Xmx in production. Runtime heap resizing causes unpredictable pauses during traffic spikes.
  • Forgetting PQ memory-mapped space. Each PQ pipeline uses at least 128MB of memory-mapped space. Ten pipelines means 1.28GB of additional physical memory pressure invisible in heap metrics.
  • Container JVM seeing host RAM. Without cgroup support or explicit heap flags, the JVM may allocate based on host memory. Always verify effective heap after startup.
  • Not enabling heap dump on OOM. Without -XX:+HeapDumpOnOutOfMemoryError, an OOM kills the process and leaves no artifact for root cause analysis. Enable it and point the dump path to a volume with enough space.

Signals to monitor

SignalWhy it mattersWarning sign
jvm.mem.heap_used_percentOverall heap pressure.Post-GC floor above 85% sustained.
jvm.mem.pools.old.used_in_bytesOld-gen accumulation indicates leak or undersized heap.Rising post-GC floor in old-gen pool.
jvm.gc.collectors.old.collection_time_in_millisFull-GC time directly steals from processing.GC overhead above 10% of wall time (warning), above 20% (severe).
jvm.gc.collectors.old.collection_countFrequency of full-GC cycles.Rate increasing over time.
Process RSSTotal physical memory including off-heap.RSS approaching container or host limit.
Pipeline output throughputWhether useful work is happening.Throughput dropping while heap is high and GC time is rising.

Alert on the post-GC floor, not the peak. The heap sawtooth pattern means peak usage regularly crosses 80% during normal GC cycles. A rising floor (heap level after GC completes) indicates real pressure that will not self-resolve.

How Netdata helps

  • Per-second heap metrics. Netdata collects jvm.mem.heap_used_percent, heap_used_in_bytes, and heap_max_in_bytes at per-second resolution, enough to distinguish normal GC cycling from a rising post-GC floor.
  • Old-gen pool tracking. Per-pool breakdown shows whether pressure is in young-gen (normal churn) or old-gen (accumulation or leak), which determines whether to increase heap or investigate a plugin memory leak.
  • GC time correlation. Correlating gc.collectors.old.collection_time_in_millis with pipeline output throughput in the same view shows whether throughput drops coincide with GC pauses, distinguishing a GC death spiral from an output bottleneck.
  • Container memory awareness. Cgroup-aware collection shows container RSS alongside heap metrics, making off-heap pressure and OOM-kill risk visible without separate tooling.
  • Anomaly detection on heap floor. Anomaly detection on the post-GC floor trend catches slow memory leaks and progressive old-gen accumulation before they trigger a death spiral.