$ guides / logstash / logstash-heap-usage-high-post-gc-floor ▌

Operations Guides

Logstash heap usage high: why the post-GC floor matters more than the peak

Logstash operators hit a recurring trap with JVM heap monitoring. They set an alert for heap_used_percent above 80%, following standard JVM guidance. The alert fires constantly because the JVM heap naturally cycles through allocation and collection. They silence it or raise the threshold. Then a real memory leak or GC death spiral develops, heap stays elevated for hours, and nobody notices because the alert was muted.

The problem is not the threshold. It is the metric choice. heap_used_percent is an instantaneous snapshot that catches a random point in the GC cycle. A JVM doing real work allocates aggressively between collections, so the peak naturally approaches the heap ceiling before GC reclaims dead objects. Alerting on the peak is alerting on normal behavior.

The meaningful signal is the post-GC floor: heap usage immediately after garbage collection runs. If GC reclaims memory efficiently and the floor stays stable, the JVM is healthy regardless of how high peaks reach. If the floor rises over time, live objects are accumulating toward a GC death spiral. This article covers why the floor matters, how to measure it, and why G1GC, the default collector on modern Logstash, breaks the pool-level metrics you would normally use to track it.

Why raw heap percentage misleads

Every JVM exhibits a sawtooth heap pattern. Objects are allocated continuously as the application runs. The heap fills until the garbage collector triggers, reclaims dead objects, and heap usage drops. Then allocation resumes and the cycle repeats. This is normal JVM memory management, not a problem.

heap_used_percent as reported by the Logstash Node Stats API (GET /_node/stats/jvm) is a single point-in-time reading. If you poll every 10 or 15 seconds, you catch the heap at whatever phase of the allocation cycle coincides with your poll. Sometimes you catch it near the floor, just after GC. Sometimes near the peak, just before GC. The reading tells you almost nothing about memory health in isolation.

Spikes are expected. A pipeline processing a batch of oversized JSON events might push heap to 85% momentarily, then GC reclaims it to 60%. Heap sizes that occasionally approach the maximum are acceptable when the GC pattern is healthy.

The alert fatigue cycle is predictable. Operators set heap_used_percent > 80% as a warning. It fires on every sawtooth peak during normal load. They raise the threshold to 85%, then 90%, then silence it entirely. When the real crisis arrives, a slow old-gen accumulation that pushes the post-GC floor from 65% to 85% to 92% over hours, the alert is long gone.

How the sawtooth works and why the floor matters

JVM heap is divided into generations. Young-gen holds recently allocated objects. Old-gen holds objects that have survived multiple GC cycles.

Young-gen collections (minor GC) are fast, frequent, and expected. They run dozens of times per minute under normal load and complete in single-digit milliseconds. They free short-lived objects allocated during event processing. This frequency is healthy.

Old-gen collections (major or full GC) are slow, stop-the-world pauses that freeze all pipeline processing. They should be rare. When old-gen GC starts running frequently, each cycle takes longer because there is less garbage to reclaim and more live data to scan. This is the GC death spiral: the JVM spends increasing time collecting and decreasing time processing events.

The post-GC floor is heap usage immediately after a collection completes. It represents memory occupied by live objects that GC could not reclaim. A stable floor means allocation and collection balance out. A rising floor means objects are accumulating in old-gen and GC cannot keep up.

The dangerous combination is high heap plus rising old-gen plus increasing GC duration. This trio signals the GC death spiral: the process stays alive but does progressively less useful work until the kernel OOM-kills it or it becomes functionally dead.

flowchart TD
    A["Sample heap_used_percent"] --> B{"uptime > 300s?"}
    B -->|No| C["Ignore: JVM warmup"]
    B -->|Yes| D["Observe sawtooth over multiple GC cycles"]
    D --> E{"Post-GC floor stable or rising?"}
    E -->|Stable| F["Healthy: GC reclaims memory efficiently"]
    E -->|Rising| G["Check old-gen pool and GC overhead"]
    G --> H{"Old-gen > 85% and GC > 20% wall time?"}
    H -->|Yes| I["GC death spiral: imminent OOM risk"]
    H -->|No| J["Accumulation: investigate leak or workload change"]

The diagnostic flow starts with one gate: JVM uptime. Logstash heap behavior during the first 5 minutes after startup is dominated by JIT compilation, class loading, and one-time filter initialization. Gate all heap analysis on jvm.uptime_in_millis > 300000 (300 seconds) to avoid false positives from warmup. After that gate, the question is never “is heap high right now” but “is the floor after GC trending upward.”

The G1GC pool metrics gap

On older JVM configurations using the CMS collector, the Logstash Node Stats API exposed per-pool heap breakdowns under jvm.mem.pools: young, old, and survivor. Each pool reported used_in_bytes, max_in_bytes, and peak_used_in_bytes. The old pool was the key signal: if pools.old.used_in_bytes was rising over time, old-gen accumulation was directly observable.

CMS was deprecated in JDK 9 and removed in JDK 14. Modern Logstash 8.x bundles JDK 11 (up to 8.3), JDK 17 (8.4-8.14), or JDK 21 (8.15+), all of which default to G1GC. G1GC uses a region-based memory layout rather than fixed generational pools, which broke Logstash’s per-pool reporting when G1 first became relevant (elastic/logstash#10726, Logstash 7.0/7.1 reported zeroes). Since Logstash 7.2 the monitoring integration aggregates the G1 pool names (G1 Eden Space, G1 Old Gen, G1 Survivor Space) into the young/old/survivor breakdown, so jvm.mem.pools.old.used_in_bytes and its siblings are populated on modern 7.x, 8.x, and 9.x releases.

Two practical techniques sharpen the signal:

Sample heap at high frequency. Poll heap_used_percent every 1 to 5 seconds and identify local minimums in the resulting series. Each local minimum corresponds to a point just after GC ran. Track whether these minimums are rising over time. This approximates the post-GC floor without pool-level data, though it conflates young-gen and old-gen reclamation.

# Get current heap stats
curl -sS http://127.0.0.1:9600/_node/stats/jvm?pretty | grep -E 'heap_used_percent|uptime_in_millis'

Enable GC logging. Add -Xlog:gc* (JDK 9+ syntax) to jvm.options. For production, direct output to a file with rotation: -Xlog:gc*:file=/var/log/logstash/gc.log:time,uptime:filecount=5,filesize=20m. GC log entries record old-gen occupancy before and after each collection, providing the exact post-GC old-gen floor that the pool gauges only approximate.

Where this shows up in production

Several patterns operators commonly misinterpret:

Normal sawtooth at 75%. A steady-state Logstash processing thousands of events per second shows heap oscillating between 60% and 78%. The floor stays flat, GC runs efficiently. This is healthy. A steady 75% with efficient GC does not indicate a leak.

Cold-start ramp. During the first 60 to 120 seconds after restart, heap fills rapidly as JIT compilation, class loading, and filter initialization allocate memory. Heap may spike above 85% briefly before settling into its working set. The uptime gate filters this out.

Large-event burst. A sudden influx of oversized events (stack traces, deeply nested JSON) causes a transient heap spike. If GC reclaims the memory and the floor returns to baseline, this is self-correcting. If the floor rises after the burst, the events may be promoting objects to old-gen faster than GC can collect them.

Logstash 9.0 buffer type change. pipeline.buffer.type is a real logstash.yml setting introduced in Logstash 8.14 with a default of direct for the Beats, TCP, HTTP, and Elastic Agent inputs; in Logstash 9.0 the default changed to heap. Operators upgrading without adjusting heap size will see higher heap_used_percent because buffer allocations that were previously off-heap now count against the Java heap. This is a configuration change, not a leak, but it shifts the baseline upward and may require a heap size increase.

GC death spiral. The end-state: post-GC floor above 90%, old-gen full, GC consuming more than 20% of wall-clock time, output throughput collapsing. The process stays alive. The API may respond slowly or time out during GC pauses. Without floor-based monitoring, this pattern is invisible until the OOM killer intervenes or dashboards go stale.

Common misuses

Alerting on heap peak with a static threshold. Any threshold between 75% and 95% either fires on normal sawtooth peaks (alert fatigue) or misses the real crisis. The fix is not a better threshold. It is a different metric: post-GC floor trend.

Using absolute heap thresholds across a heterogeneous fleet. Dev instances with 1GB heap and production instances with 8GB heap should not share the same percentage threshold. The default heap in Logstash has been 1GB (-Xms1g -Xmx1g in jvm.options) for years, explicitly noted as too small for production. Aim for 4GB to 8GB for typical ingestion, with -Xms and -Xmx set equal to prevent runtime resizing.

Ignoring GC overhead. Heap percentage without GC context is ambiguous. The same 85% reading means something different when GC overhead is 2% (healthy, GC keeping up) versus 25% (death spiral in progress). Always correlate heap with GC collection time. Compute GC overhead as delta(collection_time_in_millis) / delta(wall_time_millis). Above 10% is concerning. Above 20% is severe.

# Compute GC overhead rate over a 60-second window
T1=$(curl -sS http://127.0.0.1:9600/_node/stats/jvm | python3 -c "import sys,json; gc=json.load(sys.stdin)['jvm']['gc']['collectors']; print(sum(g['collection_time_in_millis'] for g in gc.values()))")
sleep 60
T2=$(curl -sS http://127.0.0.1:9600/_node/stats/jvm | python3 -c "import sys,json; gc=json.load(sys.stdin)['jvm']['gc']['collectors']; print(sum(g['collection_time_in_millis'] for g in gc.values()))")
echo "GC overhead: $(python3 -c "print(f'{(($T2 - $T1) / 60000) * 100:.1f}%')")"

Alerting without an uptime gate. Heap during the first 5 minutes after restart is not representative. Alerts that do not check jvm.uptime_in_millis fire during every deployment and every crash recovery. Gate on uptime above 300 seconds for investigation thresholds and above 600 seconds for page-level alerts.

Signals to watch

Signal	Why it matters	Warning sign
Post-GC heap floor (trend)	Directly measures live object accumulation	Floor rising over hours or days
Old-gen pool usage	Separates old-gen leak from young-gen churn	Rising `pools.old.used_in_bytes` (may report zero on G1GC)
GC overhead rate	Measures time stolen from event processing	`delta(gc_time) / delta(wall_time)` above 10%
Old-gen collection count	Tracks frequency of stop-the-world pauses	`old.collection_count` accelerating beyond baseline
Output throughput	Confirms whether GC pressure impairs useful work	Declining while input rate stays positive
JVM uptime	Filters warmup false positives	Below 300s: do not evaluate heap readings

How Netdata helps

Per-second heap sampling captures the actual sawtooth pattern rather than random snapshots at 15-second polling intervals. Local minimums in the per-second series approximate the post-GC floor even when G1GC pool-level metrics are unavailable.
ML-based anomaly detection flags a rising post-GC floor without requiring a static threshold. The anomaly advisor learns each pipeline’s normal sawtooth baseline and surfaces deviations from the floor trend rather than from the peak.
Overlaying heap metrics with GC collection time, output throughput, and queue depth in a single timeline makes the GC death spiral pattern (rising floor, increasing GC overhead, declining output) immediately visible.
JVM uptime is available as context on every chart, filtering warmup periods from steady-state analysis without modifying alert logic.