Logstash operators hit a recurring trap with JVM heap monitoring. They set an alert for heap_used_percent above 80%, following standard JVM guidance. The alert fires constantly because the JVM heap naturally cycles through allocation and collection. They silence it or raise the threshold. Then a real memory leak or GC death spiral develops, heap stays elevated for hours, and nobody notices because the alert was muted.

The problem is not the threshold. It is the metric choice. heap_used_percent is an instantaneous snapshot that catches a random point in the GC cycle. A JVM doing real work allocates aggressively between collections, so the peak naturally approaches the heap ceiling before GC reclaims dead objects. Alerting on the peak is alerting on normal behavior.

The meaningful signal is the post-GC floor: heap usage immediately after garbage collection runs. If GC reclaims memory efficiently and the floor stays stable, the JVM is healthy regardless of how high peaks reach. If the floor rises over time, live objects are accumulating toward a GC death spiral. This article covers why the floor matters, how to measure it, and why G1GC, the default collector on modern Logstash, breaks the pool-level metrics you would normally use to track it.

Why raw heap percentage misleads

Every JVM exhibits a sawtooth heap pattern. Objects are allocated continuously as the application runs. The heap fills until the garbage collector triggers, reclaims dead objects, and heap usage drops. Then allocation resumes and the cycle repeats. This is normal JVM memory management, not a problem.

heap_used_percent as reported by the Logstash Node Stats API (GET /_node/stats/jvm) is a single point-in-time reading. If you poll every 10 or 15 seconds, you catch the heap at whatever phase of the allocation cycle coincides with your poll. Sometimes you catch it near the floor, just after GC. Sometimes near the peak, just before GC. The reading tells you almost nothing about memory health in isolation.

Spikes are expected. A pipeline processing a batch of oversized JSON events might push heap to 85% momentarily, then GC reclaims it to 60%. Heap sizes that occasionally approach the maximum are acceptable when the GC pattern is healthy.

The alert fatigue cycle is predictable. Operators set heap_used_percent > 80% as a warning. It fires on every sawtooth peak during normal load. They raise the threshold to 85%, then 90%, then silence it entirely. When the real crisis arrives, a slow old-gen accumulation that pushes the post-GC floor from 65% to 85% to 92% over hours, the alert is long gone.

How the sawtooth works and why the floor matters

JVM heap is divided into generations. Young-gen holds recently allocated objects. Old-gen holds objects that have survived multiple GC cycles.

Young-gen collections (minor GC) are fast, frequent, and expected. They run dozens of times per minute under normal load and complete in single-digit milliseconds. They free short-lived objects allocated during event processing. This frequency is healthy.

Old-gen collections (major or full GC) are slow, stop-the-world pauses that freeze all pipeline processing. They should be rare. When old-gen GC starts running frequently, each cycle takes longer because there is less garbage to reclaim and more live data to scan. This is the GC death spiral: the JVM spends increasing time collecting and decreasing time processing events.

The post-GC floor is heap usage immediately after a collection completes. It represents memory occupied by live objects that GC could not reclaim. A stable floor means allocation and collection balance out. A rising floor means objects are accumulating in old-gen and GC cannot keep up.

The dangerous combination is high heap plus rising old-gen plus increasing GC duration. This trio signals the GC death spiral: the process stays alive but does progressively less useful work until the kernel OOM-kills it or it becomes functionally dead.

flowchart TD
    A["Sample heap_used_percent"] --> B{"uptime > 300s?"}
    B -->|No| C["Ignore: JVM warmup"]
    B -->|Yes| D["Observe sawtooth over multiple GC cycles"]
    D --> E{"Post-GC floor stable or rising?"}
    E -->|Stable| F["Healthy: GC reclaims memory efficiently"]
    E -->|Rising| G["Check old-gen pool and GC overhead"]
    G --> H{"Old-gen > 85% and GC > 20% wall time?"}
    H -->|Yes| I["GC death spiral: imminent OOM risk"]
    H -->|No| J["Accumulation: investigate leak or workload change"]

The diagnostic flow starts with one gate: JVM uptime. Logstash heap behavior during the first 5 minutes after startup is dominated by JIT compilation, class loading, and one-time filter initialization. Gate all heap analysis on jvm.uptime_in_millis > 300000 (300 seconds) to avoid false positives from warmup. After that gate, the question is never “is heap high right now” but “is the floor after GC trending upward.”

The G1GC pool metrics gap

On older JVM configurations using the CMS collector, the Logstash Node Stats API exposed per-pool heap breakdowns under jvm.mem.pools: young, old, and survivor. Each pool reported used_in_bytes, max_in_bytes, and peak_used_in_bytes. The old pool was the key signal: if pools.old.used_in_bytes was rising over time, old-gen accumulation was directly observable.

CMS was deprecated in JDK 9 and removed in JDK 14. Modern Logstash 8.x uses the JVM default collector, which is G1GC. G1GC uses a region-based memory layout rather than fixed generational pools, and the Logstash monitoring integration does not correctly report per-pool statistics under G1GC.

This means jvm.mem.pools.old.used_in_bytes, jvm.mem.pools.young.used_in_bytes, and their max_in_bytes values may all report zero on modern Logstash installations using G1GC. The top-level metrics (heap_used_percent, heap_used_in_bytes, heap_max_in_bytes) remain accurate, but the generational breakdown that operators traditionally used to distinguish young-gen churn from old-gen accumulation is unavailable through the API.

Two practical workarounds:

Sample heap at high frequency. Poll heap_used_percent every 1 to 5 seconds and identify local minimums in the resulting series. Each local minimum corresponds to a point just after GC ran. Track whether these minimums are rising over time. This approximates the post-GC floor without pool-level data, though it conflates young-gen and old-gen reclamation.

# Get current heap stats
curl -sS http://127.0.0.1:9600/_node/stats/jvm?pretty | grep -E 'heap_used_percent|uptime_in_millis'

Enable GC logging. Add -Xlog:gc* (JDK 9+ syntax) to jvm.options. For production, direct output to a file with rotation: -Xlog:gc*:file=/var/log/logstash/gc.log:time,uptime:filecount=5,filesize=20m. GC log entries record old-gen occupancy before and after each collection, providing the exact post-GC old-gen floor that the API cannot expose.

Where this shows up in production

Several patterns operators commonly misinterpret:

Normal sawtooth at 75%. A steady-state Logstash processing thousands of events per second shows heap oscillating between 60% and 78%. The floor stays flat, GC runs efficiently. This is healthy. A steady 75% with efficient GC does not indicate a leak.

Cold-start ramp. During the first 60 to 120 seconds after restart, heap fills rapidly as JIT compilation, class loading, and filter initialization allocate memory. Heap may spike above 85% briefly before settling into its working set. The uptime gate filters this out.

Large-event burst. A sudden influx of oversized events (stack traces, deeply nested JSON) causes a transient heap spike. If GC reclaims the memory and the floor returns to baseline, this is self-correcting. If the floor rises after the burst, the events may be promoting objects to old-gen faster than GC can collect them.

Logstash 9.0 buffer type change. In Logstash 8.x, the default pipeline.buffer.type is direct, placing input plugin buffers in off-heap direct memory. In Logstash 9.0, the default reportedly changes to heap. Operators upgrading without adjusting heap size will see higher heap_used_percent because buffer allocations that were previously off-heap now count against the Java heap. This is a configuration change, not a leak, but it shifts the baseline upward and may require a heap size increase.

GC death spiral. The end-state: post-GC floor above 90%, old-gen full, GC consuming more than 20% of wall-clock time, output throughput collapsing. The process stays alive. The API may respond slowly or time out during GC pauses. Without floor-based monitoring, this pattern is invisible until the OOM killer intervenes or dashboards go stale.

Common misuses

Alerting on heap peak with a static threshold. Any threshold between 75% and 95% either fires on normal sawtooth peaks (alert fatigue) or misses the real crisis. The fix is not a better threshold. It is a different metric: post-GC floor trend.

Using absolute heap thresholds across a heterogeneous fleet. Dev instances with 1GB heap and production instances with 8GB heap should not share the same percentage threshold. The default heap in Logstash has been 1GB (-Xms1g -Xmx1g in jvm.options) for years, explicitly noted as too small for production. Aim for 4GB to 8GB for typical ingestion, with -Xms and -Xmx set equal to prevent runtime resizing.

Ignoring GC overhead. Heap percentage without GC context is ambiguous. The same 85% reading means something different when GC overhead is 2% (healthy, GC keeping up) versus 25% (death spiral in progress). Always correlate heap with GC collection time. Compute GC overhead as delta(collection_time_in_millis) / delta(wall_time_millis). Above 10% is concerning. Above 20% is severe.

# Compute GC overhead rate over a 60-second window
T1=$(curl -sS http://127.0.0.1:9600/_node/stats/jvm | python3 -c "import sys,json; gc=json.load(sys.stdin)['jvm']['gc']['collectors']; print(sum(g['collection_time_in_millis'] for g in gc.values()))")
sleep 60
T2=$(curl -sS http://127.0.0.1:9600/_node/stats/jvm | python3 -c "import sys,json; gc=json.load(sys.stdin)['jvm']['gc']['collectors']; print(sum(g['collection_time_in_millis'] for g in gc.values()))")
echo "GC overhead: $(python3 -c "print(f'{(($T2 - $T1) / 60000) * 100:.1f}%')")"

Alerting without an uptime gate. Heap during the first 5 minutes after restart is not representative. Alerts that do not check jvm.uptime_in_millis fire during every deployment and every crash recovery. Gate on uptime above 300 seconds for investigation thresholds and above 600 seconds for page-level alerts.

Signals to watch

SignalWhy it mattersWarning sign
Post-GC heap floor (trend)Directly measures live object accumulationFloor rising over hours or days
Old-gen pool usageSeparates old-gen leak from young-gen churnRising pools.old.used_in_bytes (may report zero on G1GC)
GC overhead rateMeasures time stolen from event processingdelta(gc_time) / delta(wall_time) above 10%
Old-gen collection countTracks frequency of stop-the-world pausesold.collection_count accelerating beyond baseline
Output throughputConfirms whether GC pressure impairs useful workDeclining while input rate stays positive
JVM uptimeFilters warmup false positivesBelow 300s: do not evaluate heap readings

How Netdata helps

  • Per-second heap sampling captures the actual sawtooth pattern rather than random snapshots at 15-second polling intervals. Local minimums in the per-second series approximate the post-GC floor even when G1GC pool-level metrics are unavailable.
  • ML-based anomaly detection flags a rising post-GC floor without requiring a static threshold. The anomaly advisor learns each pipeline’s normal sawtooth baseline and surfaces deviations from the floor trend rather than from the peak.
  • Overlaying heap metrics with GC collection time, output throughput, and queue depth in a single timeline makes the GC death spiral pattern (rising floor, increasing GC overhead, declining output) immediately visible.
  • JVM uptime is available as context on every chart, filtering warmup periods from steady-state analysis without modifying alert logic.