The Logstash process is up. systemd reports it as active. But event throughput has collapsed to near zero, and CPU is pinned at full utilization. This is a garbage-collection death spiral: the JVM spends most of its wall-clock time in GC, reclaiming almost nothing, while the pipeline starves for compute.

The failure is misleading because the process appears alive. Process-liveness checks pass. The monitoring API on port 9600 may still respond, albeit slowly. But no useful work is happening. The JVM has entered a feedback loop: each GC cycle reclaims less than the last, so GC runs more frequently, steals CPU from event processing, causes events to accumulate, fills the heap faster, and triggers even more GC.

The loop does not self-correct. Unlike a transient allocation spike that resolves after GC catches up, a death spiral means the heap is dominated by live objects the collector cannot reclaim. The only immediate resolution is to restart the JVM. After recovery, address the root cause: typically undersized heap, excessive in-flight events, or a memory leak.

What this means

The sequence:

  1. Heap fills with live objects (events in-flight, filter state, plugin buffers).
  2. GC runs more frequently and for longer durations to find reclaimable memory.
  3. Each GC pause is stop-the-world: all worker threads freeze.
  4. Workers process fewer events per unit of wall-clock time.
  5. Events accumulate in the queue and in-flight batches faster than they drain.
  6. More live objects on the heap trigger the next GC cycle sooner.
  7. The loop tightens until the JVM spends most of its time collecting and almost none processing.
flowchart TD
    A["Heap fills with live objects"] --> B["GC frequency increases"]
    B --> C["GC pauses steal CPU from workers"]
    C --> D["Event processing slows"]
    D --> E["Events accumulate in-flight"]
    E --> A
    C --> F["Old-gen GC duration grows"]
    F --> G["Full stop-the-world pauses"]
    G --> H["Throughput collapses"]

The process may persist in this state for minutes or hours without crashing. An OOM kill from the kernel is more recoverable than a prolonged death spiral, because the restart clears the heap entirely.

Distinguishing from other failure modes

The GC death spiral is commonly confused with two other patterns. The key differentiator is CPU behavior combined with GC time:

SignalGC death spiralOutput bottleneckCPU-bound filters
GC overheadHigh, above 20% of wall timeNormalNormal
Process CPUHigh (GC threads burn cycles)Low (workers wait on I/O)High (filter compute)
Output errors/retriesAbsent or secondaryPresent (429s, timeouts)Absent
Hot threadsGC or memory pathsOutput wait pathsFilter or grok code
Output plugin durationMay be normalHigh and risingNormal

If CPU is low and throughput is low, you have an output bottleneck, not a GC problem. If CPU is high but GC overhead is normal, you have expensive filters (grok backtracking, Ruby code), not heap pressure.

Common causes

CauseWhat it looks likeFirst thing to check
Heap too small for workloadSawtooth floor rises quickly after start, old-gen fills within minutesConfigured heap in jvm.options against actual event volume
Excessive in-flight eventsDeath spiral triggers during traffic spikes; pipeline.workers x pipeline.batch.size is large relative to heappipeline.workers and pipeline.batch.size in pipeline config
Large events or field explosionsJSON parsing creates deeply nested structures, individual events consume disproportionate heapInspect sample events for size; check for unmapped JSON fields
Memory leak in filter pluginPost-GC floor rises monotonically over hours or days regardless of traffic levelCompare heap floor trend across days; check ruby filter or stateful plugins
Config change expanding event sizeDeath spiral begins after a config deployment that adds fields, clones, or splitsCorrelate onset with recent config changes and reload timestamps

Quick checks

Run these read-only checks to confirm or rule out a GC death spiral.

# Check JVM heap and GC stats
curl -sS http://127.0.0.1:9600/_node/stats/jvm?pretty

# Check API responsiveness (slow response indicates GC pressure)
time curl --max-time 10 -sS -o /dev/null -w "%{http_code} %{time_total}s\n" http://127.0.0.1:9600/_node/stats/jvm

# Check pipeline throughput
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty

# Check process CPU
curl -sS http://127.0.0.1:9600/_node/stats/process?pretty

# Check what threads are consuming CPU or blocked
curl -sS 'http://127.0.0.1:9600/_node/hot_threads?threads=10&human=true'

# Check for recent OOM kills (kernel may have killed and restarted the process)
dmesg -T | grep -i 'out of memory\|oom'

# Check configured heap size
grep -E '^-Xm[sx]' /etc/logstash/jvm.options

If the API takes more than 5 seconds to respond on a running process, suspect severe GC pressure. If it times out entirely, the JVM is likely in a prolonged stop-the-world pause.

How to diagnose it

  1. Confirm GC overhead. The defining metric is GC overhead: the fraction of wall-clock time spent in garbage collection. Compute it as delta(collection_time_in_millis) / delta(wall_time_millis) across both collectors (young and old). Take two samples 60 seconds apart:
# Compute GC overhead over a 60-second window
T1=$(curl -sS http://127.0.0.1:9600/_node/stats/jvm | \
  python3 -c "import sys,json; gc=json.load(sys.stdin)['jvm']['gc']['collectors']; print(gc['young']['collection_time_in_millis'] + gc['old']['collection_time_in_millis'])")
sleep 60
T2=$(curl -sS http://127.0.0.1:9600/_node/stats/jvm | \
  python3 -c "import sys,json; gc=json.load(sys.stdin)['jvm']['gc']['collectors']; print(gc['young']['collection_time_in_millis'] + gc['old']['collection_time_in_millis'])")
python3 -c "print(f'GC overhead: {($T2 - $T1) / 60000 * 100:.1f}%')"

Above 10% is concerning. Above 20% is severe. Above 50% of wall-clock time in GC means the JVM is functionally dead.

  1. Check the post-GC heap floor. A single heap_used_percent reading is misleading because of the normal sawtooth pattern. The signal that matters is the floor: the heap level immediately after a GC cycle. A rising floor means live objects are accumulating and the collector cannot reclaim them. Check jvm.mem.pools.old.used_in_bytes relative to jvm.mem.pools.old.max_in_bytes. If old-gen is above 85% of its max after GC, the death spiral is underway.

  2. Verify throughput collapse. Check flow.output_throughput in pipeline stats. In a death spiral, output rate drops to near zero while input rate may still be positive (until backpressure propagates). The gap between input and output widens as events accumulate.

  3. Confirm high CPU comes from GC, not filters. Check process.cpu.percent. The distinguishing signal is GC overhead from step 1: if it is elevated, GC is the CPU consumer. If CPU is high but GC overhead is normal, suspect CPU-bound filters (grok backtracking, Ruby code) instead.

  4. Capture hot threads. If the API is responsive enough, pull a hot threads snapshot. Look for threads in GC or memory management paths rather than filter or output code. Take multiple snapshots seconds apart to distinguish sustained GC from transient filter work.

  5. Rule out output bottleneck. Check output plugin duration and error counts. If output errors and retries are absent and output duration is normal, the problem is not downstream. The throughput collapse is caused by the JVM, not the destination.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
GC overhead (delta collection_time_in_millis / delta wall_time)Direct measure of time stolen from event processingAbove 10% concerning, above 20% severe
Post-GC heap floor (old.used_in_bytes after GC)Rising floor means live objects accumulating, collector cannot reclaimFloor trending toward 85-90% of old-gen max
Old-gen GC frequency (old.collection_count delta)Frequent old-gen GCs indicate heap full of long-lived objectsMore than 1 per minute sustained
Output throughput (flow.output_throughput)Functional health, not process livenessNear zero while input rate is positive
API response timeProxy for stop-the-world pause severityAbove 5 seconds or intermittent timeouts
Process CPU (process.cpu.percent)High CPU from GC threads distinguishes from output bottleneckHigh CPU with low useful throughput

Fixes

Immediate: restart the JVM

Restart is the correct immediate action because the JVM will not recover from a death spiral. Once old-gen is dominated by un-reclaimable objects, no amount of GC will fix it.

systemctl restart logstash

Data safety caveat: if persistent queue is enabled, queued events survive the restart. If using the default memory queue, events in-flight at the time of restart are lost. Assess the tradeoff: a process doing zero useful work is already losing all incoming data to backpressure or queue overflow.

Raise heap size

After restart, increase the JVM heap in jvm.options. Set -Xms and -Xmx to the same value to prevent heap resizing overhead at runtime.

Elastic’s guidance for Logstash 8.x recommends heap between 4GB and 8GB for typical ingestion workloads. Do not assume bigger is always better: oversized heaps cause their own GC pathologies where the collector cannot complete a cycle within reasonable pause times, producing the same death spiral from the opposite direction.

Reduce in-flight event pressure

The product of pipeline.workers and pipeline.batch.size determines how many events are held in memory simultaneously. Each in-flight batch consumes heap proportional to event size. If the death spiral was triggered by a traffic spike, lowering batch_size reduces peak heap pressure at the cost of some throughput.

Default values are pipeline.workers = CPU core count and pipeline.batch.size = 125. If your heap is tight, halving batch size is a safer first adjustment than reducing workers, because fewer workers also reduces parallelism and can create its own throughput bottleneck.

Investigate root cause

After stabilization, identify what filled the heap:

  • Large events: JSON parsing can produce field explosions where a single nested object creates megabytes of heap usage. Inspect sample events for size and check for unmapped or deeply nested fields.
  • Memory leak in filters: Ruby filters or stateful plugins may accumulate objects without releasing them. Track the post-GC floor across days. A monotonically rising floor regardless of traffic level points to a leak.
  • Config change: Correlate the onset of the death spiral with recent config deployments. Adding clone, split, or enrichment filters can multiply event count or size.
  • Persistent queue overhead: Each pipeline with PQ enabled adds page buffer memory for head and tail pages. With many pipelines, this overhead can consume a meaningful portion of the heap budget.

Enable heap dump on OOM

Add -XX:+HeapDumpOnOutOfMemoryError to jvm.options. If the JVM eventually OOMs instead of spiraling indefinitely, you get a heap dump for post-mortem analysis. Without this flag, the OOM produces no forensic artifact.

Prevention

  • Monitor the post-GC floor, not peak heap. Alerting on heap_used_percent > 80% fires during every normal GC cycle and trains operators to ignore it. The meaningful signal is old-gen pool usage after GC. A rising floor is the leading indicator.
  • Alert on GC overhead directly. Compute delta(collection_time_in_millis) / delta(wall_time) and alert above 10%. This catches the death spiral before throughput collapses.
  • Set Xms equal to Xmx. Heap resizing at runtime is expensive and can itself trigger GC pressure during critical moments.
  • Size batch_size and workers for your heap. The in-flight count (pipeline.workers * pipeline.batch.size) multiplied by average event size should fit comfortably within heap with room for filter state and plugin buffers.
  • Watch for event size changes. A source that starts sending larger payloads (deeper JSON, longer text fields) can push a previously stable pipeline into heap pressure without any config change on the Logstash side.
  • Enable GC logging. Add GC log flags to jvm.options for forensic detail beyond what the stats API provides. GC logs show pause durations, the cause of each collection, and old-gen occupancy at collection time.

How Netdata helps

  • Per-second JVM heap metrics let you see the sawtooth pattern and, critically, the post-GC floor in real time. A rising floor is the earliest indicator that a death spiral is developing, often visible before throughput drops.
  • GC time tracking with per-second resolution means you can compute GC overhead continuously rather than from sparse manual samples. Spikes in old-gen collection time are immediately visible alongside heap pressure.
  • Correlation between GC spikes and throughput drops shortens diagnosis. When flow.output_throughput collapses at the same moment GC time spikes, the causal chain is visible on a single timeline rather than requiring cross-referencing separate tools.
  • ML-based anomaly detection on heap usage patterns can flag a rising post-GC floor as anomalous even when absolute percentages are below traditional thresholds, giving lead time before the spiral tightens.
  • Old-gen pool monitoring separates normal young-gen churn (fast, frequent, expected) from dangerous old-gen accumulation (slow, rare, the precursor to death spiral).