$ guides / tomcat / tomcat-gc-death-spiral ▌

Operations Guides

Tomcat GC death spiral: full GCs dominating and throughput collapsing

Your Tomcat JVM is up, the connector port is bound, but requests crawl or hang. Throughput graphs stutter: brief bursts of activity separated by flatlines where no requests complete. CPU is pinned near saturation.

The mechanism is a positive feedback loop. Live data on the heap has grown until each GC cycle reclaims almost nothing. The JVM compensates by running GC more frequently, which consumes more CPU, which leaves less wall clock for application threads, which causes requests to take longer, which causes more concurrent threads to stay active, which allocates more memory. The loop ends in OutOfMemoryError or effective livelock where GC runs continuously and the application makes no progress.

The single most important distinguishing signal: CPU is HIGH. This is the opposite of thread pool exhaustion, where CPU is low because threads are parked waiting on a backend. If you see a slow Tomcat with maxed CPU, suspect GC before you suspect your database.

What this means

In a healthy JVM, the heap usage graph looks like a sawtooth: it climbs as objects accumulate, then drops sharply when GC runs. The valley of that sawtooth (the post-GC baseline) is what matters. A rising valley means live data is growing. When the valley approaches -Xmx, every GC cycle struggles to free space, pauses lengthen, and the sawtooth flattens into a line near the top.

With G1GC (the default since JDK 9), the collector is designed to avoid Full GCs entirely through concurrent marking and mixed collections. A Full GC in G1 is a stop-the-world fallback that means the concurrent collector could not keep up. Any Full GC with G1 warrants investigation. Multiple Full GCs per minute is a crisis.

The throughput signature is distinctive. During a GC pause, no requests complete. Between pauses, the application briefly runs and processes a burst of requests before the next pause. This produces an oscillating throughput graph that averages to “degraded” but is actually alternating between zero and near-normal. Average latency metrics are misleading here because they smooth over the multi-second pauses.

flowchart TD
    A["Tomcat slow or unresponsive"] --> B{"CPU high?"}
    B -- No --> C["Thread pool exhaustion: threads parked on I/O"]
    B -- Yes --> D{"Post-GC heap near max?"}
    D -- No --> E["App CPU-bound: check jstack"]
    D -- Yes --> F["GC death spiral confirmed"]
    F --> G["Capture heap dump: jmap -dump:format=b"]
    G --> H["Restart JVM to recover"]
    H --> I["Analyze heap dump offline"]

Common causes

Cause	What it looks like	First thing to check
Memory leak (collections, caches without eviction)	Post-GC old gen climbs monotonically over hours or days; Full GCs reclaim little	Heap dump dominator tree
HTTP session accumulation	`activeSessions` grows without plateau; sessions never expire or bots create one per request	JMX `activeSessions` per context
`-Xmx` too small for the live set	Post-GC heap baseline near max even after restart; Full GCs resume quickly	Compare post-GC old gen to `-Xmx`
Humongous allocation (G1-specific)	Full GCs with humongous allocation or evacuation failure in GC log	GC log for “Humongous” regions
Large response object graphs cached in memory	Old gen grows in steps correlated with specific request patterns	Heap dump for large byte arrays

Quick checks

These are safe, read-only commands. Run them before you restart, because the current heap state is the evidence you need.

Note: pgrep -f 'catalina.startup.Bootstrap' returns the PID of the Tomcat JVM. If you run multiple Tomcat instances on the same host, specify the PID explicitly.

# Watch GC in real time (1s interval). FGC = Full GC count, FGCT = Full GC time in seconds.
jstat -gcutil $(pgrep -f 'catalina.startup.Bootstrap') 1000

# Check heap summary and memory pool breakdown.
jcmd $(pgrep -f 'catalina.startup.Bootstrap') GC.heap_info

# Quick look at what is filling the heap (top object types by count and size).
jcmd $(pgrep -f 'catalina.startup.Bootstrap') GC.class_histogram | head -20

# Check for OutOfMemoryError in logs. The JVM may already be in a compromised state.
grep -c "OutOfMemoryError" /var/log/tomcat/catalina.out

# Confirm thread pool is not the bottleneck. This rules out thread exhaustion.
jstack $(pgrep -f 'catalina.startup.Bootstrap') | grep -c "http-nio-8080-exec"

The following JMX queries require a JMX port to be exposed on the JVM (for example, -Dcom.sun.management.jmxremote.port=9090). For local debugging without JMX configuration, prefer jcmd and jstat above.

# Check GC collection count and cumulative time (G1 Old Generation collector).
java -jar jmxterm.jar -l localhost:9090 -n -v silent -e \
  "get -b java.lang:type=GarbageCollector,name=\"G1 Old Generation\" CollectionCount CollectionTime"

# Check post-GC old gen usage. This is the critical number for leak detection.
java -jar jmxterm.jar -l localhost:9090 -n -v silent -e \
  "get -b java.lang:type=MemoryPool,name=\"G1 Old Gen\" Usage"

# Check active session count per context. Sessions are a common heap consumer.
java -jar jmxterm.jar -l localhost:9090 -n -v silent -e \
  "get -b Catalina:type=Manager,host=localhost,context=/ activeSessions"

# Check JVM CPU load. Sustained above 90% with low throughput points to GC-driven CPU.
java -jar jmxterm.jar -l localhost:9090 -n -v silent -e \
  "get -b java.lang:type=OperatingSystem ProcessCpuLoad"

How to diagnose it

Confirm the death spiral signature. You need three signals aligned: post-GC heap near max, Full GC frequency high (multiple per minute), and CPU high. If CPU is low, you are looking at thread pool exhaustion, not GC. See the related guide on telling a blocked backend from a GC spiral.
Capture a heap dump before restarting. This is the single most important step. The heap state is your only evidence of what is retaining memory. Once you restart, that evidence is gone.

# Capture full heap dump WITHOUT triggering an extra Full GC.
# Use format=b (not :live) to avoid forcing another GC on an already struggling JVM.
jmap -dump:format=b,file=/tmp/heap.hprof $(pgrep -f 'catalina.startup.Bootstrap')

# Equivalent on JDK 9+ (preferred):
jcmd $(pgrep -f 'catalina.startup.Bootstrap') GC.heap_dump /tmp/heap.hprof

Warning: even without :live, a heap dump of a large heap pauses the JVM and can take tens of seconds. It also writes a file roughly equal to the used heap size, so confirm you have disk space. The :live variant forces a Full GC before dumping, which gives a cleaner picture of reachable objects but can crash a JVM that is already on the edge. Prefer format=b on a spiraling JVM.

Restart the JVM to recover service. The heap dump is captured. Now restore availability. A clean restart clears the heap and breaks the feedback loop, at least temporarily. If the live set is legitimate (not a leak), the spiral will resume once the application warms up.
Analyze the heap dump offline. Open it in Eclipse MAT or VisualVM. Run the dominator tree or leak suspects report. Look for a single collection, cache, or session store that accounts for a disproportionate share of retained memory.
Correlate with GC logs. If GC logging was enabled at startup (-Xlog:gc*:file=... for JDK 9+), review the timeline leading up to the spiral. Identify when post-GC old gen started climbing and what changed around that time: a deployment, a traffic pattern shift, a cache warmup, a new query returning large result sets.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Post-GC old gen utilization	The valley of the sawtooth. Rising valley means live data is growing.	Valley trending upward, approaching 80% of max
Full GC count and frequency	With G1, any Full GC is abnormal. Multiple per minute is crisis-level.	Any Full GC with G1; frequency increasing
GC time as fraction of wall clock	Measures how much CPU GC steals from the application.	Above 5% concerning, above 20% critical
JVM CPU utilization	Distinguishes GC-driven CPU from application-driven CPU.	Sustained above 90% with low request throughput
Request throughput	Oscillating pattern between zero and bursts is the death spiral signature.	Throughput stuttering, not just dropping
Active session count	Sessions are a common heap consumer. Unbounded growth is a leak.	Monotonic growth without plateau
Memory pool old gen name	Pool name varies by GC algorithm. Must match your collector.	“G1 Old Gen” for G1, “PS Old Gen” for Parallel

Fixes

Memory leak in application code

The dominant retainer in the heap dump is usually a collection, cache, or map that grows without bound. Fix the application: add eviction (LRU, TTL, size cap), use weak or soft references where appropriate, or fix the logic that adds without removing. Deploy the fix and monitor post-GC old gen over the following days to confirm the valley stabilizes.

Session accumulation

If sessions are the retainer, check three things. Session timeout configuration (default 30 minutes may be too long for your traffic). Whether bots are creating sessions: clients without cookies cause getSession(true) to create a new session on every request. Whether maxActiveSessions is set: the default is unlimited. Set a reasonable maxActiveSessions and ensure session creation requires authentication where possible.

Undersized heap

If post-GC old gen baseline was near max even right after a fresh restart, the heap is genuinely too small for the live working set. Increase -Xmx. But verify the live set is legitimate first. A leak with a bigger heap just delays the spiral. Check whether the live set is proportional to traffic (expected) or grows independently of traffic (leak).

Humongous allocations (G1)

If the GC log shows humongous allocations or evacuation failures, the application is creating objects larger than half the G1 region size. These go directly to old gen and fragment the heap. G1 region size is automatically selected as a power of 2 between 1 MB and 32 MB based on heap size. Options: increase -XX:G1HeapRegionSize so fewer allocations qualify as humongous, or refactor the application to avoid large contiguous allocations such as big byte arrays, large strings, or oversized response buffers.

Cache without eviction

If the heap dump shows a cache as the dominator, the cache has no eviction policy or the policy is not keeping up. Add size-based or time-based eviction. For ConcurrentHashMap-based caches common in legacy code, replace with a bounded cache such as Caffeine or Guava Cache.

Prevention

Monitor post-GC old gen, not instantaneous heap. Alert on the valley rising, not on the peak. The sawtooth peak hitting 80% is normal. The valley hitting 80% is a leak.
Alert on any Full GC with G1. Full GCs should not happen with G1 in normal operation. A single Full GC is worth investigating.
Track the GC overhead ratio (cumulative GC time divided by wall clock time). Below 5% is healthy. Above 10% is a warning. Above 20% is critical.
Enable GC logging at deployment time, not after the first incident. For JDK 9+: -Xlog:gc*:file=/var/log/tomcat/gc.log:time,uptime:filecount=5,filesize=10M. You cannot retroactively get the GC log if it was never enabled.
Set -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/tomcat/ so the JVM captures a heap dump automatically before it dies. This gives you the evidence you need without being on the box at the right moment.
Set -XX:MaxMetaspaceSize to prevent silent Metaspace growth leading to OS OOM kill. This is a different failure than heap exhaustion but produces similar symptoms: the process dies.
In containers, ensure -Xmx plus native memory overhead fits within the cgroup memory limit. The OS OOM killer uses RSS, not heap, to decide what to kill. Container CPU limits can also make GC pauses longer because GC threads are descheduled mid-collection.
Watch for synchronized Full GCs across a server pool. If all instances start with identical heap configuration and traffic, they can spiral simultaneously and cause a cascading outage. Stagger restarts or vary GC tuning per instance.

How Netdata helps

Per-second JVM heap and memory pool metrics reveal the sawtooth pattern and, more importantly, the post-GC baseline trend that exposes a leak before it becomes a spiral.
GC collection count and time per collector (young versus old) at per-second resolution make Full GC frequency and duration immediately visible without manual jstat polling.
JVM CPU utilization alongside GC metrics lets you confirm the GC-driven CPU signature (high CPU plus high GC time equals memory problem, not CPU problem) within seconds.
Tomcat JMX request throughput and processing time correlate directly with GC events, making the oscillating throughput pattern visible as a GC correlation.
Anomaly detection on GC patterns flags deviations from the established baseline before the death spiral fully develops, giving you runway to capture a heap dump proactively.