The string java.lang.OutOfMemoryError: Java heap space in catalina.out means the JVM could not satisfy an allocation because live (reachable) objects plus the requested size exceeded the maximum heap (-Xmx). This is not a transient GC spike or a young-gen promotion failure. The heap is genuinely full of data the collector cannot reclaim.

Tomcat’s heap holds HTTP sessions, request/response buffers, application objects, and in-process caches. When live data outgrows -Xmx, GC frequency rises, each cycle reclaims less, and application threads get less time between collections. Eventually an allocation fails and the JVM throws OutOfMemoryError: Java heap space. The process may limp on in a degraded state, serving some requests and failing others, until restarted.

Capture a heap dump before restarting. Once you restart, the evidence is gone. The dump tells you whether the cause is a session leak, a cache without eviction, an oversized response object graph, or an undersized heap. This article covers confirming the heap is genuinely full (rather than a different memory region or an OS-level kill), capturing a usable dump under pressure, and sizing -Xmx after the fix.

What this means

Three memory regions produce different OOM error strings and need different fixes. Distinguish them before acting.

Error stringRegionRoot cause category
OutOfMemoryError: Java heap spaceJava heap (bounded by -Xmx)Live data grew past max heap
OutOfMemoryError: MetaspaceMetaspace (native, class metadata)Classloader leak on hot redeploy
OutOfMemoryError: unable to create new native threadNative thread stacksThread leak or ulimit exhaustion

This article is about the first. The diagnostic decision flow:

flowchart td
    A[OutOfMemoryError in catalina.out] --> B{Which message?}
    B -->|Java heap space| C[Heap region]
    B -->|Metaspace| D[Classloader leak path]
    B -->|unable to create new native thread| E[Thread leak path]
    C --> F{Is -Xmx reasonable for workload?}
    F -->|No| G[Undersized heap]
    F -->|Yes| H[Live data leak or oversized graph]
    H --> I[Capture heap dump before restart]
    I --> J[Analyze dominant retainers]

A genuine heap OOM leaves fingerprints: post-GC old-gen usage near -Xmx, increasing Full GC frequency, and the JVM surviving after the error in a degraded state. If the process simply vanished with no JVM error, check the kernel log for an OOM kill. That is a different problem where the container cgroup limit or system RSS budget was hit before the heap itself filled.

Common causes

CauseWhat it looks likeFirst thing to check
Session accumulationactiveSessions climbs monotonically; sessions cluster in old genJMX Catalina:type=Manager activeSessions, expiredSessions
Unbounded in-process cachePost-GC old-gen rises smoothly; one map dominates the dumpHeap histogram for top types
Oversized response object graphThroughput drops before OOM; large allocations during serializationBytes-sent rate vs request rate
Undersized -XmxPost-GC baseline routinely above 75% at peak; OOM only at peakManager Status XML max vs ergonomics default
Container cgroup limit lower than -XmxProcess dies silently, no JVM error in catalina.outdmesg, container memory limit

Quick checks

These are read-only and safe to run on a live instance. Note: jmap -histo:live is the one exception; it forces a Full GC.

# Confirm the exact error string and surrounding context
grep -B2 -A20 "OutOfMemoryError" $CATALINA_BASE/logs/catalina.out | tail -60

# Confirm whether the process died or is limping
pgrep -f 'catalina.startup.Bootstrap' || echo "DOWN"

# If the process is gone, check whether the kernel killed it (RSS, not heap)
dmesg -T | grep -i -A5 -B5 "out of memory\|oom-kill\|killed process"

# Heap snapshot via Manager Status XML (free/total/max)
curl -s -u $USER:$PASS 'http://localhost:8080/manager/status?XML=true' | \
  grep -oE '(free|total|max)="[0-9]+"'

# Old-gen pressure and Full GC count via jstat (refresh every 1s)
jstat -gcutil $(pgrep -f 'catalina.startup.Bootstrap') 1000

# Heap info via jcmd (current region occupancy)
jcmd $(pgrep -f 'catalina.startup.Bootstrap') GC.heap_info

# Histogram of top heap types (WARNING: :live forces a Full GC)
jmap -histo:live $(pgrep -f 'catalina.startup.Bootstrap') | head -30

If the JVM is already in a death spiral, prefer jmap -histo (without :live) for a quick read, then move directly to a full dump.

How to diagnose it

  1. Confirm the error is the heap variant. The full string is java.lang.OutOfMemoryError: Java heap space. GC overhead limit exceeded indicates GC thrashing rather than a hard allocation failure; most operators treat it as the same failure class. Metaspace and unable to create new native thread are different regions.

  2. Check whether the process is alive or dead. If it is gone with no JVM error in catalina.out, the OS OOM killer took it for RSS reasons. Non-heap memory (thread stacks, Metaspace, native buffers) pushed total process memory past the cgroup or host limit. That is a separate problem from heap OOM.

  3. Look at the trajectory, not the instantaneous value. A single heap reading is noise. The signal that matters is the post-GC baseline of old gen: the valley of the sawtooth. If the valley is rising over hours or days, live data is accumulating. Instantaneous usage hitting 80% before a young GC is normal.

  4. Correlate GC behavior. With G1GC (the JDK 9+ default), any Full GC is a warning sign. If FGC in jstat -gcutil is climbing and FGCT (Full GC time) is a meaningful fraction of wall clock, the heap is under sustained pressure.

  5. Capture the heap dump before restarting. This is the irreversible step. If you restart first, the evidence is gone.

# Full heap dump. WARNING: the live form forces a stop-the-world Full GC.
# On a 4GB heap this can pause the JVM for seconds to tens of seconds.
PID=$(pgrep -f 'catalina.startup.Bootstrap')
jmap -dump:live,format=b,file=/var/tmp/tomcat-heap-$(date +%s).hprof $PID

If the JVM is too far gone to respond to jmap, an automatic dump from -XX:+HeapDumpOnOutOfMemoryError (if configured in CATALINA_OPTS) is the fallback.

  1. Restart the JVM to restore service. Restarting is recovery, not a fix. Without the dump and analysis, the OOM will recur on the same trajectory.

  2. Analyze the dump offline. Open the .hprof in Eclipse MAT or VisualVM. Run the dominator tree or leak suspect report. The report points at the single retainer keeping the largest chunk of heap alive, typically a session map, a static cache, a queue, or one large response buffer.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Post-GC old-gen utilizationLive data baseline; ignores ephemeral garbageValley rising over hours, or above 75% of -Xmx sustained
Full GC count and timeG1 should rarely Full GC; each one is an eventFGC climbing in jstat; FGCT above 5% of wall clock
Active session countSessions are usually the largest heap consumerMonotonic growth without plateau
GC overhead ratioDistinguishes GC-bound from app-bound CPUGC time above 10% of wall clock
Process RSS vs heapCatches OOM-kill-before-JVM-OOMRSS approaching cgroup limit while heap looks fine
Bytes sent per requestFlags response object graph explosionSudden 10x rise in bytesSent delta vs requestCount delta

Thresholds for this failure pattern: post-GC heap above 75-85% of max sustained is the ticket zone, above 90% is page-worthy. The steady-state target is post-GC heap below 60% of -Xmx at peak.

Fixes

If the dump shows a session leak

The most common heap-OOM cause in Tomcat. Sessions live until the Manager expires them (default 30 minutes), and any client that does not send a session cookie creates a new session if the application calls getSession(true). Bots and load-test clients that ignore cookies are the usual culprit.

  • Set maxActiveSessions on the Manager element to bound the count.
  • Disable automatic session creation in JSPs with <%@ page session="false" %> where sessions are not needed.
  • Verify session expiry is actually firing: expiredSessions on the Manager MBean should be climbing over time.
  • For REST endpoints that do not need sessions, ensure the framework is not creating them implicitly.

If the dump shows an unbounded cache or collection

A static map, a hand-rolled cache, or a queue that grows without eviction will fill old gen monotonically. The dominator tree in MAT points at the exact field.

  • Add eviction: size bound or TTL.
  • If the collection is a queue, ensure consumers are keeping up. Unbounded queues under backpressure are a classic leak.
  • For thread-local accumulation, ensure remove() is called in a finally block.

If the dump shows an oversized response object graph

A serialization path that materializes an entire table, a deep entity graph with lazy loading triggered en masse, or a CSV export built in memory. The dump shows one or a few very large byte[], char[], or String instances.

  • Stream the response instead of buffering it. Use chunked transfer encoding.
  • Paginate large reads. Do not let a single request hold the full result set in memory.
  • Watch the bytes-sent signal: it spikes before the OOM if the cause is response size.

If the dump is clean and -Xmx is simply too small

If the dominator tree shows only expected application objects and the post-GC baseline has been creeping up only because traffic grew, the heap is undersized. Increase -Xmx and set -Xms equal to -Xmx for production to avoid heap resize pauses.

In containers, check MaxRAMPercentage. The default is 25% of the container memory limit, which is often too low for a Tomcat that owns its container. Also confirm the container memory limit is at least -Xmx plus non-heap overhead (Metaspace, thread stacks, native buffers). Otherwise the OS OOM killer fires before the JVM ever sees heap pressure.

Security note: DoS-via-OOM

Several recent Tomcat CVEs manifest as heap OOM under crafted input. If the OOM coincides with abnormal HTTP/2 traffic, multipart upload storms, or WebSocket floods, check the Tomcat security page for your version. The fixes are version upgrades, not heap tuning.

Prevention

  • Enable automatic heap dumps on OOM. Set -XX:+HeapDumpOnOutOfMemoryError and -XX:HeapDumpPath=/var/tmp/ so the next OOM produces a dump without operator intervention.
  • Alert on post-GC old-gen baseline, not instantaneous heap. Alerting on raw sawtooth peaks creates constant false positives.
  • Alert on any Full GC with G1GC. One is a warning; recurring is a leak.
  • Alert on monotonic session count growth. Session accumulation is the single most common Tomcat heap leak.
  • In containers, set MaxRAMPercentage explicitly. Ensure the container limit covers -Xmx plus non-heap overhead.
  • Run a soak test with a heap dump at the end in staging. Catch leaks before they hit production.

How Netdata helps

Netdata’s per-second JVM metrics turn the post-GC baseline into a visible trend rather than a guess from a jstat snapshot.

  • Heap utilization per region, sampled per second, makes the sawtooth and its rising valley visible without manual jstat polling during an incident.
  • GC collection count and time per generation show Full GC frequency alongside heap pressure, confirming the death-spiral pattern before the OOM lands.
  • Old-gen pool utilization post-GC is the leading indicator this article keeps returning to. Trending it over days surfaces leaks while they are still cheap to fix.
  • Active session count per context correlates session accumulation with heap growth, which is usually the smoking gun in Tomcat heap OOMs.
  • Process RSS alongside heap catches the inverse case where RSS is the killer and the heap looks fine.
  • Anomaly detection on the post-GC baseline flags the gradual rise that a static threshold misses, before it becomes a page.