Tomcat RSS growing while heap looks flat: native memory and the OOM killer

The Tomcat JVM dies overnight. systemctl status shows exit code 137. Heap dashboards look flat the entire time. No hs_err_pid.log, no heap dump, no OutOfMemoryError in catalina.out. The kernel, not the JVM, ended the process.

The Linux OOM killer scores victims by RSS plus an adjustment (oom_score_adj), not by Java heap. Everything the JVM keeps resident outside the heap counts: Metaspace, thread stacks, direct ByteBuffers, JIT code cache, JNI allocations, and glibc arenas. A Tomcat whose heap sawtooth looks healthy can still have RSS climbing toward the container limit. When RSS crosses memory.max, the kernel sends SIGKILL. The JVM cannot trap it, log it, or dump anything.

The signal most operators watch (heap) is the wrong one for this failure. This guide covers how to confirm an OOM kill, attribute RSS growth to a specific native region with NativeMemoryTracking, and bound native memory so the JVM and the container agree on a budget.

What this means

Java heap is one region of process memory. RSS, which the kernel charges against the cgroup and what the OOM killer scores on, is the sum of everything resident: heap plus every native region the JVM and its libraries have touched. Heap metrics (HeapMemoryUsage, Manager <jvm><memory> XML, the jstat old-gen columns) never include the regions below.

Native memory in a Tomcat process lives in several regions:

• Metaspace: loaded class metadata. The classic classloader-leak sink. Grows on hot redeploy, never shrinks within a JVM lifetime. If -XX:MaxMetaspaceSize is unset (the default), the JVM does not throw OutOfMemoryError: Metaspace; it grows until the OS kills it.
• Thread stacks: roughly -Xss (commonly 1MB) reserved per thread. With maxThreads=200 plus GC, JIT, JMX, and application threads, you can carry 300+ MB of stack alone. A maxThreads=200/-Xss1m config reserves about 200MB of off-heap memory before the first request; touched pages count toward RSS.
• Direct ByteBuffers: the Java object lives on the heap, but the backing native buffer is off-heap. If -XX:MaxDirectMemorySize is unset, the JVM allows roughly -Xmx of direct memory on top of -Xmx of heap.
• JIT Code Cache: compiled methods. Bounded by -XX:ReservedCodeCacheSize. A full Code Cache does not kill the JVM, but it stops JIT compilation and slows you down.
• JNI: any native library the application or its dependencies load (APR, image codecs, crypto, JNI shims). The JVM has no visibility into JNI allocations.
• glibc arenas: malloc’d memory from the C library, including some JDK internals. Each arena can hold freed memory rather than returning it to the OS, so RSS climbs while NMT “committed” does not.

flowchart TD
  HEAP["JVM heap (-Xmx)
what HeapMemoryUsage shows"]
  META["Metaspace
class metadata"]
  STACK["Thread stacks
-Xss per thread"]
  DIRECT["Direct ByteBuffers
MaxDirectMemorySize"]
  CC["Code Cache
ReservedCodeCacheSize"]
  JNI["JNI / native libs"]
  ARENA["glibc arenas
malloc"]
  HEAP --> RSS["Process RSS"]
  META --> RSS
  STACK --> RSS
  DIRECT --> RSS
  CC --> RSS
  JNI --> RSS
  ARENA --> RSS
  RSS --> SCORE["oom_score"]
  SCORE --> SIGKILL["SIGKILL when cgroup over memory.max"]

The OOM killer’s decision runs on oom_score_adj plus RSS. On Kubernetes with cgroup v2, memory.oom.group=1 (set by the kubelet since Kubernetes 1.28) means the entire pod cgroup dies, not just the largest scorer. Exit code 137 (128 + SIGKILL 9) is the standard signature. Because SIGKILL is uncatchable, the JVM produces no hs_err_pid.log and no heap dump. The only record is in the kernel log.

Common causes

Quick checks

# Confirm an OOM kill in the kernel log
dmesg -T | grep -iE "killed process|out of memory"
# systemd journal variant
journalctl -k --since "2 hours ago" | grep -i oom

# Process RSS right now
grep VmRSS /proc/$(pgrep -f 'catalina.startup.Bootstrap')/status

# Container memory limit (cgroup v2 first, v1 fallback)
cat /sys/fs/cgroup/memory.max 2>/dev/null \
  || cat /sys/fs/cgroup/memory/memory.limit_in_bytes 2>/dev/null

# Heap snapshot (proves heap is not the problem)
curl -s -u "$USER:$PASS" 'http://localhost:8080/manager/status?XML=true' \
  | grep -oP '(free|total|max)="[0-9]+"'

# Per-pool breakdown including Metaspace and Code Cache
curl -s -u "$USER:$PASS" 'http://localhost:8080/manager/status?XML=true' \
  | grep -oP 'name="[^"]*" usageUsed="[0-9]+" usageMax="[0-9]+"'

# Thread count (each thread reserves ~-Xss of stack)
ls /proc/$(pgrep -f 'catalina.startup.Bootstrap')/task | wc -l

# Direct buffer usage via JMX
java -jar jmxterm.jar -l localhost:9090 -n -v silent -e \
  "get -b java.nio:type=BufferPool,name=direct Count MemoryUsed TotalCapacity"

How to diagnose it

1. Confirm the kill in the kernel log. dmesg -T | grep -i "killed process" is the smoking gun. It lists the victim PID and its oom_score. If the JVM is gone with no kernel evidence, suspect a JVM crash (hs_err_pid.log) or a container runtime kill instead.

2. Prove heap was flat at death. Pull the heap series for the window from your monitoring. If HeapMemoryUsage post-GC baseline was steady while RSS climbed, heap is exonerated. This rules out the GC death spiral pattern.

3. Enable NativeMemoryTracking. Add -XX:NativeMemoryTracking=summary to JAVA_OPTS. NMT has overhead (Oracle documents a 5%-10% throughput impact), so use it for diagnosis, then remove. Restart the JVM with NMT on.

4. Read NMT. jcmd <pid> VM.native_memory summary reports committed and reserved bytes per category: Java Heap, Class (Metaspace), Thread, Code, GC, Internal, and Other. The Total line is your best NMT-side proxy for RSS, but it undercounts glibc arenas and JNI.

5. Baseline and diff. jcmd <pid> VM.native_memory baseline, reproduce load, then jcmd <pid> VM.native_memory summary.diff. The delta shows which category is actually growing. A Class delta that never shrinks across a redeploy is a classloader leak. An Internal or Other delta points at native libraries or direct buffers.

6. Cross-check with pmap. When NMT and RSS disagree (RSS climbing faster than NMT committed), the missing memory is glibc arenas or JNI. Run pmap -x <pid> twice with load between, diff the output, and look at anonymous (anon) region growth. Growing anon blocks not represented in NMT point at native allocations.

7. Check direct buffers. JMX bean java.nio:type=BufferPool,name=direct reports Count and MemoryUsed. Netty and other NIO-heavy libraries are common direct-buffer leakers. If direct memory is the consumer and -XX:MaxDirectMemorySize is unset, you are allowing roughly -Xmx of off-heap direct memory on top of heap.

8. Audit thread count. Thread stacks are native. ls /proc/<pid>/task | wc -l against your expected baseline (maxThreads plus GC, JIT, JMX, and application overhead). Unbounded thread growth feeds RSS linearly.

9. Compare -Xmx against memory.max. If -Xmx is 90% of the container limit, you have already spent the native budget before the JVM starts. The standard remediation is -XX:MaxRAMPercentage=75 (or lower) so the JVM picks a heap that leaves native headroom. UseContainerSupport is the default since JDK 10; very old JDK 8 builds pre-8u372 cannot read cgroup v2 limits and will size heap against the host, which is a separate, fast-kill bug.

Metrics and signals to monitor

Fixes

A restart buys time but does not change the next death.

Bound Metaspace

Set -XX:MaxMetaspaceSize=512m (or whatever your application actually needs, measured under load). Default is unbounded. With a bound set, the JVM throws OutOfMemoryError: Metaspace before the OS has to kill anything, and you get a stack trace to work from.

Bound direct memory

Set -XX:MaxDirectMemorySize explicitly. If you set -Xmx4g and leave MaxDirectMemorySize unset, the JVM allows up to roughly 4GB of direct buffers on top of heap. A 4GB container with -Xmx4g is a guaranteed OOM kill waiting for any Netty workload.

Cap thread count

Verify maxThreads is sized to your workload, and that application code is not spawning unbounded thread pools. Lower -Xss only after validating stack depth under production load; the wrong value turns into StackOverflowError in deep call paths. Every thread you save is up to -Xss of native memory back.

Cap glibc arenas

Set MALLOC_ARENA_MAX=2 (or up to 4) in the Tomcat service environment. The glibc default is 8 * CPU_count, and freed memory stays attached to arenas rather than being returned to the OS. On a 16-core host, the default can leave hundreds of MB unreclaimed. JDK-8193521 is closed “Won’t Fix”; the JVM will not set this for you.

Size heap for the container

If running in containers, use -XX:MaxRAMPercentage=75.0 (or lower for native-heavy workloads) instead of a fixed -Xmx. UseContainerSupport is default since JDK 10. Leave 25-30% of memory.max for Metaspace, thread stacks, direct buffers, and arenas.

Upgrade from affected JDK versions

If you are on JDK 21.0.3, 21.0.4, or 21.0.5, the C2 compiler leaks memory through objects that the JVM never reclaims (JDK-8340824, also tracked as JDK-8343322). Tomcat containers configured with -Xmx1g have been observed climbing to multiples of that in RSS before being killed. Fix is to upgrade to 21.0.7 or later (Temurin ships the backport from 21.0.7), or downgrade to 21.0.2.

Prevention

• Treat memory.max as the budget, not -Xmx. Size heap to leave 25-30% for native regions.
• Always set -XX:MaxMetaspaceSize and -XX:MaxDirectMemorySize. Unset means unbounded, and unbounded means OS OOM kill.
• Set MALLOC_ARENA_MAX in the service environment. The default glibc behavior is hostile to long-running JVMs.
• Track process RSS alongside heap. RSS is the signal the OOM killer uses; if you only watch heap, you will be surprised.
• Avoid hot redeploys in production. The reliable fix for classloader leaks is a clean JVM restart, not a redeploy.
• On Kubernetes with cgroup v2, remember that memory.oom.group=1 kills the entire pod. A sidecar with a memory leak can take down your Tomcat container.

How Netdata helps

• Per-second RSS and container memory.usage from cgroup v1/v2, plotted alongside HeapMemoryUsage. The “heap flat, RSS climbing” divergence is the early-warning signature for this failure.
• JMX memory pools including Metaspace and Code Cache, so you can attribute native growth before it becomes a kill.
• Kernel OOM events surfaced and correlated with metric drops, so a 137 exit on the JVM lines up with the kill record in dmesg.
• Thread count and thread-pool utilization side by side, so thread-stack-driven RSS growth is visible alongside application load.
• LoadedClassCount trending across deploys to catch classloader leaks early.
• GC time as a fraction of wall clock, so you can confirm the JVM is not in a death spiral while you investigate native growth.

Related guides

• Tomcat 5xx error rate: separating server failures from crawler 404s
• Tomcat accept queue overflow: acceptCount, somaxconn, and Recv-Q
• Tomcat access log setup: adding %D and %T for per-request latency
• Tomcat java.net.BindException: Address already in use: the connector never starts
• Tomcat average latency lies: why you need p95/p99 from the access log
• Tomcat threads blocked forever: the missing outbound timeout
• Tomcat classloader leak on redeploy: why the old WebappClassLoader never dies
• Tomcat connection refused: maxConnections and acceptCount both exhausted
• Tomcat accepts connections but never responds: the TCP-connect trap
• Tomcat file descriptor usage: OpenFileDescriptorCount vs the ulimit
• Tomcat frequent Full GC: pause time, G1, and the 5% overhead rule
• Tomcat GC death spiral: full GCs dominating and throughput collapsing