$ guides / tomcat / tomcat-heap-dump-before-restart ▌

Operations Guides

Tomcat heap dump before restart: capturing evidence with jmap and jstack

When Tomcat is failing, the instinct is to restart. That instinct is right for recovery and wrong for diagnosis. The JVM process holds the only copy of the evidence: the object graph that explains the heap exhaustion, the thread stack traces that explain the pool stall, the GC state that explains the death spiral. Once the process exits, that evidence is gone. You are left restarting blind with nothing but an OutOfMemoryError line in catalina.out.

This guide covers the capture procedure you run in the narrow window between “Tomcat is broken” and “Tomcat is restarted.” The goal is two artifacts: a heap dump (.hprof) and one or more thread dumps (jstack output) for offline analysis with Eclipse MAT or VisualVM. The procedure is the same whether the trigger is an OutOfMemoryError: Java heap space, a GC death spiral, or unexplained thread pool exhaustion. See the GC overhead pattern and thread pool exhaustion for broader diagnostic context.

The procedure assumes the JVM process is still alive. If the process has already been OOM-killed or has crashed, the heap is gone. Your remaining evidence is the GC log, the kernel OOM log, and any automatic heap dump produced by -XX:+HeapDumpOnOutOfMemoryError if it was configured.

What this captures

Heap dump (.hprof). A full snapshot of the JVM heap at capture time: every live object, every reference, every class instance count and retained size. This tells you what is consuming memory and who is retaining it. Load it in Eclipse MAT and run the “Leak Suspects” report to find the dominant retainer. Without this, you are guessing at the leak source.

Thread dump (jstack output). A snapshot of every JVM thread and its current stack trace, including thread state (RUNNABLE, BLOCKED, WAITING, TIMED_WAITING). For thread pool exhaustion, it reveals whether threads are stuck on a socket read, a database connection acquire, a lock, or a GC pause. Multiple thread dumps taken a few seconds apart are more useful than one, because they distinguish a thread that is permanently stuck from one that is just slow.

Capture both. A heap dump without a thread dump tells you what is retained but not what is actively blocking. A thread dump without a heap dump tells you what threads are doing but not whether the heap is the constraint.

Prerequisites

JDK, not JRE. jmap and jstack ship with the full JDK. They are absent from JRE-only distributions. This bites teams running minimal container images. Verify before you need them.
Disk space. A heap dump can be as large as the JVM’s maximum heap (-Xmx). A 4 GB heap produces a dump approaching 4 GB. On a full disk, the dump fails partway through and is useless. Check available space on the target volume before capturing.
Process access. You must run jmap and jstack as the same user that owns the JVM process, or as root. The tools attach via the JVM Attach API, which requires matching credentials.
The process must still be alive. If the JVM has exited, the Attach API has nothing to attach to. Check with pgrep first.
Local shell access to the host. In Kubernetes, you need kubectl exec into the pod, or a sidecar that can run the tools. Remote JMX is not sufficient for jmap and jstack, which are local attach tools.

Procedure

Run these steps in order. The order matters: thread dumps are cheap and fast, heap dumps are slow and disruptive.

1. Confirm the process is alive and find the PID

# Find the Tomcat JVM process
pgrep -f 'catalina.startup.Bootstrap'

For embedded Tomcat (Spring Boot), match on your application JAR or main class instead. If pgrep returns nothing, the process is already gone and this procedure cannot proceed. Check dmesg | grep -i oom for an OOM kill, or the container restart count.

Record the PID. Every subsequent command uses it.

2. Capture thread state first

jstack is fast and does not trigger a Full GC. Capture it before anything that might pause or destabilize the JVM.

# Capture three thread dumps, 5 seconds apart, for comparison
for i in 1 2 3; do
  jstack <pid> > /tmp/tomcat-threads-$i.txt
  sleep 5
done

The three-dump variant is worth the extra 10 seconds. Comparing stack traces across dumps tells you which threads are permanently stuck (same frame in all three) versus which are just slow (frames moving).

If jstack fails to attach, try the jcmd equivalent:

# jcmd alternative for thread dump
jcmd <pid> Thread.print > /tmp/tomcat-threads-1.txt

jcmd is the supported successor to jstack. Both work on current JDKs, but jmap and jstack carry an “experimental and unsupported” label in the Oracle documentation and may not be present in future JDK releases.

3. Check disk space before the heap dump

# Check available space on the target volume
df -h /tmp

# Check the JVM's max heap to estimate dump size
jcmd <pid> VM.flags | grep -i MaxHeapSize

The dump will approach the size of the live heap, up to -Xmx. If /tmp does not have enough space, write to a mounted volume that does. In containers, /tmp is often an overlay filesystem backed by a small ephemeral layer. Mount a persistent volume or an emptyDir with adequate capacity.

4. Capture the heap dump

Use one of the two commands below. The first is the traditional jmap form. The second is the jcmd equivalent, which is the recommended path on modern JDKs.

# Option A: jmap (traditional, still works on JDK 17/21)
jmap -dump:format=b,file=/tmp/tomcat-heap.hprof <pid>

# Option B: jcmd (recommended)
jcmd <pid> GC.heap_dump /tmp/tomcat-heap.hprof

Warning: the live suboption triggers a Full GC. jmap -dump:live,format=b,file=<path> <pid> forces a stop-the-world Full GC before dumping, which on a large heap can freeze the JVM for tens of seconds. On a JVM already in a GC death spiral, this pause can trip health checks and cause a load balancer to drain the node. Use live only if you specifically want the post-GC object set; omit it if you want the full heap as it currently sits.

# Full heap, no forced GC (preferred for forensics)
jmap -dump:format=b,file=/tmp/tomcat-heap.hprof <pid>

# Post-GC heap only (triggers Full GC, disruptive)
jmap -dump:live,format=b,file=/tmp/tomcat-heap.hprof <pid>

The output file must not already exist. Both jmap and jcmd fail if the target file is present. Use a unique filename including the PID and timestamp:

# Unique filename to avoid the "file exists" failure
jmap -dump:format=b,file=/tmp/tomcat-heap-$(date +%Y%m%d-%H%M%S)-<pid>.hprof <pid>

Wait for the command to return. A heap dump of a multi-gigabyte heap can take minutes. Do not interrupt it. A partial dump cannot be opened by MAT.

5. Verify the dumps are complete

# Check file sizes (heap dump should be large, thread dumps small)
ls -lh /tmp/tomcat-heap-*.hprof /tmp/tomcat-threads-*.txt

# Check the heap dump magic bytes (should start with "JAVA PROFILE")
head -c 20 /tmp/tomcat-heap-*.hprof | xxd | head -2
# Expected: 4a 41 56 41 20 50 52 4f 46 49 4c 45 ...

A heap dump smaller than a few hundred megabytes for a JVM with a multi-gigabyte -Xmx is suspicious. It may indicate the dump failed silently or the file was truncated by a full disk.

Confirm the thread dumps contain real application frames. Open the jstack output and look for org.apache.catalina.connector.CoyoteAdapter.service or your application code. A thread dump full of JVM-internal frames with no application code suggests you captured during a GC pause or startup, not during the failure.

If you captured three dumps and all three are byte-identical, either the JVM was completely frozen (likely a GC pause or deadlock) or jstack returned an error. Check for “Unable to attach” messages at the top of the file.

6. Copy the artifacts off the host

# From a Kubernetes pod (specify the exact filename; kubectl cp does not glob)
kubectl cp <namespace>/<pod>:/tmp/tomcat-heap-20240101-120000-12345.hprof ./tomcat-heap.hprof

Do not analyze the dump on the production host. Eclipse MAT needs heap at least as large as the dump file to parse it, and running MAT on the same host as the failing JVM adds memory pressure to an already pressured system.

7. Now you can restart

Once the heap dump and thread dumps are safely copied off the host, restart the JVM to recover service. The evidence is preserved.

flowchart TD
    A[Tomcat failing: OOM or GC spiral] --> B{Process alive?}
    B -- No --> C[Evidence gone. Check GC log, OOM killer, auto-dump]
    B -- Yes --> D[Find PID: pgrep catalina.startup.Bootstrap]
    D --> E[jstack: capture 3 thread dumps, 5s apart]
    E --> F[Check disk space vs -Xmx]
    F --> G[jmap or jcmd: capture heap dump]
    G --> H[Verify file size and hprof magic]
    H --> I[Copy artifacts off host]
    I --> J[Restart JVM to recover]
    J --> K[Analyze offline: MAT or VisualVM]

Common pitfalls

jmap or jstack not found in containers. Most production container images ship only the JRE. Running jmap inside the pod fails with executable file not found in $PATH. You have three options: build the image from a JDK base (for example eclipse-temurin:17-jdk instead of -jre), use jcmd if it happens to be present, or capture the dump from outside the container by exec’ing into the host’s JDK installation against the container’s PID (which requires host PID namespace sharing and is fragile). The clean fix is to build diagnostic-capable images or run a sidecar with the JDK.

The eclipse-temurin JRE images for JDK 17 and 21 ship neither jcmd, jmap, nor jstack (their bin contains only java, jfr, jrunscript, keytool, and - on 21 - jwebserver), so a JDK-based image or a sidecar is required for heap capture; kill -3 still works for a thread dump.

Disk fills before the dump completes. A heap dump can be as large as -Xmx. In a container with a small writable layer, the dump fills the overlay filesystem, the pod gets evicted, and you lose both the dump and the process. Always write to a mounted volume with capacity greater than -Xmx, and check df before capturing.

HeapDumpOnOutOfMemoryError does not fire before OOM kill. The flag -XX:+HeapDumpOnOutOfMemoryError is the right production default, but it only triggers when the JVM throws OutOfMemoryError. In a container with a memory cgroup limit lower than -Xmx, the kernel OOM killer can terminate the process before the JVM ever sees the OOM condition. No OutOfMemoryError, no automatic dump. This is why manual capture matters: the kernel does not wait for the JVM to be polite.

jmap -dump:live makes a bad situation worse. The live option forces a Full GC before dumping. On a heap that is already at 95% and barely collecting, this Full GC can run for tens of seconds or minutes, during which the JVM is unresponsive. If health checks fail, the orchestrator may kill the pod before the dump completes. Prefer the non-live form for forensics unless you specifically need the post-GC snapshot.

SIGQUIT (kill -3) is not a substitute for jstack. kill -3 <pid> writes a thread dump to the JVM’s stdout, which for Tomcat usually means catalina.out. It works when jstack cannot attach, but the output is less structured and interleaves with application logging. Use it as a fallback, not a first choice.

OpenJ9 and other non-HotSpot JVMs use different tooling. jmap and jstack are HotSpot tools. On OpenJ9 (used in some IBM and Semeru images), the heap dump format is .phd, not .hprof, and the capture mechanism differs. If your Tomcat runs on OpenJ9, the commands in this guide do not apply directly. Check the OpenJ9 documentation for the equivalent -Xdump options.

OpenJ9 writes heap dumps as .phd (capture with jcmd <pid> GC.heap_dump or the -Xdump:heap trigger). Eclipse MAT cannot open .phd by itself: it needs the IBM DTFJ feature installed (IBM publishes a MAT build with it pre-installed).

Signals to monitor

These signals tell you when to initiate the capture procedure. If you wait for a hard crash, the evidence is gone.

Signal	Why it matters	Warning sign
Post-GC heap utilization	Rising valley means live data growing toward OOM	Post-GC heap above 85% of `-Xmx` and trending up
Full GC frequency	A Full GC on G1 means the concurrent cycle is not keeping up	More than one Full GC per minute
GC overhead ratio	GC consuming CPU that should serve requests	GC time above 20% of wall clock
Thread pool utilization	Threads at max with low CPU means blocked, not busy	`currentThreadsBusy` at `maxThreads` sustained over 60s
`OutOfMemoryError` in catalina.out	JVM has already failed to allocate	Any occurrence; capture immediately if process survives

See the monitoring checklist for the full signal catalog.

How Netdata helps

The capture procedure is reactive. Continuous monitoring tells you when to trigger it.

Per-second JVM heap metrics show the sawtooth pattern and, more importantly, the post-GC baseline trend. A rising valley over hours or days is the leading indicator that a heap dump will soon be necessary.
GC collection count and time, broken down by young and old generation, surface the transition from normal collection to a death spiral. Sustained Full GCs on G1 are a page-worthy trigger for capture.
Thread pool utilization (currentThreadsBusy relative to maxThreads) distinguishes a heap problem from a thread problem. If threads are at max but heap is fine, the issue is blocking, and jstack is the priority capture, not jmap.
Anomaly detection on heap and GC metrics flags deviations from the baseline before thresholds are crossed.
Cgroup memory usage in containerized deployments shows the gap between JVM heap and the container limit, indicating how close the kernel OOM killer is to firing before the JVM throws OutOfMemoryError.