Tomcat process not running: crashes, OOM-kills, and failed restarts

The Tomcat JVM is gone. pgrep -f org.apache.catalina.startup.Bootstrap returns nothing, the service check is red, and your health probe is timing out. Before you restart anything, find out why it died, because the same cause will kill it again, often within minutes.

“Process not running” conflates three distinct events: the JVM crashed inside its own runtime, the kernel OOM-killed it with SIGKILL, or the supervisor (systemd, init, container runtime) failed to bring it back. Each leaves evidence in a different place, and only one of them leaves anything in catalina.out. This guide separates those three classes plus the “alive but nonfunctional” D-state trap.

For embedded Tomcat (Spring Boot and similar), substitute your application jar for org.apache.catalina.startup.Bootstrap in every pgrep below.

What this means

“Process not running” means the JVM that hosts Tomcat is absent from the process table. Three things produce that state, and they are not interchangeable.

A JVM crash is an unexpected termination inside the JVM: a segfault in JIT-compiled code, a JNI native library fault, or an internal VM error. HotSpot attempts to write an hs_err_pid<pid>.log before it exits. If one exists with a timestamp near the outage, you have a crash, not an OOM kill.

An OS OOM kill is the Linux kernel sending SIGKILL because system or cgroup memory was exhausted. SIGKILL cannot be caught, so the JVM runs no shutdown hooks, writes no heap dump, and produces no hs_err_pid file. The only evidence is in the kernel log.

A failed restart means the process exited (for any reason, including a clean SIGTERM) and the supervisor either did not attempt a restart or attempted one that failed. The service may sit in a failed state with no live PID while your monitor reports “down”.

Two traps sit alongside these:

• A process in D state (uninterruptible sleep) is alive in the process table but nonfunctional, usually blocked in a kernel I/O path such as a stale NFS mount or wedged storage. pgrep matches it, but Tomcat is not serving. kill -9 has no effect until the kernel call returns.
• A running process does not prove Tomcat is serving. The connector may have failed to bind (port conflict, bad TLS configuration), leaving the JVM up with no listening HTTP port. Confirm with a port or health-endpoint check.

flowchart TD
    A["pgrep catalina.Bootstrap"] -->|absent| B{"dmesg shows OOM kill?"}
    A -->|present| C{"Process state?"}
    B -->|yes| D["OS OOM killer: SIGKILL, no hs_err"]
    B -->|no| E{"hs_err_pid*.log present?"}
    E -->|yes| F["JVM crash: native or VM fault"]
    E -->|no| G["Check supervisor journal for failed restart"]
    C -->|D state| H["Uninterruptible sleep: stuck on I/O"]
    C -->|R or S| I["Process alive: verify connector bound"]

Common causes

Quick checks

These are read-only and safe to run during an incident. They are ordered by evidence decay: kernel logs rotate, crash artifacts get cleaned up, and the supervisor journal is easiest to lose.

# Standalone Tomcat: is the JVM process present?
pgrep -af 'org.apache.catalina.startup.Bootstrap' || echo "DOWN"

# Embedded (Spring Boot): match the application jar instead
pgrep -af 'myapp.jar' || echo "DOWN"

# If CATALINA_PID is set, cross-check the PID file vs. the live process.
# Note: PID reuse can produce a false positive if the PID was recycled.
cat "$CATALINA_PID" 2>/dev/null && kill -0 "$(cat "$CATALINA_PID")" 2>/dev/null || echo "PID FILE STALE OR MISSING"

# Did the kernel OOM-kill it? SIGKILL leaves no JVM log.
dmesg -T | grep -iE 'oom|killed process'

# Equivalent on systemd hosts:
journalctl -k | grep -iE 'oom|killed process'

# Did the JVM write a crash artifact? Check working dir and CATALINA_BASE/logs
ls -lt "$CATALINA_BASE"/hs_err_pid*.log 2>/dev/null | head
ls -lt "$CATALINA_BASE/logs"/hs_err_pid*.log 2>/dev/null | head

# If a PID exists, is it runnable or stuck in D state?
ps -o pid,ppid,stat,etime,cmd -p "$(pgrep -f 'catalina.startup.Bootstrap')" 2>/dev/null

# Is the HTTP connector actually bound? Process alive does not mean serving.
# Replace 8080 with your configured connector port.
ss -tlnp 'sport = :8080'

# What does the supervisor think happened?
systemctl status tomcat 2>/dev/null || systemctl status tomcat9 2>/dev/null
journalctl -u tomcat --since '1 hour ago' --no-pager | tail -50

How to diagnose it

1. Confirm the process is really gone. Run pgrep directly on the host, not through a wrapper that may mask exit codes. A stale PID file is not proof of death.
2. Classify the exit before touching the service. Check the kernel log first, because OOM kills leave nothing in Tomcat’s own logs. Then look for hs_err_pid files for JVM crashes. Then inspect the supervisor journal for failed restarts.
3. If the kernel log shows Out of memory: Killed process referencing the JVM, you have an OS OOM kill. The JVM could write nothing. Move to memory sizing: heap versus total RSS versus the cgroup limit.
4. If an hs_err_pid<pid>.log matches the outage window, you have a JVM crash. Open it and read the top: the signal name, the Problematic frame (which often names a native library), and the VM arguments.
5. If neither exists and the service is failed, the JVM exited with a code systemd treated as failure, or the restart policy did not fire. Check Restart=, SuccessExitStatus=, StartLimitBurst, and the exit code in the journal.
6. If a PID exists but sits in D state, do not restart yet. Identify the blocked kernel operation with cat /proc/<pid>/stack and cat /proc/<pid>/wchan. Stale NFS and wedged storage are the usual culprits.
7. Restart only after you have a cause class. Otherwise the condition repeats, often quickly.

Metrics and signals to monitor

Fixes

OS OOM kill

The OOM killer scores victims primarily by RSS (resident set size), not by Java heap usage. A Tomcat whose heap is well within -Xmx can still be killed when native memory (thread stacks, Metaspace, direct buffers, JNI allocations) pushes total RSS past the system or cgroup limit. Confirm the kill, then close the gap between what the JVM can consume and what the kernel allows.

• Compare -Xmx plus expected native overhead against the cgroup memory limit in containers, or free host memory on bare metal. If the container limit sits below -Xmx plus native overhead, the kernel kills the JVM before the heap ever fills.
• Always set -XX:MaxMetaspaceSize. Without it, a classloader leak grows native memory until the OS kills the process with no JVM-level error.
• Consider -XX:+ExitOnOutOfMemoryError (available since JDK 8u92) so a genuine Java-level OutOfMemoryError exits cleanly and lets the supervisor restart, instead of the JVM limping on in a compromised state.

JVM crash

An hs_err_pid<pid>.log means the JVM itself faulted. Read the top of the file: the signal line (SIGSEGV, SIGBUS, SIGABRT), the Problematic frame (which often names a native library), and the loaded native libraries list. Typical drivers are a JNI library built against a different JVM or glibc, a broken TLS native provider, or a JIT bug.

• Match the crash to a frame. If Problematic frame points at a native library, rebuild or upgrade that library.
• If the frame is inside the JVM (libjvm.so), check whether you are on a known-bad JDK build and apply the vendor patch.
• The APR/native connector was removed in Tomcat 10.1. If you are still carrying APR on 9.0.x, a native mismatch is a likely crash source.

Until the root cause is fixed, Restart=on-failure keeps the service alive, but treat repeated identical crashes as a hard blocker rather than a workaround.

Failed restart

systemd stops a Java service with SIGTERM. The JVM shuts down gracefully and exits 143 (128 plus 15). Without SuccessExitStatus=143 in the unit, systemd records that exit as failure, and the next start may be delayed or blocked by StartLimitBurst and StartLimitIntervalSec.

• Add SuccessExitStatus=143 so a clean SIGTERM shutdown is not recorded as failed.
• Confirm Restart=on-failure (or Restart=always) with a sane RestartSec. Without a restart policy, a one-off crash becomes a manual outage.
• If the unit is stuck in failed after hitting the start limit, clear it with systemctl reset-failed before restarting. That is recovery, not a fix.

D-state hang

A process stuck in D is waiting on the kernel. kill -9 does nothing until the syscall returns, so do not assume Tomcat is the problem.

• Read /proc/<pid>/stack and /proc/<pid>/wchan to find the blocked operation.
• Stale NFS handles and wedged SAN paths are the usual cause. Fixing the storage or network is the only real remedy.
• Recovering the underlying resource (unmount, fence the host) is a last resort when the kernel call truly cannot complete.

Prevention

• Size total memory, not just heap. Plan for -Xmx plus thread stacks (-Xss times peak thread count), Metaspace, direct buffers, and JNI overhead. The OOM killer cares about RSS.
• Set -XX:MaxMetaspaceSize. A classloader leak should fail loudly inside the JVM, not die silently under SIGKILL.
• Run under a supervisor with a real restart policy. Restart=on-failure, a modest RestartSec, and SuccessExitStatus=143 prevent a one-off exit from becoming an outage.
• Pin the crash artifact location. Add -XX:ErrorFile=$CATALINA_BASE/logs/hs_err_pid%p.log to JAVA_OPTS so crash files land in a known, rotated directory rather than the JVM working directory, which under systemd is often /.
• Treat the connector check as the real availability signal. Port bound and health endpoint returning, with process-alive as a supporting floor.
• Keep cgroup limits above -Xmx plus native overhead. Otherwise the kernel OOM-kills before the JVM ever sees an OutOfMemoryError.

How Netdata helps

• Per-second process presence, CPU, and memory for the JVM, so a crash or OOM kill surfaces immediately rather than at the next polling interval.
• Kernel OOM events shown alongside process metrics, so a SIGKILL with no Tomcat log is still visible and correlated with the RSS spike that preceded it.
• JVM heap, Metaspace, GC, and thread metrics via JMX, letting you watch the memory pressure that becomes an OOM kill before the kernel intervenes.
• Anomaly detection on RSS and heap baselines, which flags the slow drift that ends in a silent kill and leaves nothing in catalina.out.
• systemd unit state and restart counters, so a service stuck in failed or hitting its start limit is visible next to the process metrics.