Apache Tomcat Operations Guides on Netdata

How Tomcat actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

Tomcat looks like a single process serving HTTP, but inside it is a nested container hierarchy with a three-tier buffering model in front of your servlet code. Most production incidents are not bugs in Tomcat; they are mismatches between what the operator thought was happening and what the connector and thread pool were doing. The thread pool exhausts while CPU sits at 5%. The JVM is alive but every request hangs. Sessions fill the heap while request rate looks normal. Clients see connection timeouts while Tomcat logs nothing.

Tomcat 'appears to have started a thread but has failed to stop it': undeploy leak warnings

Mon, 01 Jan 0001 00:00:00 +0000

The exact log line is the symptom. When you undeploy or redeploy a Tomcat webapp and see this in catalina.out:

SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
The web application [myapp] appears to have started a thread named [AbandonedConnectionCleanupThread]
but has failed to stop it. This is very likely to create a memory leak.

that is not generic noise. During Context stop, WebappClassLoaderBase.clearReferencesThreads() walked every live thread in the JVM, found at least one whose context classloader is still the webapp’s WebappClassLoader, and named it. The bracketed thread name is the breadcrumb back to the library or application code that started it.

Tomcat 401 flood on the Manager app: credential brute force and LockOutRealm

Mon, 01 Jan 0001 00:00:00 +0000

A 401 flood is a sustained spike of HTTP 401 responses against /manager, /host-manager, or any realm-secured endpoint. The signature is concentration: many 401s from a small set of source IPs, or many 401s aimed at a single username. Scattered 401s from mistyped passwords are noise; concentration and sustain are the signal.

Against the Manager app, a 401 flood is almost always credential brute force or stuffing and a precursor to compromise. A successful Manager login lets an attacker deploy WAR files (RCE), enumerate sessions, and undeploy applications. An internet-exposed Manager with weak or default credentials can be fully taken over within minutes of the flood beginning.

Tomcat 5xx error rate: separating server failures from crawler 404s

Mon, 01 Jan 0001 00:00:00 +0000

Your Tomcat error rate alert fired. The dashboard shows a spike in errorCount. The JVM is healthy, the thread pool has headroom, heap is fine. The access log tells a different story: 404s from a crawler storm, not 500s from a failing app. The page was a false alarm.

This is the Tomcat error-rate trap. The JMX errorCount attribute exposed via Catalina:type=GlobalRequestProcessor,name="http-nio-8080" counts every response with status >= 400. It does not distinguish 4xx from 5xx. A crawler hitting dead URLs inflates it identically to an application throwing unhandled exceptions. If you alert on this counter as a ratio of total requests, crawler noise will page you.

Tomcat accept queue overflow: acceptCount, somaxconn, and Recv-Q

Mon, 01 Jan 0001 00:00:00 +0000

Clients start seeing “connection refused” or TCP timeouts, but the Tomcat JVM looks healthy. The thread pool has capacity. Heap is stable. JMX shows nothing wrong. The manager status page reports normal request counts. The application logs are quiet.

The problem is in a place Tomcat cannot see: the OS-level TCP accept queue, also called the listen backlog. When this queue fills, the kernel refuses new connections by sending RST or silently dropping the completed handshake. No JMX counter tracks this. No Tomcat log records it. The only direct signal is ss output showing Recv-Q climbing toward Send-Q on the listening socket.

Tomcat accepts connections but never responds: the TCP-connect trap

Mon, 01 Jan 0001 00:00:00 +0000

A client opens a TCP connection to Tomcat on port 8080. The three-way handshake completes. The client sends an HTTP request. Nothing comes back. The connection hangs until the client times out. Meanwhile, the load balancer health check still reports the instance as healthy, because its check is a bare TCP connect that succeeds every time.

A successful connect() only proves the kernel accepted the socket. It says nothing about whether Tomcat has a worker thread available, whether the JVM is mid-garbage-collection, or whether any deployed application can serve a response. On NIO Tomcat, the default connector since 8.5, the OS accept queue and the connector poller can hold thousands of connections even when the worker thread pool is completely exhausted. A port check is nearly useless as a health signal.

Tomcat access log setup: adding %D and %T for per-request latency

Mon, 01 Jan 0001 00:00:00 +0000

Tomcat’s common and combined access log patterns record the request line, status, and byte count. They do not record per-request latency. Without per-request timing in the access log, you are limited to the cumulative average that JMX exposes via processingTime / requestCount on the GlobalRequestProcessor MBean. That average hides the tail: a few 15-second requests averaged against a thousand 10-millisecond requests looks fine while real users time out.

Adding %D or %T to the AccessLogValve pattern puts per-request latency alongside the status code, URL, and client IP. This enables p95/p99 computation, 5xx-only latency filtering, and per-endpoint slow-request investigation. The change is one pattern string. The traps are version-specific units and synchronous logging overhead.

Tomcat activeSessions growing without plateau: the session leak

Mon, 01 Jan 0001 00:00:00 +0000

The chart shows activeSessions from Catalina:type=Manager,host=localhost,context=/<app> climbing in a near-straight line. No plateau. You restarted Tomcat this morning, and by midafternoon the count is already back to where it was before the restart. Sessions are being created faster than they expire, and with the default 30-minute session-timeout, expiry is slow enough that the slope looks gentle until you compare it to the request rate.

This is a session leak, not a traffic spike. A spike produces a step up that plateaus at roughly peak_creation_rate * session_timeout. A leak has no plateau: the line keeps rising after traffic drops because sessions that should have expired are still pinned, or because new sessions are being minted faster than the background expiry loop can clear them.

Tomcat average latency lies: why you need p95/p99 from the access log

Mon, 01 Jan 0001 00:00:00 +0000

Your latency dashboard says Tomcat is healthy at 250ms. Your users say requests take 15 seconds. Both are right. The JMX average your dashboard consumes is structurally incapable of representing the tail.

Catalina:type=GlobalRequestProcessor exposes processingTime (cumulative milliseconds since startup) and requestCount. Divide the deltas over a window and you get a mean. A workload where p50 is 100ms but p99 is 15s produces an average of roughly 250ms. 1% of your users wait an eternity, but the chart looks fine. Stuck threads averaged into fast requests are the classic blind spot, and Tomcat gives you nothing from JMX that breaks the mean open.

Tomcat classloader leak on redeploy: why the old WebappClassLoader never dies

Mon, 01 Jan 0001 00:00:00 +0000

Every hot redeploy leaves the previous WebappClassLoader in memory. After enough redeploys the JVM hits OutOfMemoryError: Metaspace, or if -XX:MaxMetaspaceSize is unset, the OS OOM-kills the process with no JVM error at all. Heap looks flat. GC looks fine. RSS climbs until the process dies.

Each Tomcat webapp gets its own WebappClassLoader for isolation. On undeploy, the classloader and every class it loaded should be collected. If anything still holds a reference to a class loaded by the webapp, a ThreadLocal value, a JDBC driver registered with DriverManager, a Timer thread that was never cancelled, a log appender, or a static field in a shared library, the classloader is pinned. All its classes stay in Metaspace forever.

Tomcat connection refused: maxConnections and acceptCount both exhausted

Mon, 01 Jan 0001 00:00:00 +0000

Clients are getting “connection refused” or connections hanging until timeout. The Tomcat JVM is up, the HTTP port is bound, heap looks fine, and there is nothing in catalina.out. Manager status shows the connector alive. From Tomcat’s perspective, nothing is wrong. From the kernel’s perspective, the listening socket’s accept queue is full and new SYNs are being dropped or RST’d.

This happens when two limits stack: the NIO poller has hit maxConnections (default 8192 for NIO) and the OS accept queue, bounded by acceptCount (default 100) and clamped by net.core.somaxconn, is also full. No JMX counter exposes this state. Tomcat logs nothing. Detection is either client-side connection error monitoring or ss -tnl showing Recv-Q stuck at the accept queue limit.

Tomcat context in FAILED state: the app is down but Tomcat looks healthy

Mon, 01 Jan 0001 00:00:00 +0000

Your monitoring says Tomcat is up. The JVM process is running, the HTTP connector accepts TCP connections, and a curl to port 8080 returns an HTTP response. But every request to the application returns 404, and users cannot reach any endpoint.

This is the signature of a context in FAILED or STOPPED state. Tomcat continues running. The connectors keep listening. Other deployed contexts keep serving. But the failed context’s servlet mappings are inactive, so every request to it returns 404. From the outside, it looks like a missing route or an undeployed application.

Tomcat file descriptor usage: OpenFileDescriptorCount vs the ulimit

Mon, 01 Jan 0001 00:00:00 +0000

Tomcat’s JVM holds file descriptors for every open socket, every log file, every JAR on the classpath, the NIO selector itself, and various internal pipes. The pool is finite and process-scoped. When it runs out, Tomcat stops accepting connections and stops writing to logs in the same instant. The error is java.net.SocketException: Too many open files, and by the time you see it the failure is already total.

Operators have two views into this resource. JMX exposes OpenFileDescriptorCount and MaxFileDescriptorCount from the java.lang:type=OperatingSystem MBean. The OS exposes the same count through /proc/<pid>/fd and the limit through ulimit -n and /proc/<pid>/limits. The two views should agree within one descriptor. When they do not, you are usually looking at the wrong process, the wrong container namespace, or at lsof output (which counts more than file descriptors).

Tomcat frequent Full GC: pause time, G1, and the 5% overhead rule

Mon, 01 Jan 0001 00:00:00 +0000

A Tomcat instance that was steady for hours or days starts pausing. Latency p99 climbs, request throughput drops in bursts, and the JVM is alive but barely making progress. The thread pool is not the bottleneck: the wall clock is being eaten by garbage collection. In the GC log you see “Pause Full” lines arriving every few seconds, each stopping the world for hundreds of milliseconds to multiple seconds.

On G1GC, the default collector since JDK 9, this is not a tuning quirk. G1 does its old-generation work concurrently. A Full GC in G1 is a fallback, not a scheduled event, and it means the concurrent machinery failed to keep up with allocation or live-data growth. One Full GC during a traffic spike is worth a glance. Multiple per minute is a crisis that ends in OutOfMemoryError or effective livelock unless you intervene.

Tomcat GC death spiral: full GCs dominating and throughput collapsing

Mon, 01 Jan 0001 00:00:00 +0000

Your Tomcat JVM is up, the connector port is bound, but requests crawl or hang. Throughput graphs stutter: brief bursts of activity separated by flatlines where no requests complete. CPU is pinned near saturation.

The mechanism is a positive feedback loop. Live data on the heap has grown until each GC cycle reclaims almost nothing. The JVM compensates by running GC more frequently, which consumes more CPU, which leaves less wall clock for application threads, which causes requests to take longer, which causes more concurrent threads to stay active, which allocates more memory. The loop ends in OutOfMemoryError or effective livelock where GC runs continuously and the application makes no progress.

Tomcat Ghostcat (CVE-2020-1938): AJP connector exposure and arbitrary file read

Mon, 01 Jan 0001 00:00:00 +0000

Ghostcat (CVE-2020-1938) is a configuration-driven vulnerability in Apache Tomcat’s AJP connector. Before the 9.0.31 / 8.5.51 / 7.0.100 fixes, the AJP protocol carried no authentication. Any client that could reach port 8009 could craft AJP requests that made Tomcat read and return arbitrary files from the server, including WEB-INF/web.xml and server.xml. In some configurations the same primitive enables remote code execution through JSP inclusion.

The fix was not a patch to existing behavior. It was a set of new defaults: the AJP connector binds to the loopback address and requires a secret attribute with secretRequired=true. Deployments that carried an older server.xml forward, or operators who re-enabled AJP without setting the new attributes, remain exposed on instances that look perfectly healthy to a normal HTTP health check.

Tomcat heap dump before restart: capturing evidence with jmap and jstack

Mon, 01 Jan 0001 00:00:00 +0000

When Tomcat is failing, the instinct is to restart. That instinct is right for recovery and wrong for diagnosis. The JVM process holds the only copy of the evidence: the object graph that explains the heap exhaustion, the thread stack traces that explain the pool stall, the GC state that explains the death spiral. Once the process exits, that evidence is gone. You are left restarting blind with nothing but an OutOfMemoryError line in catalina.out.

Tomcat heap usage: watch the post-GC baseline, not the sawtooth peak

Mon, 01 Jan 0001 00:00:00 +0000

Every JVM heap graph looks the same: a jagged sawtooth that climbs steadily, drops sharply, and repeats. If you alert on “heap > 80%”, you will page on every pre-GC peak. That alert fires dozens of times per hour on a healthy Tomcat, training your team to ignore heap warnings until a real OutOfMemoryError arrives and nobody saw it coming.

The heap is designed to fill between garbage collection cycles. High instantaneous usage is normal. The signal that matters is the valley after each collection: the post-GC baseline. When that baseline is stable, your live data set is bounded. When it rises over hours or days, live data is accumulating and a leak is underway.

Tomcat hot redeploy vs clean restart: when autoDeploy leaks and when it doesn't

Mon, 01 Jan 0001 00:00:00 +0000

When you deploy a new WAR, Tomcat gives you two paths. Let autoDeploy watch webapps/ and swap the application in place, or stop the JVM and start a clean one. Both deliver the new code. They differ in what they leave behind, measured in Metaspace.

The root cause is the WebappClassLoader. Each Context gets its own. On a hot redeploy the old one is supposed to be garbage collected. Often it is not. A lingering ThreadLocal, a DriverManager registration, a timer thread, or a static field can pin the old classloader and every class it loaded. On a clean restart the JVM dies, and every classloader dies with it. There is nothing to pin.

Tomcat HTTP Status 503 Service Unavailable: the connector is out of threads

Mon, 01 Jan 0001 00:00:00 +0000

A 503 in a Tomcat topology means something between the user and the servlet gave up. The instinct is to blame the connector thread pool, and that is often the right place to look, but the mechanism is more layered than it appears. Treating “503” and “out of threads” as the same condition leads to misdiagnosis.

With the default NIO connector, Tomcat decouples worker threads from TCP connections. A request thread pool can be saturated while the connector keeps accepting connections, and the connection pool can be saturated while the JVM still passes a TCP health check. The path from a full thread pool to a visible 503 passes through two more buffers before any client sees a failure.

Tomcat java.lang.OutOfMemoryError: Java heap space: the heap is genuinely full

Mon, 01 Jan 0001 00:00:00 +0000

The string java.lang.OutOfMemoryError: Java heap space in catalina.out means the JVM could not satisfy an allocation because live (reachable) objects plus the requested size exceeded the maximum heap (-Xmx). This is not a transient GC spike or a young-gen promotion failure. The heap is genuinely full of data the collector cannot reclaim.

Tomcat’s heap holds HTTP sessions, request/response buffers, application objects, and in-process caches. When live data outgrows -Xmx, GC frequency rises, each cycle reclaims less, and application threads get less time between collections. Eventually an allocation fails and the JVM throws OutOfMemoryError: Java heap space. The process may limp on in a degraded state, serving some requests and failing others, until restarted.

Tomcat java.lang.OutOfMemoryError: Metaspace: the classloader leak hot redeploys cause

Mon, 01 Jan 0001 00:00:00 +0000

A java.lang.OutOfMemoryError: Metaspace after a redeploy is the canonical classloader leak symptom in long-running Tomcat instances. The JVM still has heap headroom, GC looks healthy, and the process may have been up for weeks. Then a deploy lands and Tomcat dies, or in containerized setups the kernel OOM-kills it with no JVM-level error at all.

Recovery is the same in every case: restart the JVM. That clears Metaspace and brings the application back, but it does not fix anything. If you hot-deploy again, the leak returns and Metaspace climbs one step higher per cycle until the next crash.

Tomcat java.net.BindException: Address already in use: the connector never starts

Mon, 01 Jan 0001 00:00:00 +0000

You restart Tomcat. The JVM comes up. The process check goes green. Then catalina.out shows:

SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector[HTTP/1.1-8080]]
java.net.BindException: Address already in use

The JVM is alive, but the HTTP connector never bound its port. No traffic is served. Health checks that only verify the PID report healthy while every client gets connection refused.

This is the “Connector Binding vs Process Alive” trap: the JVM stays up after a connector init failure, so process-based monitoring passes while nothing listens. The failure modes are narrow. A stale Tomcat still holding 8080 or 8443, a port conflict with another service, or a previous shutdown that did not release the socket. Each has a different fix, and conflating them produces the wrong remediation.

Tomcat java.net.SocketException: Too many open files: file descriptor exhaustion

Mon, 01 Jan 0001 00:00:00 +0000

java.net.SocketException: Too many open files in a Tomcat log is a hard cliff, not gradual degradation. The JVM goes from serving traffic to unable to accept a new TCP connection, open a log file, or load a JAR resource. The error is frequently misread as a disk fault (log writes break) or a network fault (accept() fails), when the real constraint is the per-process file descriptor limit.

Every socket, open file, NIO selector, and JAR handle the JVM touches consumes one file descriptor against the process ulimit -n ceiling. A busy Tomcat with thousands of keepalive connections, a deep classpath, and several rotating log files can sit at hundreds of FDs at idle and climb into the thousands under load. When the count hits the limit, the kernel refuses the next open() or accept() syscall and the JVM throws the exception.

Tomcat JDBC connection leak: numActive stuck at max even when idle

Mon, 01 Jan 0001 00:00:00 +0000

The signature of a Tomcat JDBC connection leak is specific: numActive sits at maxActive and refuses to fall, even when request rate is near zero. New requests that need a database connection block on getConnection() until maxWait elapses, then throw a pool exhaustion error. The database reports a flock of idle Sleep connections from Tomcat that never close.

The cascade is what takes the service down. Each request blocking on getConnection() also holds an HTTP worker thread. As leaked connections accumulate, more threads stall on the pool, the thread pool fills, and requests queue in the acceptor. The JVM is healthy, the database is healthy, CPU is low, and the service is effectively down.

Tomcat JDBC connection pool exhaustion: numActive at maxActive and threads blocking

Mon, 01 Jan 0001 00:00:00 +0000

HTTP requests start timing out, currentThreadsBusy climbs toward maxThreads, CPU stays low, and the database looks healthy. The Tomcat JDBC pool has hit its ceiling: numActive equals maxActive, waitCount is greater than zero, and every request that needs a database connection is parked in getConnection().

The JDBC pool and the HTTP worker thread pool are coupled. A request holds a worker thread for its entire duration. When it blocks in getConnection(), that worker stays occupied. Database pool exhaustion drives thread pool exhaustion, and thread pool exhaustion drives accept queue buildup and 503s. By the time users see errors, both resources are already exhausted.

Tomcat keepalive and connectionTimeout: coordinating timers with your reverse proxy

Mon, 01 Jan 0001 00:00:00 +0000

When Tomcat sits behind nginx, HAProxy, Apache httpd, or a cloud load balancer, the proxy-to-Tomcat connection is almost always persistent keep-alive. Both ends maintain idle-timeout timers that independently decide when to close. If those timers are not coordinated, the proxy periodically tries to reuse a connection Tomcat has already closed, and the user gets a 502 or a connection reset.

The proxy and Tomcat each behave correctly on their own. The problem is a race between two independently configured clocks, and both sets of defaults are tuned for direct client-to-server traffic, not proxy-to-origin. The one rule that eliminates the race: the inner layer (Tomcat) must always time out after the outer layer (the proxy).

Tomcat Manager app exposed: default credentials, WAR upload, and RCE

Mon, 01 Jan 0001 00:00:00 +0000

The Tomcat Manager and Host Manager web applications are administrative interfaces bundled with standalone Tomcat distributions. The Manager app deploys and undeploys WAR files, lists and invalidates HTTP sessions, reloads applications, and exposes server status. The deploy capability is a direct path to remote code execution: an attacker uploads a WAR containing a webshell or reverse-shell payload, Tomcat auto-deploys it, and the code runs inside the JVM with the filesystem and network access of the Tomcat process.

Tomcat maxConnections saturation: the NIO poller stops accepting

Mon, 01 Jan 0001 00:00:00 +0000

Clients report connection timeouts or “connection refused” errors. The Tomcat JVM is running, the HTTP port is bound, GC looks normal, and the worker thread pool may be mostly idle. What you are looking at is maxConnections saturation: the NIO poller has filled to its configured ceiling and the acceptor has stopped registering new sockets.

The default maxConnections for NIO is 8192 (since Tomcat 9.0.30; it was 10000 for NIO on older 8.5.x and early 9.x releases). When connectionCount approaches that number, Tomcat stops accepting new connections until existing ones close. New SYNs pile into the OS TCP backlog, bounded by acceptCount (default 100). When that queue fills, the kernel sends RST and clients see a hard refusal.

Tomcat MaxMetaspaceSize unset: the silent OS OOM-kill with no Java error

Mon, 01 Jan 0001 00:00:00 +0000

The Tomcat process is gone. catalina.out ends mid-line or shows a normal shutdown request that never executed. There is no OutOfMemoryError, no HeapDumpOnOutOfMemoryError file, and no JFR recording. The systemd unit reports code=killed, status=9/KILL or simply a vanished PID. Your heap dashboard looked healthy right up to the moment the process disappeared.

This is the silent Metaspace kill. The JVM never throws a Java-level error because the kernel terminates it first. When -XX:MaxMetaspaceSize is left at its default (effectively unlimited), class metadata lives in native memory that is not bounded by -Xmx, not covered by heap alerts, and not reclaimed when the heap looks fine. Metaspace grows until process RSS hits the OS limit or the container cgroup limit, and the Linux OOM-killer ends the JVM with SIGKILL.

Tomcat maxThreads and minSpareThreads: sizing the executor correctly

Mon, 01 Jan 0001 00:00:00 +0000

The <Executor> thread pool, or the connector-level pool when no explicit executor is referenced, is the most critical bounded resource in a Tomcat instance. maxThreads sets the ceiling on concurrent request processing. minSpareThreads sets the floor of pre-warmed threads. Every active HTTP request occupies one worker thread for its entire processing duration, unless you use async servlets. When the pool is full, connections keep arriving but requests wait.

In a typical servlet workload, threads spend most of their lifetime blocked on I/O: database queries, downstream HTTP calls, cache lookups. A pool sized to CPU count is almost always too small. A pool sized to peak concurrent requests times worst-case processing time is closer to correct, but the queueing behavior between connections, the poller, and worker threads makes the accounting subtle.

Tomcat monitoring checklist: the signals every production instance needs

Mon, 01 Jan 0001 00:00:00 +0000

Tomcat’s capacity model rests on a small number of bounded resources: a worker thread pool, a connection poller, the JVM heap, Metaspace, and the OS file descriptor table. Most production outages are one of these hitting its limit while the JVM process keeps running. A monitoring setup that only tracks CPU and memory will miss the dominant failure mode: thread pool exhaustion with the JVM at low CPU and a healthy process.

Tomcat monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Most Tomcat monitoring setups start with a process check, a health endpoint probe, and maybe a 5xx alert. That catches crashes. It does not catch the failures that actually page teams: thread pool exhaustion, GC death spirals, accept queue overflow, classloader leaks after hot redeploys. This article maps the signals that matter across four maturity levels so you can audit what you have, identify what you are missing, and prioritize what to add next.

Tomcat NoClassDefFoundError after redeploy: stale classloaders serving half-loaded classes

Mon, 01 Jan 0001 00:00:00 +0000

A hot redeploy on Tomcat succeeds, the new context reports STARTED, and within minutes requests start failing with java.lang.NoClassDefFoundError for classes that are visibly present in the new WAR. Sometimes it is ClassNotFoundException. Sometimes the error names a class that was renamed or removed between versions. Restarting the JVM clears it. The next redeploy brings it back.

The root mechanism is the WebappClassLoader lifecycle. Tomcat creates a new classloader for each web application context and replaces it on every redeploy. The old classloader should be garbage collected along with every class it loaded. When something holds a strong reference to it (a thread, a ThreadLocal, a JDBC driver registration, a logging appender, a static field, a shutdown hook), it stays alive. Threads still bound to it try to resolve classes against it. Those classes either no longer exist in the new context or conflict with what the old classloader already loaded, and the JVM throws NoClassDefFoundError or ClassNotFoundException.

Tomcat OutOfMemoryError: GC overhead limit exceeded: GC running but freeing nothing

Mon, 01 Jan 0001 00:00:00 +0000

java.lang.OutOfMemoryError: GC overhead limit exceeded means the heap is functionally full. GC has been running almost continuously and recovering almost nothing. This is the terminal stage of a GC death spiral, thrown just before a hard Java heap space OOM.

The exact rule the JVM applies: more than 98% of CPU time spent in GC and less than 2% of heap recovered, sustained across five consecutive collections. When both conditions hold, the JVM aborts rather than burn CPU indefinitely. Instantaneous heap usage can still read under -Xmx when this fires. By that point, the live set is effectively pinned to the ceiling.

Tomcat OutOfMemoryError: unable to create new native thread: the JVM can't spawn threads

Mon, 01 Jan 0001 00:00:00 +0000

When Tomcat throws java.lang.OutOfMemoryError: unable to create new native thread, do not reach for heap dumps or GC tuning. This is not a heap problem. The JVM called pthread_create and the kernel refused. The heap can be 20% full when this fires.

Heap OOMs need -Xmx increases, leak hunting, or GC tuning. Native thread OOMs need ulimit -u increases, cgroup pids.max adjustments, -Xss tuning, or finding the code that creates threads without bound.

Tomcat process not running: crashes, OOM-kills, and failed restarts

Mon, 01 Jan 0001 00:00:00 +0000

The Tomcat JVM is gone. pgrep -f org.apache.catalina.startup.Bootstrap returns nothing, the service check is red, and your health probe is timing out. Before you restart anything, find out why it died, because the same cause will kill it again, often within minutes.

“Process not running” conflates three distinct events: the JVM crashed inside its own runtime, the kernel OOM-killed it with SIGKILL, or the supervisor (systemd, init, container runtime) failed to bring it back. Each leaves evidence in a different place, and only one of them leaves anything in catalina.out. This guide separates those three classes plus the “alive but nonfunctional” D-state trap.

Tomcat request processing time climbing: reading processingTime correctly

Mon, 01 Jan 0001 00:00:00 +0000

Your monitoring shows Tomcat request processing time trending upward. Before you chase the trend, understand what the processingTime attribute actually represents.

processingTime on the GlobalRequestProcessor MBean is a cumulative counter of wall-clock milliseconds across all requests processed since Tomcat started. It is not a per-request value, not a rate, and not a percentile. Reading it raw produces a number that only increases until the JVM restarts. To get anything useful, compute a delta over a time window and divide by the delta of requestCount over the same window.

Tomcat request throughput dropping: requestCount rate below baseline

Mon, 01 Jan 0001 00:00:00 +0000

The Tomcat requestCount counter on GlobalRequestProcessor is climbing slower than expected. The drop tells you fewer requests are being processed, not why. The cause can be upstream of Tomcat (load balancer routed traffic away, health check started failing, CDN absorbing load) or inside Tomcat (worker thread pool exhausted, JVM in a GC death spiral, deadlock consuming the pool).

The first diagnostic fork: is traffic still arriving at this host? If the answer is no, you have a routing, load balancer, or health check problem, and chasing thread dumps wastes time. If yes, Tomcat is failing to keep up, and the question becomes whether the failure is I/O bound (threads blocked on a slow backend) or CPU bound (GC thrashing or a compute-bound code path).

Tomcat RSS growing while heap looks flat: native memory and the OOM killer

Mon, 01 Jan 0001 00:00:00 +0000

The Tomcat JVM dies overnight. systemctl status shows exit code 137. Heap dashboards look flat the entire time. No hs_err_pid.log, no heap dump, no OutOfMemoryError in catalina.out. The kernel, not the JVM, ended the process.

The Linux OOM killer scores victims by RSS plus an adjustment (oom_score_adj), not by Java heap. Everything the JVM keeps resident outside the heap counts: Metaspace, thread stacks, direct ByteBuffers, JIT code cache, JNI allocations, and glibc arenas. A Tomcat whose heap sawtooth looks healthy can still have RSS climbing toward the container limit. When RSS crosses memory.max, the kernel sends SIGKILL. The JVM cannot trap it, log it, or dump anything.

Tomcat session timeout and maxActiveSessions: bounding session memory

Mon, 01 Jan 0001 00:00:00 +0000

HTTP sessions are heap-resident state. Each active session holds references to its attribute objects, and those objects remain live until the session is invalidated or expires. Two configuration parameters bound this heap usage: the session timeout (how long a session can live) and maxActiveSessions (how many can exist simultaneously).

At their defaults, neither bounds memory. The session timeout defaults to 30 minutes. maxActiveSessions defaults to -1, meaning no limit. Under sustained traffic, bot-driven session creation, or a traffic spike, sessions accumulate until the heap fills. The result is either an OutOfMemoryError or a GC death spiral where the JVM spends most of its CPU collecting and barely processing requests.

Tomcat sessions eating the heap: bots, getSession(true), and unbounded growth

Mon, 01 Jan 0001 00:00:00 +0000

Post-GC heap baseline climbing steadily, Full GC frequency increasing, and eventually an OutOfMemoryError or a GC death spiral that leaves the JVM effectively unresponsive. Thread dumps look normal, CPU is dominated by GC threads, and a restart clears the problem temporarily before the cycle repeats within hours or days.

A common and overlooked root cause is HTTP session accumulation. Under Tomcat’s default StandardManager, every active session lives in JVM heap. When sessions are created faster than they expire, they fill the heap, promote to Old Gen, and drive major GCs. Sessions are frequently the largest heap consumer in a Tomcat app, and unbounded session count is a classic memory leak vector.

Tomcat SEVERE Error deploying web application: reading startup failures in catalina.out

Mon, 01 Jan 0001 00:00:00 +0000

The line you searched for looks something like this in catalina.out:

SEVERE [...] org.apache.catalina.startup.HostConfig.deployWAR Error deploying web application [...]
java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/yourapp]]

That stack is a wrapper. The class that actually failed, the missing dependency, the listener that threw, or the NoSuchMethodError from a library clash is not in catalina.out. It is in localhost.YYYY-MM-DD.log in the same $CATALINA_BASE/logs directory. The Tomcat default logging.properties routes org.apache.catalina.core.ContainerBase.[Catalina].[localhost] to the 2localhost.org.apache.juli.AsyncFileHandler, which writes that file. The SEVERE line in catalina.out only tells you the deploy failed and that the context is now a FailedContext placeholder rather than a StandardContext.

Tomcat shutdown port 8005: the default SHUTDOWN command is an instant kill

Mon, 01 Jan 0001 00:00:00 +0000

The shutdown port is one of Tomcat’s oldest features. The stock server.xml ships with <Server port="8005" shutdown="SHUTDOWN">, which opens a raw TCP listener. Anything that can reach that port and send the bytes SHUTDOWN triggers an immediate, orderly shutdown. No authentication, no HTTP, no handshake. The string is well-known because it is the shipped default.

This is a documented feature, not a CVE. But it is a configuration risk that operators routinely misjudge. If port 8005 is reachable from an untrusted network and still carries the default command, a single netcat line stops the JVM. The mitigation is straightforward: bind the listener to localhost, replace the command with a long random string, or disable the port entirely with port="-1".

Tomcat Slowloris slow-client attack: connections climbing while bytes stay near zero

Mon, 01 Jan 0001 00:00:00 +0000

Your Tomcat HTTP connector’s connection count keeps climbing. The bytes received rate is nearly flat. CPU is normal, heap looks healthy, GC is quiet, and the thread pool is nowhere near saturated. Yet legitimate users are timing out, and new connections are starting to get refused.

This is the Slowloris slow-client pattern. An attacker opens many TCP connections to Tomcat and dribbles data across them, one byte at a time, or sends a partial HTTP request and never finishes the headers. The connection stays open, consuming a slot in the NIO poller or, on older BIO connectors, a worker thread. The request never completes, so it never reaches your application code. From Tomcat’s perspective, the JVM is idle. From the user’s perspective, the site is down.

Tomcat stuck threads: StuckThreadDetectionValve and 'may be stuck' warnings

Mon, 01 Jan 0001 00:00:00 +0000

The warning shows up in catalina.out: a worker thread “has been active for [N] milliseconds … and may be stuck.” By the time you see it, the thread has already been blocked for the full threshold duration. If the valve is running with its default 600-second threshold, that is 10 minutes of a worker thread doing nothing useful. With maxThreads=200, a handful of these threads eats real capacity.

The StuckThreadDetectionValve is Tomcat’s built-in canary for thread-pool exhaustion. It is not enabled by default. When enabled, it logs a WARN-level message whenever a request-processing thread exceeds a configurable threshold, and it includes the blocking stack trace. That stack trace is the single most useful piece of evidence you will get during a thread-starvation incident, because it tells you exactly where the thread is parked.

Tomcat thread dumps: reading jstack to find what the pool is waiting on

Mon, 01 Jan 0001 00:00:00 +0000

When currentThreadsBusy reaches maxThreads, Tomcat stops processing new requests even though the JVM is healthy. The thread pool gauge tells you the pool is full. It does not tell you why. The only diagnostic that shows exactly what every worker thread is doing at that moment is a JVM thread dump.

jstack <pid> (or the modern equivalent jcmd <pid> Thread.print) is the primary first response for thread-related Tomcat failures. A single dump shows whether each http-nio-exec thread is idle and parked in the task queue, blocked in a native socket read against a slow backend, waiting to acquire a database connection, or contending on a Java monitor. Many threads parked in the same stack frame is the stuck path. A cluster of threads BLOCKED on the same lock is contention, or a deadlock.

Tomcat thread pool exhaustion: currentThreadsBusy at maxThreads and requests hanging

Mon, 01 Jan 0001 00:00:00 +0000

Tomcat is up: the JVM process is running, the HTTP port is open, health checks pass. But requests hang or return 503. The dashboard shows currentThreadsBusy pinned at maxThreads (default 200). CPU is low. Heap is stable. No OutOfMemoryError in the logs.

This is thread pool exhaustion. Every worker thread is occupied and no thread is available to process new requests. The JVM is healthy; the application is just not being served.

Tomcat threads blocked forever: the missing outbound timeout

Mon, 01 Jan 0001 00:00:00 +0000

The JVM is healthy. Heap is fine, no Full GCs, CPU sits at 5%, and the process answers a TCP connect on 8080 within milliseconds. But every HTTP request hangs, the access log has gone quiet, and clients are timing out. When you finally grab a thread dump, most of the http-nio-8080-exec threads are parked in the same stack frame: java.net.SocketInputStream.socketRead0, blocked against a backend that will never reply.

This is the missing-outbound-timeout failure. It is one of the most common causes of Tomcat thread pool exhaustion, and it is almost never Tomcat’s fault. The threads are waiting forever because somewhere in the application or its libraries, an outbound call was made with no connect timeout and no read timeout. In Java, “no timeout” does not mean a sensible default. It means infinite.

Tomcat threads busy but CPU idle: telling a blocked backend from a GC spiral

Mon, 01 Jan 0001 00:00:00 +0000

When users report that Tomcat is “hung” but the JVM is up, ports are open, and the operating system looks fine, the fastest split you can make is to look at one number alongside the thread pool state: JVM CPU. Two different failure modes produce currentThreadsBusy == maxThreads, and they have opposite CPU signatures.

If CPU is low while every worker thread is busy, those threads are parked on I/O: a slow database, a hung downstream HTTP service, an unresolvable DNS lookup, or a stalled NFS mount. The JVM is healthy; a backend is not. Restarting Tomcat is the wrong move, because the threads re-block the moment traffic returns.

Tomcat virtual threads (JDK 21+): why currentThreadsBusy reports -1

Mon, 01 Jan 0001 00:00:00 +0000

You upgraded to JDK 21, enabled virtual threads, and the one Tomcat gauge every dashboard and every alert leans on (currentThreadsBusy) now reads -1. maxThreads still shows 200 but is meaningless. The thread pool utilization panel that used to be your primary saturation signal is blank or broken, and your SLO alerts are either firing constantly or silently dead.

This is expected, by-design behavior, not a bug. When a connector runs on virtual threads (useVirtualThreads="true" on NioEndpoint, or StandardVirtualThreadExecutor, on Tomcat 10.1.x/11.0.x with JDK 21+, or Tomcat 9.0.84+ with useVirtualThreads), Tomcat’s classic bounded-thread-pool model stops applying. There is no fixed pool to measure. currentThreadsBusy returns -1, maxThreads is inherited from defaults and not honored, and the connection layer becomes your only proxy for active request load.

Tomcat weak TLS: disabling TLSv1.0/1.1 and hardening ciphers on 8443

Mon, 01 Jan 0001 00:00:00 +0000

An HTTPS connector on 8443 that still negotiates TLSv1.0 or TLSv1.1 is active downgrade surface. Clients that can force TLSv1.0 can exploit protocol-level weaknesses, and ciphers that exist only to support legacy protocols weaken posture for every client. The fix has two parts: restrict the protocol list to TLSv1.2 and TLSv1.3, and prune the cipher list so weak suites are not offered.

This is a configuration change, not a runtime incident, but it has runtime consequences. A misconfigured SSLHostConfig prevents the connector from starting. A Tomcat upgrade can silently re-enable TLSv1.3 ciphers you intended to exclude. The procedure covers the version split between Tomcat 8.5/9 (where sslEnabledProtocols is deprecated but functional) and Tomcat 10/11 (where it is removed), the cipher attribute split in 9.0.115/10.1.52/11.0.18, and external verification.