$ guides / tomcat ▌

TOMCAT · OPERATIONS PLAYBOOK

Tomcat keeps accepting connections long after it has run out of threads to serve them

A Java servlet container where the NIO connector will happily queue thousands of connections while every one of its 200 worker threads sits blocked on a slow database call — so the process looks alive, CPU reads near idle, and nothing is being served. We trace how a request actually flows from the accept queue to a worker to your servlet, why the heap and Metaspace fail in completely different ways, and what to do when each limit is hit.

> Start with the monitoring checklist → # Jump to the full guide list

Tomcat's defaults get you to production quickly, then hand you a set of limits that most teams only meet during an incident.

The defaults work. Until currentThreadsBusy reaches maxThreads (still 200) and requests hang while the JVM sits at low CPU, because the real problem is a backend call with no timeout. Until a hot redeploy leaks a WebappClassLoader, Metaspace climbs step by step, and with MaxMetaspaceSize unset the OS OOM-killer takes the process with no Java error at all. Until live data outgrows -Xmx and java.lang.OutOfMemoryError: Java heap space arrives after GC has already spent minutes freeing nothing. Until the NIO poller hits maxConnections, the accept queue (acceptCount, still 100) fills, and the kernel starts refusing connections with nothing logged. Until the default ulimit of 1024 file descriptors runs out and Tomcat can neither accept a socket nor rotate a log.

These guides are written for engineers who already run Tomcat, not for people learning what a servlet is. The goal is the mental model of how the container actually behaves under load, the failure patterns that keep recurring, the monitoring story that catches them before they page anyone, and the runbooks you wish someone had handed you before your last incident.

How Tomcat actually runs in production

Tomcat is not one thing that is 'up' or 'down'. It is a chain: the OS hands sockets to a connector, an NIO poller multiplexes thousands of live connections, a bounded worker pool actually processes requests, a container hierarchy routes each one to a webapp, and everything the app allocates lands in the heap or in native memory the JVM manages separately. Most production failures live between these layers, not inside any one of them.

clients + reverse proxy

Browsers, API clients, and the nginx or load balancer in front of Tomcat. Each connection is a socket and a file descriptor; a proxy's keepalive pool means <code>connectionCount</code> reflects the proxy, not end users. Mismatched keepalive timers surface here as connection resets.

CLIENT

connector + acceptor

The <code><Connector></code> binds the port (8080/8443) and an acceptor thread pulls sockets off the OS listen backlog. That backlog is <code>acceptCount</code> (default 100), itself capped by <code>net.core.somaxconn</code>. If the port is already taken the connector never starts — <code>BindException</code> — while the JVM stays alive.

NIO poller + maxConnections

The poller multiplexes every accepted connection up to <code>maxConnections</code> (default 8192 for NIO). A connection slot is not a worker — a live connection can sit here, idle or slow, consuming a poller slot and an FD without ever occupying a thread.

POLL

worker thread pool (executor)

The real capacity model. Up to <code>maxThreads</code> (default 200) worker threads each process one request start to finish. <code>currentThreadsBusy</code> is the headline gauge — when it equals <code>maxThreads</code>, new requests queue behind the poller and time out, no matter how healthy the JVM is.

WORKER

engine · host · context · servlet

The container hierarchy that routes a request to the right webapp <code>Context</code> and <code>Servlet</code>. A context in FAILED or STOPPED serves 404s while the process and connectors look perfectly healthy — easily misread as a missing route rather than a dead app.

ROUTE

webapp classloader

Each <code>Context</code> gets its own <code>WebappClassLoader</code>. A hot redeploy should discard the old one; a <code>ThreadLocal</code>, an un-deregistered JDBC driver, or an uncancelled timer pins it, so every deploy accumulates another full copy of the app's classes in Metaspace.

CLASSLOAD

JVM heap + GC

Objects, caches, and HTTP sessions (<code>StandardManager</code>, on heap) live here under G1GC. The instantaneous sawtooth is meant to fill before GC runs; only the post-GC baseline is the truth. A rising baseline is a leak; a spiralling one ends in <code>OutOfMemoryError: Java heap space</code>.

HEAP

JVM native memory + OS

Metaspace, thread stacks, direct and mmap buffers, and file descriptors all live outside the heap. The Linux OOM killer decides on RSS, not heap, so a native-heavy Tomcat can be killed with heap 'fine'. ulimits bound both file descriptors and threads.

NATIVE

Why this matters: 'Tomcat is hung' can mean the worker pool is exhausted behind a slow backend, the poller is full of slow clients, a GC death spiral is eating CPU, Metaspace is silently OOM-killing the process, a context failed to start, or the file-descriptor ceiling has been hit. The symptom rhymes but each layer has a different signal — and a different fix.

The failures you'll actually see

Most Tomcat incidents fall into a small set of recurring patterns. Recognise the shape, and triage gets dramatically faster.

CRITICAL

Thread pool exhaustion

currentThreadsBusy reaches maxThreads (default 200) and every worker is occupied. The NIO poller keeps accepting connections up to maxConnections while requests queue for a free thread, then time out. The JVM is healthy and CPU is often low — the app just is not being served. It is almost always a slow or hung backend, not Tomcat. Take a thread dump before any restart; the threads re-accumulate until the backend is fixed.

currentThreadsBusy at or equal to maxThreads (default 200)
Requests queuing then timing out while CPU sits near idle
jstack shows http-nio-exec threads blocked in socketRead0 or connection acquire
A restart clears it, then it returns within minutes

Investigate →

CRITICAL

Heap exhaustion and the GC death spiral

The post-GC heap baseline climbs toward -Xmx, Full GCs become frequent and long, throughput oscillates between zero and brief bursts, and CPU runs high on the GC threads — the opposite signature of thread starvation. It ends in java.lang.OutOfMemoryError: Java heap space (or GC overhead limit exceeded just before). Capture a heap dump before the recovery restart; fix the retainer offline.

java.lang.OutOfMemoryError: Java heap space in catalina.out
Rising post-GC heap valley over hours or days, not the sawtooth peak
Frequent or lengthening Full GC pauses with CPU high
Throughput collapsing to near zero between GC bursts

Investigate →

CRITICAL

Connection refused at the door

The NIO poller reaches maxConnections (default 8192) and the OS accept queue (acceptCount, still 100) fills, so the kernel RSTs new connections. There is no JMX counter and Tomcat logs nothing — clients just see connection refused. This is the last buffer before users are locked out entirely, and it is invisible unless you watch it at the OS level.

Client-side connection refused while the process is still up
connectionCount at or near maxConnections (8192 NIO)
ss -tnl showing Recv-Q at the listen socket's backlog limit
No corresponding Tomcat log line at all

Investigate →

ACTIVE

Stuck threads on a hung backend

A worker blocks past a threshold on a downstream call with no timeout — a database, an HTTP API, DNS, or NFS — and the StuckThreadDetectionValve logs that a thread has been active too long and may be stuck. Each stuck thread is one fewer worker; enough of them and the pool exhausts. The valve is not enabled by default and emits a log line, not a first-class gauge, so it is easy to miss until threads run out.

'has been active for [N] milliseconds ... and may be stuck' in the logs
currentThreadsBusy creeping up with no matching rise in traffic
jstack showing many threads parked in the same backend call
Latency bimodal: most requests fast, a tail hanging for seconds

Investigate →

IMMINENT

The Metaspace redeploy leak

Each hot redeploy should let the old WebappClassLoader be collected. A ThreadLocal, an un-deregistered JDBC driver, an uncancelled timer, or a logging appender pins it, so every deploy accumulates another full copy of the app's classes. Metaspace rises step by step; with MaxMetaspaceSize unset the OS OOM-kills the process with no Java error at all.

java.lang.OutOfMemoryError: Metaspace, or a silent OS OOM-kill in dmesg
Metaspace and LoadedClassCount stepping up with each redeploy
'appears to have started a thread ... but has failed to stop it' warnings
NoClassDefFoundError appearing after a redeploy

Investigate →

IMMINENT

File descriptor exhaustion

Every socket, JAR handle, and log file is a file descriptor, and NIO selectors add more. The default ulimit of 1024 or 4096 is far too low; at the limit Tomcat can neither accept a connection nor open or rotate a log — a hard cliff that is routinely misread as a disk or network fault. FD growth without matching connection growth is a leak building toward that cliff.

java.net.SocketException: Too many open files
OpenFileDescriptorCount approaching MaxFileDescriptorCount
FD count growing without matching connection growth (a leak)
New connections refused while the process stays up

Investigate →

Choosing a tool

Best Apache Tomcat Monitoring Tools in 2026

A ranked review of the tools teams actually shortlist here, what each one is genuinely good at, and how the pricing behaves as you scale.

Read the buyer's guide →

Tomcat monitoring maturity levels

Tomcat observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your instance matters. Most production instances should land at the second level.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor. With these signals you can answer one question: is Tomcat alive and is it serving requests? You will not learn what broke, but you will learn that something broke before users do. Survival is enough for dev instances and non-critical apps.

JVM process alive Is the Bootstrap/app JVM actually running, and not stuck in D state?
Connector serves a real request An application endpoint returns 200 — not just a TCP connect succeeding.
Web application context STARTED A FAILED context returns 404 while Tomcat looks healthy.
OutOfMemoryError in the log Any OOM string in catalina.out is a page, day one.
5xx error rate Server-side failures, isolated from 4xx crawler noise.

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production instances should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident on its own: thread starvation, GC pressure, backlogs, resource limits, error spikes.

Thread pool utilisation currentThreadsBusy / maxThreads — the headline gauge; keep peak under 0.6.
Post-GC heap baseline The valley after GC, not the sawtooth peak.
GC time ratio Cumulative GC time / wall clock; over 10% is trouble.
Request throughput requestCount rate as a reset-aware delta against a time-of-day baseline.
Average request processing time processingTime delta divided by requestCount delta.
4xx vs 5xx split errorCount lumps them together; separate to alert safely.
File descriptor usage ratio Cliff-edge; alert past 0.8 of the ulimit.
Active session count Monotonic growth without a plateau is a leak.
JDBC pool numActive / maxActive The second bounded resource behind the thread pool.

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone up. Threads slowly sticking on a degrading backend, Metaspace stepping up across deploys, RSS drifting away from heap, the accept queue filling during spikes. None of these page you on day one. They become page-out incidents on day thirty.

Stuck thread detection StuckThreadDetectionValve warnings, ideally per endpoint.
Metaspace usage Tracked separately from heap; steps up per redeploy.
Process RSS vs heap RSS climbing while heap is flat means native growth.
connectionCount vs maxConnections Poller saturation, distinct from worker saturation.
Accept queue depth (Recv-Q) OS-only signal; a sustained queue means Tomcat cannot accept fast enough.
Per-endpoint p95/p99 latency From the access log with %D; the average hides the tail.
LoadedClassCount across deploys Rising across redeploys confirms a classloader leak.
JDBC pool waitCount Threads blocking in getConnection() — a double cliff.

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. Native memory growth, GC allocation pressure, container CPU throttling stretching pauses, a certificate about to expire, virtual-thread pools that no longer report the gauge every dashboard relies on. Most teams never need every signal here. Add the ones your incident history says you do.

Native memory tracking (NMT) jcmd VM.native_memory for off-heap growth the heap metrics never show.
Allocation and promotion rate What drives GC frequency before the baseline moves.
G1 region occupancy Humongous allocations and region exhaustion.
cgroup CPU throttling Container throttling stretches GC pauses invisibly.
Scheduled thread dumps Periodic jstack to catch transient stuck states.
Keepalive / connectionTimeout alignment Coordinated with the reverse proxy's keepalive to avoid resets.
TLS certificate expiry A silent, total outage; an external check, not a broker metric.
Virtual-thread connection signals On JDK 21+ currentThreadsBusy reports -1; watch connections instead.

Operating mistakes worth avoiding

The traps Tomcat teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

Alerting on the heap sawtooth instead of the post-GC baseline

Instantaneous heap is meant to climb until GC runs, so a 'heap > 80%' alert fires constantly on a perfectly healthy JVM. Only the post-GC valley matters — a baseline that rises over days is the real leak. Correlate heap with GC events or read <code>jstat -gcutil</code>; the Manager status XML gives free/total snapshots only.

Trusting a TCP-connect health check

A successful TCP connect only proves the OS accepted the socket into the queue — it says nothing about whether a worker thread will ever serve it. A thread-exhausted or GC-spiralling Tomcat accepts connections and answers none. Health checks must exercise the servlet pipeline through a real application endpoint, not just probe the port.

Leaving MaxMetaspaceSize unset

Without <code>-XX:MaxMetaspaceSize</code>, Metaspace grows until the OS OOM-killer takes the process — no <code>OutOfMemoryError</code>, no Tomcat log, just a dead JVM you find in <code>dmesg</code>. Setting a bound converts a silent kill into a catchable <code>OutOfMemoryError: Metaspace</code> you can alert on, and Metaspace must be watched separately because heap metrics never cover it.

Reading average latency instead of percentiles

<code>processingTime / requestCount</code> is a mean: a p50 of 100ms with a p99 of 15s still averages to a 'healthy' 200ms while one request in a hundred waits fifteen seconds. Stuck threads averaged against fast requests vanish entirely. Percentiles need the access log with <code>%D</code> or <code>%T</code> — neither is in the default or 'combined' pattern.

Leaving outbound calls without timeouts

The default socket timeouts in many HTTP clients, <code>HttpURLConnection</code>, and JDBC drivers are infinite, so a single hung downstream pins a worker thread forever and pool exhaustion becomes inevitable. Thread-pool exhaustion is almost always a backend problem, not a Tomcat one. Set connect and read timeouts on every outbound call, and add the StuckThreadDetectionValve to catch the ones you missed.

Running the default file-descriptor ulimit

The default ulimit of 1024 (or 4096) is exhausted quickly — every socket, JAR, log file, and NIO selector is an FD — and at the limit Tomcat can neither accept a connection nor rotate a log, which is routinely misdiagnosed as a disk or network fault. Production wants 65535, raised in both the OS limits and the systemd unit, with an alert past 80% usage.

Alerting on errorCount as if it were 5xx

The JMX <code>errorCount</code> lumps every status at or above 400 together, so 404 crawler noise inflates it and it can never safely page. A 503 from Tomcat means thread or connector saturation; a 500 is an application exception; a flood of 401/403 is auth probing. Splitting 5xx from 4xx requires parsing the access log by status code.

Leaving the Manager app, AJP, and shutdown port at defaults

The Manager app can deploy a WAR — remote code execution — so default credentials in <code>tomcat-users.xml</code> are the most exploited Tomcat vector; the AJP connector was unauthenticated before 9.0.31 (Ghostcat, <code>CVE-2020-1938</code>); and anyone who reaches port 8005 can send <code>SHUTDOWN</code> and stop the server. Restrict them with <code>RemoteAddrValve</code>, bind to localhost, or remove them entirely in production.

Tomcat runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

Process, connector, and availability

▸

Thread pool and request capacity

▸

Heap, GC, and OutOfMemoryError

▸

Metaspace, classloaders, and redeploy leaks

▸

Connectors, connections, and the accept queue

▸

Latency, throughput, and error rates

▸

Stuck threads, backends, and JDBC pools

▸

HTTP sessions and session memory

▸

File descriptors, native memory, and OS limits

▸

Deployment, contexts, and app lifecycle

▸

Manager app, AJP, TLS, and ports

WHERE TO GO NEXT

Setting up Tomcat monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.

> Start with the checklist > Back to Operations Guides