Tomcat monitoring maturity model: from survival to expert

Most Tomcat monitoring setups start with a process check, a health endpoint probe, and maybe a 5xx alert. That catches crashes. It does not catch the failures that actually page teams: thread pool exhaustion, GC death spirals, accept queue overflow, classloader leaks after hot redeploys. This article maps the signals that matter across four maturity levels so you can audit what you have, identify what you are missing, and prioritize what to add next.

The model is cumulative. Each level includes everything below it. Signals are drawn from the Tomcat-specific failure archetypes operators encounter in production, not from generic JVM advice.

flowchart TD
  L1["Level 1: Survival
process, connector, context, OOM, 5xx"]
  L2["Level 2: Operational
threads, heap, GC, throughput, latency"]
  L3["Level 3: Mature
stuck threads, Metaspace, RSS, accept queue"]
  L4["Level 4: Expert
NMT, allocation rate, G1 regions, cgroup"]
  L1 -->|adds| L2
  L2 -->|adds| L3
  L3 -->|adds| L4

How to read these levels

Each level assumes the previous level’s signals are in place. The tables list the signal, the failure mode it catches, and the source. The note at the end of each level explains the gap that motivates the next level.

Deployment variants matter. Standalone Tomcat and embedded Tomcat (Spring Boot) expose the same JMX MBeans, but the Manager application is absent in embedded deployments, and application properties replace server.xml. Clustered Tomcat with session replication adds its own monitoring surface (DeltaManager and BackupManager MBeans, cluster heartbeat) that is outside this model.

Virtual threads on JDK 21+ with Tomcat change the thread pool model fundamentally. When virtual threads are enabled, currentThreadsBusy reports -1 and maxThreads is not meaningful. Connection-based monitoring (connectionCount minus keepalive estimate) becomes the proxy for active work, and the standard thread pool utilization thresholds at Level 2 no longer apply.

Level 1: survival

The absolute minimum to know Tomcat is alive and serving.

A TCP connect success does not mean Tomcat is healthy. The OS accept queue holds connections even when Tomcat has no threads to process them. Your health check must complete an HTTP request through the servlet pipeline, not just open a socket.

A context in FAILED state at startup may not produce HTTP errors. Requests to that context return 404, which can be confused with a missing route. Tomcat does not restart failed applications by default. They remain in FAILED state until manually redeployed or the JVM restarts.

What this misses: thread pool saturation (process up, port open, requests hang or 503), GC death spirals (process alive, latency climbing to seconds), Metaspace exhaustion from classloader leaks, and accept queue overflow (clients get connection refused, Tomcat logs nothing).

Level 2: operational

What a competent production team monitors. These signals catch the top Tomcat failure archetype: thread pool exhaustion.

The thread pool signal deserves emphasis. currentThreadsBusy / maxThreads is the single most important Tomcat-specific metric. When it reaches 1.0, requests are queuing. With NIO, connections are still accepted up to maxConnections (default 10000 for NIO, 8192 for APR/native), but no thread is available to process them. Users experience timeouts while the JVM sits at low CPU, looking healthy.

Alerting on post-GC heap means tracking the valley of the sawtooth, not the peak. High instantaneous heap usage before GC is normal. A rising post-GC baseline over hours or days indicates a memory leak. Alerting on raw “heap above 80 percent” produces constant false positives because the value spends most of its time there by design.

The default access log pattern (%h %l %u %t "%r" %s %b) omits processing time. Without adding %D (milliseconds in Tomcat 9.x and 10.x), you cannot compute per-request latency percentiles. JMX processingTime is a cumulative total, not an average. Dividing by requestCount yields an arithmetic mean that hides tail latency: a few 30-second requests averaged with a thousand 10-millisecond requests looks fine.

What this misses: threads stuck indefinitely (no StuckThreadDetectionValve configured), Metaspace growth from classloader leaks, the OS accept queue depth, per-endpoint latency breakdowns, and the full process RSS (heap metrics exclude non-heap memory).

Level 3: mature

Full coverage for a production-grade deployment. These signals close the gaps that cause the most confusing incidents: the ones where Tomcat looks healthy but users are failing.

Stuck thread detection requires explicit configuration. The StuckThreadDetectionValve is not enabled by default. Add it to server.xml or context.xml with a threshold. The default of 600 seconds is too high for most production applications. Without it, stuck threads are indistinguishable from normal busy threads until the pool drains.

Metaspace is the silent killer. If MaxMetaspaceSize is not set, Metaspace grows toward the JVM’s computed ceiling. In containers with tight memory limits, the cgroup OOM killer can terminate the process before the JVM reaches its own limit, leaving no OutOfMemoryError in the log. When the JVM does hit its Metaspace limit, it throws OutOfMemoryError: Metaspace. Teams that watch heap but ignore Metaspace are blindsided by this after N hot redeploys. In containerized environments with immutable deploys, this is rarely an issue. In environments that hot-deploy, it is inevitable.

The accept queue is invisible to JMX. When maxConnections is reached and acceptCount (default 100) fills, the kernel rejects TCP connections with RST. Tomcat logs nothing. The only detection is OS-level ss -tnl monitoring showing non-zero Recv-Q, or client-side connection error rates.

Two security signals belong at this level. Manager application access from unexpected sources warrants alerting, since the Manager allows WAR upload and remote code execution. AJP connector posture matters too: the AJP connector should bind to localhost with secretRequired=true. An exposed AJP port on 0.0.0.0 without authentication is the Ghostcat (CVE-2020-1938) attack surface, which allows unauthenticated file read and potential RCE.

What this misses: the native memory breakdown behind RSS growth, allocation and promotion rates that predict future heap pressure, and container-level CPU throttling that extends GC pauses.

Level 4: expert

Signals that teams add after their third or fourth major Tomcat incident. These require deeper JVM expertise and higher instrumentation overhead, but they catch failures hours or days before they become outages.

Native memory tracking explains the gap between heap usage and RSS. When heap looks stable but RSS keeps climbing, the growth is in thread stacks, native buffers, direct memory, or JNI allocations. NMT breaks this down by category. The tradeoff: enabling NMT adds overhead. Reserve it for diagnosis or run it permanently only on instances where you have CPU headroom.

Allocation rate (bytes per second into young generation) is a better predictor of GC pressure than absolute heap usage. A sudden increase means the heap fills faster, GC runs more frequently, and pauses lengthen. Promotion rate (objects surviving to old gen) predicts old gen pressure specifically. Neither is exposed via standard JMX. They require GC log analysis or JDK Flight Recorder.

Cgroup CPU throttling is the hidden latency source in containerized Tomcat. CFS quota throttling can deschedule GC threads mid-collection, extending a 50-millisecond pause to seconds. The nr_throttled counter in the cgroup CPU controller reveals this. Java 17+ has proper cgroup v2 support; older Java versions may misread container limits.

Scheduled thread dumps, not just incident-driven ones, reveal contention patterns that build slowly. A thread dump taken every few minutes during peak load, compared across days, shows whether threads are converging on the same lock or backend call before the pool drains.

What most teams get wrong regardless of level

A few gaps persist across all maturity levels because they are instrumentation or configuration failures, not missing metrics:

• Monitoring the PID, not the thread pool. JVM alive does not mean Tomcat is serving. Thread pool exhaustion leaves the process up, the port open, and requests hanging or returning 503.
• Alerting on instantaneous heap. The sawtooth is normal GC behavior. Track the post-GC baseline. Alerting on raw utilization above 80 percent produces constant false positives.
• Average latency instead of percentiles. JMX processingTime / requestCount is an arithmetic mean. Configure %D in the access log and compute p95 and p99 to see actual user experience.
• No timeouts on outbound connections. Default socket timeouts in java.net.HttpURLConnection, Apache HttpClient, and most JDBC drivers are infinite. One hanging backend consumes threads forever. This is a code problem, but without StuckThreadDetectionValve, monitoring cannot surface it.
• Confusing thread exhaustion with connection exhaustion. Thread pool full does not mean connections are refused. With NIO, connections continue to be accepted up to maxConnections. Connections are only refused when both maxConnections and acceptCount are exceeded.

How Netdata helps

Netdata collects JVM and Tomcat metrics at per-second resolution and correlates them with OS-level signals, which is where the gap between symptom and root cause usually closes:

• Thread pool utilization (currentThreadsBusy / maxThreads) collected per-second and correlated with request processing time and throughput distinguishes backend-driven saturation from load-driven saturation within seconds.
• JVM heap, GC pause time, and GC frequency tracked continuously expose the post-GC baseline trend without manual jstat polling or GC log parsing.
• Metaspace and class loading counters captured across redeploy events surface classloader leaks before they become silent OOM kills.
• OS-level file descriptor count, CPU, and RSS alongside JVM metrics give the full memory picture (heap, non-heap, native) in a single correlated view.
• Anomaly detection on connector error rates, session counts, and connection counts flags deviations from baseline before static thresholds trip.

Related guides

• Tomcat monitoring checklist: the signals every production instance needs
• How Tomcat actually works in production: a mental model for operators
• Tomcat process not running: crashes, OOM-kills, and failed restarts
• Tomcat accepts connections but never responds: the TCP-connect trap
• Tomcat java.net.BindException: Address already in use: the connector never starts
• Tomcat thread pool exhaustion: currentThreadsBusy at maxThreads and requests hanging
• Tomcat HTTP Status 503 Service Unavailable: the connector is out of threads
• Tomcat maxThreads and minSpareThreads: sizing the executor correctly
• Tomcat threads busy but CPU idle: telling a blocked backend from a GC spiral
• Tomcat virtual threads (JDK 21+): why currentThreadsBusy reports -1
• Tomcat java.lang.OutOfMemoryError: Java heap space: the heap is genuinely full
• Tomcat OutOfMemoryError: GC overhead limit exceeded: GC running but freeing nothing