Tomcat request throughput dropping: requestCount rate below baseline

The Tomcat requestCount counter on GlobalRequestProcessor is climbing slower than expected. The drop tells you fewer requests are being processed, not why. The cause can be upstream of Tomcat (load balancer routed traffic away, health check started failing, CDN absorbing load) or inside Tomcat (worker thread pool exhausted, JVM in a GC death spiral, deadlock consuming the pool).

The first diagnostic fork: is traffic still arriving at this host? If the answer is no, you have a routing, load balancer, or health check problem, and chasing thread dumps wastes time. If yes, Tomcat is failing to keep up, and the question becomes whether the failure is I/O bound (threads blocked on a slow backend) or CPU bound (GC thrashing or a compute-bound code path).

This article assumes you collect requestCount as a per-second or per-minute rate, not a raw cumulative value. A cumulative counter plotted as-is is useless for alerting and triggers false positives after every restart. The rate computation must be reset-aware: the counter drops to zero on JVM restart, so a naive delta produces a huge negative spike that a flat alert threshold interprets as a recovery.

What this means

requestCount is a cumulative long counter exposed by the Catalina:type=GlobalRequestProcessor,name="<connector>" MBean. It increments once per request the connector processes, including health checks and static asset requests. It resets to zero when the JVM restarts. To monitor throughput, compute a delta over a fixed window and divide by the window length to get a rate.

A rate below baseline has two interpretations:

1. Fewer requests are arriving. The drop is real demand reduction. The cause is upstream of Tomcat: the load balancer removed the instance from rotation, a routing change redirected traffic, a CDN is absorbing load, or scheduled maintenance shifted the diurnal pattern.
2. Requests are arriving but Tomcat is processing fewer of them. Tomcat has a processing constraint. The four classic causes are worker thread pool exhaustion, GC death spiral, a backend dependency failing or hanging, or an application-level deadlock.

The single most important diagnostic question is which of these two you are in. Correlating the requestCount rate with currentThreadsBusy and JVM CPU answers it in seconds. If upstream traffic metrics show requests still arriving but currentThreadsBusy is parked at maxThreads, you are in case two, and the bottleneck is almost always a backend.

flowchart TD
    A["requestCount rate below baseline"] --> B{"Upstream traffic still arriving at host?"}
    B -- "No" --> C["LB / DNS / health check routing"]
    B -- "Yes" --> D{"currentThreadsBusy at maxThreads?"}
    D -- "No" --> E{"GC pause time elevated?"}
    E -- "Yes" --> F["GC death spiral"]
    E -- "No" --> G["Check error rate, app context state"]
    D -- "Yes" --> H{"JVM CPU high?"}
    H -- "Yes" --> I["CPU-bound or GC thrashing"]
    H -- "No" --> J["I/O blocked - backend or deadlock"]
    J --> K["Thread dump identifies the blocker"]

Common causes

Quick checks

Run these read-only. None change Tomcat state.

# Snapshot request count, thread pool, errors, and processing time from Manager XML
curl -s -u "$USER:$PASS" 'http://localhost:8080/manager/status?XML=true' | \
  grep -oP '(requestCount|currentThreadsBusy|maxThreads|currentThreadCount|errorCount|processingTime)="[0-9]+"'

# Sample twice, 10 seconds apart, to compute a live rate:
# (requestCount_now - requestCount_10s_ago) / 10 = requests per second

# Thread pool ratio via JMX
# jmxterm has no -e execute flag (-e is exit-on-failure); pipe the command on stdin.
echo "get -b Catalina:type=ThreadPool,name=\"http-nio-8080\" currentThreadsBusy maxThreads" | \
  java -jar jmxterm.jar -l localhost:9090 -n -v silent

# CPU snapshot: low CPU with threads at max = I/O blocked; high CPU = GC or compute bound
ps -p "$(pgrep -f 'catalina.startup.Bootstrap')" -o %cpu,%mem,etime

# GC activity at 1-second cadence
# Note: jstat remains a supported JDK tool; -gcutil works on JDK 8 through current releases.
jstat -gcutil "$(pgrep -f 'catalina.startup.Bootstrap')" 1000

# Accept queue depth on the HTTP port. Non-zero Recv-Q means connections are backing up
ss -tnl 'sport = :8080'

# 5xx rate from the access log over the current day
awk '$9 ~ /^5/' /var/log/tomcat/localhost_access_log.$(date +%Y-%m-%d).txt | wc -l

If currentThreadsBusy == maxThreads and JVM CPU is low, take a thread dump immediately. It identifies which backend the threads are stuck on.

# Capture a thread dump (safe and read-only, but large)
jstack "$(pgrep -f 'catalina.startup.Bootstrap')" > /tmp/tomcat-threads-$(date +%s).txt
# Find the dominant stack frame across worker threads
grep -A 3 "http-nio-8080-exec" /tmp/tomcat-threads-*.txt | grep "at " | sort | uniq -c | sort -rn | head

How to diagnose it

1. Confirm the drop is real, not a counter reset. Compare JVM uptime against the time the rate dropped. If the JVM restarted, requestCount jumped to zero and a hand-rolled delta will misfire. A Prometheus-style rate() handles resets correctly; a shell delta does not.
2. Verify traffic is still arriving upstream. Compare against the load balancer request rate, upstream proxy logs, or a client-side metric. If upstream traffic is also down, this is a routing or demand problem, not a Tomcat problem. Stop here.
3. Check currentThreadsBusy / maxThreads. If the ratio is near 1.0 and sustained, the worker pool is the bottleneck. The throughput drop is a consequence of threads not being freed, not of fewer requests arriving.
4. Look at JVM CPU. Low CPU with a saturated thread pool points to I/O blocking (database, downstream HTTP, DNS, file). High CPU points to GC thrashing or a compute-bound code path.
5. Check GC activity. Run jstat -gcutil <pid> 1000 for a few seconds. If Full GC count (FGC) is climbing and Full GC time (FGCT) dominates wall clock, you are in a GC death spiral. Throughput oscillates between zero during pauses and brief bursts between cycles, which produces a misleading average.
6. Take a thread dump. When threads are saturated, the thread dump is the diagnostic. Look for the dominant stack trace pattern across http-nio-8080-exec threads. If many share the same blocked frame, that frame is the backend or lock consuming your pool.
7. Check errorCount rate. A rising error rate alongside the throughput drop points to backend failures. errorCount increments on any status >= 400, so it mixes 4xx and 5xx; use the access log if you need to separate them.
8. Check the accept queue. ss -tnl 'sport = :8080'. A non-zero Recv-Q means connections are backing up in the OS backlog because Tomcat cannot accept them fast enough. This signal is invisible to JMX.

Metrics and signals to monitor

Fixes

Slow backend or backend failure

The dominant production cause. Threads block on a database query, a downstream HTTP call, or DNS with no timeout configured. The fix is upstream of Tomcat: restore the backend, configure socket and query timeouts on every outbound path, or shed load.

• Do not restart Tomcat first. If the backend is still degraded, threads re-accumulate immediately after restart and you are back in the same state within minutes.
• Do take a thread dump before restarting. The stack trace identifies the offending backend. Once the backend is healthy, throughput recovers on its own.

Tradeoff: lowering outbound connectionTimeout fails requests faster but increases the error rate. A circuit breaker is usually a better tool than shorter timeouts for genuinely degraded backends.

GC death spiral

If GC is the cause, the JVM is already compromised. Capture a heap dump before restarting or the evidence is gone.

# Heap dump of all reachable and unreachable objects. Does not trigger a Full GC
# unless you append :live (which forces one). Large heaps make this slow and can
# pause the JVM. Run before restart.
jmap -dump:format=b,file=/tmp/heap-$(date +%s).hprof "$(pgrep -f 'catalina.startup.Bootstrap')"

Restart clears memory and restores throughput, but the leak recurs. Analyze the heap dump offline (Eclipse MAT) to find the dominant retainer. See Tomcat GC death spiral and Tomcat heap dump before restart.

Application deadlock

jstack prints “Found one Java-level deadlock” when one exists. A deadlock is the one case where restart is the only immediate fix. The longer-term fix is in application code (lock ordering, lock acquisition across paths). For related redeploy-driven failure modes, see Tomcat classloader leak on redeploy.

Load balancer or routing

If traffic stopped arriving at the host, fix the routing layer. Check the LB health check configuration against your health endpoint. A health check that exercises heavyweight dependencies (database, downstream calls) can fail under load and cause the LB to drain the instance, which from inside Tomcat looks exactly like a throughput problem.

Prevention

• Use time-of-day baselines, not flat rolling averages. Diurnal traffic makes flat averages produce false positives at off-peak hours and miss real drops during peak. Compare against the same hour and same day of week.
• Alert on currentThreadsBusy / maxThreads sustained above 0.8, not just on throughput. Thread pool saturation is the leading indicator; the throughput drop is the lagging symptom.
• Configure timeouts on every outbound connection. Default socket timeouts in java.net.HttpURLConnection, Apache HttpClient, and most JDBC drivers are effectively infinite. A hung backend will consume threads forever.
• Enable StuckThreadDetectionValve with a threshold appropriate to your latency profile. The default 600 seconds is too long for most user-facing services; 60 seconds is usually better. Its output is a log message, not a JMX metric, so wire log monitoring to it.
• Filter health checks out of throughput baselines if your LB probes the connector frequently. requestCount includes them, and a change in probe interval can look like a traffic change.
• Monitor the accept queue (ss Recv-Q). It is the only signal that catches connection refusal before clients report errors.

How Netdata helps

• The Tomcat collector exposes requestCount, currentThreadsBusy, maxThreads, processingTime, and errorCount at per-second resolution, so the throughput rate is computed from real deltas rather than sparse scrapes.
• Per-second collection makes counter resets on restart visible as a clean step, and the rate calculation handles them correctly without manual delta logic.
• ML-based anomaly detection on the requestCount rate adapts to diurnal patterns, which avoids the false positives that flat thresholds produce on real workloads.
• Correlating a throughput drop with thread pool saturation, GC pause time, and JVM CPU in a single view shortens the “is this an I/O problem or a GC problem” question to a few seconds.
• The same chart surface shows errorCount rising alongside throughput, which points to backend failure rather than a routing change.
• OS-level accept queue depth from the ss collector can be correlated with connector-level metrics, closing the gap between JMX-visible state and kernel-level connection refusal.

Related guides

• Tomcat accept queue overflow: acceptCount, somaxconn, and Recv-Q
• Tomcat java.net.BindException: Address already in use: the connector never starts
• Tomcat classloader leak on redeploy: why the old WebappClassLoader never dies
• Tomcat connection refused: maxConnections and acceptCount both exhausted
• Tomcat accepts connections but never responds: the TCP-connect trap
• Tomcat frequent Full GC: pause time, G1, and the 5% overhead rule
• Tomcat GC death spiral: full GCs dominating and throughput collapsing
• Tomcat heap dump before restart: capturing evidence with jmap and jstack
• Tomcat heap usage: watch the post-GC baseline, not the sawtooth peak
• How Tomcat actually works in production: a mental model for operators
• Tomcat HTTP Status 503 Service Unavailable: the connector is out of threads
• Tomcat process not running: crashes, OOM-kills, and failed restarts