Tomcat 5xx error rate: separating server failures from crawler 404s

Your Tomcat error rate alert fired. The dashboard shows a spike in errorCount. The JVM is healthy, the thread pool has headroom, heap is fine. The access log tells a different story: 404s from a crawler storm, not 500s from a failing app. The page was a false alarm.

This is the Tomcat error-rate trap. The JMX errorCount attribute exposed via Catalina:type=GlobalRequestProcessor,name="http-nio-8080" counts every response with status >= 400. It does not distinguish 4xx from 5xx. A crawler hitting dead URLs inflates it identically to an application throwing unhandled exceptions. If you alert on this counter as a ratio of total requests, crawler noise will page you.

The only reliable way to build a 5xx-only error rate is to parse the access log by status code.

What this means

Tomcat increments errorCount in RequestInfo.updateCounters() whenever the response status is >= 400. The source comment reads “// number of response codes >= 400”. This behavior has been stable since at least Tomcat 7 and is identical across 8.5, 9.x, and 10.1.x. Micrometer’s TomcatMetrics binder exposes the same lumped counter as tomcat.global.error, and a per-servlet variant as tomcat.servlet.error, both read from the same JMX errorCount attributes (global request processor and per-servlet request info).

Because the threshold is >= 400, the counter includes:

• 400 Bad Request from malformed clients
• 401 Unauthorized and 403 Forbidden from auth probes, WAF blocks, or credential stuffing
• 404 Not Found from crawlers, broken links, and scanners
• 405 Method Not Allowed from misconfigured clients
• 500 Internal Server Error from application exceptions
• 503 Service Unavailable from thread pool or connector saturation

Any other status >= 400 returned by application-level gateway or proxy code is counted as well. Standard Tomcat rarely generates 502 or 504 from its own HTTP connector; those codes typically come from a reverse proxy in front of Tomcat and would not appear in Tomcat’s access log.

For paging, the only statuses that indicate server-side failure are 5xx. Elevated 4xx rates mean broken clients, changed API contracts, or probing, not a broken server.

SSL handshake failures are not counted in errorCount at all. They never create a request object, so they never reach the counter. If your symptom is handshake failures, both the access log and errorCount will miss it. Check the connector’s TLS error logging or the reverse proxy instead.

Common causes

Quick checks

All commands are read-only.

# Check overall error count from JMX (cumulative, mixes 4xx and 5xx)
java -jar jmxterm.jar -l localhost:9090 -n -v silent -e \
  "get -b Catalina:type=GlobalRequestProcessor,name=\"http-nio-8080\" errorCount requestCount"

# Status code breakdown from access log
awk '{print $9}' /var/log/tomcat/localhost_access_log.$(date +%Y-%m-%d).txt | \
  sort | uniq -c | sort -rn

# 5xx count specifically
awk '$9 ~ /^5/' /var/log/tomcat/localhost_access_log.$(date +%Y-%m-%d).txt | wc -l

# 5xx as a percentage of total requests in the current log
total=$(wc -l < /var/log/tomcat/localhost_access_log.$(date +%Y-%m-%d).txt)
fivexx=$(awk '$9 ~ /^5/' /var/log/tomcat/localhost_access_log.$(date +%Y-%m-%d).txt | wc -l)
echo "scale=2; $fivexx * 100 / $total" | bc

# Top URIs producing 500s
awk '$9 == "500"' /var/log/tomcat/localhost_access_log.$(date +%Y-%m-%d).txt | \
  awk '{print $7}' | sort | uniq -c | sort -rn | head

# Thread pool state (rules out 503 from saturation)
java -jar jmxterm.jar -l localhost:9090 -n -v silent -e \
  "get -b Catalina:type=ThreadPool,name=\"http-nio-8080\" currentThreadsBusy maxThreads"

# Recent exceptions in catalina.out (heuristic: catches common framework packages)
grep -cE "^(java\.|javax\.|org\.).*Exception" /var/log/tomcat/catalina.out

# Application context state
curl -s -u $USER:$PASS 'http://localhost:8080/manager/text/list'

The $9 field index assumes the default common pattern %h %l %u %t "%r" %s %b. The time field [...] contains a space (the timezone offset), and the quoted request line splits into three fields, so the status code lands at $9. If your pattern differs, adjust the field index accordingly. Log paths vary by distribution; adjust /var/log/tomcat/ to match your install.

How to diagnose it

Separate three questions: is the error rate real (5xx), is it Tomcat’s fault (503) or the application’s (500), and is the JMX counter lying to you (crawler 404s).

1. Compute the 5xx-only rate from the access log, not from errorCount. Use the awk commands above. If the 5xx rate is near zero but errorCount spiked, the spike is 4xx noise and should not page.

2. If 5xx rate is elevated, classify by exact status. A 503 means Tomcat itself refused to process the request. A 500 means the application threw an exception. A 502 or 504 in Tomcat’s access log is unusual for the HTTP connector and typically points to application-level gateway code, not a standard Tomcat response.

3. For 503s, check thread pool saturation. Pull currentThreadsBusy and maxThreads from the ThreadPool MBean. If currentThreadsBusy == maxThreads sustained, the 503s are from pool exhaustion. See the thread pool exhaustion guide for the full cascade.

4. For 500s, correlate with application logs. The access log tells you which URI and how often. catalina.out or the application’s own log tells you the exception. A 500 that appears only on one endpoint after a deploy is a deployment bug, not a capacity problem.

5. For mass 401/403, check source IPs. Concentrated traffic from one IP is a scanner or credential stuffing attempt. Distributed 401/403 after a config change is a WAF or auth filter misconfiguration. Neither is a server failure and neither should page.

6. Filter health-check endpoints from the ratio before alerting. A load balancer hitting /health every few seconds can dominate request volume. If that endpoint ever returns non-200, it will swing the error rate without representing user impact.

flowchart TD
    A[errorCount spike] --> B{5xx rate elevated?}
    B -- No --> C[4xx noise: crawlers, probes]
    C --> C1[Do not page. Tune alert or filter.]
    B -- Yes --> D{Which 5xx?}
    D -- 503 --> E[Thread pool saturation]
    D -- 500 --> F[Application exception]
    D -- 502/504 --> G[Application gateway or proxy]
    E --> E1[Check currentThreadsBusy vs maxThreads]
    F --> F1[Correlate with catalina.out stack traces]
    G --> G1[Check application proxy logic or reverse proxy]

Metrics and signals to monitor

Fixes

If the alert is firing on 4xx noise

Stop alerting on errorCount as a ratio. The counter is structurally incapable of separating client errors from server failures. Replace the page with an access-log-derived 5xx-only rate. Keep the errorCount ratio as a trend-only signal or a low-urgency ticket if you want visibility into 4xx probe activity.

If crawler 404s are the dominant noise source, filter known crawler user agents out of the ratio, or rate-limit aggressive crawlers at the reverse proxy. Do not suppress 404 responses in the application. A 404 is a correct response for a missing resource.

If 503s are real (thread pool saturation)

The fix is not to lower the alert threshold. The fix is to address the thread exhaustion. Check for slow backends, missing timeouts on outbound calls, or a maxThreads value too small for the workload. A thread dump (jstack <pid>) shows exactly what the busy threads are waiting on. See the 503 guide and the connection refused guide for the full diagnosis path.

If 500s are real (application exceptions)

The access log tells you which URI is failing and how often. The application log tells you why. A 500 spike after a deploy is almost always a deployment bug: a missing dependency, a bad config value, or an incompatible library version. Roll back or fix forward. A 500 spike without a deploy is a backend or data issue: a slow query timing out, a downstream service returning unexpected data, or a resource that disappeared.

If health checks are inflating the ratio

Use the AccessLogValve’s conditionUnless attribute to suppress logging for health-check requests. Set a request attribute in a filter or valve for health endpoints, then configure the valve to skip logging when that attribute is present. This keeps health checks out of both the access log and any error-rate computation derived from it.

<!-- In server.xml or a Valve configuration -->
<Valve className="org.apache.catalina.valves.AccessLogValve"
       conditionUnless="excludeFromAccessLog" ... />

Then in a filter for /health and similar endpoints:

request.setAttribute("excludeFromAccessLog", "true");

This works for any endpoint you want to exclude from access-log-derived metrics, not just health checks.

Prevention

• Alert on access-log 5xx rate, not errorCount. This is the single most important change. The JMX counter is fine for trending, unsafe for paging.
• Filter health-check endpoints from the ratio. Use conditionUnless on the AccessLogValve or filter at the log-parsing layer.
• Include %D in your access log pattern. The default common pattern omits processing time. Without %D (request time in milliseconds), you cannot correlate 5xx spikes with latency spikes from the same log.
• Correlate 5xx spikes with deploy events. A 5xx burst minutes after a deploy is a deployment bug until proven otherwise. Tag your deploys in your monitoring timeline.
• Track 4xx separately, as a ticket or trend. Rising 404s from crawlers are noise for paging but useful for capacity planning and for detecting broken links or API contract changes.
• Verify the MBean name matches your connector. http-nio-8080 is the common case. Embedded Tomcat in Spring Boot may expose beans under a different domain or name. If your JMX query returns nothing, confirm the object name with jmxterm beans first.

How Netdata helps

• The Tomcat collector surfaces errorCount and requestCount alongside per-second thread pool, heap, and GC metrics. An errorCount spike with flat currentThreadsBusy and flat GC pause time is almost certainly 4xx noise, visible within seconds of onset.
• Per-second resolution distinguishes a sustained 5xx failure from a sub-minute burst that minute-bucketed tools would smear into a false trend.
• Pair the collector with log parsing so the dashboard shows both the coarse error trend and the precise per-status breakdown from the access log.

Related guides

• Tomcat accept queue overflow: acceptCount, somaxconn, and Recv-Q
• Tomcat java.net.BindException: Address already in use: the connector never starts
• Tomcat average latency lies: why you need p95/p99 from the access log
• Tomcat classloader leak on redeploy: why the old WebappClassLoader never dies
• Tomcat connection refused: maxConnections and acceptCount both exhausted
• Tomcat accepts connections but never responds: the TCP-connect trap
• Tomcat frequent Full GC: pause time, G1, and the 5% overhead rule
• Tomcat GC death spiral: full GCs dominating and throughput collapsing
• Tomcat heap dump before restart: capturing evidence with jmap and jstack
• Tomcat heap usage: watch the post-GC baseline, not the sawtooth peak
• How Tomcat actually works in production: a mental model for operators
• Tomcat HTTP Status 503 Service Unavailable: the connector is out of threads