Tomcat 401 flood on the Manager app: credential brute force and LockOutRealm

A 401 flood is a sustained spike of HTTP 401 responses against /manager, /host-manager, or any realm-secured endpoint. The signature is concentration: many 401s from a small set of source IPs, or many 401s aimed at a single username. Scattered 401s from mistyped passwords are noise; concentration and sustain are the signal.

Against the Manager app, a 401 flood is almost always credential brute force or stuffing and a precursor to compromise. A successful Manager login lets an attacker deploy WAR files (RCE), enumerate sessions, and undeploy applications. An internet-exposed Manager with weak or default credentials can be fully taken over within minutes of the flood beginning.

What this means

Tomcat returns 401 when a request to a realm-secured resource arrives without credentials, with invalid credentials, or with credentials for the wrong realm. A script-driven attacker replays username/password pairs and treats every 401 as “try the next pair”.

The diagnostic question is not “are there 401s?” but “is the 401 rate concentrated, sustained, and aimed at administrative surfaces?” A flood is one or more of:

• Many 401s per minute from one IP or a tight set of IPs.
• Many 401s targeting /manager/html, /manager/text, /manager/jmxproxy, /host-manager/html, or any custom realm-secured endpoint.
• Many 401s for the same username (per-user brute force) or many distinct usernames (distributed enumeration).

If LockOutRealm is configured, sustained 401s against one username should produce lockout events in catalina.out and a non-zero locked-user count on the realm MBean. A flood with no corresponding lockouts is itself a signal: either LockOutRealm is not wrapping the active realm, or you are hitting a known bypass.

flowchart TD
    A["401 rate spike in access log"] --> B{"Concentrated on /manager/* ?"}
    B -- yes --> C{"Few IPs, many requests each?"}
    B -- no --> D["Check app auth change or broken client"]
    C -- yes --> E["Per-user brute force"]
    C -- no --> F["Distributed credential stuffing"]
    E --> G{"LockOutRealm engaging?"}
    F --> H["IP-based rate limit at edge; LockOutRealm will not help"]
    G -- yes --> I["Contain IPs, rotate creds, audit"]
    G -- no --> J["Check case-sensitivity CVE or realm mis-config"]
    I --> K{"Any 200 on /manager/html from a flooding IP?"}
    K -- yes --> L["Treat as compromise"]
    K -- no --> M["Monitor and harden"]

Common causes

Quick checks

These are read-only. Adjust $CATALINA_BASE and the access log filename to match your install.

# Confirm Manager is reachable and what code it returns locally
# Expected: 401 (auth required) if Manager is deployed; 404 if not
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8080/manager/html

# Count 401s in today's access log
grep '" 401 ' "$CATALINA_BASE/logs/localhost_access_log.$(date +%Y-%m-%d).txt" | wc -l

# Group 401s by source IP (field 1 is %h in the default Common Log Format)
grep '" 401 ' "$CATALINA_BASE/logs/localhost_access_log.$(date +%Y-%m-%d).txt" \
  | awk '{print $1}' | sort | uniq -c | sort -rn | head -20

# Group 401s by requested path (extract path from the request line)
grep '" 401 ' "$CATALINA_BASE/logs/localhost_access_log.$(date +%Y-%m-%d).txt" \
  | awk -F'"' '{split($2, r, " "); print r[2]}' | sort | uniq -c | sort -rn | head -20

# Inspect User-Agents on 401s (requires the combined pattern; CLF has no User-Agent)
grep '" 401 ' "$CATALINA_BASE/logs/localhost_access_log.$(date +%Y-%m-%d).txt" \
  | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn | head -20

# Confirm LockOutRealm is configured and which realm it wraps
grep -A 6 'LockOutRealm' "$CATALINA_BASE/conf/server.xml"

# Look for lockout events
grep -iE 'lockout|locked' "$CATALINA_BASE/logs/catalina.out" | tail -20

# Check Manager credentials for default usernames
grep -iE 'username=.*(tomcat|admin|manager|role1|root)' "$CATALINA_BASE/conf/tomcat-users.xml"

# Confirm what interface the HTTP connector is bound to
ss -tnl 'sport = :8080'

The default Common Log Format pattern is %h %l %u %t "%r" %s %b. In that pattern, %u (remote user) logs as - on failed auth, so you cannot see attempted usernames from the access log alone. To capture usernames you need a custom valve or temporary authentication debug logging. The User-Agent awk index ($6) is correct only for the combined pattern (%h %l %u %t "%r" %s %b "%{Referer}i" "%{User-Agent}i"); in plain CLF there is no User-Agent field.

How to diagnose it

1. Establish flood versus baseline. Pull 24 hours of 401 counts per hour. A flood is a clear departure from baseline, not a slow drift.
2. Group by source IP. A small number of IPs producing most of the 401s is brute force. A flat distribution across many IPs, each sending few requests, is distributed credential stuffing.
3. Group by requested path. Concentration on /manager/html, /manager/text/deploy, or /host-manager/html is the highest-signal indicator. 401s on application endpoints may indicate a broken auth change rather than an attack.
4. Group by username if available. Distribution tells you whether you face per-user brute force (one username, many passwords) or username enumeration (many usernames, one password). The access log will not give you this by default; enable it via a custom valve or auth debug.
5. Correlate with lockouts. If LockOutRealm is configured, sustained per-user brute force should produce lockouts. Many 401s with no lockouts suggests a case-sensitivity bypass (see CVE-2026-43513 below) or that LockOutRealm is not actually wrapping the active realm.
6. Look for the transition from 401 to 200. The most important forensic signal is a previously flooding IP that suddenly gets a 200 on /manager/html. That is a successful compromise and must be treated as a security incident, not a tuning problem.
7. Check for follow-on Manager activity. After a successful auth, look for POST /manager/text/deploy, /manager/html/upload, or any session listing. Any of these is evidence of attacker action.

The LockOutRealm picture

LockOutRealm is a wrapper realm that extends CombinedRealm and imposes account lockout after repeated failed authentications. You enable it by nesting the real realm (UserDatabaseRealm, DataSourceRealm, JNDIRealm) inside a <Realm className="org.apache.catalina.realm.LockOutRealm"> element in server.xml.

Default attribute values:

The stock server.xml has shipped the default UserDatabaseRealm wrapped inside a LockOutRealm since Tomcat 7.0.1, and every release of the 8.5.x, 9.0.x, 10.1.x and 11.x lines has done the same since.

What LockOutRealm does well

• Per-user lockout after a threshold of failures. A script trying a thousand passwords for admin gets cut off at 5 (default), then again every 5 minutes.
• A short, deterministic lockout window (default 5 minutes) that slows automated attacks without permanently locking legitimate users.
• Failure cache eviction at cacheSize=1000. Under a distributed attack against many distinct usernames, older failure records evict, which can let a previously-failing username start fresh.

What LockOutRealm does not protect against

• Distributed brute force where each request uses a different username. LockOutRealm locks per-user, not per-source-IP. An attacker rotating usernames (admin, administrator, root, tomcat, guest) with one common password never hits any single user’s failure threshold. Access-log IP grouping is the only detection for this pattern.
• Legitimate-user lockout as a side effect. If an attacker floods a known-good username, the legitimate user cannot authenticate for lockOutTime after the threshold is hit. Tune lockOutTime, or implement an out-of-band unlock procedure, if operators depend on quick re-login.
• Case-sensitivity bypass (CVE-2026-43513, disclosed May 2026, severity Low). On versions where the lockout key was treated as case-sensitive, an attacker could cycle capitalizations (admin, Admin, ADMIN, aDmIn) and each variant counted as a distinct username, multiplying the effective failure budget. For realms where the backing store (JNDI, DataSource) authenticates case-insensitively, this allowed a real bypass. Fixed in Tomcat 9.0.118, 10.1.55, and 11.0.22 by the caseSensitive attribute, which defaults to false. The CVE advisory lists Tomcat 8.5.0 through 8.5.100 as affected; 8.5 reached end-of-life on 31 March 2024 (final release 8.5.100) and no official Apache 8.5.x fix was published - only third-party extended-support vendors patch it. Apache rates the issue Low; the fixed versions for supported lines are 9.0.118, 10.1.55, and 11.0.22.
• Username enumeration timing side channels. LockOutRealm makes enumeration harder but does not eliminate timing differences between existing and non-existing usernames. No Tomcat CVE covers this timing side channel; treat it as an inherent property of bind-based directory realms rather than a versioned defect.

Metrics and signals to monitor

Fixes

Immediate containment

If the flood is in progress:

1. Block the flooding source IPs at the firewall or reverse proxy, not at Tomcat. Per-request blocking at the application layer still costs you a thread per attempt.
2. If the Manager app is internet-exposed, remove it from the public path immediately. Either undeploy it, or front it with a network ACL restricting /manager/* and /host-manager/* to a VPN or bastion range.
3. If a 200 from a previously flooding IP has already occurred, treat it as a compromise: rotate all Manager credentials, audit tomcat-users.xml, list webapps/ for unexpected WARs, and review the access log for any POST to deploy endpoints.

Making LockOutRealm effective

1. Confirm LockOutRealm is actually wrapping the active realm in server.xml. A common misconfiguration is leaving the old UserDatabaseRealm as a sibling element instead of nesting it inside LockOutRealm.
2. If you are on a vulnerable version (pre-9.0.118, pre-10.1.55, pre-11.0.22), upgrade. The case-sensitivity bypass is a meaningful amplifier for any attacker who knows the lockout threshold.
3. Consider lowering failureCount for high-value administrative realms (for example, failureCount="3") and tuning lockOutTime to balance attacker friction against operator lockout risk.
4. LockOutRealm is per-user, not per-IP. For distributed attacks you need IP-based rate limiting at the reverse proxy or a WAF in front of Tomcat.

Removing the attack surface

The strongest fix is to not expose the Manager app on the network at all.

• In production, undeploy the Manager and Host Manager webapps if you deploy via CI/CD and do not need runtime redeploy.
• If you need Manager, restrict it with a RemoteAddrValve in the app’s context.xml to allow only known management ranges, and require client-certificate authentication in addition to the realm.
• Disable the shutdown port by setting port="-1" on the <Server> element in server.xml. Note that with this set, catalina.sh stop will not work; use systemctl stop or send SIGTERM to the JVM instead.
• If AJP is enabled, ensure it binds to localhost and has secretRequired="true".

Prevention

• Do not ship the Manager app on internet-facing instances. Default installations include Manager. Remove or restrict it.
• Rotate default credentials. Any username in tomcat-users.xml matching tomcat, admin, manager, role1, or root with a default or dictionary password is a target.
• Front Tomcat with a reverse proxy or WAF that rate-limits 401s per source IP. This catches the distributed brute force pattern that LockOutRealm cannot.
• Keep Tomcat patched. CVE-2026-43513 is low severity in isolation but high impact when combined with weak credentials.
• Alert on successful Manager access from unexpected sources. A 200 on /manager/html from a non-management IP should page, even if the 401 flood that preceded it was small.
• Log enough to investigate. The default CLF pattern does not include User-Agent or response time. At minimum, add User-Agent so you can cluster attacker tooling.

How Netdata helps

• Per-second HTTP status code dimensions on the Tomcat connector surface a 401 spike within seconds, not on the next minute rollup.
• Individual status code breakdown separates 401s from the aggregate 4xx rate.
• JMX collection surfaces LockOutRealm locked-user count and authentication failure counters, so you can correlate the 401 flood against lockout engagement and detect the case-sensitivity bypass signature: many 401s, few or no lockouts.
• Source-IP concentration views distinguish single-IP brute force from distributed credential stuffing.
• Anomaly detection on the 401 baseline trips an alert even when the absolute rate is below any fixed threshold, which matters for low-traffic instances where twenty 401s in a minute is already serious.
• A 401 spike followed by a 200 on /manager/html on the same timeline makes the attack-to-compromise transition visible.

Related guides

• Tomcat 5xx error rate: separating server failures from crawler 404s
• Tomcat accept queue overflow: acceptCount, somaxconn, and Recv-Q
• Tomcat access log setup: adding %D and %T for per-request latency
• Tomcat java.net.BindException: Address already in use: the connector never starts
• Tomcat average latency lies: why you need p95/p99 from the access log
• Tomcat threads blocked forever: the missing outbound timeout
• Tomcat classloader leak on redeploy: why the old WebappClassLoader never dies
• Tomcat connection refused: maxConnections and acceptCount both exhausted
• Tomcat accepts connections but never responds: the TCP-connect trap
• Tomcat file descriptor usage: OpenFileDescriptorCount vs the ulimit
• Tomcat frequent Full GC: pause time, G1, and the 5% overhead rule
• Tomcat GC death spiral: full GCs dominating and throughput collapsing