$ guides / activemq / activemq-authentication-failures ▌

Operations Guides

ActiveMQ authentication failures: credential rotation fallout and brute force

Your broker log starts filling with SecurityException: User name [X] or password is invalid or Authentication failed for user. Sometimes it is a slow drip from a single client. Sometimes it is a flood from dozens of source IPs. Both look similar in the log, but they are very different incidents: one is usually a forgotten service after a credential rotation, the other is either a production outage (real consumers and producers dropped) or an active attack.

ActiveMQ makes this harder to diagnose than it should be. A failed login closes the connection immediately. Most JMS clients reconnect in a loop, so one bad credential generates a continuous stream of failed connections, log noise, thread churn, and GC pressure. The broker ships with well-known default credentials (admin/admin) and has no built-in rate limiting on authentication attempts, so nothing on the broker side slows down either a broken client or a brute-force scan.

This article covers classifying what you are seeing, finding the source, and stopping the churn. The hub at /guides/activemq/ covers the broader failure pattern catalogue; here we focus narrowly on authentication failures.

What this means

Authentication in ActiveMQ Classic happens at connection setup, before any destinations are touched. The check is done by whatever security plugin you have configured, typically simpleAuthenticationPlugin (users inline in activemq.xml) or a JAAS login module such as PropertiesLoginModule (users in users.properties/groups.properties). When the check fails, the broker logs the failure and closes the connection. There is no retry counter, no lockout, no backoff on the broker side.

That design has three operational consequences:

Every failed login becomes a log line plus a closed socket. A client with a bad password and aggressive reconnect logic produces a sawtooth of connect-fail-close cycles indefinitely.
Nothing distinguishes “wrong password” from “password was rotated and this client still has the old one” in the log message itself. You classify by source, username, timing, and scope.
Nothing throttles an attacker. A brute-force scan and a broken client look identical at the per-attempt level; only the pattern differs.

The classification question is always the same: is this one client with a stale credential, many clients after a change you made, or traffic you do not recognize at all?

Common causes

Cause	What it looks like	First thing to check
Credential rotation missed a service	Sporadic failures, one or a few source IPs, one username, starting right after a password change	Which clients still hold the old credential; when failures started relative to the rotation
Rotation done on 5.12+ without `reload=true` (JAAS)	Failures continue even after the properties file was edited and looks correct	Whether the broker was restarted, or `reload=true` is set in `login.config` (see AMQ-5876 behavior change)
Misconfigured or null-username client	`User name [null] or password is invalid`, single source, constant rate	Whether the client is expected to send credentials at all, or whether `anonymousAccessAllowed` should be set
Network connector credential mismatch	Failures whose “client” is another broker in your Network of Brokers	The credentials on the `<networkConnector>` of the peer broker
Web console credential mismatch	Web console shows a security error on `vm://localhost`; failures reference the web console user	The `connectionFactory` credentials in the web console config after rotation
Brute-force scan	High rate, many source IPs or a single non-allowlisted IP cycling through usernames	Whether the source is routable from anywhere it should not be; whether transport connectors are exposed
Compromised default credentials in use elsewhere	Failures referencing `admin` or other defaults you thought were removed	Whether default accounts still exist and where they are being tried from

Quick checks

All read-only and safe to run during an incident.

# Count authentication failures in the broker log
grep -c "User name .* or password is invalid\|Authentication failed for user" /opt/activemq/data/activemq.log

# Failures grouped by username (top offenders)
grep -o "User name \[[^]]*\]" /opt/activemq/data/activemq.log | sort | uniq -c | sort -rn | head

# Recent failures with timestamps (spot the start time and rate)
grep "or password is invalid" /opt/activemq/data/activemq.log | tail -50

# Who is currently connected to the OpenWire transport
ss -tn state established '( dport = :61616 or sport = :61616 )'

# Connection churn: watch established count on the transport
# A sawtooth here matches failed-login reconnect loops
watch -n 2 "ss -tn state established '( sport = :61616 )' | wc -l"

# Are management interfaces exposed beyond localhost?
ss -tlnp | grep -E '8161|1099'

# Is the broker transport reachable from outside the expected network?
ss -tlnp | grep 61616

# Which auth plugin is in play, and whether defaults are still present
grep -A 20 "authenticationPlugin\|simpleAuthenticationPlugin\|jaasAuthenticationPlugin" /opt/activemq/conf/activemq.xml

# If using JAAS properties login: is reload enabled?
cat /opt/activemq/conf/login.config

Correlate the start time of the failures with your change log. This single step resolves most sporadic cases: if failures began within minutes of a credential rotation, you have rotation fallout, not an attack.

How to diagnose it

Classify the failure pattern before touching anything.

flowchart TD
  A[Auth failures in broker log] --> B{Single source or many?}
  B -->|Single source, one username| C{Started right after a credential change?}
  C -->|Yes| D[Rotation fallout: one missed client]
  C -->|No| E[Misconfigured client or null username]
  B -->|Many sources| F{Sources recognized as your services?}
  F -->|Yes| G[Rotation missed many clients: production outage risk]
  F -->|No| H{Usernames cycling or high rate?}
  H -->|Yes| I[Brute-force scan or attack]
  H -->|No| J[Check network connectors and web console]

Establish the rate and shape. Count failures per minute. A broken client with default failover transport behavior produces a steady, periodic drip. A scan produces bursts and username cycling. A post-rotation outage produces a step change from zero to many across multiple source IPs at the same time.
Identify the source IPs. The log line gives you the username; pair it with ss output and any connection logging to attribute failures to hosts. If the “client” IP is another broker, suspect network connector credentials. If it is the broker host itself, suspect the web console or an embedded client.
Check what changed. Credential rotations, activemq.xml edits, users.properties edits, client deployments. On ActiveMQ 5.12.0 and later with JAAS PropertiesLoginModule, the properties files are loaded once at broker startup by default; edits do not take effect until restart unless reload=true is set in login.config. This is the most common way a “successful” rotation produces a wall of auth failures: the file is correct, the broker is running the old credentials, and every client using the new password fails. On 5.11.1 and earlier the files were reloaded on every authentication, so teams upgrading across that boundary get surprised.
Check the non-obvious clients. Network connectors authenticate too: if Broker A requires auth, the <networkConnector> on Broker B must present valid credentials, and a rotation that missed them produces the same log pattern. The web console connects through a connectionFactory bean in its embedded config; if its credentials are not updated, browsing destinations fails with a security error and adds to the failure count.
Classify attack vs outage. Non-allowlisted sources, usernames you do not use, dictionary-style cycling, or a high rate from the public internet means attack. Known service IPs failing with one username after a change means outage. Treat a confirmed broad outage as a page: real consumers and producers are down, and queues will be backing up behind them.
Confirm recovery. After the fix, failures should return to baseline and the connection count should stabilize. Watch CurrentConnectionsCount and consumer counts on critical queues, not just the log.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Authentication failure rate (log-derived)	The core signal: distinguishes drip from flood, outage from attack	Step change after a change; sustained rate from non-allowlisted sources
Transport accept rate	Failed logins close connections; clients reconnect, so churn shows up here	Accept rate many times baseline with no real traffic growth
`CurrentConnectionsCount`	Sawtooth or suppressed count while failures are ongoing means clients cannot stay connected	Oscillation matching the failure cadence
Consumer count on critical queues	Broad auth failure after rotation drops real consumers	Consumer count below expected while enqueue continues
Queue depth / message age on critical queues	The downstream cost of an auth outage: backlog builds while consumers are locked out	Growth starting at the same time as the failure burst
Connections to management ports (8161, 1099)	Unexpected sources on the web console or JMX indicate probing of a different attack surface	Established connections from outside the management network
Thread count and FD usage	Reconnect churn creates threads and sockets; a long-running flood can pressure both	Growth tracking the failure rate rather than real client count

Fixes

Rotation fallout: one or a few missed clients

Update the client configuration with the new credential and restart or redeploy that client. The broker side needs nothing. If the client is one you cannot immediately change (third party, embedded appliance), temporarily keep the old account active with scoped authorization until the client is updated, then remove it.

Rotation fallout: JAAS properties not reloaded (5.12+)

Either restart the broker to load the edited users.properties/groups.properties, or add reload=true to the login module entry in login.config so future edits take effect on change. The restart is disruptive to connected clients; plan it. Setting reload=true after the fact still requires one restart to pick up the login.config change itself.

Misconfigured or null-username clients

User name [null] or password is invalid means the client sent no credentials. Fix the client to send them, or, if the traffic is legitimately anonymous (some internal STOMP/MQTT producers), set anonymousAccessAllowed="true" on simpleAuthenticationPlugin deliberately and scope authorization so anonymous users can only reach what they need. Do not enable anonymous access just to silence the log.

Network connector credentials

Update the userName/password on the <networkConnector> of every peer broker that connects to the authenticated broker, and restart or recreate the connector. Verify the bridge re-establishes and demand forwarding resumes; a failed bridge also strands messages, so check store-and-forward backlog afterward.

Brute-force or attack traffic

The broker cannot help you here; there is no rate limiting to tune. The fix is at the edge:

Restrict reachability. Firewall transport connectors and management ports to known networks. Transport connectors should never be publicly routable; the web console (since 5.16) binds to 127.0.0.1 by default and remote JMX is disabled by default, but many deployments re-open these for convenience.
Remove default credentials. admin/admin and any other shipped defaults must not exist in production.
Patch known auth-adjacent CVEs. On 6.0.0 through 6.1.1, the default configuration exposed the Jolokia and REST APIs without authentication (CVE-2024-32114, fixed in 6.1.2); an attacker on those versions never touches the auth layer at all, so absence of auth failures is not evidence of absence of attack. On 5.15.0-5.15.13 and 5.16.0 with LDAP anonymous bind, password verification could be bypassed entirely (CVE-2021-26117, fixed in 5.15.14/5.16.1). If you are on affected versions, log-based detection is unreliable; upgrade first.

Reconnection churn from any cause

While you fix the root cause, the churn itself costs threads, FDs, and log volume. Configure clients with reconnect backoff (the failover transport supports backoff parameters) so a broken credential produces a slow drip instead of a hammer. Do not restart the broker to “clear” the failures; the clients will simply reconnect and resume failing.

Prevention

Inventory credentials before rotating. Rotation fallout is almost always an inventory problem: an account shared by more services than anyone remembered. Keep a per-account consumer/producer list and treat rotations as deployments, with a rollback plan.
Rotate in two steps where possible. Add the new credential, migrate clients, then remove the old one. Watching the auth failure rate during the overlap tells you exactly which clients you missed.
Set reload=true on JAAS properties login modules if your rotation process edits files without restarting brokers.
Remove defaults and lock down exposure. No admin/admin, no publicly routable transport connectors, management interfaces on localhost or a management network only.
Alert on the pattern, not just the presence. Sporadic single-source failures are a ticket. A broad multi-source burst after a change, or high-rate failures from non-allowlisted sources, is a page.
Give clients reconnect backoff so future credential mistakes cost log lines, not broker stability.
Watch the downstream signals. After any rotation, consumer count and queue depth on critical queues are your ground truth for whether real traffic survived.

How Netdata helps

Log-derived auth failure rate over time turns “the log looks busy” into a measurable rate, so you can see the step change at rotation time and distinguish a drip from a flood.
Connection count and accept-rate correlation exposes the reconnect churn signature: failures in the log rising while established connections oscillate is the fingerprint of failed-login reconnect loops.
Consumer count per destination alongside failure bursts tells you immediately whether the failures dropped real consumers (outage) or just a noisy misconfigured client (ticket).
Queue depth and enqueue/dequeue rates quantify the blast radius of an auth outage: backlog growth starting at the failure burst confirms business impact and gives you the recovery signal when clients reconnect.
Per-port connection visibility (OpenWire, web console, JMX) helps attribute unexpected management-plane access to sources outside your allowlist.