RabbitMQ ACCESS_REFUSED - Login was refused: authentication failures and credential rotation

Your application logs are filling with ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN, and the broker log shows matching PLAIN login refused lines. No messages flow from the affected clients, and depending on the client library’s retry behavior, the broker may also be spending CPU on a tight connect-authenticate-reject loop.

This error means the broker rejected the username/password pair (or the backend that validates them) during AMQP connection establishment. It happens before any vhost, exchange, or queue is touched. The connection is closed immediately, which is why a single misconfigured deployment can turn into connection churn that looks like a capacity problem.

Establish scope first. One client failing sporadically is almost always a configuration problem, usually a stale credential after rotation. A sustained flood of failures from many unknown sources is a security event, not an operational one. Treat them differently from minute one.

What this means

During the AMQP 0-9-1 handshake, the client offers a username and password via the PLAIN mechanism. RabbitMQ checks them against its configured authentication backend chain (internal database by default, LDAP or other backends if configured). On failure, the broker rejects the login and closes the connection. Clients that advertise the authentication_failure_close capability receive a clean connection.close frame carrying the ACCESS_REFUSED reason; older clients may just see the TCP connection drop.

Two consequences matter operationally:

• The connection never reaches a usable state. The client cannot declare, publish, or consume anything. From the application’s point of view the broker is “down” even though every other client is fine.
• Retries amplify the failure. Most client libraries reconnect immediately on connection loss. Each retry costs the broker a TCP accept, an AMQP negotiation, and an Erlang process, all torn down seconds later. A few hundred misconfigured pods retrying in a tight loop becomes a connection storm.

flowchart TD
  A[Client opens AMQP connection] --> B{Broker checks credentials}
  B -->|valid| C[Connection established]
  B -->|invalid or backend error| D[ACCESS_REFUSED, connection closed]
  D --> E[Client retries immediately]
  E --> A
  D --> F[Connection churn rises]
  F --> G[FD, socket, CPU pressure on broker]

The log line tells you which mechanism and usually which user failed. The text after PLAIN login refused distinguishes “bad password” from “user can only connect via localhost” from backend errors, so read the full message before assuming the password is wrong.

Common causes

One subtlety: LDAP backend failures look identical to bad passwords in the client error. If the LDAP server is slow or down, perfectly valid credentials produce ACCESS_REFUSED. Always check backend health before concluding the password is wrong.

Quick checks

All of these are read-only and safe to run during an incident.

# Count authentication failures in the broker log
grep -c "login refused" /var/log/rabbitmq/rabbit@$(hostname).log

# Look at the most recent refused logins with full detail
grep "login refused" /var/log/rabbitmq/rabbit@$(hostname).log | tail -50

# Test a specific username/password pair against the broker
rabbitmqctl authenticate_user <username> '<password>'

# Confirm the user exists
rabbitmqctl list_users

# See what the user is allowed to do (for the permission errors that follow login)
rabbitmqctl list_user_permissions <username>

# Check whether failed logins are driving reconnect churn
curl -s -u guest:guest http://localhost:15672/api/overview | jq '.churn_rates | {created: .connection_created_details.rate, closed: .connection_closed_details.rate}'

# Count connections by state and find the noisiest sources
rabbitmqctl list_connections name state peer_host user_provided_name | head -30

# Watch FD and socket headroom if the churn is heavy
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, fd_used, fd_total, sockets_used, sockets_total}'

Failed AMQP logins show up in GET /api/connections as very short-lived connections that close immediately. A source IP with dozens of connections each living a second or two is your retrying client.

How to diagnose it

1. Read the full log line. The text after PLAIN login refused is the diagnosis in miniature. user 'guest' can only connect via localhost is a different fix from a generic refusal, and a message naming the LDAP backend points away from the password entirely.

2. Measure the rate and identify the source. Sporadic failures (as a rule of thumb, under roughly 10/minute from one source) are a misconfiguration hunt. Over roughly 100/minute from a single source is an automated client in a retry loop or an attack. Note the source IPs and the usernames being attempted.

3. Test the credential out of band. Run rabbitmqctl authenticate_user with the exact pair the application is using, copied from the live secret store rather than from memory. If it succeeds but the app still fails, the app is sending something different than you think: a stale mounted secret, an environment variable overridden at runtime, or a truncated value.

4. Rule out the backend. If you use LDAP, check that the server is reachable from the broker node and responding quickly. A known operational gotcha: Active Directory closes idle LDAP connections after about 15 minutes, and the plugin’s default idle timeout is effectively infinite, so the pool can hold dead connections and produce sporadic refusals until they cycle out. Setting auth_ldap.idle_timeout below AD’s 900-second window (for example 600000 ms) addresses this.

5. Assess the blast radius. While you fix the credential, check connection churn, fd_used / fd_total, and the Erlang run queue. If the failing clients retry without backoff, the broker is burning CPU on handshakes and accumulating short-lived connections. That secondary load can outlive the original fix if you do not also stop the retry loop.

6. Classify unknown-source floods separately. Failures from IPs you do not recognize, across many usernames, with no matching deployment, are a brute-force pattern. This is a security ticket: identify the exposure path (is port 5672 reachable from the internet or a shared network?), block the source at the network level, and review user tags and passwords.

Metrics and signals to monitor

AMQP authentication failures are not exposed as a broker metric. They exist only in the log file, so you need log collection or an external counter to alert on them.

Fixes

Stale credentials after rotation

Update the secret the application actually reads, then restart or reload the consumers so they pick it up. Verify with rabbitmqctl authenticate_user against the new value before restarting, so you catch a wrong secret without another churn cycle. If you mount secrets as files, remember that updating the file does not reconnect existing AMQP connections; clients must detect the auth failure and re-read the credential, or be restarted.

Disabled, deleted, or wrong user

Recreate the user or reset its password with rabbitmqctl, then re-grant permissions per vhost. Check rabbitmqctl list_user_permissions <username> afterward, because a recreated user starts with no permissions and you will trade a login failure for a stream of ACCESS_REFUSED permission errors on declare and publish.

guest from a remote host

Do not relax the localhost restriction on guest. Create a dedicated user for the application, grant it permissions on the vhosts it needs, and update the client configuration. The default guest account exists for local administration and first boot, not for application traffic.

LDAP backend failures

Fix reachability first: network path, firewall rules, service account lockout on the directory side. Then address the idle-connection gotcha by setting auth_ldap.idle_timeout below your directory’s idle disconnect window. If failures align with LDAP latency rather than outages, review pool sizing and directory load, since every new AMQP connection blocks on the backend during handshake.

Brute-force flood

Block the offending sources at the firewall or load balancer while you investigate exposure. Review rabbitmqctl list_users for weak or default accounts and confirm the AMQP listener is not reachable from untrusted networks. Do not tune the broker to absorb the flood; remove the exposure.

Stop the retry loop regardless of cause

Whatever the root cause, add exponential backoff with jitter to client reconnection logic. A client that retries immediately on ACCESS_REFUSED will never succeed (the credential is still wrong) and converts a configuration bug into broker load. Backoff turns a connection storm back into a quiet log line.

Prevention

• Rotation with overlap. When rotating credentials, create the new credential and deploy it before revoking the old one, or accept a controlled blip. Rotating by changing the password in place guarantees every connected client fails at once.
• Reconnect handling in clients. Require backoff and jitter in every service’s AMQP client configuration. Failed logins close the connection immediately, so retry policy is your only control over churn.
• One user per service, least privilege. Per-service accounts make the failing username in the log line immediately actionable and limit what a leaked credential can do. Audit with list_user_permissions after any change.
• No guest over the network. Keep guest localhost-only and do not create production users named guest with guest passwords.
• Remove anonymous authentication. Check which SASL mechanisms are enabled and remove ANONYMOUS if present; it maps unauthenticated clients to a default user and has no place in production.
• Alert on the failure rate. Collect the auth failure count from logs and alert above roughly 10/minute sustained. Pair it with connection churn so a retrying misconfigured client pages before it pressures FDs.
• Watch the auth backend as a dependency. If you authenticate against LDAP, monitor its reachability and latency like any other critical dependency, because its failures surface as broker login failures.

How Netdata helps

• Netdata’s RabbitMQ collector tracks connection churn, total connections, and channels per second, so an auth-failure-driven retry storm is visible as a churn spike even though the failure itself lives only in the log.
• Correlating churn against fd_used, socket usage, and Erlang process counts shows whether the retry loop is approaching a resource limit that would start rejecting healthy clients.
• The run queue and scheduler utilization charts reveal the CPU cost of handshake floods, which matters most when TLS is terminated at the broker.
• Short-lived connection patterns help separate “one app with a stale secret” from “many sources hammering the listener,” the distinction that routes you to a config fix versus a security response.
• Alerting on churn-rate deviation from baseline catches the misconfigured client within minutes of a bad rotation, instead of at the next capacity review.

Related guides

• RabbitMQ connection storm: reconnect loops, FD pressure, and CPU spent on handshakes
• RabbitMQ connection blocked / blocking: publishers frozen by a resource alarm
• RabbitMQ connection in flow state: credit-based backpressure explained
• RabbitMQ channel leak and churn: the channel-per-message anti-pattern
• RabbitMQ Erlang distribution port saturation: the single link that carries all cluster traffic
• RabbitMQ consumers connected but not acknowledging: the consumer black hole