RabbitMQ permission denied on vhost: configure, write, and read access errors

Your application connected to RabbitMQ, authenticated successfully, and then failed the moment it tried to do actual work. The client log shows something like ACCESS_REFUSED - access to exchange 'events' in vhost 'billing' refused for user 'svc-billing' or 403 ACCESS_REFUSED on a queue declare, publish, or consume call.

This is a distinct failure class from a login failure. The credentials are valid; the broker rejected a specific operation because the user’s permissions on that vhost or resource do not cover it. The fix is completely different: a login failure means wrong credentials or a broken auth backend, while ACCESS_REFUSED means the authorization layer is doing its job and the user, vhost, or permission set is wrong.

The most common trigger is an application deployed with the wrong user or pointed at the wrong vhost. Less commonly, it is an attempted privilege escalation from a compromised credential. Either way, the diagnostic path is the same: identify exactly which operation was refused, for which user, on which vhost and resource, and reconcile that against the permission set actually granted.

What this means

RabbitMQ authorization has two layers, and ACCESS_REFUSED can come from either:

  1. Vhost access. Every user must be explicitly granted access to a vhost. A newly created vhost has no user permissions at all. If a user has never been granted anything on vhost billing, any operation in that vhost fails, and the error message says “access to vhost refused”.

  2. Resource permissions. Within a vhost, each user has a triple of regular expressions: configure, write, and read. Configure covers declaring and deleting exchanges, queues, and bindings. Write covers publishing to an exchange and binding resources. Read covers consuming from queues and acknowledging messages. Each operation is checked against the relevant regex, matched against the resource name. If the regex does not match, the operation is refused with a 403.

Permissions are set per user, per vhost with rabbitmqctl set_permissions. Because the patterns are regexes matched against resource names, a user can have write access to exchange events but not audit in the same vhost, and only one of the two operations will fail. That is why the refused operation in the error message is the single most important piece of evidence.

flowchart TD
    A[Client operation fails with ACCESS_REFUSED] --> B{Did the AMQP connection open?}
    B -- No --> C[Authentication problem: wrong credentials,
disabled user, auth backend down.
Different guide.]
    B -- Yes --> D{Error names the vhost,
not a specific resource?}
    D -- Yes --> E[Vhost access missing:
user has no permissions row
on that vhost, or vhost down]
    D -- No, names exchange/queue --> F[Resource permission missing:
configure/write/read regex
does not match resource name]
    E --> G[Reconcile with
list_user_permissions]
    F --> G
    G --> H[Fix: correct user/vhost in app config,
or grant the missing regex]

Common causes

CauseWhat it looks likeFirst thing to check
Application deployed with wrong vhostEverything the app attempts fails; error names the vhost. Often appears after a config rollout or environment mix-up (staging app pointed at prod, or vice versa).Compare the vhost in the app’s connection string against rabbitmqctl list_vhosts and the intended environment.
User exists but was never granted permissions on this vhost“Access to vhost ‘/’ refused” even though the user authenticates. New vhosts grant nobody anything by default.rabbitmqctl list_user_permissions <user>: is there a row for this vhost at all?
Missing configure permission for declare operationsPublish and consume work, but declaring the exchange or queue fails. Common trap: operators grant write+read and assume that is enough. It is not; declare requires configure.Check the configure regex for the user: does it match the resource being declared?
Permission regex does not match server-generated or unexpected resource namesIntermittent or partial failures. The regex was written for a naming convention the app does not fully follow (server-named queues, reply queues, amq.* exchanges).Get the exact resource name from the refused operation in the log and test it against the regex mentally.
Permission changed but client still using cached authorizationYou granted the permission, but the client still gets refused until it reconnects. Permission changes may only take effect on reconnect.Reconnect the client or restart the app instance after changing permissions.
Vhost down or corrupted, not a permission problem“Access to vhost refused” even though permissions look correct. A vhost can be in a stopped or partially available state.GET /api/vhosts cluster_state, or rabbitmqctl list_vhosts; check vhost health separately from permissions.
Zero-permission user used for health checks after a hardening changeA monitoring or health-check account that used to do passive declares now fails after an upgrade where passive declarations started requiring at least one permission on the resource.Check broker version; grant a narrow read regex to the health-check user.
Attempted privilege escalationDenials from a credential that should only ever consume, now trying to declare exchanges or write to unexpected resources.Correlate source: which user, from which host, attempting what. Treat as a security signal, not a config ticket.

Quick checks

These are read-only and safe to run during an incident.

# What the broker actually logged (the authoritative record of what was refused)
grep -i "ACCESS_REFUSED\|access refused" /var/log/rabbitmq/rabbit@$(hostname).log | tail -20

# List vhosts that exist
rabbitmqctl list_vhosts

# List users and their tags
rabbitmqctl list_users

# Show the exact configure/write/read regexes for the user in the error
rabbitmqctl list_user_permissions svc-billing

# Show all users' permissions on a specific vhost
rabbitmqctl list_permissions -p billing

Two things to note in the log line. First, it names the operation (exchange.declare, queue.declare, basic.publish, basic.consume), and that maps directly to which of the three regexes failed: declare to configure, publish to write, consume to read. Second, it names the exact resource. Regexes match against that exact string, so capture it verbatim.

If the error names the vhost itself rather than a resource, check whether the vhost is healthy before assuming a permission problem:

# Vhost cluster state via management API
curl -s -u guest:guest http://localhost:15672/api/vhosts | jq '.[] | {name, cluster_state}'

A vhost that is stopped or partially available can surface as a refusal that looks like a permission error. Check cluster_state per node; “running” is expected everywhere outside maintenance.

How to diagnose it

  1. Capture the exact refusal. Pull the full ACCESS_REFUSED line from the broker log. Record: user, vhost, operation, resource name, and source connection. This single line usually identifies the fix.

  2. Confirm the user can authenticate. If the connection itself is being rejected (login refused, authentication failed), you are dealing with an auth problem, not authorization. Stop here and fix credentials or the auth backend. ACCESS_REFUSED happens after a successful login.

  3. Check whether the vhost exists and is healthy. rabbitmqctl list_vhosts and the /api/vhosts cluster state. A typo in the app’s vhost name, a vhost deleted out from under the app, or a down vhost all present similarly. The vhost in a connection string is URL-encoded; / becomes %2f, and mistakes here are a classic cause of “wrong vhost” deployments.

  4. Read the user’s actual permissions. rabbitmqctl list_user_permissions <user>. If there is no row for the vhost, that is the whole story: the user was created but never granted access, or the vhost was created after the user and nobody ran set_permissions.

  5. Map the refused operation to the failed regex. Declare/delete of exchanges, queues, bindings: configure regex. Publish to an exchange: write regex. Consume and ack: read regex. Then check whether the resource name from step 1 matches the corresponding regex. An empty string or ^$ grants nothing. .* grants everything.

  6. Check for a regex coverage gap rather than a missing grant. If the regex is ^events\..* and the refused resource is reply.4f2a91 (a server-generated name), the grant exists but the pattern does not cover what the client actually does. This is especially common with exclusive/reply queues, amq.* exchanges, and plugin-created resources.

  7. If permissions were recently changed, force a reconnect. Permission changes may not apply to already-open connections and channels. If you granted the permission and the app still fails, restart the application instance or bounce its connection before concluding the grant did not work.

  8. If everything looks correct, check version-specific behavior. On recent RabbitMQ versions, passive declarations (exchange.declare / queue.declare with passive=true) require the user to have at least one of configure, write, or read on the target resource. Health checks and monitoring tooling that relied on passive declares with a zero-permission user will start failing right after the upgrade.

  9. If the source and operation look wrong for that credential, treat it as a security event. A consumer credential suddenly attempting exchange declares or publishing to unfamiliar exchanges is a privilege-escalation pattern. Rotate the credential and audit what it did while it was valid.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
ACCESS_REFUSED lines in broker logThis error is not exposed as a broker metric; the log is the only source. Protocol errors like PRECONDITION_FAILED and ACCESS_REFUSED are log-only signals.More than about 5 denials per minute from a single source warrants investigation.
Connection churn rate (churn_rates.connection_created/closed)A misconfigured app often retries in a loop: connect, get refused, close, reconnect. Churn is the metric-level symptom of a permission bug.Creation rate above 3x rolling baseline sustained for 2+ minutes, correlated with ACCESS_REFUSED lines.
Channel error/closure rateAn ACCESS_REFUSED on a channel operation closes the channel but not the connection. Many client libraries then tear down and rebuild the whole connection, amplifying churn.Elevated channel churn without corresponding message flow.
Publish / deliver / ack ratesA refused publish or consume shows up as a rate dropping toward zero on the affected vhost while connection counts stay stable.Publish or ack rate drops >80% from baseline while connections remain up.
Consumer count per queueA consumer that cannot pass the read-permission check never subscribes.Zero consumers on a queue that should have them, right after a deployment.
Authentication failure log linesCompanion signal: auth failures and authz failures often travel together during credential rotation or misconfigured deployments.Sustained failures above ~10/minute from one source.

Fixes

Wrong user or vhost in the application

Fix the application configuration, not the broker. If the app connected to the prod vhost with a staging user, granting permissions to make it work papers over the real problem and widens the blast radius of a future staging bug. Correct the connection string, redeploy, and confirm the refused operations stop in the log.

User was never granted permissions on the vhost

Grant the minimal set the application actually needs:

# Grant configure/write/read regexes per user, per vhost
rabbitmqctl set_permissions -p billing svc-billing "^events\..*" "^events\..*" "^events\..*"

A user that only consumes should get an empty configure and write regex ("" "" "^events\..*"), not a full grant. The empty string denies all operations on all resources, which is exactly what you want for the permissions the app does not need. Prefer narrow patterns over .* for anything that is not an operator account.

Missing configure permission for declares

Decide who owns topology. Two clean patterns:

  • Topology owned by deployment tooling. The app gets write+read only, and exchanges/queues/bindings are declared by Terraform, Helm hooks, or an operator script running as an admin user. The app’s failure to declare is then a signal the topology was never provisioned, which is what you want.
  • Topology owned by the app. Grant configure with a regex scoped to the app’s resource namespace, e.g. ^billing\..*, so the app can manage its own exchanges and queues but nothing else.

What does not work: granting write+read and assuming declare will succeed. Declare requires configure, full stop.

Regex does not match server-generated names

If the client uses server-named queues, exclusive reply queues, or plugin-created resources, a prefix regex will not match them. Either name the resources deterministically in the application so they fall under the prefix, or widen the regex deliberately. Widen as narrowly as possible: match the actual naming scheme, not .*.

Grant applied but client still refused

Reconnect. Permission changes are not reliably picked up by already-open connections and channels, so a client that connected before the grant can keep failing. Restart the application instances or close their connections and let them reconnect. This is a common false “the fix did not work” during incidents.

Vhost down, not a permission issue

If the vhost is stopped or partially available, permissions are irrelevant until the vhost recovers. Check node health and the vhost’s cluster state. If a vhost is persistently down due to corrupted metadata, recovery involves broker-level intervention on the data directory, which is destructive and beyond a permission fix; treat it as its own incident.

Zero-permission health-check user after upgrade

Grant the health-check or monitoring user a minimal read regex on the resource it passively declares, rather than full permissions. Keep it scoped to a single known queue or exchange used for the check.

Prevention

  • Grant the minimum per application. Configure only for apps that own their topology, write only for publishers, read only for consumers. Empty strings for everything else. This turns privilege escalation attempts into noisy, loggable failures instead of silent success.
  • Provision users and vhosts together. A new vhost grants nobody anything. Make set_permissions part of the same automation that creates the vhost and the application user, so “user exists but has no permissions” cannot happen in the first place.
  • Namespace resources per application. If every app’s exchanges and queues share a prefix (billing.*, search.*), permission regexes stay simple and auditable, and cross-app access mistakes become visible.
  • Alert on authorization denials, not just auth failures. ACCESS_REFUSED is log-only, so wire log alerting on it. Repeated denials from one source after a deployment is a fast catch for misconfiguration; denials from a credential attempting new operations is a security tripwire.
  • Rotate credentials deliberately and watch for the fallout. Credential rotation without updating every consumer produces a mix of auth failures and authorization oddities. Track rotation as a deployment event and correlate with the log signals above.
  • Audit permission sets periodically. rabbitmqctl list_permissions per vhost, compared against what each application is supposed to do. Drift accumulates: temporary .* grants from a past incident have a way of becoming permanent.

How Netdata helps

  • Churn correlation. Netdata charts RabbitMQ connection and channel churn alongside message rates, so the retry loop from a permission bug (connections cycling, publish rate flat at zero) is visible as a pattern rather than two separate anomalies.
  • Rate drop detection per vhost. A refused publish or consume path shows up as publish, deliver, or ack rates collapsing on one vhost while connections stay up, which distinguishes authorization failures from connectivity failures in one view.
  • Consumer count per queue. A consumer blocked by a missing read permission never subscribes. Zero consumers on a queue that should have them is charted and alertable.
  • Deployment-window context. Because most permission incidents follow a rollout, having churn, rates, and consumer counts on a single dashboard makes the “what changed” question answerable in seconds.
  • Log-side gap awareness. ACCESS_REFUSED itself is not a broker metric. Netdata surfaces the operational symptoms (churn, zero rates, missing consumers) that tell you to go read the broker log for the exact refusal.