$ guides / apache-pulsar / apache-pulsar-authorization-failures ▌

Operations Guides

Apache Pulsar authorization failures: the AuthorizationException only the logs will show

A producer or consumer authenticates successfully but cannot produce, consume, or manage a topic. The client receives an AuthorizationException or a generic permission error. Your dashboard shows nothing unusual because Pulsar exposes no authorization failure counter in its Prometheus metrics. The only evidence is in broker logs.

This is a common blind spot. Authentication failures have a dedicated metric (pulsar_authentication_failures_total), but authorization failures do not. A client with valid credentials but missing role permissions fails silently from a metrics perspective. You find out when an application team reports errors or when someone greps the broker log.

The usual root cause is an application deployed with a role that lacks produce, consume, or manage permissions on the target tenant, namespace, or topic. Less commonly, it is a proxy role misconfiguration, a version-specific permission change (such as the subscription permission requirement introduced in Pulsar 3.0), or an attempted privilege escalation.

What this means

When authorizationEnabled=true in broker.conf, every client operation (produce, consume, lookup, manage) is checked against the authenticated role. The default provider is org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider. If the role lacks the required action on the target namespace or topic, the broker rejects the operation and logs a warning.

The rejection is logged at WARN level in ServerCnx, the Netty channel handler that processes Pulsar protocol commands. The pattern (exact wording varies by version; Pulsar 4.0 also prints the original role):

Role <role> is not authorized to perform operation <OPERATION> on topic <topic>
Role <role> or OriginalRole <role> is not authorized to perform operation <OPERATION> on topic <topic>

For namespace-level operations, the message references the namespace instead of a topic. The <OPERATION> is typically LOOKUP, PRODUCE, CONSUME, or a manage-level operation. The exception thrown to the client is org.apache.pulsar.client.api.PulsarClientException.AuthorizationException.

The critical gap: there is no pulsar_authorization_failures_total or equivalent counter in any current Pulsar version (verified against the broker’s Prometheus metrics in the 3.0.x and 4.0.x lines). The authentication metrics reference lists pulsar_authentication_success_total and pulsar_authentication_failures_total (with provider_name, auth_method, and reason labels), but nothing for authorization. Metrics-based alerting cannot detect authorization failures unless you build custom log-derived monitoring.

Common causes

Cause	What it looks like	First thing to check
Role lacks permission on namespace/topic	`AuthorizationException` on produce or consume for a specific topic, same role repeating	`pulsar-admin namespaces permissions <tenant>/<namespace>`
Proxy role not in `proxyRoles`	`cannot specify originalPrincipal when connecting without valid proxy role` in broker log	broker.conf `proxyRoles` list and proxy’s authenticated role
Subscription-level permission configured	Consume works but subscribe fails for a specific subscription	Whether `pulsar-admin namespaces subscription-permission` shows roles configured for the subscription
Transaction system topic LOOKUP denied	`not authorized to perform operation LOOKUP on topic persistent://pulsar/system/transaction_coordinator_assign`	Whether the client role has produce or consume permission (which implies LOOKUP) on the system namespace
Wildcard role mismatch	Role `svc-prod-orders` denied but `svc-*` was granted, or wildcard not enabled	`authorizationAllowWildcardsMatching` in broker.conf
Privilege escalation attempt	Unusual roles requesting manage operations, bursts from unknown IPs	Broker log patterns and admin audit logs

Quick checks

# Search broker logs for authorization denials (adjust path to your deployment)
grep -c "AuthorizationException\|not authorized\|permission denied" /var/log/pulsar/broker.log

# Show recent authorization denials with context
grep "not authorized\|AuthorizationException" /var/log/pulsar/broker.log | tail -20

# Check current permissions on a namespace
pulsar-admin namespaces permissions <tenant>/<namespace>

# Check permissions on a specific topic
pulsar-admin topics permissions persistent://<tenant>/<namespace>/<topic>

# Verify authorization is enabled
grep "^authorizationEnabled" /path/to/broker.conf

# Check superUserRoles and proxyRoles configuration
grep -E "^superUserRoles|^proxyRoles" /path/to/broker.conf

# Verify the authorization provider in use
grep "^authorizationProvider" /path/to/broker.conf

# Check if wildcard matching is enabled
grep "^authorizationAllowWildcardsMatching" /path/to/broker.conf

# Confirm authentication failures metric exists but authorization does not
# Default Prometheus port is 8000; adjust if your deployment differs
curl -s http://<broker-host>:8000/metrics | grep -E "authentication_failures|authorization_failures"

How to diagnose it

flowchart TD
    A["Client receives AuthorizationException"] --> B{"Is authentication succeeding?"}
    B -- "No" --> C["Check pulsar_authentication_failures_total
Fix credentials, token, or TLS"]
    B -- "Yes" --> D["Grep broker log for 'not authorized'"]
    D --> E["Extract role, operation, topic from log line"]
    E --> F{"Is role in superUserRoles?"}
    F -- "Yes" --> G["Check for proxy role misconfiguration
or originalPrincipal issue"]
    F -- "No" --> H["Check namespace/topic permissions"]
    H --> I{"Does role have required action?"}
    I -- "No" --> J["Grant permission via pulsar-admin"]
    I -- "Yes" --> K["Check wildcard matching,
subscription authz, or transaction topic"]

Step 1: Confirm the failure is authorization, not authentication.

If pulsar_authentication_failures_total is increasing for the same client, the problem is identity validation (expired token, bad TLS cert, wrong auth plugin), not permissions. If authentication succeeds but operations still fail, it is authorization.

curl -s http://<broker-host>:8000/metrics | grep pulsar_authentication_failures_total

Step 2: Extract the role, operation, and target from the broker log.

Parse out three fields: the role (authenticated identity), the operation (LOOKUP, PRODUCE, CONSUME, etc.), and the target (topic or namespace). These three fields narrow the diagnosis to a specific permission check.

Step 3: Check the role’s current permissions.

# Namespace-level permissions
pulsar-admin namespaces permissions <tenant>/<namespace>

# Topic-level permissions (overrides namespace)
pulsar-admin topics permissions persistent://<tenant>/<namespace>/<topic>

The output maps roles to allowed actions (produce, consume, functions, sources, sinks, packages). If the role from the log is not listed, or is listed without the required action, that is the root cause.

Step 4: Verify proxy role configuration if a Pulsar Proxy is in the path.

When a proxy forwards client connections, it authenticates with its own role and passes the client’s original principal. If the proxy’s authenticated role is not listed in proxyRoles in broker.conf, the broker rejects the connection. The log message differs from a standard denial and typically reads cannot specify originalPrincipal when connecting without valid proxy role. This is frequent in Helm-deployed clusters where the proxy role configuration is inconsistent between proxy and broker configs.

Step 5: Check for version-specific permission changes.

Subscription-level authorization is a long-standing Pulsar feature, not a Pulsar 3.0 change. If a role list is configured for the specific subscription (via pulsar-admin namespaces grant-subscription-permission), only roles in that list may subscribe; otherwise subscribing falls back to the role’s consume permission on the topic or namespace. There is no subscriptionAuthorizationEnabled broker property: PIP-167 proposed making an empty subscription permission deny by default, but its implementation (PR #15576) was merged and reverted the same day, so the change never shipped.

If the denial appeared after a version upgrade, review the release notes for authorization behavior changes.

Step 6: Check the transaction system topic if the client uses transactions.

Clients using Pulsar transactions need LOOKUP permission on persistent://pulsar/system/transaction_coordinator_assign. This produces a misleading AuthorizationException that looks like a regular topic permission issue. If the log line references this system topic, grant LOOKUP permission to the client role on the system namespace.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Broker log: `AuthorizationException` rate	The only direct signal for authorization failures	Any sustained non-zero count from a production client
`pulsar_authentication_failures_total`	Distinguishes authN from authZ. If also spiking, the problem is credentials, not permissions	Spike with `reason` label indicating token or cert failure
`pulsar_rate_in` / `pulsar_rate_out`	Authorization failures on producers cause throughput drops; on consumers, dispatch stalls	Sudden drop on specific topics while cluster aggregate looks normal
`pulsar_active_connections`	Connections succeed (authentication passes) but operations fail (authorization denies)	Stable or growing connection count with falling throughput
`pulsar_broker_lookup_failures`	LOOKUP authorization denials increment lookup failure counters	Elevated lookup failures correlating with authorization log entries

Fixes

Missing role permission on namespace or topic

The most common fix. Grant the required action to the role.

# Grant produce and consume on a namespace
pulsar-admin namespaces grant-permission <tenant>/<namespace> \
  --actions produce,consume \
  --role <role>

# Grant on a specific topic
pulsar-admin topics grant-permission persistent://<tenant>/<namespace>/<topic> \
  --actions produce,consume \
  --role <role>

Note: grant-permission does not accept comma-separated roles in a single --role flag. Each role must be granted separately.

Wildcard role matching

If you use role naming conventions (for example, svc-prod-*), wildcard matching avoids granting each role individually. Enable it in broker.conf:

authorizationAllowWildcardsMatching=true

Wildcards only work at the beginning or end of the role name. svc-* matches svc-prod-orders but not prod-svc-orders. The broker must be restarted for this change to take effect.

Proxy role configuration

Add the proxy’s authenticated role to proxyRoles in broker.conf on every broker:

proxyRoles=<proxy-role>

Also ensure the proxy’s own role does not appear in superUserRoles. A proxy role and a super-user role being the same is a security risk and can cause unexpected authorization behavior.

Transaction system topic permission

Clients using transactions need access to persistent://pulsar/system/transaction_coordinator_assign. lookup is not a grantable action (pulsar-admin accepts produce, consume, functions, sources, sinks, and packages only); the default authorization provider derives LOOKUP from produce or consume permission, so grant the role produce (and consume if it consumes) on the pulsar/system namespace, or add the role to superUserRoles:

pulsar-admin namespaces grant-permission pulsar/system \
  --actions produce,consume \
  --role <role>

Subscription-level permission

If a subscription has configured roles and consumers cannot subscribe, verify the client role is in that subscription’s role list. Subscription-level permissions are configured per namespace with the admin CLI (they existed before Pulsar 3.0; PIP-167, which would have made an unconfigured subscription deny by default, was reverted and never shipped):

# Show subscription-level permissions
pulsar-admin namespaces subscription-permission <tenant>/<namespace>

# Grant a role permission to consume from a specific subscription
pulsar-admin namespaces grant-subscription-permission <tenant>/<namespace> \
  --subscription <sub> --roles <role>

pulsar-admin namespaces revoke-subscription-permission <tenant>/<namespace> \
  --subscription <sub> --roles <role>

If no roles are configured for the subscription, subscribing falls back to the role’s consume permission on the topic or namespace.

Revoke stale permissions

During permission reconciliation, revoke access for roles that should no longer have it:

# WARNING: immediately revokes access for the specified role.
# Confirm with the application owner before running.
pulsar-admin namespaces revoke-permission <tenant>/<namespace> \
  --role <role>

Prevention

Log-based authorization alerting. Since there is no metric, build a log-derived counter. A log forwarder (Filebeat, Fluentd, Vector) can count AuthorizationException or not authorized lines per role and emit a custom metric or alert. Alert on sustained non-zero rates from production client IPs. A burst from unknown IPs warrants a security investigation.

Distinguish authN from authZ in your runbook. When a client reports access errors, check pulsar_authentication_failures_total first. If it is not increasing, the problem is authorization. This prevents wasting time on credential debugging when the issue is a missing role grant.

Codify permission grants in deployment pipelines. The most common cause is a new application deployed without the required grant-permission call. Include permission grants as a step in deployment automation, alongside topic and subscription creation.

Review permissions periodically. Stale permissions accumulate. Use pulsar-admin namespaces permissions as part of a quarterly access review. Revoke roles for decommissioned services.

Monitor for CVE-related behavior changes. If you patched CVE-2024-29834 (affects 2.7.1 through 2.10.6, 2.11.0 through 2.11.4, 3.0.0 before 3.0.4, 3.1.0 through 3.1.3, and 3.2.0 before 3.2.2) or CVE-2024-28098 (affects 2.7.1 before 2.10.6, 2.11.0 before 2.11.4, 3.0.0 before 3.0.3, 3.1.0 before 3.1.3, and 3.2.0 before 3.2.1), previously-allowed operations on partitioned topics and topic-level policies may now correctly fail with AuthorizationException. Applications that relied on the bypassed authorization need explicit permission grants after the fix.

Keep proxy and broker configs synchronized. In Helm deployments, ensure the proxy role in the broker config matches the proxy’s authenticated role. Use config maps or templating to keep these consistent across all brokers.

How Netdata helps

Netdata collects Pulsar broker metrics at per-second resolution, including pulsar_authentication_failures_total. While authorization failures have no native metric, Netdata’s log monitoring can surface AuthorizationException patterns from broker logs alongside the metrics timeline, letting you correlate a permission denial with throughput drops or connection behavior in the same view.
The pulsar_rate_in and pulsar_rate_out metrics help identify which topics are affected when an authorization failure blocks producers or consumers. A sudden rate drop on a specific namespace while cluster-wide throughput remains stable is a strong indicator of a targeted permission issue.
pulsar_active_connections tells you whether the client is connected at all. If connections are stable but throughput has dropped, the problem is downstream of authentication. Correlating connection count with throughput narrows the diagnosis.
pulsar_broker_lookup_failures can reveal LOOKUP authorization denials, which are the first operation a client performs. Elevated lookup failures coinciding with authorization log entries confirm the permission scope.
Netdata’s anomaly detection (ML-based) can flag unusual patterns in connection churn or throughput that may correlate with permission changes, even when no explicit authorization metric exists.