A producer or consumer authenticates successfully but cannot produce, consume, or manage a topic. The client receives an AuthorizationException or a generic permission error. Your dashboard shows nothing unusual because Pulsar exposes no authorization failure counter in its Prometheus metrics. The only evidence is in broker logs.

This is a common blind spot. Authentication failures have a dedicated metric (pulsar_authentication_failures_total), but authorization failures do not. A client with valid credentials but missing role permissions fails silently from a metrics perspective. You find out when an application team reports errors or when someone greps the broker log.

The usual root cause is an application deployed with a role that lacks produce, consume, or manage permissions on the target tenant, namespace, or topic. Less commonly, it is a proxy role misconfiguration, a version-specific permission change (such as the subscription permission requirement introduced in Pulsar 3.0), or an attempted privilege escalation.

What this means

When authorizationEnabled=true in broker.conf, every client operation (produce, consume, lookup, manage) is checked against the authenticated role. The default provider is org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider. If the role lacks the required action on the target namespace or topic, the broker rejects the operation and logs a warning.

The rejection is logged at WARN level in ServerCnx, the Netty channel handler that processes Pulsar protocol commands. The pattern:

Role <role> is not authorized to perform operation <OPERATION> on topic <topic>

For namespace-level operations, the message references the namespace instead of a topic. The <OPERATION> is typically LOOKUP, PRODUCE, CONSUME, or a manage-level operation. The exception thrown to the client is org.apache.pulsar.client.api.PulsarClientException.AuthorizationException.

The critical gap: there is no pulsar_authorization_failures_total or equivalent counter in any current Pulsar version (verified through 4.0.x LTS ). The authentication metrics reference lists pulsar_authentication_success_total and pulsar_authentication_failures_total (with a reason label), but nothing for authorization. Metrics-based alerting cannot detect authorization failures unless you build custom log-derived monitoring.

Common causes

CauseWhat it looks likeFirst thing to check
Role lacks permission on namespace/topicAuthorizationException on produce or consume for a specific topic, same role repeatingpulsar-admin namespaces permissions <tenant>/<namespace>
Proxy role not in proxyRolescannot specify originalPrincipal when connecting without valid proxy role in broker logbroker.conf proxyRoles list and proxy’s authenticated role
Subscription permission required (Pulsar 3.0+)Consume works but subscribe fails after upgradeWhether subscriptionAuthorizationEnabled is set and the role has consume permission
Transaction system topic LOOKUP deniednot authorized to perform operation LOOKUP on topic persistent://pulsar/system/transaction_coordinator_assignWhether the client role has LOOKUP on the system namespace
Wildcard role mismatchRole svc-prod-orders denied but svc-* was granted, or wildcard not enabledauthorizationAllowWildcardsMatching in broker.conf
Privilege escalation attemptUnusual roles requesting manage operations, bursts from unknown IPsBroker log patterns and admin audit logs

Quick checks

# Search broker logs for authorization denials (adjust path to your deployment)
grep -c "AuthorizationException\|not authorized\|permission denied" /var/log/pulsar/broker.log

# Show recent authorization denials with context
grep "not authorized\|AuthorizationException" /var/log/pulsar/broker.log | tail -20

# Check current permissions on a namespace
pulsar-admin namespaces permissions <tenant>/<namespace>

# Check permissions on a specific topic
pulsar-admin topics permissions persistent://<tenant>/<namespace>/<topic>

# Verify authorization is enabled
grep "^authorizationEnabled" /path/to/broker.conf

# Check superUserRoles and proxyRoles configuration
grep -E "^superUserRoles|^proxyRoles" /path/to/broker.conf

# Verify the authorization provider in use
grep "^authorizationProvider" /path/to/broker.conf

# Check if wildcard matching is enabled
grep "^authorizationAllowWildcardsMatching" /path/to/broker.conf

# Confirm authentication failures metric exists but authorization does not
# Default Prometheus port is 8000; adjust if your deployment differs
curl -s http://<broker-host>:8000/metrics | grep -E "authentication_failures|authorization_failures"

How to diagnose it

flowchart TD
    A["Client receives AuthorizationException"] --> B{"Is authentication succeeding?"}
    B -- "No" --> C["Check pulsar_authentication_failures_total
Fix credentials, token, or TLS"]
    B -- "Yes" --> D["Grep broker log for 'not authorized'"]
    D --> E["Extract role, operation, topic from log line"]
    E --> F{"Is role in superUserRoles?"}
    F -- "Yes" --> G["Check for proxy role misconfiguration
or originalPrincipal issue"]
    F -- "No" --> H["Check namespace/topic permissions"]
    H --> I{"Does role have required action?"}
    I -- "No" --> J["Grant permission via pulsar-admin"]
    I -- "Yes" --> K["Check wildcard matching,
subscription authz, or transaction topic"]

Step 1: Confirm the failure is authorization, not authentication.

If pulsar_authentication_failures_total is increasing for the same client, the problem is identity validation (expired token, bad TLS cert, wrong auth plugin), not permissions. If authentication succeeds but operations still fail, it is authorization.

curl -s http://<broker-host>:8000/metrics | grep pulsar_authentication_failures_total

Step 2: Extract the role, operation, and target from the broker log.

Parse out three fields: the role (authenticated identity), the operation (LOOKUP, PRODUCE, CONSUME, etc.), and the target (topic or namespace). These three fields narrow the diagnosis to a specific permission check.

Step 3: Check the role’s current permissions.

# Namespace-level permissions
pulsar-admin namespaces permissions <tenant>/<namespace>

# Topic-level permissions (overrides namespace)
pulsar-admin topics permissions persistent://<tenant>/<namespace>/<topic>

The output maps roles to allowed actions (produce, consume, functions, sources, sinks, packages). If the role from the log is not listed, or is listed without the required action, that is the root cause.

Step 4: Verify proxy role configuration if a Pulsar Proxy is in the path.

When a proxy forwards client connections, it authenticates with its own role and passes the client’s original principal. If the proxy’s authenticated role is not listed in proxyRoles in broker.conf, the broker rejects the connection. The log message differs from a standard denial and typically reads cannot specify originalPrincipal when connecting without valid proxy role. This is frequent in Helm-deployed clusters where the proxy role configuration is inconsistent between proxy and broker configs.

Step 5: Check for version-specific permission changes.

Pulsar 3.0.0 made subscription-level permissions configurable via PIP-167. Before this change, having consume permission on a namespace implicitly allowed subscribing. After the change, if subscription authorization is enabled, the role needs explicit consume permission.

If the denial appeared after a version upgrade, review the release notes for authorization behavior changes.

Step 6: Check the transaction system topic if the client uses transactions.

Clients using Pulsar transactions need LOOKUP permission on persistent://pulsar/system/transaction_coordinator_assign. This produces a misleading AuthorizationException that looks like a regular topic permission issue. If the log line references this system topic, grant LOOKUP permission to the client role on the system namespace.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Broker log: AuthorizationException rateThe only direct signal for authorization failuresAny sustained non-zero count from a production client
pulsar_authentication_failures_totalDistinguishes authN from authZ. If also spiking, the problem is credentials, not permissionsSpike with reason label indicating token or cert failure
pulsar_rate_in / pulsar_rate_outAuthorization failures on producers cause throughput drops; on consumers, dispatch stallsSudden drop on specific topics while cluster aggregate looks normal
pulsar_active_connectionsConnections succeed (authentication passes) but operations fail (authorization denies)Stable or growing connection count with falling throughput
pulsar_broker_lookup_failuresLOOKUP authorization denials increment lookup failure countersElevated lookup failures correlating with authorization log entries

Fixes

Missing role permission on namespace or topic

The most common fix. Grant the required action to the role.

# Grant produce and consume on a namespace
pulsar-admin namespaces grant-permission <tenant>/<namespace> \
  --actions produce,consume \
  --role <role>

# Grant on a specific topic
pulsar-admin topics grant-permission persistent://<tenant>/<namespace>/<topic> \
  --actions produce,consume \
  --role <role>

Note: grant-permission does not accept comma-separated roles in a single --role flag. Each role must be granted separately.

Wildcard role matching

If you use role naming conventions (for example, svc-prod-*), wildcard matching avoids granting each role individually. Enable it in broker.conf:

authorizationAllowWildcardsMatching=true

Wildcards only work at the beginning or end of the role name. svc-* matches svc-prod-orders but not prod-svc-orders. The broker must be restarted for this change to take effect.

Proxy role configuration

Add the proxy’s authenticated role to proxyRoles in broker.conf on every broker:

proxyRoles=<proxy-role>

Also ensure the proxy’s own role does not appear in superUserRoles. A proxy role and a super-user role being the same is a security risk and can cause unexpected authorization behavior.

Transaction system topic permission

Grant LOOKUP on the system namespace to roles that need transactions:

pulsar-admin namespaces grant-permission pulsar/system \
  --actions lookup \
  --role <role>

Subscription permission (Pulsar 3.0+)

If subscription authorization is enabled and consumers cannot subscribe, verify the role has explicit consume permission. The subscription permission check was added in Pulsar 3.0 and may catch teams upgrading from 2.x where consume permission implicitly covered subscription.

Revoke stale permissions

During permission reconciliation, revoke access for roles that should no longer have it:

# WARNING: immediately revokes access for the specified role.
# Confirm with the application owner before running.
pulsar-admin namespaces revoke-permission <tenant>/<namespace> \
  --role <role>

Prevention

Log-based authorization alerting. Since there is no metric, build a log-derived counter. A log forwarder (Filebeat, Fluentd, Vector) can count AuthorizationException or not authorized lines per role and emit a custom metric or alert. Alert on sustained non-zero rates from production client IPs. A burst from unknown IPs warrants a security investigation.

Distinguish authN from authZ in your runbook. When a client reports access errors, check pulsar_authentication_failures_total first. If it is not increasing, the problem is authorization. This prevents wasting time on credential debugging when the issue is a missing role grant.

Codify permission grants in deployment pipelines. The most common cause is a new application deployed without the required grant-permission call. Include permission grants as a step in deployment automation, alongside topic and subscription creation.

Review permissions periodically. Stale permissions accumulate. Use pulsar-admin namespaces permissions as part of a quarterly access review. Revoke roles for decommissioned services.

Monitor for CVE-related behavior changes. If you patched CVE-2024-29834 (affects 2.7.1 through 3.2.1, fixed in 3.0.4+) or CVE-2024-28098 (affects 2.7.1 through 3.2.0, fixed in 3.0.3+), previously-allowed operations on partitioned topics and topic-level policies may now correctly fail with AuthorizationException. Applications that relied on the bypassed authorization need explicit permission grants after the fix.

Keep proxy and broker configs synchronized. In Helm deployments, ensure the proxy role in the broker config matches the proxy’s authenticated role. Use config maps or templating to keep these consistent across all brokers.

How Netdata helps

  • Netdata collects Pulsar broker metrics at per-second resolution, including pulsar_authentication_failures_total. While authorization failures have no native metric, Netdata’s log monitoring can surface AuthorizationException patterns from broker logs alongside the metrics timeline, letting you correlate a permission denial with throughput drops or connection behavior in the same view.
  • The pulsar_rate_in and pulsar_rate_out metrics help identify which topics are affected when an authorization failure blocks producers or consumers. A sudden rate drop on a specific namespace while cluster-wide throughput remains stable is a strong indicator of a targeted permission issue.
  • pulsar_active_connections tells you whether the client is connected at all. If connections are stable but throughput has dropped, the problem is downstream of authentication. Correlating connection count with throughput narrows the diagnosis.
  • pulsar_broker_lookup_failures can reveal LOOKUP authorization denials, which are the first operation a client performs. Elevated lookup failures coinciding with authorization log entries confirm the permission scope.
  • Netdata’s anomaly detection (ML-based) can flag unusual patterns in connection churn or throughput that may correlate with permission changes, even when no explicit authorization metric exists.