$ guides / apache-pulsar / apache-pulsar-authentication-failures ▌

Operations Guides

Apache Pulsar authentication failures: expired tokens, wrong credentials, and brute force

Authentication failures in Apache Pulsar surface through pulsar_authentication_failures_total and AuthenticationException entries in broker logs. In a stable environment, this counter sits near zero. When it spikes, the temporal pattern matters more than the absolute volume: a single burst from a known client after a credential rotation is normal; sporadic low-rate failures from production IPs point to misconfiguration; a sustained flood from unknown sources signals brute force or credential compromise.

Each failure closes the TCP connection. The client reconnects, authenticates again, and either succeeds or fails. This loop consumes file descriptors, Netty direct memory buffers, and CPU on both sides. Brokers check auth data expiry every authenticationRefreshCheckSeconds (default 60 seconds). If the client supports auth refreshing, the broker sends a CommandAuthChallenge. If not, and the credential is expired, the broker disconnects. An expired token does not cause a single failure; it causes a continuous reconnect storm until the credential is refreshed.

The downstream symptoms (connection churn, rising direct memory, intermittent producer or consumer errors) are often what operators notice first. Check pulsar_authentication_failures_total before investigating connection or memory metrics.

The auth_method label distinguishes TLS, JWT, and OAuth2 failure modes. The provider_name label identifies the authentication provider class. The reason label carries the specific failure cause, though its enumerated values are not formally documented and may vary by provider and deployment.

Common causes

Cause	What it looks like	First thing to check
Expired JWT token	Sudden spike from one client or region, `auth_method=JWT`. Reconnect storm follows.	Token expiry timestamp and refresh cycle config
Expired TLS certificate	All TLS clients fail simultaneously. `auth_method=TLS`. Broker logs show handshake errors.	`openssl s_client` against broker TLS port
Wrong credentials (new deploy)	Failures start at deployment time, concentrated on one client identity.	Client auth configuration vs. broker provider settings
External auth provider down	Failures across all OAuth2 or OIDC clients. No specific credential error.	Connectivity to the OIDC or OAuth2 provider endpoint
Proxy auth forwarding misconfigured	Expired tokens not enforced on proxied connections. Clients continue with stale credentials.	`forwardAuthorizationCredentials` and `authenticateOriginalAuthData` settings
Brute force or scanning	Sustained flood of failures from many unknown source IPs. No corresponding success rate increase.	Source IP distribution from broker logs

Quick checks

# Check current auth failure counts with label breakdown
curl -s http://<broker-host>:8080/metrics | grep pulsar_authentication_failures_total

# Compare failures to successes for ratio analysis
# pulsar_authentication_success_total exists (labels: provider_name, auth_method)
curl -s http://<broker-host>:8080/metrics | grep -E "pulsar_authentication_(failures|success)_total"

# Check expired token counter (metric name pulsar_expired_token_total since Pulsar 2.11.0; pulsar_expired_token_count before)
curl -s http://<broker-host>:8080/metrics | grep pulsar_expired_token_total

# Inspect recent AuthenticationException entries in broker logs
grep "AuthenticationException" /var/log/pulsar/broker.log | tail -50

# Check TLS certificate expiry on the broker TLS port
openssl s_client -connect <broker-host>:6651 2>/dev/null | openssl x509 -enddate -noout

# Monitor connection churn (names: pulsar_connection_created_total_count / pulsar_connection_closed_total_count)
curl -s http://<broker-host>:8080/metrics | grep -E "pulsar_connection_(created|closed)"

# Check broker process health
curl -sf http://<broker-host>:8080/admin/v2/brokers/health

# Group auth failures by auth_method to isolate the affected provider
curl -s http://<broker-host>:8080/metrics | grep pulsar_authentication_failures_total | sed 's/.*auth_method="\([^"]*\)".*/\1/' | sort | uniq -c

How to diagnose it

flowchart TD
    A["Spike in auth failures"] --> B{"Rate pattern?"}
    B -->|"Sudden spike, one client"| C["Expired token or cert"]
    B -->|"Sporadic, low rate"| D["Wrong credentials"]
    B -->|"Sustained flood, unknown IPs"| E["Brute force"]
    C --> C1["Check auth_method label"]
    D --> D1["Check provider_name label"]
    E --> E1["Check source IP in broker logs"]
    C1 --> C2["Verify token expiry and refresh config"]
    D1 --> D2["Review client auth configuration"]
    E1 --> E2["Apply network-level controls"]

Group by reason label. The reason label distinguishes credential validation failures, provider errors, and expired credentials. Group failures by this label to identify the dominant failure mode.
Correlate with broker logs. AuthenticationException entries provide the full error message including the client identity. Use these to identify which client identities or source IPs are generating failures.
Check for proxy involvement. If you run Pulsar Proxy in front of brokers, verify that forwardAuthorizationCredentials=true is set in proxy.conf and authenticateOriginalAuthData=true is set in broker.conf. Without both, token expiry is not enforced on proxied connections. This is a well-documented operational gap that has affected multiple Pulsar versions.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`pulsar_authentication_failures_total`	Primary indicator of auth problems. Near zero in healthy clusters.	Any sustained rate above zero, or a spike from baseline
`pulsar_authentication_failures_total` by `auth_method`	Isolates which auth provider is failing (TLS vs JWT vs OAuth2)	One method failing while others succeed
`pulsar_authentication_failures_total` by `reason`	Distinguishes expired credentials from validation errors and provider failures	Single reason value dominating the failures
`pulsar_expired_token_total`	Counts expired tokens separately from general auth failures (Pulsar 2.11.0 and later)	Non-zero and growing indicates token lifecycle issue
`pulsar_connection_created_total_count` vs `pulsar_connection_closed_total_count`	Auth failures close connections, driving reconnect churn	Created count significantly exceeds closed count over time
`pulsar_active_connections`	Reconnect storms inflate connection count and file descriptor usage	Sudden spike or oscillating pattern
TLS certificate days to expiry	Expired certs cause total auth breakdown for TLS clients	Less than 30 days remaining

Fixes

Expired tokens

For JWT tokens, regenerate and distribute new tokens with appropriate expiry windows. Short-lived tokens (1 to 24 hours) are more secure but require automated rotation infrastructure. Long-lived tokens (7 days or more) are simpler to operate but increase the blast radius of a compromised token.

Verify that authenticationRefreshCheckSeconds is set appropriately for your token lifetime. The default of 60 seconds means the broker checks expiry once per minute. If your tokens are short-lived, clients must support the auth refresh challenge flow to avoid disconnection.

For OAuth2 or OIDC, verify that the token endpoint is reachable and returning valid tokens. Check client credentials, scopes, and the issuer URL configuration against the broker’s configured provider.

Expired TLS certificates

Rotate certificates before expiry. An expired broker certificate causes immediate TLS handshake failures for all TLS clients. After rotation, existing connections may need to re-establish if the certificate change is not transparent to the TLS session.

Proxy auth forwarding misconfiguration

This is the most insidious auth failure mode because it can result in silent acceptance of expired credentials, not rejection. If you run Pulsar Proxy:

Set forwardAuthorizationCredentials=true in proxy.conf.
Set authenticateOriginalAuthData=true in broker.conf.
Both settings are required. Either alone is insufficient.

Without both settings, the broker trusts the proxy’s authentication result and does not independently validate token expiry. Clients with expired tokens continue operating until the next full reconnect, at which point they may fail unpredictably.

CVE-2023-31007 documented a related gap where clients could stay connected after their auth data expired (affected: Pulsar through 2.9.4, 2.10.0-2.10.3, and 2.11.0; fixed: 2.9.5, 2.10.4, and 2.11.1). Later Pulsar releases improved enforcement, but the proxy configuration requirement persists. Always verify both settings after any proxy or broker configuration change.

Pulsar 4.2.0 (PR #25179) closes this gap: the proxy now closes the client connection immediately when credentials expire and forwardAuthorizationCredentials is disabled.

Wrong credentials after deployment

Identify the affected client from broker logs or the provider_name label. Compare the client auth configuration against the broker’s configured auth provider. Common issues include wrong token file path or contents, mismatched auth plugin class name between client and broker, incorrect trust store or certificate chain for TLS, and wrong tenant or namespace in the auth data.

External auth provider unavailable

If you use an external auth provider (OIDC, OAuth2, LDAP), auth failures spike when the provider is unreachable. Check network connectivity to the provider endpoint, verify the provider service is healthy, and check for DNS resolution issues. The reason label will show provider-related errors rather than credential validation errors.

Brute force or unauthorized access

Pulsar does not include built-in rate limiting for authentication failures. There is no authenticationMaxFailedAttempts or equivalent setting in the broker configuration. To protect against brute force:

Restrict broker network access to known client IP ranges using network ACLs or security groups.
Deploy a WAF or reverse proxy with rate limiting in front of the Pulsar binary protocol.
Use fail2ban or equivalent log-scanning tools to detect and block source IPs generating sustained auth failures from broker logs.
Monitor the auth failure rate by source IP by correlating with broker logs, since the Prometheus metric does not include source IP as a label.

Prevention

Automate token rotation. Deploy tokens with automated refresh mechanisms. Never rely on manual rotation for production credentials.
Monitor certificate expiry. Track days-to-expiry for all TLS certificates. Alert at 30 days and again at 7 days.
Audit proxy configuration after every change. The forwardAuthorizationCredentials and authenticateOriginalAuthData settings are easy to miss during proxy reconfiguration and the failure mode is silent.
Include auth checks in deployment pipelines. Verify client auth configuration before deploying to production. A staging environment with the same auth providers catches most misconfiguration.
Restrict broker network exposure. Brokers should only be reachable from known client networks. Direct internet exposure invites brute force.
Monitor auth failure rate as a security signal. A sudden increase from unknown sources can be the first indicator of credential compromise or network scanning.

How Netdata helps

Netdata collects pulsar_authentication_failures_total at per-second resolution. A spike that resolves in seconds is a rotation event; a sustained rate over minutes from the same source is an attack. 15 to 30 second scrape intervals can miss burst patterns that cause minutes of reconnect churn.
The auth_method, provider_name, and reason labels are preserved, allowing you to slice auth failures by provider and failure mode without custom queries or log parsers.
ML anomaly detection on the auth failure rate catches subtle increases that would not trigger a static threshold but indicate a growing misconfiguration or slow credential leak.
Correlating auth failures with connection churn metrics (pulsar_connection_created_total_count, pulsar_active_connections) in the same view reveals the downstream impact on broker resource consumption.