Your dashboards look fine. Messages are flowing, publish latency is normal, consumers are draining backlog. But the new service you just deployed cannot start: its producer hangs or fails trying to connect to its topic. Restarting it does not help. Existing clients are unaffected.
This is the classic Pulsar grey failure. Topic lookup is how every producer and consumer discovers which broker owns its topic. Existing connections keep working because they already resolved ownership. New traffic fails when lookups cannot resolve. Process health checks, message rates, and bookie metrics can all look healthy until a deploy, restart, scale event, or broker bounce forces mass re-lookup.
Use this to confirm lookup failure, separate normal redirects from real failures, and isolate the two usual root causes: metadata store latency and bundle ownership churn.
What this means
Before a producer or consumer can send or receive, the client performs a topic lookup: it asks a broker which broker owns the namespace bundle containing that topic. The broker answers with an address, or with a redirect telling the client to ask again. Only after lookup resolves does the client open its connection and start producing or consuming.
Broker Prometheus metrics commonly expose this path as:
pulsar_broker_lookup_failures: lookups that failed outrightpulsar_broker_lookup_answers: lookups answered successfullypulsar_broker_lookup_redirects: lookups answered with a redirectpulsar_broker_lookup_pending_requests: lookups waiting to be resolved
Metric names vary by version and configuration. If these do not appear in /metrics, verify the metric surface for your version before declaring the lookup path healthy. Absence of the metric is not evidence of health.
The failure versus redirect distinction matters. A redirect during a bundle unload is expected: ownership is moving and the client is told to redo lookup. A failure means the broker could not resolve ownership, usually because it could not read or write bundle ownership state in the metadata store, because ownership was in flux, or because no broker currently held the bundle.
flowchart LR C[New producer or consumer] -->|topic lookup| B[Broker] B -->|bundle ownership read| M[Metadata store] M -->|slow or failing| B B -->|lookup failure| C B -.->|redirect during bundle unload, expected| C X[Existing clients] -->|already resolved, keep working| T[Topic owner broker]
Existing connections bypass lookup entirely, so throughput dashboards hide the blast radius until something forces new lookups.
Common causes
| Cause | What it looks like | First thing to check |
|---|---|---|
| Metadata store latency or unavailability | Lookup failures rise across brokers, ZooKeeper latency rises before broker symptoms, session expiry events appear in broker logs | ZK latency and health with echo stat | nc {zk-host} 2181 if four-letter words are enabled |
| Bundle ownership churn | Failures track high unload activity and repeated ownership acquire or release logs | Bundle unload rate outside maintenance |
| Bundle stuck unloading or unassigned | Topics in one namespace bundle fail indefinitely while other namespaces are fine | Whether failures concentrate on one namespace; inspect bundle ownership through the admin API |
| GC death spiral on a broker | Long GC pauses, ZK session expiry, bundle loss, reconnect storm, more GC | Broker GC logs and heap or direct memory pressure |
| Broker startup race | A freshly started broker fails lookups for a window after startup | Whether failures align with a broker restart and whether they recover |
| Broker overload rejecting lookups | Pending lookups climb, connections are throttled, active connections are high | Pending lookup requests and throttled connection metrics |
Quick checks
These are read-only, assuming your admin and metrics endpoints are reachable. If authentication is enabled, add credentials; a 401 from an admin endpoint is not proof the broker is down.
# 1. Pull lookup metrics from each broker
curl -s http://{broker-host}:8080/metrics | grep pulsar_broker_lookup
# 2. Compute failure ratio over several samples
# ratio = lookup_failures / (lookup_failures + lookup_answers)
# sustained above 1% for more than 5 minutes is actionable
# 3. Check metadata store health from brokers and from ZK itself
curl -s http://{broker-host}:8080/metrics | grep -i zookeeper
echo stat | nc {zk-host} 2181
# 4. Check bundle unload activity
curl -s http://{broker-host}:8080/metrics | grep pulsar_lb_unload_bundle_total
# 5. Check pending lookups: unresolved lookup backlog
curl -s http://{broker-host}:8080/metrics | grep pulsar_broker_lookup_pending_requests
# 6. Watch ZooKeeper watch counts during reconnect storms
echo wchs | nc {zk-host} 2181
# 7. Check broker process health; auth failures are not the same as broker failure
curl -sf http://{broker-host}:8080/admin/v2/brokers/health
The ZK four-letter commands require 4lw.commands.whitelist or equivalent access on many deployments. If stat or wchs returns nothing, that is an allowlist problem, not proof ZK is healthy.
How to diagnose it
Work in order. Each step removes one layer.
Confirm the ratio, not the raw count. Compute
lookup_failures / (lookup_failures + lookup_answers)per broker. A sustained ratio above 1% for more than 5 minutes is a real incident. A few failures per hour can be background noise during unloads.Separate failures from redirects. Redirects rising while failures stay flat, especially during bundle unloads or a rolling restart, is normal ownership transfer. Brief redirection during unload is expected. Only failures, or redirects that never settle, indicate a deeper problem.
Check metadata store latency first. ZK degradation is a leading indicator: it degrades before broker symptoms become obvious. As a heuristic, average latency under 10 ms is healthy, sustained latency above 50 ms is a warning, and sustained latency above 100 ms often precedes session expiry and fencing. If ZK latency is elevated and lookup failures rise across multiple brokers at once, treat the metadata store as root cause until proven otherwise. Check transaction log disk, CPU, request latency, and watch counts.
Check bundle ownership churn. If ZK is healthy, look at unload rate and ownership logs. A sustained unload rate above roughly 1 per minute for more than 10 minutes outside maintenance means the load balancer is moving bundles often enough to invalidate lookup answers. Correlate with broker CPU, heap, direct memory, and GC to see whether a resource-exhausted broker is shedding bundles it cannot hold.
Localize the failures. If failures concentrate in one namespace, suspect one bundle stuck in unloading or left unassigned. Every topic hashing into that bundle can fail lookup while the rest of the cluster looks fine. Confirm by mapping failing topics to their bundle and checking ownership state through the admin API or broker logs. Some releases have had stuck-unload bugs; check your exact version and release notes before assuming.
Check for the GC death spiral. On any broker with elevated failures, read GC logs. The pattern is heap or direct memory pressure, long pause, ZK session expiry, fencing, bundle loss, reconnect storm, more metadata load, then another pause. Lookup failures are the downstream symptom. Oscillation confirms it: the broker loses ownership, recovers, then loses it again.
Check broker overload. If
pulsar_broker_lookup_pending_requestsclimbs and does not drain, the broker is queuing lookups faster than it can resolve them. Correlate with active connections and throttled connections. Saturation calls for capacity or bundle redistribution, not lookup tuning.Check whether failures follow broker startup. If a broker fails lookups for a window right after startup, lookup requests may be arriving before ownership services are ready. If failures never recover on an older version, restart the broker to restore service and check release notes for startup lookup races before the next restart.
Metrics and signals to monitor
| Signal | Why it matters | Warning sign |
|---|---|---|
pulsar_broker_lookup_failures vs pulsar_broker_lookup_answers | Core symptom: new clients cannot resolve topics | Failure ratio above 1% sustained for 5 minutes |
pulsar_broker_lookup_redirects | Separates normal ownership transfer from failure | Redirects spike without unload activity, or never settle |
pulsar_broker_lookup_pending_requests | Lookup queue depth; growth means resolution is behind | Sustained growth that does not drain |
| Metadata store latency | Root cause one; degrades before broker symptoms | Avg above 50 ms sustained; above 100 ms critical |
| Broker logs for ZK session expiry and fencing | Session loss fences the broker and orphans bundles | Any unexpected session expiry or fence event |
pulsar_lb_unload_bundle_total | Root cause two: ownership churn invalidates lookup answers | Sustained unload outside maintenance |
| Broker GC pause times | Long pauses expire sessions and trigger fencing cascades | Repeated pauses above 1 second |
pulsar_broker_throttled_connections and active connections | Broker protecting itself under load | Sustained throttling or connection count near limits |
Fixes
Metadata store latency
Reduce load on ZooKeeper and fix the resource constraint. Check the transaction log disk first; a saturated or degraded disk is a common cause. Check watch counts with echo wchs | nc {zk-host} 2181 because reconnect storms register large numbers of watches and compound latency. If an ensemble member is unhealthy, restore quorum before touching brokers.
Do not restart brokers to “fix” ZK. Brokers reconnect on their own once ZK recovers, and restarting them adds reconnection load to an already saturated metadata store.
If broker GC pauses are causing session expiries, fix heap sizing, direct memory pressure, or the allocation source. Raising the ZK session timeout can mask GC and slow real failure detection; treat it as a temporary lever only.
Bundle ownership churn
Find why the load balancer is moving bundles. Sustained unloads outside maintenance mean either a broker cannot hold its load or shedding thresholds are too aggressive for the workload.
If two brokers oscillate a hot bundle, with rapid acquire and release in logs and constant client reconnects, stop the loop before tuning. Do not flip load manager classes ad hoc during an incident unless your version documents that setting as dynamic and safe; changing load balancing behavior cluster-wide can disable rebalancing or trigger movement at the worst time. Use it only as an emergency brake with change control, then restore the original load manager after the cluster is stable.
A hot bundle containing one very heavy topic will keep triggering movement regardless of thresholds. Split that topic across partitions or move it into its own namespace if the workload allows.
Bundle stuck in unloading
If one bundle never completes unload, lookups for every topic in it can fail indefinitely. Restore service first: identify the owning broker, confirm the bundle state, and restart only the broker holding the stuck bundle if that is the supported recovery path for your version. Then check release notes and upgrade if you are on a version with a known stuck-unload fix. Do not restart the whole cluster for one stuck bundle.
Broker overload
If pending lookups climb because the broker is saturated, the fix is capacity: redistribute bundles to less loaded brokers or add brokers. Confirm saturation with active connections, throttled connections, CPU, heap, direct memory, and GC before changing lookup or connection limits.
Prevention
- Alert on the ratio, not the count.
lookup_failures / (lookup_failures + lookup_answers)above 1% sustained for 5 minutes catches the grey failure before a deployment wave turns it into an outage. - Monitor metadata store latency as a first-class signal. ZK degradation is the leading indicator for the worst Pulsar cascades. Do not wait for broker symptoms.
- Baseline bundle unload rates. Know normal rolling restart behavior versus steady state so a thrashing load balancer stands out immediately.
- Keep clients and brokers on supported versions. Lookup-path behavior has changed over time, including client handling of transient bundle-unload errors and broker startup behavior. Verify fixes against your exact release rather than assuming a version is safe.
- Load test the lookup path, not just throughput. Restart exercises and scaling events are where lookup failures hurt. Include client reconnect storms in chaos testing and watch the lookup ratio during them.
- Know your client retry behavior. Client handling of lookup failure varies by implementation and version. Some clients retry transient lookup failures; others surface them to the application. Test what your deployed client does and make sure application connection logic retries with backoff.
How Netdata helps
- Netdata collects the broker Prometheus endpoint, so lookup answers, failures, redirects where exposed, and pending lookup counts are charted per broker with no extra instrumentation. Per-broker views matter because a cluster-wide average can hide one bad broker.
- Per-second granularity matters here: lookup failure bursts during bundle unloads and reconnect storms are short, and slow scrape intervals can average them away.
- Correlating lookup failures with
pulsar_lb_unload_bundle_totalon one timeline separates expected redirect blips from real ownership churn. - Correlating lookup failures with ZooKeeper health, broker logs, and GC behavior shows whether metadata store latency or broker fencing is driving the incident.
- Long retention lets you baseline normal unload and redirect behavior during maintenance windows, which makes steady-state anomalies obvious.