$ guides / apache-pulsar / apache-pulsar-broker-down ▌

Operations Guides

Apache Pulsar broker down: telling a dead broker from a fenced one

Your alerting fired: the broker’s HTTP admin endpoint on :8080 has been unreachable for more than two minutes, and the broker was previously running, so this is not a fresh deploy or a rolling restart. Pager says “broker down.” That phrase hides two very different incidents.

The first is a hard death: the JVM crashed, got OOM-killed, or the host failed. The process is gone and the OS will tell you so in seconds. The second is a fenced broker: the process is alive, maybe even accepting TCP connections, but it has lost its ZooKeeper session and with it ownership of every namespace bundle it served. From the client’s perspective the broker is down. From the process table’s perspective it is fine. The fix, the blast radius, and the forensics are completely different for each.

This guide covers how to tell the two apart, what each state implies for your data path, and which signals confirm the diagnosis before you restart anything.

What this means

In Pulsar, brokers are stateless serving processes that own namespace bundles: hash-range slices of the topic namespace. Bundle ownership is recorded in the metadata store (ZooKeeper in most 3.x deployments) via ephemeral nodes tied to the broker’s ZK session. That design creates a sharp distinction:

Dead broker: process terminated. The ZK session dies with it, ephemeral nodes are removed, and surviving brokers take over its bundles. Recovery is automatic. Your job is finding why the process died (OOM, host failure, kill signal) and whether it will die again on restart.
Fenced broker: process alive, ZK session expired (long GC pause, ZK latency storm, network partition). The broker loses topic ownership; topics get fenced while ownership moves. A broker in this state is “up” by every process check while serving nothing. Worse, it can oscillate: lose ownership, regain it, lose it again, in the GC death spiral pattern.
Degraded but owned: process alive, session intact, but the broker is unhealthy (direct memory exhausted, GC spiral in progress, stuck BookKeeper client). /metrics may even respond. Only a deeper health check catches this.

The basic liveness signal (HTTP endpoint on :8080 responding) only catches the first case reliably. A binary process check is necessary but not sufficient.

flowchart TD
    A[Alert: :8080 unreachable or clients failing] --> B{Process running?}
    B -- No --> C[Dead broker]
    B -- Yes --> D{ZK session intact?}
    D -- No --> E[Fenced broker: lost topic ownership]
    D -- Yes --> F{Health check passes?}
    F -- No --> G[Degraded: owns bundles but cannot serve]
    F -- Yes --> H[Healthy: look elsewhere, e.g. bookies or ZK]
    C --> I[Check OOM signature, exit reason, host health]
    E --> J[Check GC pauses, ZK latency, session expiry in logs]
    G --> K[Check direct memory, thread pools, BK client]

Common causes

Cause	What it looks like	First thing to check
Direct memory exhaustion (OOM)	Process dead or hung; heap metrics looked fine before death; RSS far above heap	Broker logs for `OutOfDirectMemoryError` or “Direct buffer memory”
GC death spiral leading to session expiry	Process alive, :8080 slow or unreachable, repeated bundle ownership churn in logs	GC logs / `jstat -gc`, then ZK session events
ZooKeeper latency storm	Multiple brokers affected at once, ZK latency elevated before broker symptoms	`echo stat \| nc <zk-host> 2181`, ZK disk and CPU
Host or network failure	Process gone, no JVM error in logs, other services on host also affected	Host logs, network reachability, `dmesg`
OOM killer (cgroup or kernel)	Process gone, exit without JVM stack trace	`dmesg` / journal for oom-kill entries, container memory limits vs heap + direct memory
Rolling restart mistaken for outage	Endpoint down briefly, broker re-registers, uptime low	Broker uptime and deployment history (the “previously running > 2 min” condition filters this)

Quick checks

All of these are read-only.

# 1. Is the process alive at all?
pgrep -af PulsarBroker

# 2. Is the HTTP listener dead or just the health path?
curl -sf http://<broker-host>:8080/metrics > /dev/null && echo "UP" || echo "DOWN"

# 3. Deeper health: does the broker exercise the full produce/consume path?
curl -sf http://<broker-host>:8080/admin/v2/brokers/health

# 4. Is the metadata-store session intact? Pulsar exposes no zookeeper_connected Prometheus metric; broker log events are the signal (see #7)

# 5. GC pressure right now (if process alive)
jstat -gc <broker-pid> 1000 5

# 6. Direct memory vs heap gap
grep VmRSS /proc/<broker-pid>/status

# 7. Recent death signatures
grep -E "OutOfDirectMemoryError|Direct buffer memory|Session expired|Connection loss" /var/log/pulsar/broker.log | tail -20

# 8. Kernel OOM killer
dmesg -T | grep -i "killed process" | tail

# 9. Is the cluster seeing the broker as gone?
curl -s http://<other-broker>:8080/admin/v2/brokers/<cluster>

Note on check 3: GET /admin/v2/brokers/health is not a cheap ping. It creates a producer and a non-durable subscription on a system health-check topic, sends a message, reads it back, and tears down. That is why it catches fenced and degraded brokers: it exercises the full path through the managed ledger and BookKeeper. Do not point a load balancer at it with a one-second interval on a loaded cluster; concurrent calls have historically exposed race conditions in older 2.x versions, and each call costs a produce/consume round trip.

How to diagnose it

Confirm the alert scope. Check whether other brokers also lost their :8080 endpoints or ZK sessions. Multiple brokers affected simultaneously points at ZooKeeper or shared infrastructure, not at individual broker death. A single broker points local.
Dead or alive? pgrep -af PulsarBroker. If the process is gone, you are in the dead-broker path: skip to step 5. If it is alive, the liveness alert already told you something important: the process is up but not serving.
Alive: check the metadata-store session. Pulsar does not expose a ZooKeeper-connection Prometheus metric; the signal is Session expired / Connection loss lines in the broker log (corroborate server-side with echo stat | nc <zk-host> 2181). Such an event means the broker was fenced: it lost bundle ownership and its topics moved to surviving brokers. Correlate the timestamp with GC pauses (jstat history, GC logs) and ZK latency. A GC pause longer than the ZK session timeout is the classic trigger.
Alive with session intact: run the deep health check. GET /admin/v2/brokers/health. A failure here with a live session means the broker owns bundles but cannot complete a produce/consume round trip: suspect direct memory exhaustion in progress, a stuck BookKeeper client, or thread pool saturation. Check RSS vs heap, and Netty allocator stats via /admin/v2/broker-stats/allocator-stats/default (valid for the default and ml-cache allocators).
Dead: find the death signature. In order: broker log tail for OutOfDirectMemoryError or OutOfMemoryError; dmesg for the kernel OOM killer; container runtime events for cgroup OOM or eviction; host logs for hardware or power events. “Process gone with no JVM error” plus an oom-kill entry means your memory accounting was wrong: heap + MaxDirectMemorySize + JVM overhead exceeded the limit. Heap is usually monitored, direct memory is not, and the sum is what the kernel enforces.
Verify client impact separately from broker state. A fenced broker’s topics are already being served by other brokers after bundle reassignment; clients see reconnection blips, not an outage, assuming the rest of the cluster is healthy. Check pulsar_lb_unload_bundle_total on surviving brokers and lookup failures (pulsar_broker_lookup_failures) to confirm the handover completed. Sustained lookup failures after the event mean the problem has moved to the metadata store or ownership churn, not the dead broker.
Only then restart. A dead broker with an explained cause (host failure, fixed OOM) can be restarted or replaced. A fenced broker oscillating in a GC spiral should not be restarted blindly into the same configuration; capture GC evidence first, because the restart erases it and the spiral will recur.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
:8080 endpoint reachability (process liveness)	The binary dead/alive signal	Unreachable > 2 min on a previously running broker: page
Broker metadata-store session events (log `Session expired` / `Connection loss`)	Distinguishes fenced from dead on a live process	Session events appearing, or repeated flapping
`/admin/v2/brokers/health` result	Exercises produce/consume path; catches degraded-but-alive	Non-200 while `/metrics` still responds
GC pause duration/frequency (JMX or GC logs)	Leading cause of session expiry and fencing	Pauses approaching the ZK session timeout
Process RSS minus heap	Proxy for direct memory growth (Pulsar 2.11+ also exposes `jvm_memory_direct_bytes_used` on `/metrics`)	Gap growing steadily; RSS near container limit
`pulsar_lb_unload_bundle_total`	Confirms ownership handover after a broker is fenced or dies	Unloads without a corresponding broker event; thrashing rate
`pulsar_broker_lookup_failures`	Shows whether clients can still find topic owners	Failure ratio > 1% sustained after the event
`pulsar_active_connections`	Reconnect storms after fencing; also feeds direct memory pressure	Sharp drop (fencing) then spike (thundering herd reconnect)

On Pulsar 3.0+ there is also a passive option: PIP-271 added a pulsar_health gauge at /metrics with values 1 (healthy/success), 0 (failed), -1 (unknown). It is disabled by default (healthCheckMetricsUpdateTimeInSeconds=-1); set it to a positive interval in seconds (e.g. 30-60) to make the broker run the full health check on that schedule and update the gauge. This gives you health state without paying for a full produce/consume health check on every scrape.

Fixes

Dead broker: direct memory or heap OOM

Restart the process (or let the orchestrator do it), then fix the accounting before it happens again. Pulsar’s broker defaults are -Xms2g -Xmx2g -XX:MaxDirectMemorySize=4g. The container or host memory limit must cover heap + direct + roughly 20% JVM overhead. If RSS-minus-heap was climbing before death, increase MaxDirectMemorySize headroom or reduce connection count and message sizes. If heap itself was the ceiling, check managedLedgerCacheSizeMB and topic count per broker.

Fenced broker: GC-induced session expiry

Do not restart first. Capture jstat history or GC logs, then address the pause source: heap sizing, cache sizing, or a memory leak in functions/connectors. As a stopgap you can raise zooKeeperSessionTimeoutMillis to buy time, but treat that as masking, not fixing: longer timeouts also slow real failure detection.

Fenced broker: ZooKeeper-side latency storm

The broker is a victim, not the cause. Check ZK server disk (transaction log), CPU, and watch counts (echo wchs | nc <zk-host> 2181). Note that on ZooKeeper 3.5+, four-letter words like wchs work only if whitelisted via 4lw.commands.whitelist; if they are blocked, use mntr via the admin server or your ZK metrics instead. Do not mass-restart brokers: they reconnect on their own once ZK recovers, and restarting adds reconnection load to an already saturated metadata store.

Topics stuck fenced after ownership moves

If producers report fenced-topic errors long after the broker event, the fence release is lagging. The topicFencingTimeoutSeconds broker config controls how long a topic stays fenced before being force-closed; the default is 0, which disables forced closing of fenced topics entirely. Setting it low (e.g. 5 seconds) releases fencing faster but carries data-consistency risk, and fenced-topic bugs have been reported across multiple 2.x/3.x lines.

Host or infrastructure failure

Replace the host and let the cluster rebalance. Confirm surviving brokers absorbed the bundles cleanly via bundle counts and lookup failure rates before calling it done.

Prevention

Monitor direct memory explicitly. Pulsar 2.11+ exposes jvm_memory_direct_bytes_used and jvm_memory_direct_bytes_max on /metrics; on older versions use JMX (java.nio:type=BufferPool,name=direct) or track RSS-minus-heap. Alert on the trend, not just the ceiling.
Alert on GC pauses relative to the ZK session timeout, not on absolute pause length. A 10-second pause is harmless with a 30-second session timeout and fatal with a 10-second one.
Size container limits honestly: heap + MaxDirectMemorySize + JVM overhead, with headroom. This eliminates the most common “broker died with no log” incident.
Treat ZK latency as a leading indicator. Sustained latency above 10ms is an early warning for fencing events cluster-wide; above 100ms, failure is minutes away.
Gate liveness pages with “previously running > 2 min” so rolling restarts and cold starts do not page you.
Run controlled broker kills in staging so the team knows what a normal bundle handover looks like in metrics before a real one happens.

How Netdata helps

Netdata’s Pulsar collector scrapes the broker’s Prometheus endpoint on :8080, so broker liveness, pulsar_zookeeper_connected, bundle unload counters, and lookup failures land in one place at per-second resolution.
Host-level metrics from the same agent (RSS per process, dmesg OOM events, disk and NIC health) let you see the RSS-minus-heap gap and kernel OOM kills next to Pulsar’s own view of the broker, which is exactly the correlation that separates dead from fenced.
JVM and GC charts alongside ZK session state make the GC-spiral-to-fencing chain visible in one timeline instead of two tools.
Bundle unload spikes correlated with connection count drops confirm ownership handover after a broker event, telling you whether the cluster healed or the problem moved.