Your alerting fired: the broker’s HTTP admin endpoint on :8080 has been unreachable for more than two minutes, and the broker was previously running, so this is not a fresh deploy or a rolling restart. Pager says “broker down.” That phrase hides two very different incidents.

The first is a hard death: the JVM crashed, got OOM-killed, or the host failed. The process is gone and the OS will tell you so in seconds. The second is a fenced broker: the process is alive, maybe even accepting TCP connections, but it has lost its ZooKeeper session and with it ownership of every namespace bundle it served. From the client’s perspective the broker is down. From the process table’s perspective it is fine. The fix, the blast radius, and the forensics are completely different for each.

This guide covers how to tell the two apart, what each state implies for your data path, and which signals confirm the diagnosis before you restart anything.

What this means

In Pulsar, brokers are stateless serving processes that own namespace bundles: hash-range slices of the topic namespace. Bundle ownership is recorded in the metadata store (ZooKeeper in most 3.x deployments) via ephemeral nodes tied to the broker’s ZK session. That design creates a sharp distinction:

  • Dead broker: process terminated. The ZK session dies with it, ephemeral nodes are removed, and surviving brokers take over its bundles. Recovery is automatic. Your job is finding why the process died (OOM, host failure, kill signal) and whether it will die again on restart.
  • Fenced broker: process alive, ZK session expired (long GC pause, ZK latency storm, network partition). The broker loses topic ownership; topics get fenced while ownership moves. A broker in this state is “up” by every process check while serving nothing. Worse, it can oscillate: lose ownership, regain it, lose it again, in the GC death spiral pattern.
  • Degraded but owned: process alive, session intact, but the broker is unhealthy (direct memory exhausted, GC spiral in progress, stuck BookKeeper client). /metrics may even respond. Only a deeper health check catches this.

The basic liveness signal (HTTP endpoint on :8080 responding) only catches the first case reliably. A binary process check is necessary but not sufficient.

flowchart TD
    A[Alert: :8080 unreachable or clients failing] --> B{Process running?}
    B -- No --> C[Dead broker]
    B -- Yes --> D{ZK session intact?}
    D -- No --> E[Fenced broker: lost topic ownership]
    D -- Yes --> F{Health check passes?}
    F -- No --> G[Degraded: owns bundles but cannot serve]
    F -- Yes --> H[Healthy: look elsewhere, e.g. bookies or ZK]
    C --> I[Check OOM signature, exit reason, host health]
    E --> J[Check GC pauses, ZK latency, session expiry in logs]
    G --> K[Check direct memory, thread pools, BK client]

Common causes

CauseWhat it looks likeFirst thing to check
Direct memory exhaustion (OOM)Process dead or hung; heap metrics looked fine before death; RSS far above heapBroker logs for OutOfDirectMemoryError or “Direct buffer memory”
GC death spiral leading to session expiryProcess alive, :8080 slow or unreachable, repeated bundle ownership churn in logsGC logs / jstat -gc, then ZK session events
ZooKeeper latency stormMultiple brokers affected at once, ZK latency elevated before broker symptomsecho stat | nc <zk-host> 2181, ZK disk and CPU
Host or network failureProcess gone, no JVM error in logs, other services on host also affectedHost logs, network reachability, dmesg
OOM killer (cgroup or kernel)Process gone, exit without JVM stack tracedmesg / journal for oom-kill entries, container memory limits vs heap + direct memory
Rolling restart mistaken for outageEndpoint down briefly, broker re-registers, uptime lowBroker uptime and deployment history (the “previously running > 2 min” condition filters this)

Quick checks

All of these are read-only.

# 1. Is the process alive at all?
pgrep -af PulsarBroker

# 2. Is the HTTP listener dead or just the health path?
curl -sf http://<broker-host>:8080/metrics > /dev/null && echo "UP" || echo "DOWN"

# 3. Deeper health: does the broker exercise the full produce/consume path?
curl -sf http://<broker-host>:8080/admin/v2/brokers/health

# 4. Is the ZK session intact?
curl -s http://<broker-host>:8080/metrics | grep pulsar_zookeeper_connected

# 5. GC pressure right now (if process alive)
jstat -gc <broker-pid> 1000 5

# 6. Direct memory vs heap gap
grep VmRSS /proc/<broker-pid>/status

# 7. Recent death signatures
grep -E "OutOfDirectMemoryError|Direct buffer memory|Session expired|Connection loss" /var/log/pulsar/broker.log | tail -20

# 8. Kernel OOM killer
dmesg -T | grep -i "killed process" | tail

# 9. Is the cluster seeing the broker as gone?
curl -s http://<other-broker>:8080/admin/v2/brokers/<cluster>

Note on check 3: GET /admin/v2/brokers/health is not a cheap ping. It creates a producer and a non-durable subscription on a system health-check topic, sends a message, reads it back, and tears down. That is why it catches fenced and degraded brokers: it exercises the full path through the managed ledger and BookKeeper. Do not point a load balancer at it with a one-second interval on a loaded cluster; concurrent calls have historically exposed race conditions in older 2.x versions, and each call costs a produce/consume round trip.

How to diagnose it

  1. Confirm the alert scope. Check whether other brokers also lost their :8080 endpoints or ZK sessions. Multiple brokers affected simultaneously points at ZooKeeper or shared infrastructure, not at individual broker death. A single broker points local.

  2. Dead or alive? pgrep -af PulsarBroker. If the process is gone, you are in the dead-broker path: skip to step 5. If it is alive, the liveness alert already told you something important: the process is up but not serving.

  3. Alive: check the ZK session. pulsar_zookeeper_connected should be 1. A 0, or Session expired / Connection loss lines in the broker log, means the broker was fenced: it lost bundle ownership and its topics moved to surviving brokers. Correlate the timestamp with GC pauses (jstat history, GC logs) and ZK latency. A GC pause longer than the ZK session timeout is the classic trigger.

  4. Alive with session intact: run the deep health check. GET /admin/v2/brokers/health. A failure here with a live session means the broker owns bundles but cannot complete a produce/consume round trip: suspect direct memory exhaustion in progress, a stuck BookKeeper client, or thread pool saturation. Check RSS vs heap, and Netty allocator stats via /admin/v2/broker-stats/allocator-stats/default.

  5. Dead: find the death signature. In order: broker log tail for OutOfDirectMemoryError or OutOfMemoryError; dmesg for the kernel OOM killer; container runtime events for cgroup OOM or eviction; host logs for hardware or power events. “Process gone with no JVM error” plus an oom-kill entry means your memory accounting was wrong: heap + MaxDirectMemorySize + JVM overhead exceeded the limit. Heap is usually monitored, direct memory is not, and the sum is what the kernel enforces.

  6. Verify client impact separately from broker state. A fenced broker’s topics are already being served by other brokers after bundle reassignment; clients see reconnection blips, not an outage, assuming the rest of the cluster is healthy. Check pulsar_lb_unload_bundle_total on surviving brokers and lookup failures (pulsar_broker_lookup_failures) to confirm the handover completed. Sustained lookup failures after the event mean the problem has moved to the metadata store or ownership churn, not the dead broker.

  7. Only then restart. A dead broker with an explained cause (host failure, fixed OOM) can be restarted or replaced. A fenced broker oscillating in a GC spiral should not be restarted blindly into the same configuration; capture GC evidence first, because the restart erases it and the spiral will recur.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
:8080 endpoint reachability (process liveness)The binary dead/alive signalUnreachable > 2 min on a previously running broker: page
pulsar_zookeeper_connectedDistinguishes fenced from dead on a live processTransition 1 to 0, or flapping
/admin/v2/brokers/health resultExercises produce/consume path; catches degraded-but-aliveNon-200 while /metrics still responds
GC pause duration/frequency (JMX or GC logs)Leading cause of session expiry and fencingPauses approaching the ZK session timeout
Process RSS minus heapProxy for direct memory growth, which Prometheus does not exposeGap growing steadily; RSS near container limit
pulsar_lb_unload_bundle_totalConfirms ownership handover after a broker is fenced or diesUnloads without a corresponding broker event; thrashing rate
pulsar_broker_lookup_failuresShows whether clients can still find topic ownersFailure ratio > 1% sustained after the event
pulsar_active_connectionsReconnect storms after fencing; also feeds direct memory pressureSharp drop (fencing) then spike (thundering herd reconnect)

On Pulsar 3.0+ there is also a passive option: PIP-271 added a pulsar_health gauge at /metrics with values 1 (active), 0 (inactive), -1 (unknown). It is disabled by default; enable it via healthCheckMetricsUpdateTimeInSeconds. This gives you health state without paying for a full produce/consume health check on every scrape.

Fixes

Dead broker: direct memory or heap OOM

Restart the process (or let the orchestrator do it), then fix the accounting before it happens again. Pulsar’s broker defaults are -Xms2g -Xmx2g -XX:MaxDirectMemorySize=4g. The container or host memory limit must cover heap + direct + roughly 20% JVM overhead. If RSS-minus-heap was climbing before death, increase MaxDirectMemorySize headroom or reduce connection count and message sizes. If heap itself was the ceiling, check managedLedgerCacheSizeMB and topic count per broker.

Fenced broker: GC-induced session expiry

Do not restart first. Capture jstat history or GC logs, then address the pause source: heap sizing, cache sizing, or a memory leak in functions/connectors. As a stopgap you can raise zooKeeperSessionTimeoutMillis to buy time, but treat that as masking, not fixing: longer timeouts also slow real failure detection.

Fenced broker: ZooKeeper-side latency storm

The broker is a victim, not the cause. Check ZK server disk (transaction log), CPU, and watch counts (echo wchs | nc <zk-host> 2181). Note that on ZooKeeper 3.5+, four-letter words like wchs work only if whitelisted via 4lw.commands.whitelist; if they are blocked, use mntr via the admin server or your ZK metrics instead. Do not mass-restart brokers: they reconnect on their own once ZK recovers, and restarting adds reconnection load to an already saturated metadata store.

Topics stuck fenced after ownership moves

If producers report fenced-topic errors long after the broker event, the fence release is lagging. The topicFencingTimeoutSeconds broker config controls how long a topic stays fenced before being force-closed; setting it low (e.g. 5 seconds) releases fencing faster but carries data-consistency risk, and fenced-topic bugs have been reported across multiple 2.x/3.x lines.

Host or infrastructure failure

Replace the host and let the cluster rebalance. Confirm surviving brokers absorbed the bundles cleanly via bundle counts and lookup failure rates before calling it done.

Prevention

  • Monitor direct memory explicitly. It is not in Pulsar’s Prometheus output. Use JMX (java.nio:type=BufferPool,name=direct) or track RSS-minus-heap. Alert on the trend, not just the ceiling.
  • Alert on GC pauses relative to the ZK session timeout, not on absolute pause length. A 10-second pause is harmless with a 30-second session timeout and fatal with a 10-second one.
  • Size container limits honestly: heap + MaxDirectMemorySize + JVM overhead, with headroom. This eliminates the most common “broker died with no log” incident.
  • Treat ZK latency as a leading indicator. Sustained latency above 10ms is an early warning for fencing events cluster-wide; above 100ms, failure is minutes away.
  • Gate liveness pages with “previously running > 2 min” so rolling restarts and cold starts do not page you.
  • Run controlled broker kills in staging so the team knows what a normal bundle handover looks like in metrics before a real one happens.

How Netdata helps

  • Netdata’s Pulsar collector scrapes the broker’s Prometheus endpoint on :8080, so broker liveness, pulsar_zookeeper_connected, bundle unload counters, and lookup failures land in one place at per-second resolution.
  • Host-level metrics from the same agent (RSS per process, dmesg OOM events, disk and NIC health) let you see the RSS-minus-heap gap and kernel OOM kills next to Pulsar’s own view of the broker, which is exactly the correlation that separates dead from fenced.
  • JVM and GC charts alongside ZK session state make the GC-spiral-to-fencing chain visible in one timeline instead of two tools.
  • Bundle unload spikes correlated with connection count drops confirm ownership handover after a broker event, telling you whether the cluster healed or the problem moved.