Most Pulsar incidents are misdiagnosed in the first thirty minutes because the operator is looking at the wrong layer. A publish latency spike gets chased on the broker when the real bottleneck is a bookie journal disk. A “consumer problem” turns out to be a cursor that stopped advancing. A broker that looks healthy on its HTTP endpoint is fenced off from ZooKeeper and losing topic ownership in a loop.

Pulsar’s architecture makes this confusion easy: serving and storage are separate fleets with separate failure modes, coordinated by a third system neither can function without. If you carry a mental model from Kafka or RabbitMQ into a Pulsar incident, you will misread the signals.

This article is the mental model to hold before any runbook: what each layer owns, how a message moves on write and on read, and which abstractions (managed ledgers, bundles, cursors, the entry cache, the journal) explain the failure patterns you will see in production.

What it is and why it matters

Apache Pulsar is a distributed messaging system built on a two-layer architecture:

  1. Brokers (serving layer). Stateless processes that handle producer and consumer connections, route messages, enforce policies (retention, TTL, backlog quotas), manage subscriptions, and coordinate topic ownership. Brokers hold no durable message data.
  2. BookKeeper bookies (storage layer). Stateful processes that durably persist messages into append-only ledger segments on local disks. Bookies know nothing about topics; they store opaque ledger entries. Data durability lives here entirely.

A third component coordinates both:

  1. The metadata store. In production this is typically ZooKeeper (still the default in Pulsar 3.x). It stores all cluster metadata: topic policies, schema registry, bundle-to-broker ownership, ledger metadata, and cursor mark-delete positions. Pulsar 3.3.0 introduced experimental Oxia support (PIP-335) as an eventual ZooKeeper replacement; etcd is also supported but less common.

Why this matters operationally: every Pulsar symptom has a home layer, and the fix lives in the same layer as the cause. A broker restart will never fix a journal disk. Adding bookies will never fix a broker GC pause. The fastest way to shorten an incident is to map the symptom to the layer in the first minute, and that requires knowing exactly what each layer owns.

How it works

Brokers: stateless, but not simple

Brokers own “bundles”: hash-range slices of the topic namespace. Topic lookup resolves namespace to bundle (via a hash of the topic name) to the owning broker. Each bundle is owned by exactly one broker at a time. When a broker dies, its bundles are reassigned to surviving brokers and clients reconnect.

Internally, a broker maintains:

  • Netty event loops for all network I/O, using off-heap (direct) memory for zero-copy buffers.
  • A managed ledger cache (off-heap) holding recently written entries, so tailing consumers read from memory instead of disk.
  • Dispatcher threads that push messages to connected consumers.
  • Connection pools to the bookies and to the metadata store.
  • Per-topic managed ledger instances that wrap BookKeeper ledgers into a continuous append stream.

The broker default JVM settings are worth memorizing because they define two separate memory failure domains: -Xms2g -Xmx2g -XX:MaxDirectMemorySize=4g. Heap and direct memory exhaust independently, and standard heap monitoring does not see the direct side.

Bookies: where durability actually lives

Each bookie maintains:

  • A write-ahead journal on a dedicated disk: sequential writes, fsync’d per entry or per batch. This is the critical path for every persistent message and the heartbeat of write throughput.
  • Entry logs: sequential files containing the actual message payloads, interleaved across ledgers.
  • Ledger indexes: mapping ledger/entry IDs to positions in entry logs.
  • A read cache for hot data, and thread pools for journal writes, entry log writes, and read requests.

Each topic’s managed ledger writes to an “ensemble” of bookies with a configured write quorum (Qw) and ack quorum (Qa). A write is durable, and acknowledged to the broker, once Qa bookies have confirmed it. One consequence that surprises operators: because Qa bookies must ack, a single slow bookie in the ensemble can bottleneck every topic whose ledger includes it.

Journal and ledger storage must be on separate physical disks. If entry log reads compete with journal fsyncs on one device, write latency becomes erratic for the entire cluster. This is the most common architecture mistake in Pulsar deployments.

The metadata store: the silent dependency

ZooKeeper is the single point of truth for cluster topology. Every broker and bookie maintains a ZK session, with ephemeral nodes for leadership and service discovery. Its failure modes cascade: when ZK latency exceeds the session timeout, brokers get fenced, lose bundle ownership, and trigger mass client reconnections, which generate more ZK load. If the metadata store is down, Pulsar is down, regardless of how healthy brokers and bookies look.

flowchart LR
  P[Producer] -->|publish| B[Broker]
  B -->|append to managed ledger| ML[Managed ledger]
  ML -->|parallel write to Qw bookies| BK1[Bookie 1]
  ML -->|parallel write to Qw bookies| BK2[Bookie 2]
  ML -->|parallel write to Qw bookies| BK3[Bookie 3]
  BK1 -->|journal fsync| J1[(Journal disk)]
  BK2 -->|journal fsync| J2[(Journal disk)]
  BK3 -->|journal fsync| J3[(Journal disk)]
  J1 & J2 & J3 -->|Qa acks| ML
  ML -->|ack| P
  B -->|ownership, ledger metadata, cursors| ZK[(Metadata store)]
  BK1 & BK2 & BK3 -->|registration| ZK
  C[Consumer] -->|subscribe| B
  B -->|dispatch from cache or bookies| C
  C -->|ack, cursor advances| B

The write path

Producer to broker on a Netty channel; the broker appends to the topic’s managed ledger; the managed ledger issues a parallel write to Qw bookies; each bookie writes to its journal and fsyncs; once Qa acks arrive, the broker acknowledges to the producer.

Two operational consequences follow. First, producer-visible write latency is bounded below by bookie journal fsync time: if bookie_journal_JOURNAL_SYNC spikes, pulsar_broker_publish_latency spikes, and producers block or time out. This is the single most common production failure in Pulsar. Second, durability is entirely a BookKeeper concern. A broker crash loses no acknowledged data, because acknowledgment only follows Qa durable copies. Losing one bookie’s disks after acknowledgment is survivable; the data-loss window opens when bookie failures outpace AutoRecovery’s re-replication.

The read path

Consumer connects to the broker owning the topic; the broker serves from the managed ledger cache if the data is there; on a cache miss it reads from the bookies; the dispatcher pushes messages to the consumer; the consumer acknowledges; the cursor advances (mark-delete position in the metadata store, individual acks in a per-cursor BookKeeper ledger).

The cache is the fork in the road. Tailing consumers read from broker memory: fast, and invisible to bookies. Consumers that fall behind force bookie disk reads, adding I/O load to the storage layer while it is handling foreground writes. This is how a slow consumer becomes a cluster-wide latency problem: backlog grows, cache misses grow, bookie read I/O grows, journal and entry log contention rises, publish latency rises for everyone.

The abstractions you need before the runbooks

Managed ledger. Pulsar’s abstraction on top of BookKeeper. Each persistent topic has one. It tracks the current open ledger (where new messages are written), a list of closed ledgers (historical data), and one cursor per subscription tracking the acknowledged position. When a managed ledger rolls to a new segment, or when a new broker takes over a topic, the old ledger is fenced and a new one created. Fencing is normal during planned failover; unexpected fencing loops signal ownership oscillation.

Bundles. The load-balancing unit. Bundle splitting happens when a bundle gets too hot; bundle unloading is how the load balancer moves work between brokers. Each unload briefly disconnects the clients of every topic in that bundle. A sustained high unload rate (pulsar_lb_unload_bundle_total) outside of maintenance means the load balancer is thrashing. Bundle count only grows via splitting; it never shrinks.

Cursors, dispatch, and backlog. Each subscription has a dispatcher pushing messages to connected consumers, and a cursor recording progress. The backlog is the gap between the newest published message and the furthest-behind cursor. Backlog is the pressure gauge of your consumer fleet, but it has a trap: abandoned subscriptions hold cursors that prevent data deletion, so storage grows silently even with no consumer attached. A stable high backlog is often fine; a monotonically growing one is not.

The unacked message limit. Messages dispatched but not acknowledged count against maxUnackedMessagesPerSubscription (default 200,000) and maxUnackedMessagesPerConsumer (default 50,000). At the limit, the broker silently stops dispatching: consumers stay connected, backlog may look flat, and no forward progress happens. Nothing errors, which makes this one of the most insidious failure modes.

Entry cache. The broker’s off-heap cache of recently written entries. Undersized cache or memory pressure forces reads to bookies, which is a latency cliff and a load amplifier. High miss rates are normal for 10 to 30 minutes after a broker restart (cold cache) and alarming at any other time.

BookKeeper journal. Synchronous, fsync-heavy, latency-critical. If you memorize one thing from this article: the journal disk is the write path. Monitor it per device, not as aggregate disk usage.

Where it shows up in production

The architecture produces a small set of recurring failure archetypes. Recognizing the shape early is most of the battle:

  • The write stall. Journal disk cannot keep up with fsync requests. Pending adds queue on the bookie, brokers wait for acks, producers block. The most common performance failure in Pulsar.
  • The GC death spiral. Broker JVM under memory pressure, GC pauses stop the world, ZK sessions expire, bundles unload, clients reconnect, metadata load increases, more GC. Self-reinforcing.
  • The metadata deadlock. ZK latency spikes; bundle ownership updates hang; requests stall without explicit errors. Watch explosions during consumer reconnect storms accelerate it.
  • The backlog avalanche. Consumers fall behind, backlog grows and consumes bookie disk, cache fills with old entries, disk approaches threshold, bookie goes read-only. Feedback loop.
  • The bookie cascade. One bookie fails, recovery I/O stresses survivors, another appears to time out, more recovery triggers. Continues until someone intervenes.
  • Direct memory exhaustion. RSS far exceeds heap, standard JVM monitoring looks fine, broker hangs or dies with OutOfDirectMemoryError. Invisible until it happens.

The layers also compete for a fixed set of resources, and knowing which resource maps to which layer saves diagnostic time:

ResourceWhere it hurts first
Disk I/OBookie journal (writes) and ledger disks (reads); must be separate devices
Direct memoryBroker Netty buffers and managed ledger cache; invisible to heap metrics
HeapBroker connection state, cursor tracking, subscription metadata
File descriptorsEvery client connection plus bookie ledger/index files; production brokers need 100K+
NetworkBroker-to-bookie cross-talk; WAN if geo-replication is enabled
Metadata store sessionsEvery broker and bookie holds a session; ZK becomes the bottleneck in large clusters

Deployment variants change the picture. Standalone runs ZK, broker, and bookie in one JVM (monitor them logically separately). Geo-replication adds replicator cursors and cross-region backlog. Tiered storage adds offload behavior and changes retention semantics.

Common misuses

  • Monitoring aggregate disk metrics on bookies. Journal and ledger disks have completely different criticality. Aggregate utilization hides journal saturation until writes stall.
  • Treating “Pulsar is up” as sufficient. Bookies have independent failure modes (journal stall, compaction stall, under-replication) that are invisible from broker metrics until they cascade. Monitor BookKeeper as a separate system.
  • Alerting on p50 latency. Publish latency p50 can look perfect while p99 spikes into producer timeouts. Alert on P99.
  • Assuming the application manages subscription lifecycles. It usually does not. Track subscription count over time and alert on stale subscriptions before they fill disks.
  • Restarting brokers during a ZK event. Brokers reconnect on their own once ZK recovers; restarting adds reconnection load to an already overloaded coordination layer.

Signals to watch in production

These are the signals that most directly reflect the architecture above, and what deviation means.

SignalWhy it mattersWarning sign
bookie_journal_JOURNAL_SYNC P99The physical limit of write throughput; every write waits on journal fsyncSustained 2x degradation from baseline (SSD P99 above roughly 5ms)
bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZEEarliest write-path saturation warning; rises before sync latency spikesSustained depth above zero for more than 10 seconds
bookkeeper_server_ADD_ENTRY_IN_PROGRESSBookie write queue depth; growth means the disk cannot drain writesQueue not draining within 30 seconds after a burst
pulsar_broker_publish_latency P99The producer-facing SLI; dominated by the slowest bookie in the quorumSustained 2x baseline
bookie_SERVER_STATUS1 = writable, 0 = read-only (usually disk), -1 = unregisteredAny 0 with remaining writable bookies below ensemble size
pulsar_subscription_back_logConsumer health and disk-fill predictorMonotonic growth for more than 15 minutes with active consumers
pulsar_subscription_unacked_messagesApproaching the silent dispatch freezeAbove 50% of maxUnackedMessagesPerSubscription
pulsar_ml_cache_hits_rate vs missesWhether consumer reads hit memory or bookie disksMiss rate above 20% sustained after warm-up
auditor_NUM_UNDER_REPLICATED_LEDGERSData-loss risk after bookie failureNon-zero and not trending back to zero
Metadata store (ZK) latencyLeading indicator for cluster-wide cascadesSustained above 10ms; above 100ms expect session expiry and fencing
pulsar_lb_unload_bundle_totalLoad balancer health; each unload drops clientsSustained above 1/minute outside maintenance
pulsar_active_connections trendFD and direct memory pressure; slow growth means a leakUnexplained growth over weeks

Note one instrumentation gap that follows directly from the architecture: Pulsar does not expose broker direct memory as a Prometheus metric. You need JMX (java.nio:type=BufferPool,name=direct) or process-level monitoring (RSS minus heap) to see the memory domain that kills brokers most often.

How Netdata helps

The two-layer split makes diagnosis fundamentally a correlation problem: broker symptoms with bookie causes, or bookie symptoms with metadata-store causes. Netdata shortens that mapping:

  • Per-second collection on brokers, bookies, and ZooKeeper side by side, so a journal sync latency spike and the resulting broker publish latency spike line up on the same timeline instead of in separate tools.
  • Bookie journal metrics (bookie_journal_JOURNAL_SYNC, force write queue, add-entry in progress) collected per bookie, so the single slow bookie dragging a quorum shows up as an outlier rather than being averaged away.
  • Per-subscription backlog, unacked, and redelivery metrics, so the silent dispatch freeze (unacked at limit, backlog flat, no errors) is distinguishable from a dead consumer.
  • Bookie disk usage and server status tracked together, so the read-only transition is explained by the disk curve that preceded it.
  • Bundle unload rate and connection churn alongside metadata store latency, making the GC death spiral and ZK latency storm signatures visible as they form rather than after clients start timing out.
  • Process-level memory (RSS) per component, which exposes the direct memory gap that JVM heap metrics and Pulsar’s own Prometheus endpoint both miss.