$ guides / apache-pulsar ▌

APACHE PULSAR · OPERATIONS PLAYBOOK

Apache Pulsar's two-layer split: stateless brokers, a storage layer where every write waits on one journal fsync, and a metadata store that can fence the whole fleet

A messaging system that separates serving from storage: brokers own nothing durably and can be moved or fenced at any moment, BookKeeper bookies persist every message behind a journal fsync that gates the entire write path, and ZooKeeper coordinates all of it. We trace how that design behaves under load, where a problem on one layer surfaces as a symptom on another, and what to do when it does.

> Start with the monitoring checklist → # Jump to the full guide list

Pulsar's separation of serving from storage is its great strength and the reason its incidents are so hard to place: the symptom you see on the broker almost always started one layer down.

The defaults work. Until one bookie's journal disk cannot fsync fast enough and every producer whose topic touches that bookie slows down at once — because a write is not acknowledged until the ack quorum has it on disk. Until a broker's off-heap memory fills and Netty throws OutOfDirectMemoryError, killing the broker while the heap dashboard still reads green. Until a GC pause outlasts the ZooKeeper session timeout, the broker is fenced, loses its bundles, and every client on it reconnects at once. Until a consumer stalls, backlog crosses its backlog quota, and the policy you never set turns a consumer problem into a producer outage. Until a bookie fills to diskUsageThreshold, flips to read-only, and new ledgers fail with NotEnoughBookiesException.

These guides are written for engineers who already run Pulsar, not for people learning what a topic is. The goal is the mental model of how brokers, BookKeeper, and the metadata store actually behave under load, the failure patterns that keep recurring across the two layers, the monitoring story that catches them before they page anyone, and the runbooks you wish someone had handed you before your last incident.

How Apache Pulsar actually runs in production

Pulsar is not one system but three tiers that fail independently: a stateless broker fleet that owns topics only by lease, a BookKeeper storage layer where durability and the write path actually live, and a metadata store that coordinates everything. A message travels down through all of them and back up to consumers — and most production failures live in the seams between tiers, not inside any one of them.

producers / clients

Every producer and consumer first does a topic lookup to find which broker owns the topic, then holds a TCP connection. Each connection costs a file descriptor and Netty direct memory for buffers. Client libraries reconnect automatically — which is a feature until a bundle unload or fenced broker makes thousands of them reconnect at once.

CLIENT

brokers + bundle ownership

Brokers are stateless. Each owns a set of namespace bundles — hash-range slices of the topic space — assigned by the load balancer and recorded in the metadata store. A broker owns a topic only by lease; when it dies, is overloaded, or loses its session, its bundles move and its topics are fenced and reopened elsewhere.

BROKER

managed ledger + entry cache

Each topic is a managed ledger: a continuous append stream wrapping a sequence of BookKeeper ledgers. Recently written entries sit in an off-heap entry cache so connected consumers read from broker memory. A cache miss sends the read down to the bookies, adding disk I/O on the storage layer.

LEDGER

BookKeeper bookies

The storage layer, and where durability lives. Bookies know nothing about topics — they store opaque ledger entries. Each managed ledger writes to an ensemble of bookies with a write quorum (Qw) and ack quorum (Qa); the producer is acknowledged only after Qa bookies confirm. The slowest bookie in the write set sets the publish latency.

BOOKIE

journal + entry log disks

Each bookie writes a sequential, per-entry-fsync'd write-ahead journal on one dedicated disk, and interleaved entry logs (the actual payloads) on another. The journal fsync is the physical heartbeat of write throughput; sharing it with any other workload — including the entry-log reads — is the single most common architecture mistake.

STORAGE

subscriptions + cursors

Consumers read through subscriptions, each holding a cursor (mark-delete position) in the metadata store. Delivered-but-unacknowledged messages count against a per-subscription limit; hit it and dispatch silently freezes. An abandoned subscription's cursor pins every message after it, so storage cannot be reclaimed.

CONSUME

metadata store (ZooKeeper)

ZooKeeper (still the default in 3.x; Oxia and etcd exist) is the single source of truth: bundle-to-broker ownership, ledger metadata, cursors, schemas, policies. Its latency is a leading indicator for the whole cluster — when it crosses the session timeout, brokers are fenced and the cluster destabilises.

METADATA

JVM runtime (heap + direct memory)

Brokers and bookies are JVMs. Heap holds connection and cursor state; direct (off-heap) memory holds Netty buffers and the entry cache and is invisible to heap metrics. A full GC pause that outlasts the ZK session timeout is what turns memory pressure into a fenced broker and the GC death spiral.

JVM

Why this matters: 'Pulsar is slow' or 'publishes are timing out' can come from one saturated journal disk, an off-heap memory crash the heap dashboard hides, a ZooKeeper latency storm fencing brokers, a single hot broker, a stalled consumer tripping a backlog quota, a bookie gone read-only, or ownership oscillating between brokers. The symptom rhymes but each tier has a different signal — and a different fix.

The failures you'll actually see

Most Pulsar incidents fall into a small set of recurring patterns that cross the broker/BookKeeper/metadata seams. Recognise the shape, and triage gets dramatically faster.

CRITICAL

The write stall

A bookie's journal disk cannot fsync fast enough. The journal force-write queue grows, add-entry operations queue behind it, and because a write is not acked until the ack quorum has it on disk, every producer whose topic touches that bookie slows down. Publish latency climbs, throughput collapses, and the whole write path waits on one disk. This is the single most common performance failure in Pulsar.

bookie_journal_JOURNAL_SYNC P99 well above the device baseline
JOURNAL_FORCE_WRITE_QUEUE_SIZE sustained above zero
bookkeeper_server_ADD_ENTRY_IN_PROGRESS not draining after bursts
pulsar_broker_publish_latency rising on topics using that bookie

Investigate →

CRITICAL

The off-heap broker crash

Netty uses direct (off-heap) memory for every network buffer and the managed ledger cache. Under a connection storm, large messages, or an oversized cache, direct memory fills and Netty throws OutOfDirectMemoryError — the broker can no longer allocate buffers for any I/O and is dead in all but PID. Heap monitoring shows plenty of room, so standard JVM dashboards miss it entirely.

io.netty.util.internal.OutOfDirectMemoryError in broker logs
Process RSS far larger than configured heap
Broker health check failing while the process still exists
All topics on the broker unavailable at once

Investigate →

CRITICAL

The ZooKeeper session cascade

ZooKeeper latency rises — watch explosion, transaction-log disk I/O, a GC pause on a ZK server. Brokers miss heartbeats, their sessions expire, and they are fenced and lose bundle ownership. Every client reconnects at once, the reconnect metadata storm pushes ZK latency higher, and more sessions expire. The cluster thrashes even though every broker process is running fine.

Session expired events on multiple brokers simultaneously
Metadata store latency elevated before any broker symptom
Bundle reassignment and client reconnections spiking together
Broker lookup failures and client-side LookupException rising

Investigate →

ACTIVE

The backlog avalanche

A consumer falls behind. Backlog grows, so reads miss the entry cache and go to the bookies, whose disks now serve reads and writes; write latency rises, dispatch slows, more consumers fall behind, more reads hit the bookies. Left alone, backlog fills bookie disk or crosses its quota — at which point the policy either holds producers, errors them, or silently evicts the oldest messages.

pulsar_subscription_back_log growing continuously with consumers connected
pulsar_rate_out flat or falling while pulsar_rate_in is normal
Managed ledger cache miss rate climbing, bookie read latency rising
Publish latency rising once bookie disks serve both reads and writes

Investigate →

CRITICAL

Not enough bookies to write

The broker rolls to a new ledger periodically and on recovery, but cannot satisfy the ensemble and quorum requirement — often after a bookie filled to diskUsageThreshold and went read-only, or after too many bookies were restarted at once. Ledger creation fails with NotEnoughBookiesException and new messages cannot be written durably. Rack-awareness makes it subtler: enough bookies numerically, not enough distinct failure domains.

NotEnoughBookiesException / BKException on ledger create in broker logs
One or more bookies at bookie_SERVER_STATUS == 0 (read-only)
bookie_ledger_writable_dirs == 0 on affected bookies
Publish errors on topics whose ledgers need to roll

Investigate →

IMMINENT

Ownership oscillation and fencing loops

Two brokers keep trading a bundle: A opens the managed ledgers, the balancer unloads it, B fences the ledgers with LedgerFencedException and opens new ones, the balancer moves it back. Each cycle seals a ledger, opens another, and reconnects every client on those topics. Normal rebalancing converges in minutes; this can run indefinitely, and if a new ledger cannot be created after fencing, the topic enters a broken state.

LedgerFencedException with no matching planned bundle transfer
Rapid acquired / released ownership log lines on two brokers
Bundle unload rate high with constant client reconnections
Spiky publish latency on the contested topics

Investigate →

Choosing a tool

Best Apache Pulsar Monitoring Tools Ranked (2026)

A ranked review of the tools teams actually shortlist here, what each one is genuinely good at, and how the pricing behaves as you scale.

Read the buyer's guide →

Apache Pulsar monitoring maturity levels

Pulsar observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your cluster matters. Most production deployments should land at the second level — and monitor BookKeeper as a first-class concern, not an afterthought.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor. With these signals you can answer one question: are all three tiers alive and is data flowing? You will not learn what broke, but you will learn that something broke before users do. Survival is enough for dev clusters and non-critical pipelines.

Broker process up HTTP admin endpoint on :8080 answering, not just the JVM alive?
Bookie process up + writable bookie_SERVER_STATUS == 1; read-only means it stopped accepting writes.
Metadata store (ZooKeeper) up ruok / quorum healthy — the coordinator every tier depends on.
Message rates in and out pulsar_rate_in / pulsar_rate_out — is data actually flowing?
Bookie ledger disk usage Approaching diskUsageThreshold flips the bookie to read-only.

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production clusters should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident on its own: write stalls, backlogs, ZooKeeper degradation, connection pressure, grey lookup failures.

Per-namespace publish / dispatch rate Per-topic rates catch partial failures cluster averages hide.
Subscription backlog on critical topics Alert on rate-of-change, not absolute size.
Broker publish latency P99 The producer SLI; P50 looks fine while P99 spikes.
Bookie journal sync latency P99 The physical write-throughput limit; the cluster heartbeat.
Active connections + FD ratio Each connection costs an FD and direct memory; watch the trend.
Metadata store latency The #1 leading indicator; > 50ms sustained is a warning.
Broker lookup failures New clients can't find their topic while old ones work — grey failure.
Bookie disk usage trend Runway to read-only, not just the current value.

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone up. A subscription quietly not acking, the entry cache thrashing, ledgers left under-replicated, one broker owning far too many topics, a bookie write queue that no longer drains. None of these page you on day one. They become page-out incidents on day thirty.

Per-subscription backlog + redelivery Redelivery storms mean receiving without progressing.
Unacked messages vs the freeze limit At maxUnackedMessagesPerSubscription, dispatch freezes silently.
Managed ledger cache hit / miss Misses send consumer reads down to the bookies.
Under-replicated ledger count Data at risk after a bookie failure; should trend to zero.
Add-entry in-progress + force-write queue Write-path saturation before latency spikes.
Bundle unload rate Steady-state unloads should be rare; high means thrashing.
Topic count per broker One hot broker while cluster averages look healthy.
TLS certificate expiry Not a broker metric; an expired cert is a total, silent outage.

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. Direct memory, GC pauses, fencing churn, entry-log GC lag, dead-letter arrival. Most teams never need every signal here. Add the ones your incident history says you do — and remember Pulsar exposes several of these only via JMX or the logs, not Prometheus.

Broker direct (off-heap) memory Via JMX; the invisible killer heap dashboards never show.
GC pause times A pause past the ZK session timeout fences the broker.
Journal add-entry latency percentiles Finer-grained than sync latency for disk diagnosis.
Ledger fencing events / ownership churn Fencing with no bundle transfer means split ownership.
Entry-log active vs total space BookKeeper GC falling behind on space reclamation.
Dead-letter queue arrival rate A ledger of application processing failures.
Message expiration rate TTL deleting unread messages — silent consumer-side loss.
ZooKeeper watch count Watch explosions crush metadata latency (wchs).

Operating mistakes worth avoiding

The traps Pulsar teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

Watching rates but not subscription backlog

Teams monitor publish and consume rates but not backlog directly. Backlog is the accumulation signal — it grows silently until it fills bookie disk, hits retention, or trips a <code>backlog quota</code> that throttles producers. By the time it fires an alert, the damage has already cascaded. Monitor per-subscription backlog and its growth rate; alert on sustained growth, not an absolute threshold.

Ignoring broker direct (off-heap) memory

Standard JVM monitoring shows heap. Pulsar brokers use significant direct memory for Netty buffers and the managed ledger cache, and Pulsar does NOT expose it as a Prometheus metric. Teams see 'heap is fine at 60%' and cannot explain why the broker hung or crashed with <code>OutOfDirectMemoryError</code>. Set <code>MaxDirectMemorySize</code> explicitly and watch it via JMX, or track process RSS minus heap.

Not separating journal and storage disk metrics

Bookies use separate disks — they must — for the write-ahead journal and the entry logs. Teams monitor aggregate disk I/O and miss that the journal disk is saturated. Journal saturation directly blocks the entire write path; it is the most impactful failure point in the stack. Monitor per-device I/O and use the <code>journalIndex</code> label to isolate the journal.

Missing ZooKeeper degradation until the cascade

ZK latency creeps up over days. Brokers occasionally lose ownership and recover, and it gets dismissed as transient — until latency crosses the session timeout and the whole cluster destabilises in a session cascade. Treat sustained metadata latency above 10ms as an early warning; it is the number-one leading indicator for cluster-wide failures.

No visibility into consumer acknowledgment patterns

Teams see consumers connected and messages dispatched and assume progress. They miss that consumers are not acking — driving redelivery storms, or hitting <code>maxUnackedMessagesPerSubscription</code> and freezing dispatch entirely. Compare acknowledgment rate to dispatch rate; the gap, plus the redelivery rate, is the real indicator.

Reactive capacity planning on bookie disk

Teams wait until a bookie disk is 90% full to expand. Bookie disk growth is predictable from retention policy and publish rate, and a full bookie flips to read-only — which can cascade into cluster-wide write failure. Plan expansion at 70%, estimate runway to 90%, and factor in that a consumer outage accelerates the fill.

Monitoring backlog size but not backlog age

A backlog of 1,000 messages from five seconds ago is nothing; 1,000 messages from five hours ago is an SLA breach. Size is volume, age is latency. Age is also what reveals cursor leaks and abandoned subscriptions that pin storage. Without it, teams react to normal batch spikes and miss the degradation that actually matters.

Alerting on average latency instead of P99

Publish latency P50 can look beautiful while P99 spikes — and P99 is what producers experience during bursts and what triggers their timeouts. Teams that alert on mean or P50 miss the degradation entirely. Alert on P99 for publish and journal-sync latency, and remember broker-side latency excludes client-to-broker network and batching.

Apache Pulsar runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

Broker health, connections, and memory

▸

BookKeeper journal and the write path

▸

Bookie disk, ledger storage, and GC

▸

Replication, recovery, and durability

▸

Managed ledgers and the broker cache

▸

Backlog, subscriptions, and consumers

▸

Metadata store and ZooKeeper

▸

Namespace bundles and load balancing

▸

Geo-replication and RPO

▸

Authentication, authorization, and TLS

WHERE TO GO NEXT

Setting up Apache Pulsar monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.

> Start with the checklist > Back to Operations Guides