Most Pulsar outages do not arrive without warning. The journal force write queue grows before publish latency spikes. ZooKeeper latency drifts upward for days before brokers lose session ownership. Direct memory climbs for weeks before the broker dies with an OutOfDirectMemoryError while heap dashboards look fine. The signals were there. The team just was not collecting them yet.

This is a four-level maturity model for Pulsar monitoring, from the bare minimum that tells you the cluster is alive to the deep signals operators add after their second or third incident. Use it to audit your current coverage and decide what to instrument next. The levels are cumulative: each one assumes everything below it is already in place.

The model covers the three components that matter: brokers (serving layer), BookKeeper bookies (storage layer), and the metadata store (ZooKeeper in most production deployments). A setup that only watches brokers is blind to the two failure domains behind the worst Pulsar incidents: bookie write-path stalls and ZooKeeper degradation.

flowchart TD
  L1["Survival: process up, bookie writable, rates flowing, disk not full"]
  L2["Operational: rates, backlog, publish P99, journal sync P99, ZK latency"]
  L3["Mature: unacked, ML cache, under-replicated ledgers, force write queue"]
  L4["Expert: GC pauses, direct memory, recovery progress, allocator stats"]
  L1 -->|adds| L2
  L2 -->|adds| L3
  L3 -->|adds| L4

Level 1: Survival

The question this level answers: is the cluster alive, and can it accept writes right now?

This is the minimum that distinguishes “Pulsar is down” from “Pulsar is up.” It will not tell you why anything failed, and it gives no advance warning. If any of these checks fail, the cluster is in immediate trouble.

SignalSourceWhat it tells you
Broker process healthGET http://<broker>:8080/metrics or /admin/v2/brokers/healthBroker process is alive and its HTTP listener works. Necessary but not sufficient: a broker can respond while unable to reach bookies or ZooKeeper.
Bookie process healthGET http://<bookie>:8000/metricsBookie process is alive.
Bookie server statusbookie_SERVER_STATUS1 = writable, 0 = read-only (typically disk full). If enough bookies go read-only, write quorums cannot be met and writes to affected ledgers fail.
ZooKeeper liveness4-letter command ruok on port 2181The coordination layer is answering. Note that recent ZooKeeper versions require 4-letter words to be explicitly allowed via 4lw.commands.whitelist.
Message ratespulsar_rate_in, pulsar_rate_outData is actually flowing. Zero publish rate when producers should be active means producer failure or a broker that cannot accept writes.
Bookie disk usagebookie_ledger_dir_{path}_usage, bookie_ledger_writable_dirsHeadroom before the bookie flips itself read-only at its configured disk usage threshold (default 0.95).

Two cautions at this level. First, broker liveness is a weak signal: the common degraded states (ZK session expired, GC death spiral, direct memory exhaustion) all leave the process running. Second, bookie cold starts take time. After a restart, journal replay can take 5-10 minutes for large journals, and the bookie may report read-only during replay. Do not page on transient states during startup.

Level 2: Operational

The question this level answers: is the cluster healthy under its current workload, and where is pressure building?

This is the professional baseline. A team at Level 2 can detect the most common production failure (the bookie journal write stall) and the most common consumer failure (growing backlog) before users notice.

SignalSourceWhat it tells you
Per-namespace publish and dispatch ratespulsar_rate_in, pulsar_rate_outDeviation from baseline during expected-traffic periods is the earliest sign of producer or consumer failure. Divergence between the two means backlog is growing.
Subscription backlogpulsar_subscription_back_logConsumer health. Alert on sustained growth, not absolute size. A high but stable backlog is fine; a monotonically growing one is not.
Broker publish latencypulsar_broker_publish_latency P99The primary SLI for producer experience. Healthy SSD-backed clusters typically see P99 under 10ms. Alert on 2x degradation from your rolling baseline, not a fixed number.
Bookie journal sync latencybookie_journal_JOURNAL_SYNC P99The physical limit of write throughput. Every write waits for journal fsync. SSDs should hold P99 under 5ms, HDDs under 20ms. If this spikes, the entire write path stalls from bookie through broker to producer.
Broker and bookie memoryJMX heap metricsHeap above 85% sustained with rising GC frequency precedes the GC death spiral.
Active connectionspulsar_active_connectionsLoad on the FD table and Netty direct memory. Unexplained growth over weeks indicates a connection leak.
ZooKeeper latencyZK stat command, admin server, or broker-side metricsHealthy is under 10ms average. Above 50ms sustained stalls broker metadata operations. Above 100ms, session expirations and cascading bundle unloads are minutes away.
Authentication failurespulsar_authentication_failures_totalExpired tokens, misconfigured clients, or unauthorized attempts. Should be near zero in a stable environment.
Broker lookup failurespulsar_broker_lookup_failures vs pulsar_broker_lookup_answersThe grey failure: the cluster looks up, but new producers and consumers cannot resolve topic ownership. Sustained failure rate above 1% of lookups indicates metadata store or bundle ownership problems.

Note the metric availability gotchas. Lookup metrics may not be present in all Pulsar versions, and per-topic rates depend on exposeTopicLevelMetricsInPrometheus. On clusters with thousands of topics, per-topic metrics can overwhelm Prometheus. Verify against your own /metrics output before writing alerts.

Level 3: Mature

The question this level answers: is the cluster degrading in ways that will become an incident in hours or days?

Level 3 signals are leading indicators. They fire before user-visible latency or errors, which is what buys you time to act during business hours instead of at 3 a.m.

SignalSourceWhat it tells you
Per-subscription backlog and redeliverypulsar_subscription_back_log, pulsar_subscription_msg_rate_redeliverRedelivery above 10% of dispatch rate means consumers receive but fail to process: poison messages, downstream failures, or ack timeout misconfiguration.
Unacked messagespulsar_subscription_unacked_messagesWhen this hits maxUnackedMessagesPerSubscription (default 200,000), the broker silently stops dispatching. Consumers stay connected, backlog may look flat, and no forward progress happens. Alert at 50% of the limit.
Managed ledger cache efficiencypulsar_ml_cache_hits_rate, pulsar_ml_cache_misses_rate, pulsar_ml_cache_evictionsHit rate above 80% means consumers read from broker memory. High misses plus high evictions mean the cache is undersized and reads are falling through to bookie disks.
Under-replicated ledgersauditor_NUM_UNDER_REPLICATED_LEDGERS on the auditor nodeData-loss risk. Must be zero in steady state. After a bookie failure it spikes and should trend back to zero as AutoRecovery works. A growing count means recovery is failing or bookies are failing faster than recovery completes.
Geo-replication backlog and delaypulsar_replication_backlog, pulsar_replication_delay_in_seconds, pulsar_replication_disconnected_countBacklog that never drains is your RPO window growing. Compare the delay metric against your declared RPO, not against zero.
Add entry in progressbookkeeper_server_ADD_ENTRY_IN_PROGRESSThe bookie write queue depth. Should drain to near zero within seconds after a burst. Sustained growth means the journal disk cannot keep up and the write path is about to stall.
Journal force write queuebookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZEThe earliest write-path saturation warning. It rises before journal sync latency spikes and before add-entry queues grow. Should sit at zero.
Bundle unload ratepulsar_lb_unload_bundle_totalExpected during rolling restarts. Sustained unloading above 1 per minute outside maintenance means the load balancer is thrashing, and each unload drops client connections for the affected topics.
Throttled connectionspulsar_broker_throttled_connectionsThe broker is protecting itself from overload by limiting intake. Should be zero.
Topic count per brokerpulsar_topics_countHotspot detection. Any broker owning more than 2x the cluster average for 30+ minutes indicates imbalance the load balancer is not fixing.
TLS certificate expirationopenssl x509 -enddate against cert files or live endpointsAn expired cert is a total, preventable communication failure. Ticket at 30 days, escalate at 7.
Bookie entry log spacebookie_ENTRY_LOG_SPACE_BYTES vs bookie_ACTIVE_ENTRY_LOG_SPACE_BYTESTotal much larger than active means BookKeeper GC is falling behind on reclaiming deleted data, and disk will fill despite logical deletions.

Level 3 is also where you start correlating failure patterns: journal force write queue plus add-entry-in-progress plus publish latency is the write throttle loop; GC pauses plus ZK latency plus bundle unloads plus flapping connections is the GC death spiral.

Level 4: Expert

The question this level answers: why did it degrade, and what will fail next?

These signals come from incident retrospectives. Several are not exposed through Pulsar’s Prometheus endpoint at all, which is exactly why most teams lack them until after the incident that needed them.

SignalSourceWhat it tells you
GC pause timesJMX java.lang:type=GarbageCollector, GC logsPulsar does not expose JVM heap or GC metrics on its standard Prometheus endpoint. Long full GC pauses approaching the ZK session timeout (tens of seconds by default) risk session expiry; even shorter pauses show up as publish latency spikes. Correlate pauses with latency spikes and session events.
Direct memoryJMX java.nio:type=BufferPool,name=directThe invisible killer. Brokers default to -XX:MaxDirectMemorySize=4g in the shipped pulsar_env.sh, Netty buffers and the managed ledger cache live here, and exhaustion crashes the broker while heap looks healthy. Also watch process RSS minus heap as a proxy.
Journal add entry percentilesbookie_journal_JOURNAL_ADD_ENTRYTime to record an entry in the journal, distinct from the fsync itself. Percentile drift here separates journal queueing problems from raw disk problems.
Ledger recovery progressauditor_NUM_UNDER_REPLICATED_LEDGERS trend, auditor_UNDER_REPLICATED_LEDGERS_TOTAL_SIZENot just the count but the direction and rate. Recovery I/O competes with foreground traffic on surviving bookies, so watch journal sync latency on the survivors while recovery runs. This is how you see the bookie cascade forming.
Entry log fragmentationRatio of active to total entry log spaceFragmentation slows reads and compaction. Track the ratio over time, not just at disk-full time.
DLQ arrival ratePer-topic metrics on {topic}-{subscription}-DLQ topicsMessages landing in the dead letter topic after exhausting redeliveries. Any arrival rate above zero on topics where messages must be processed is a consumer correctness problem.
ZK watch countecho wchs | nc <zk-host> 2181Watch explosions during consumer reconnect storms are a leading cause of ZK latency storms. Track the count as a time series so you can see the storm forming. Requires the 4-letter word whitelist mentioned above.
Netty allocator statsGET /admin/v2/broker-stats/allocator-stats/defaultDirect buffer pool utilization and fragmentation. Use when connection count is stable but direct memory keeps growing: the signature of a buffer leak.
Message expiration ratepulsar_subscription_msg_rate_expiredMessages deleted by TTL before consumers read them. Silent data loss from the consumer’s perspective. Should be zero on any topic where loss is unacceptable.
Cross-broker load distributionPer-broker rates, pulsar_lb_unload_bundle_totalCluster averages hide a single hot broker. Compare per-broker throughput, topic counts, and latency directly.

Because several Level 4 signals require JMX or admin API scraping rather than the Prometheus endpoint, plan the collection mechanism before you need it. Retrofitting a JMX exporter during a GC death spiral is not fun.

How to use this model

Audit honestly. Most teams sit between Level 1 and Level 2: they have liveness and rates, but no journal sync latency and no backlog alerting. The highest-value upgrades, in order of incident frequency they prevent:

  • Journal write-path trio (force write queue, journal sync P99, add-entry-in-progress). Catches the single most common Pulsar production failure before throughput collapses.
  • Backlog and unacked. Catches the silent consumer stall, including the dispatch freeze at the unacked limit that produces no errors anywhere.
  • ZooKeeper latency and session events. The number one leading indicator of cluster-wide cascade failures.
  • Direct memory. The gap that turns “heap is fine” dashboards into a 3 a.m. OOM.

Also decide what “expected traffic” means before alerting on rates and backlog. Idle clusters, cold starts after broker restarts (cache warmup takes 10-30 minutes), bookie journal replay, and rolling upgrades all produce Level 2 and 3 signals that look alarming but are normal. Gate alerts on uptime and traffic floors, or you will train the team to ignore the alerts that matter.

How Netdata helps

  • Bookie write-path correlation: Netdata collects BookKeeper journal metrics (bookie_journal_JOURNAL_SYNC, force write queue, add-entry-in-progress) alongside per-disk I/O utilization, so you can confirm in one view whether a journal latency spike is disk saturation or something else.
  • Broker and bookie in the same dashboard: the write throttle loop spans both layers. Seeing broker publish latency P99 next to bookie journal sync P99, per node, collapses the “is it the broker or the storage” question into seconds.
  • Backlog and redelivery per subscription: Netdata charts pulsar_subscription_back_log, unacked counts, and redelivery rates together, which is the exact correlation needed to distinguish “no consumers” from “stuck consumers.”
  • Anomaly detection on noisy gauges: signals like bookkeeper_server_ADD_ENTRY_IN_PROGRESS are burst-prone and defy static thresholds. ML-based anomaly flags catch the sustained non-zero drift without paging on normal traffic spikes.
  • System context for the silent failures: Netdata pairs Pulsar metrics with host-level signals (disk latency per device, TCP retransmits, process RSS) that the playbook calls out as expert-level evidence, including the RSS-minus-heap gap that exposes direct memory growth.