Apache Pulsar monitoring maturity model: from survival to expert

Most Pulsar outages do not arrive without warning. The journal force write queue grows before publish latency spikes. ZooKeeper latency drifts upward for days before brokers lose session ownership. Direct memory climbs for weeks before the broker dies with an OutOfDirectMemoryError while heap dashboards look fine. The signals were there. The team just was not collecting them yet.

This is a four-level maturity model for Pulsar monitoring, from the bare minimum that tells you the cluster is alive to the deep signals operators add after their second or third incident. Use it to audit your current coverage and decide what to instrument next. The levels are cumulative: each one assumes everything below it is already in place.

The model covers the three components that matter: brokers (serving layer), BookKeeper bookies (storage layer), and the metadata store (ZooKeeper in most production deployments). A setup that only watches brokers is blind to the two failure domains behind the worst Pulsar incidents: bookie write-path stalls and ZooKeeper degradation.

flowchart TD
  L1["Survival: process up, bookie writable, rates flowing, disk not full"]
  L2["Operational: rates, backlog, publish P99, journal sync P99, ZK latency"]
  L3["Mature: unacked, ML cache, under-replicated ledgers, force write queue"]
  L4["Expert: GC pauses, direct memory, recovery progress, allocator stats"]
  L1 -->|adds| L2
  L2 -->|adds| L3
  L3 -->|adds| L4

Level 1: Survival

The question this level answers: is the cluster alive, and can it accept writes right now?

This is the minimum that distinguishes “Pulsar is down” from “Pulsar is up.” It will not tell you why anything failed, and it gives no advance warning. If any of these checks fail, the cluster is in immediate trouble.

Signal	Source	What it tells you
Broker process health	`GET http://<broker>:8080/metrics` or `/admin/v2/brokers/health`	Broker process is alive and its HTTP listener works. Necessary but not sufficient: a broker can respond while unable to reach bookies or ZooKeeper.
Bookie process health	`GET http://<bookie>:8000/metrics`	Bookie process is alive.
Bookie server status	`bookie_SERVER_STATUS`	`1` = writable, `0` = read-only (typically disk full), `-1` = unregistered. If enough bookies go read-only, write quorums cannot be met and writes to affected ledgers fail.
ZooKeeper liveness	4-letter command `ruok` on port 2181	The coordination layer is answering. Note that recent ZooKeeper versions require 4-letter words to be explicitly allowed via `4lw.commands.whitelist`.
Message rates	`pulsar_rate_in`, `pulsar_rate_out`	Data is actually flowing. Zero publish rate when producers should be active means producer failure or a broker that cannot accept writes.
Bookie disk usage	`bookie_ledger_dir_{path}_usage`, `bookie_ledger_writable_dirs`	Headroom before the bookie flips itself read-only at its configured disk usage threshold (default 0.95).

Two cautions at this level. First, broker liveness is a weak signal: the common degraded states (ZK session expired, GC death spiral, direct memory exhaustion) all leave the process running. Second, bookie cold starts take time. After a restart, journal replay can take 5-10 minutes for large journals, and the bookie may report read-only during replay. Do not page on transient states during startup.

Level 2: Operational

The question this level answers: is the cluster healthy under its current workload, and where is pressure building?

This is the professional baseline. A team at Level 2 can detect the most common production failure (the bookie journal write stall) and the most common consumer failure (growing backlog) before users notice.

Signal	Source	What it tells you
Per-namespace publish and dispatch rates	`pulsar_rate_in`, `pulsar_rate_out`	Deviation from baseline during expected-traffic periods is the earliest sign of producer or consumer failure. Divergence between the two means backlog is growing.
Subscription backlog	`pulsar_subscription_back_log`	Consumer health. Alert on sustained growth, not absolute size. A high but stable backlog is fine; a monotonically growing one is not.
Broker publish latency	`pulsar_broker_publish_latency` P99	The primary SLI for producer experience. Healthy SSD-backed clusters typically see P99 under 10ms. Alert on 2x degradation from your rolling baseline, not a fixed number.
Bookie journal sync latency	`bookie_journal_JOURNAL_SYNC` P99	The physical limit of write throughput. Every write waits for journal fsync. SSDs should hold P99 under 5ms, HDDs under 20ms. If this spikes, the entire write path stalls from bookie through broker to producer.
Broker and bookie memory	JMX heap metrics	Heap above 85% sustained with rising GC frequency precedes the GC death spiral.
Active connections	`pulsar_active_connections`	Load on the FD table and Netty direct memory. Unexplained growth over weeks indicates a connection leak.
ZooKeeper latency	ZK `stat` command, admin server, or broker-side metrics	Healthy is under 10ms average. Above 50ms sustained stalls broker metadata operations. Above 100ms, session expirations and cascading bundle unloads are minutes away.
Authentication failures	`pulsar_authentication_failures_total`	Expired tokens, misconfigured clients, or unauthorized attempts. Should be near zero in a stable environment.
Broker lookup failures	`pulsar_broker_lookup_failures` vs `pulsar_broker_lookup_answers`	The grey failure: the cluster looks up, but new producers and consumers cannot resolve topic ownership. Sustained failure rate above 1% of lookups indicates metadata store or bundle ownership problems.

Note the metric availability gotchas. Lookup metrics may not be present in all Pulsar versions, and per-topic rates depend on exposeTopicLevelMetricsInPrometheus (default true in 2.6 through 4.x; setting it to false switches the exporter to namespace-level aggregates). On clusters with thousands of topics, per-topic metrics can overwhelm Prometheus. Verify against your own /metrics output before writing alerts.

Level 3: Mature

The question this level answers: is the cluster degrading in ways that will become an incident in hours or days?

Level 3 signals are leading indicators. They fire before user-visible latency or errors, which is what buys you time to act during business hours instead of at 3 a.m.

Signal	Source	What it tells you
Per-subscription backlog and redelivery	`pulsar_subscription_back_log`, `pulsar_subscription_msg_rate_redeliver`	Redelivery above 10% of dispatch rate means consumers receive but fail to process: poison messages, downstream failures, or ack timeout misconfiguration.
Unacked messages	`pulsar_subscription_unacked_messages`	When this hits `maxUnackedMessagesPerSubscription` (default 200,000), the broker silently stops dispatching. Consumers stay connected, backlog may look flat, and no forward progress happens. Alert at 50% of the limit.
Managed ledger cache efficiency	`pulsar_ml_cache_hits_rate`, `pulsar_ml_cache_misses_rate`, `pulsar_ml_cache_evictions`	Hit rate above 80% means consumers read from broker memory. High misses plus high evictions mean the cache is undersized and reads are falling through to bookie disks.
Under-replicated ledgers	`replication_auditor_NUM_UNDER_REPLICATED_LEDGERS` on the auditor node	Data-loss risk. Must be zero in steady state. After a bookie failure it spikes and should trend back to zero as AutoRecovery works. A growing count means recovery is failing or bookies are failing faster than recovery completes.
Geo-replication backlog and delay	`pulsar_replication_backlog`, `pulsar_replication_delay_in_seconds`, `pulsar_replication_disconnected_count`	Backlog that never drains is your RPO window growing. Compare the delay metric against your declared RPO, not against zero.
Add entry in progress	`bookkeeper_server_ADD_ENTRY_IN_PROGRESS`	The bookie write queue depth. Should drain to near zero within seconds after a burst. Sustained growth means the journal disk cannot keep up and the write path is about to stall.
Journal force write queue	`bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE`	The earliest write-path saturation warning. It rises before journal sync latency spikes and before add-entry queues grow. Should sit at zero.
Bundle unload rate	`pulsar_lb_unload_bundle_total`	Expected during rolling restarts. Sustained unloading above 1 per minute outside maintenance means the load balancer is thrashing, and each unload drops client connections for the affected topics.
Throttled connections	`pulsar_broker_throttled_connections`	The broker is protecting itself from overload by limiting intake. Should be zero.
Topic count per broker	`pulsar_topics_count`	Hotspot detection. Any broker owning more than 2x the cluster average for 30+ minutes indicates imbalance the load balancer is not fixing.
TLS certificate expiration	`openssl x509 -enddate` against cert files or live endpoints	An expired cert is a total, preventable communication failure. Ticket at 30 days, escalate at 7.
Bookie entry log space	`bookie_ENTRY_LOG_SPACE_BYTES` vs `bookie_ACTIVE_ENTRY_LOG_SPACE_BYTES`	Total much larger than active means BookKeeper GC is falling behind on reclaiming deleted data, and disk will fill despite logical deletions.

Level 3 is also where you start correlating failure patterns: journal force write queue plus add-entry-in-progress plus publish latency is the write throttle loop; GC pauses plus ZK latency plus bundle unloads plus flapping connections is the GC death spiral.

Level 4: Expert

The question this level answers: why did it degrade, and what will fail next?

These signals come from incident retrospectives. Several are not exposed through Pulsar’s Prometheus endpoint at all, which is exactly why most teams lack them until after the incident that needed them.

Signal	Source	What it tells you
GC pause times	JMX `java.lang:type=GarbageCollector`, GC logs	Pulsar 2.11+ exposes JVM heap (`jvm_memory_bytes_used`) and GC counters (`jvm_gc_collection_seconds`) on `/metrics`, but not per-pause durations, so GC logs/JMX remain the source for pause length. Long full GC pauses approaching the ZK session timeout (tens of seconds by default) risk session expiry; even shorter pauses show up as publish latency spikes. Correlate pauses with latency spikes and session events.
Direct memory	JMX `java.nio:type=BufferPool,name=direct`, or broker `/metrics` `jvm_memory_direct_bytes_used` (Pulsar 2.11+)	The invisible killer. Brokers default to `-XX:MaxDirectMemorySize=4g` in the shipped `pulsar_env.sh`, Netty buffers and the managed ledger cache live here, and exhaustion crashes the broker while heap looks healthy. Also watch process RSS minus heap as a proxy.
Journal add entry percentiles	`bookie_journal_JOURNAL_ADD_ENTRY`	Time to record an entry in the journal, distinct from the fsync itself. Percentile drift here separates journal queueing problems from raw disk problems.
Ledger recovery progress	`replication_auditor_NUM_UNDER_REPLICATED_LEDGERS` trend, `replication_auditor_UNDER_REPLICATED_LEDGERS_TOTAL_SIZE`	Not just the count but the direction and rate. Recovery I/O competes with foreground traffic on surviving bookies, so watch journal sync latency on the survivors while recovery runs. This is how you see the bookie cascade forming.
Entry log fragmentation	Ratio of active to total entry log space	Fragmentation slows reads and compaction. Track the ratio over time, not just at disk-full time.
DLQ arrival rate	Per-topic metrics on `{topic}-{subscription}-DLQ` topics	Messages landing in the dead letter topic after exhausting redeliveries. Any arrival rate above zero on topics where messages must be processed is a consumer correctness problem.
ZK watch count	`echo wchs \| nc <zk-host> 2181`	Watch explosions during consumer reconnect storms are a leading cause of ZK latency storms. Track the count as a time series so you can see the storm forming. Requires the 4-letter word whitelist mentioned above.
Netty allocator stats	`GET /admin/v2/broker-stats/allocator-stats/default`	Direct buffer pool utilization and fragmentation. Use when connection count is stable but direct memory keeps growing: the signature of a buffer leak.
Message expiration rate	`pulsar_subscription_msg_rate_expired`	Messages deleted by TTL before consumers read them. Silent data loss from the consumer’s perspective. Should be zero on any topic where loss is unacceptable.
Cross-broker load distribution	Per-broker rates, `pulsar_lb_unload_bundle_total`	Cluster averages hide a single hot broker. Compare per-broker throughput, topic counts, and latency directly.

There is no dedicated fragmentation metric; derive it from bookie_ENTRY_LOG_SPACE_BYTES vs bookie_ACTIVE_ENTRY_LOG_SPACE_BYTES across BookKeeper versions.

Because several Level 4 signals require JMX or admin API scraping rather than the Prometheus endpoint, plan the collection mechanism before you need it. Retrofitting a JMX exporter during a GC death spiral is not fun.

How to use this model

Audit honestly. Most teams sit between Level 1 and Level 2: they have liveness and rates, but no journal sync latency and no backlog alerting. The highest-value upgrades, in order of incident frequency they prevent:

Journal write-path trio (force write queue, journal sync P99, add-entry-in-progress). Catches the single most common Pulsar production failure before throughput collapses.
Backlog and unacked. Catches the silent consumer stall, including the dispatch freeze at the unacked limit that produces no errors anywhere.
ZooKeeper latency and session events. The number one leading indicator of cluster-wide cascade failures.
Direct memory. The gap that turns “heap is fine” dashboards into a 3 a.m. OOM.

Also decide what “expected traffic” means before alerting on rates and backlog. Idle clusters, cold starts after broker restarts (cache warmup takes 10-30 minutes), bookie journal replay, and rolling upgrades all produce Level 2 and 3 signals that look alarming but are normal. Gate alerts on uptime and traffic floors, or you will train the team to ignore the alerts that matter.

How Netdata helps

Bookie write-path correlation: Netdata collects BookKeeper journal metrics (bookie_journal_JOURNAL_SYNC, force write queue, add-entry-in-progress) alongside per-disk I/O utilization, so you can confirm in one view whether a journal latency spike is disk saturation or something else.
Broker and bookie in the same dashboard: the write throttle loop spans both layers. Seeing broker publish latency P99 next to bookie journal sync P99, per node, collapses the “is it the broker or the storage” question into seconds.
Backlog and redelivery per subscription: Netdata charts pulsar_subscription_back_log, unacked counts, and redelivery rates together, which is the exact correlation needed to distinguish “no consumers” from “stuck consumers.”
Anomaly detection on noisy gauges: signals like bookkeeper_server_ADD_ENTRY_IN_PROGRESS are burst-prone and defy static thresholds. ML-based anomaly flags catch the sustained non-zero drift without paging on normal traffic spikes.
System context for the silent failures: Netdata pairs Pulsar metrics with host-level signals (disk latency per device, TCP retransmits, process RSS) that the playbook calls out as expert-level evidence, including the RSS-minus-heap gap that exposes direct memory growth.