Apache Pulsar monitoring checklist: the signals every production cluster needs

Pulsar fails differently from most systems you operate. A cluster can report every process “up” while the write path stalls on a bookie journal disk, while a broker GC-pauses its ZooKeeper session away, or while a subscription silently freezes because its unacked message count hit a limit. Monitoring that only checks “is the broker running” misses almost every real Pulsar incident.

This checklist organizes the signals that matter into four maturity levels: survival, operational, mature, and expert. Each level builds on the previous one. If you are standing up monitoring for a new cluster, start at Level 1 and work down. If you are auditing an existing deployment, jump to the level you think you are at and check for gaps.

One framing note before the list: Pulsar is a two-layer system. Brokers (the serving layer) are stateless and handle connections, routing, and policies. BookKeeper bookies (the storage layer) are stateful and own durability. ZooKeeper (or the configured metadata store) coordinates both. Most cascades start in one layer and surface in another, which is why the checklist covers all three.

flowchart LR
  P[Producers] --> B[Brokers]
  C[Consumers] --> B
  B -->|write quorum Qw| BK[Bookies]
  B -->|metadata, ownership| ZK[ZooKeeper]
  BK -->|journal fsync| D[Journal disk]
  BK -->|entry log| L[Ledger disks]

Level 1: survival

The bare minimum. If any of these fail, the cluster is in immediate trouble. Everything here warrants a page when the gating conditions are met (sustained, previously running, real traffic).

Broker process health. curl -sf http://<broker>:8080/metrics responding, or GET /admin/v2/brokers/health returning 200. Necessary but not sufficient: a broker can answer HTTP while unable to reach bookies or ZooKeeper.
Bookie process health. curl -sf http://<bookie>:8000/metrics responding. Severity scales with quorum math: if the remaining bookies still satisfy your ensemble requirement, it is a ticket; below that, it is a page.
ZooKeeper process up. echo ruok | nc <zk-host> 2181 returning imok. ZK is the single point of truth for cluster topology; its failure cascades into everything.
Bookie server status. bookie_SERVER_STATUS on the bookie metrics endpoint. 1 = writable, 0 = read-only (usually disk full), -1 = unregistered. A read-only bookie cannot accept writes; if enough bookies go read-only, write quorum fails cluster-wide. Page on 0 only when sustained past startup, remaining writable bookies fall below ensemble size, and publish traffic is non-zero.
Message flow. pulsar_rate_in and pulsar_rate_out per topic. Zero publish rate during expected traffic means producers are failing or being rejected. Non-zero rate-in with zero rate-out means consumers are down or dispatch is blocked.
Bookie disk usage. bookie_ledger_dir_{path}_usage (percent) and bookie_ledger_writable_dirs (count). At the configured threshold (default around 95%) the bookie flips read-only. Monitor per volume, not aggregate: one full volume among many still degrades the bookie.

Level 2: operational

The professional baseline. This is what a competent team monitors in production, and it is where most clusters should land.

Signal	Why it matters	Warning sign
`pulsar_rate_in` / `pulsar_rate_out` (per namespace)	Heartbeat of message flow; divergence between the two means backlog is accumulating	Sustained gap, or drop below 10% of baseline during expected traffic
`pulsar_broker_publish_latency` P99	Primary SLI for producer experience: broker processing plus bookie write path plus quorum acks	P99 above 2x rolling baseline; healthy SSD-backed clusters typically see P99 under 10ms
`bookie_journal_JOURNAL_SYNC` P99	The single most critical latency metric in the stack; every write waits on journal fsync	P99 above 5ms on SSD, 20ms on HDD, or 2x your established baseline
`pulsar_subscription_back_log`	Consumer health and disk-fill predictor	Growing continuously for > 15 minutes with consumers connected; trend matters, not absolute value
`pulsar_active_connections`	FD pressure, direct memory pressure, connection leak detection	Approaching 80% of the FD limit, or unexplained growth over weeks
Broker and bookie memory (heap + direct)	GC death spiral and invisible direct-memory OOM	Heap > 85% with rising GC frequency; process RSS far above heap
Metadata store (ZK) latency	Leading indicator that degrades before any broker or bookie symptom	Sustained average above 10ms; above 50ms is warning; above 100ms is failure within minutes
`pulsar_authentication_failures_total`	Expired credentials, misconfigured clients, or unauthorized access	Sustained rate above 1/minute from production clients
Bookie disk usage growth rate	Capacity runway, not just current state	Growth trend projecting past 90% within your planning horizon
`pulsar_broker_lookup_failures`	New producers/consumers cannot connect while existing ones keep working: the classic grey failure	Failure ratio above 1% of lookups sustained > 5 minutes

Two instrumentation notes for this level. First, publish latency is broker-side only; it does not include client-to-broker network time. For true end-to-end latency, instrument at the application. Second, Pulsar does not expose JVM heap, GC, or direct memory through its Prometheus endpoint. You need JMX or process-level monitoring for those, and skipping them is one of the most common monitoring gaps in Pulsar deployments.

Level 3: mature

Everything above, plus the signals that let you see a cascade forming instead of discovering it in progress.

Subscription unacked messages. pulsar_subscription_unacked_messages against maxUnackedMessagesPerSubscription (default 200,000). At 100% of the limit the broker stops dispatching to that subscription with no explicit error. Alert above 50% of the limit sustained; the blockedSubscriptionOnUnackedMsgs flag in topic stats is the definitive “dispatch frozen” indicator.
Redelivery rate. pulsar_subscription_msg_rate_redeliver versus dispatch rate. Sustained redelivery above 10% of dispatch means consumers receive messages but fail to process them: poison messages, downstream failures, or ack timeout misconfiguration.
Managed ledger cache efficiency. pulsar_ml_cache_hits_rate versus pulsar_ml_cache_misses_rate, plus pulsar_ml_cache_evictions. Miss rate above 20% sustained (excluding the 10-30 minute cold start after a broker restart) means consumer reads are falling through to bookie disks, adding I/O load to the storage layer.
Bookie write-path pressure gauges. bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE and bookkeeper_server_ADD_ENTRY_IN_PROGRESS. The force-write queue should sit near zero; sustained depth is the earliest warning of journal disk saturation, rising before sync latency spikes. Add-entry in-progress should drain within seconds after a burst.
Under-replicated ledgers. auditor_NUM_UNDER_REPLICATED_LEDGERS on the auditor node, or bookkeeper shell listunderreplicated. Zero in steady state; a spike after a bookie failure is expected but must trend back to zero. A growing count means recovery is failing or bookies are failing faster than recovery completes.
Bundle unload rate. pulsar_lb_unload_bundle_total. Rare in steady state (under one per hour); above one per minute sustained outside maintenance means the load balancer is thrashing, and each unload drops client connections for the affected topics.
Throttled connections. pulsar_broker_throttled_connections. Should be zero; any sustained non-zero value means the broker is at capacity and protecting itself by limiting intake.
Topic count per broker. pulsar_topics_count compared across brokers. Any broker owning more than 2x the cluster average for over 30 minutes is a hotspot the load balancer failed to fix.
Bookie cache hit rates. bookie_write_cache_hits / bookie_write_cache_misses and bookie_read_cache_hits / bookie_read_cache_misses (DbLedgerStorage). Read cache miss rate above 50% sustained means bookies are serving cold reads from disk.
Entry log space ratio. bookie_ACTIVE_ENTRY_LOG_SPACE_BYTES versus the reclaimed-space counters (bookie_RECLAIMED_COMPACTION_SPACE_BYTES, bookie_RECLAIMED_DELETION_SPACE_BYTES). Active space staying high while reclaimed space does not grow, sustained, with disk usage above 70%, means BookKeeper GC is falling behind on space reclamation.
Geo-replication lag (if configured). pulsar_replication_backlog and pulsar_replication_delay_in_seconds. The delay metric is your actual RPO window; compare it against your declared RPO, not against zero.
TLS certificate expiration. Days-to-expiry on broker, bookie, and ZK certificates. Ticket at 30 days, escalate at 7. An expired cert is an instant, preventable outage.

Level 4: expert

The deep signals teams add after being burned. None of these are required for competent operation; all of them shorten diagnosis during the incidents that reach past the first three levels.

GC pause times correlated with latency spikes. The GC death spiral signature: pauses long enough to expire ZK sessions, causing bundle unloads, client reconnections, more metadata load, more GC. Look for the oscillation pattern: broker loses ownership, recovers, loses again.
Journal add-entry latency percentiles. bookie_journal_JOURNAL_ADD_ENTRY alongside JOURNAL_SYNC and JOURNAL_QUEUE_LATENCY to separate queueing from disk time.
Bookie read latency outliers across the fleet. bookkeeper_server_READ_ENTRY_REQUEST and bookie_BOOKIE_READ_ENTRY compared per bookie. One bookie consistently slower than its peers is a latent failure: the ensemble still meets quorum, but that bookie will drag the write path or cascade later.
ZooKeeper watch count. echo wchs | nc <zk-host> 2181. Watch explosions during mass consumer reconnect storms are a direct ZK killer and a leading indicator of the ZK latency storm pattern.
Netty buffer pool stats. GET /admin/v2/broker-stats/allocator-stats/default. Direct memory usage is not a Prometheus metric; this and JMX buffer pool metrics are how you see the off-heap pressure that standard JVM monitoring misses.
Message expiration rate. pulsar_subscription_msg_rate_expired. Non-zero on a topic where message loss is unacceptable means consumers are falling behind their TTL window and data is being silently deleted.
Connection churn. pulsar_connection_created_total_count versus pulsar_connection_closed_total_count. The difference should track active connections; a growing divergence over weeks is a leak.
Cross-broker load distribution. Per-broker publish latency and throughput spread, not cluster averages. Cluster averages in Pulsar are dangerous: one hot broker bottlenecks its topics while the fleet-wide numbers look fine.
System namespace visibility. Topics in pulsar/system (__change_events, transaction logs) have their own backlogs and disk usage. When system topics back up, the cluster behaves strangely (policies not propagating, transactions failing) while standard alerts stay quiet.

Signals most teams skip

These are the gaps that recur across deployments, drawn from the composite failure patterns in the playbook.

No direct memory monitoring. Heap dashboards look healthy right up to the OutOfDirectMemoryError. Monitor RSS versus heap, JMX buffer pools, or Netty allocator stats.
Journal and ledger disks monitored in aggregate. The journal disk is the write-path critical point and must be isolated in metrics (use the journalIndex label). Journal and ledger storage on the same disk is the number one architecture mistake; monitoring them as one number is the number two mistake.
ZooKeeper treated as invisible infrastructure. ZK latency degrades for days before the cascade. It is the top leading indicator for cluster-wide failure and is frequently unmonitored.
Alerting on backlog size instead of backlog growth. A high stable backlog is fine. A monotonically growing one is a problem at any size.
No unacked-message visibility. The dispatch freeze at the unacked limit is silent: consumers stay connected, backlog can look flat, and nothing moves.
Disk alerts at 90%. Bookie disk fill is predictable from publish rate and retention. Plan expansion at 70%, urgent at 85-90%. Consumer outages accelerate fill because backlog growth consumes bookie disk.
Single-bookie latency ignored. One degraded disk hides inside fleet averages until the bookie cascade pattern starts: failure, recovery I/O stress, next failure.

Also know what normal looks like so you do not page on it: high cache misses for 10-30 minutes after a broker restart, brief under-replication after a bookie restart, read-only bookie status during cold journal replay (5-10 minutes for large journals), bundle unloads during rolling upgrades, and a large “backlog” on a newly created subscription reading from earliest.

How Netdata helps

Netdata collects the broker (:8080/metrics), bookie (:8000/metrics), and ZooKeeper Prometheus endpoints, so all three layers of the checklist land on one dashboard with per-second resolution.
Correlating pulsar_broker_publish_latency against bookie_journal_JOURNAL_SYNC per bookie is the fastest way to answer “broker problem or storage problem” when producers complain about write latency.
Backlog growth rate, unacked message saturation, and redelivery rate can be charted per subscription, which surfaces the silent consumer stall before it reaches the unacked limit.
Bookie disk usage trending alongside subscription backlog growth makes capacity runway visible instead of a 90%-full surprise.
ML-based anomaly detection on rate and latency metrics flags deviation from baseline, which matches how most of these thresholds are actually defined (relative, not absolute).