Apache Pulsar Operations Guides on Netdata

Apache Pulsar abandoned subscriptions: cursor leaks that pin storage forever

Mon, 01 Jan 0001 00:00:00 +0000

Bookie disk usage climbs steadily across the cluster, but publish rates are flat and there is no traffic spike. Per-subscription backlogs look fine for the subscriptions you know about. When you try to delete an old topic to reclaim space, the operation fails with a message about active subscriptions. The topic has not had a consumer in weeks.

This is the signature of a silent subscription cursor leak. Applications create dynamically-named durable subscriptions and then disconnect without unsubscribing. Each abandoned subscription leaves behind a cursor pinned at its last acknowledged position. Pulsar cannot delete any message after that cursor’s position until the subscription acknowledges past it, which never happens because no consumer is connected. Storage grows monotonically, and the growth is invisible in per-subscription backlog dashboards because those dashboards only track subscriptions the team knows about.

Apache Pulsar active connections climbing: connection leaks and file descriptor exhaustion

Mon, 01 Jan 0001 00:00:00 +0000

pulsar_active_connections has been drifting upward for weeks. Not spiking, not crashing, just rising a few connections a day. Then one morning new producers start failing to connect, or the broker dies with OutOfDirectMemoryError, and the postmortem shows the leak was visible the whole time.

Every TCP connection costs the broker one file descriptor and a slice of Netty direct memory. Connections that are established but never closed accumulate silently until one of those two resources hits its ceiling, at which point the failure is abrupt: refused connections or a crash, with no graceful degradation in between.

Apache Pulsar authentication failures: expired tokens, wrong credentials, and brute force

Mon, 01 Jan 0001 00:00:00 +0000

Authentication failures in Apache Pulsar surface through pulsar_authentication_failures_total and AuthenticationException entries in broker logs. In a stable environment, this counter sits near zero. When it spikes, the temporal pattern matters more than the absolute volume: a single burst from a known client after a credential rotation is normal; sporadic low-rate failures from production IPs point to misconfiguration; a sustained flood from unknown sources signals brute force or credential compromise.

Each failure closes the TCP connection. The client reconnects, authenticates again, and either succeeds or fails. This loop consumes file descriptors, Netty direct memory buffers, and CPU on both sides. Brokers check auth data expiry every authenticationRefreshCheckSeconds (default 60 seconds). If the client supports auth refreshing, the broker sends a CommandAuthChallenge. If not, and the credential is expired, the broker disconnects. An expired token does not cause a single failure; it causes a continuous reconnect storm until the credential is refreshed.

Apache Pulsar authorization failures: the AuthorizationException only the logs will show

Mon, 01 Jan 0001 00:00:00 +0000

A producer or consumer authenticates successfully but cannot produce, consume, or manage a topic. The client receives an AuthorizationException or a generic permission error. Your dashboard shows nothing unusual because Pulsar exposes no authorization failure counter in its Prometheus metrics. The only evidence is in broker logs.

This is a common blind spot. Authentication failures have a dedicated metric (pulsar_authentication_failures_total), but authorization failures do not. A client with valid credentials but missing role permissions fails silently from a metrics perspective. You find out when an application team reports errors or when someone greps the broker log.

Apache Pulsar AutoRecovery stalled: under-replicated ledgers that never heal

Mon, 01 Jan 0001 00:00:00 +0000

Under-replicated ledgers are the durability canary for your Pulsar cluster. When a bookie fails, ledger fragments on it lose redundancy and the under-replicated count spikes. AutoRecovery is supposed to drive that count back to zero. When the count stays flat or grows instead of trending down, recovery has stalled.

A stall is not slow recovery. Slow recovery means the count is decreasing, just not fast enough. A stall means it is frozen or climbing. The diagnostic paths and fixes are completely different.

Apache Pulsar backlog age vs size: the latency depth alone cannot show

Mon, 01 Jan 0001 00:00:00 +0000

Backlog size tells you how much data is waiting. Backlog age tells you how long it has been waiting. Monitoring only one leaves you blind to entire classes of consumer health failures.

A stable backlog of 1,000 entries is unremarkable if those entries are 5 seconds old. The same 1,000 entries from 5 hours ago means a cursor is stuck, a consumer is down, or an application has silently stopped acknowledging. Size alone cannot distinguish these scenarios. Age can.

Apache Pulsar backlog quota exceeded: producers held or rejected when consumers stall

Mon, 01 Jan 0001 00:00:00 +0000

Producers are timing out or receiving errors, but the root cause is not on the producer side. A subscription backlog has crossed a configured quota, and the backlog policy has kicked in. Depending on the policy, producers are now blocked, rejected, or the oldest messages are being silently deleted. The symptom is a producer outage or data loss, but the root cause is a consumer that stopped keeping up.

The blast radius depends entirely on a namespace-level policy that many teams set once and forget. The same growing backlog can produce three different outcomes: producer timeouts, producer exceptions, or silent message eviction. Knowing which policy is active on the affected namespace is the first thing you need before touching anything.

Apache Pulsar bookie add-entry queue not draining: writes arriving faster than the disk can commit

Mon, 01 Jan 0001 00:00:00 +0000

When bookkeeper_server_ADD_ENTRY_IN_PROGRESS grows and does not drain to near-zero within seconds of a traffic burst, the bookie write path is saturated. Writes are arriving faster than the journal disk can fsync them, or the write thread is blocked by GC. Left unaddressed, this causes broker-side timeouts, producer failures, and throughput collapse.

This metric is a gauge, not a counter. Absolute value matters less than trend. Spikes during traffic bursts are normal; sustained positive growth is not. The queue fills and stays filled because the bookie cannot commit entries as fast as they arrive. By the time broker publish latency spikes, the bookie has already been saturated for seconds or minutes.

Apache Pulsar bookie disk filling: runway to read-only and how to reclaim space

Mon, 01 Jan 0001 00:00:00 +0000

Bookie disk usage decides whether your cluster can accept writes. When a bookie’s ledger directories fill past the configured threshold (default 95%), the bookie transitions to read-only mode and stops accepting new entries. If enough bookies go read-only, the write quorum for affected ledgers cannot be satisfied, and producers start seeing errors.

BookKeeper needs disk headroom to compact entry logs and reclaim space from deleted ledgers. At 95% (diskUsageThreshold), the bookie goes read-only and suspends GC entirely. At that point, shortening retention or deleting topics will not reclaim space because the compaction that rewrites entry logs is itself suspended.

Apache Pulsar bookie failure cascade: recovery I/O that topples surviving bookies

Mon, 01 Jan 0001 00:00:00 +0000

Bookies are failing one at a time. Each time one drops, AutoRecovery starts replicating its ledgers to the survivors. The recovery reads and writes add I/O load to bookies already handling foreground traffic. Journal sync latency climbs, publish latency follows, then the next bookie starts timing out. It fails, and the cycle accelerates.

This is the bookie failure cascade. The root cause is not the initial bookie failure; it is the recovery mechanism competing with production traffic on bookies near their I/O ceiling. With default settings, AutoRecovery triggers immediately (lostBookieRecoveryDelay=0) and reads entries in batches of 100 (rereplicationEntryBatchSize=100) from surviving bookies, then writes them locally.

Apache Pulsar bookie journal and ledger storage on one disk: the #1 architecture mistake

Mon, 01 Jan 0001 00:00:00 +0000

Every BookKeeper bookie has two storage responsibilities with fundamentally different I/O profiles. The journal is a write-ahead log: sequential writes, fsync’d per entry or batch, on the critical path of every producer acknowledgment. Entry logs and ledger indexes (managed by DbLedgerStorage, the default storage backend) store message payloads and serve random reads whenever consumers catch up on historical data.

These workloads are incompatible on a single physical disk. When they share a device, consumer catch-up reads and BookKeeper garbage collection interrupt the journal’s sequential write pattern. Fsync latency spikes. Every producer writing through that bookie stalls. Because Pulsar’s write quorum requires acknowledgments from multiple bookies, one saturated bookie can bottleneck the entire topic.

Apache Pulsar bookie read latency high: catch-up reads competing with the write path

Mon, 01 Jan 0001 00:00:00 +0000

When bookkeeper_server_READ_ENTRY_REQUEST or bookie_BOOKIE_READ_ENTRY P99 latency rises well above baseline, the cause is usually not an isolated disk problem. It is consumers draining backlog (catch-up reads) generating a high volume of cold reads that bypass both the broker’s managed ledger cache and the bookie’s read cache, hitting ledger storage disks directly. If journal and ledger directories share a physical device, those read I/O operations compete with journal fsyncs and the write path degrades too.

Apache Pulsar bookie read-only: disk full and bookie_SERVER_STATUS at zero

Mon, 01 Jan 0001 00:00:00 +0000

bookie_SERVER_STATUS == 0 means the bookie has transitioned to read-only mode and is no longer accepting writes. In most cases this is a self-protective response to ledger disk usage crossing diskUsageThreshold (default 0.95). The bookie keeps serving reads, but any topic whose ensemble includes this bookie may fail to meet its write quorum, causing broker-side write errors and producer timeouts.

The real danger is the cascade. If enough bookies in the ensemble go read-only simultaneously, the write quorum (Qa) cannot be satisfied for new ledger creation, and all writes to affected topics fail. This can look like a cluster-wide outage even though the root cause is disk exhaustion on a few nodes.

Apache Pulsar broker down: telling a dead broker from a fenced one

Mon, 01 Jan 0001 00:00:00 +0000

Your alerting fired: the broker’s HTTP admin endpoint on :8080 has been unreachable for more than two minutes, and the broker was previously running, so this is not a fresh deploy or a rolling restart. Pager says “broker down.” That phrase hides two very different incidents.

The first is a hard death: the JVM crashed, got OOM-killed, or the host failed. The process is gone and the OS will tell you so in seconds. The second is a fenced broker: the process is alive, maybe even accepting TCP connections, but it has lost its ZooKeeper session and with it ownership of every namespace bundle it served. From the client’s perspective the broker is down. From the process table’s perspective it is fine. The fix, the blast radius, and the forensics are completely different for each.

Apache Pulsar broker GC death spiral: heap pressure, stop-the-world pauses, and lost topic ownership

Mon, 01 Jan 0001 00:00:00 +0000

A broker is losing topic ownership, recovering, and losing it again. Clients reconnect in bursts. Lookup failures climb. Publish latency is erratic even though traffic is flat and CPU looks busy for no obvious reason. The broker process never actually dies, which is why restarts and instance health checks keep “fixing” it for ten minutes at a time.

This is the GC death spiral: the broker JVM is under memory pressure, full GC pauses stop the world long enough to miss ZooKeeper heartbeats, the session expires, the broker is fenced and loses its bundles, and the resulting client reconnect and metadata churn allocates even more memory, which triggers more GC. The loop feeds itself. The signature is oscillation, not a clean failure.

Apache Pulsar broker hotspot: one broker owning far more topics than the rest

Mon, 01 Jan 0001 00:00:00 +0000

You have a multi-broker Pulsar cluster where one broker carries a disproportionate share of topic ownership. Cluster-wide averages look acceptable, but that single broker shows elevated GC pressure, higher publish latency, more active connections, or a larger heap footprint than its peers. The imbalance may have built for hours or days without triggering an alert because aggregate metrics hid it.

In Pulsar, topics are not assigned to brokers individually. They are assigned at the namespace bundle level. Each namespace is sharded into bundles (hash-range slices of the topic namespace), and each bundle is owned by exactly one broker at a time. A topic lands in a bundle by hashing its name. When a bundle accumulates too many topics, too much throughput, or too many sessions, the load balancer should split it and redistribute ownership. When this does not happen, one broker ends up owning a hot bundle and becomes a bottleneck.

Apache Pulsar broker lookup failures: new clients cannot find their topic

Mon, 01 Jan 0001 00:00:00 +0000

Your dashboards look fine. Messages are flowing, publish latency is normal, consumers are draining backlog. But the new service you just deployed cannot start: its producer hangs or fails trying to connect to its topic. Restarting it does not help. Existing clients are unaffected.

This is the classic Pulsar grey failure. Topic lookup is how every producer and consumer discovers which broker owns its topic. Existing connections keep working because they already resolved ownership. New traffic fails when lookups cannot resolve. Process health checks, message rates, and bookie metrics can all look healthy until a deploy, restart, scale event, or broker bounce forces mass re-lookup.

Apache Pulsar bundle unload thrashing: the load balancer that never converges

Mon, 01 Jan 0001 00:00:00 +0000

The primary signal is pulsar_lb_unload_bundle_total, a counter exposed on the broker Prometheus metrics endpoint. In steady state it barely moves: bundle ownership changes are rare, ideally less than once per hour. When this counter climbs past one unload per minute and there is no rolling upgrade, no broker restart, and no planned maintenance, the load balancer is thrashing.

Each unload transfers a namespace bundle (a hash-range slice of topics) from one broker to another. Each transfer drops the TCP connections for every producer and consumer on those topics. Clients reconnect automatically, but reconnection generates metadata operations against ZooKeeper, new ledger creation in BookKeeper, and a latency blip measured in tens of milliseconds per affected topic. When unloads happen every few seconds, those blips stack into sustained degradation: elevated publish latency, client reconnection storms, and metadata store pressure that feeds back into more instability.

Apache Pulsar consumers connected but not acknowledging: the silent consumer stall

Mon, 01 Jan 0001 00:00:00 +0000

Your Pulsar cluster looks healthy. Brokers are up, bookies are writable, producers are publishing at normal rates. But for one topic or subscription, pulsar_rate_out has gone flat. Messages are flowing in but nothing is coming out. The backlog is climbing. Consumers show as connected, but the acknowledgment rate is zero.

No error surfaces in broker logs. The broker serves connections and bookie writes are fast. The problem lives in the consumer application or in the interaction between consumer configuration and broker dispatch policy.

Apache Pulsar dead letter queue filling: maxRedeliveryCount and the DLQ topic

Mon, 01 Jan 0001 00:00:00 +0000

The dead letter topic is where Pulsar parks messages that exhausted their retry budget. When you set DeadLetterPolicy.builder().maxRedeliverCount(N).build() on a Shared or Key_Shared subscription, the broker stops redelivering a message after N failed attempts, routes it to {topic}-{subscription}-DLQ, and auto-acks it in the origin subscription so backlog clears.

A growing DLQ arrival rate is a ledger of application processing failures. A few poison messages parked for inspection is the DLQ working as designed. A steady stream of thousands of messages per second is a systemic failure. Unmonitored, the DLQ topic itself becomes the incident: messages accumulate, storage grows, and nobody notices until disk fills or a downstream team asks why their data is missing.

Apache Pulsar ensemble, write quorum, and ack quorum: what E, Qw, and Qa actually guarantee

Mon, 01 Jan 0001 00:00:00 +0000

Every persistent message in Apache Pulsar passes through BookKeeper’s quorum system, governed by three parameters: ensemble size (E), write quorum (Qw), and ack quorum (Qa). Configured per namespace and inherited by topics, they determine how many bookies receive each entry, how many must confirm a durable write before the producer is acknowledged, and how the cluster behaves when bookies fail, slow down, or are taken offline.

The nuance is in what each parameter does not guarantee. Write quorum is not a durability floor for acknowledged entries. Ack quorum is not the number of copies that will exist after the write completes. Ensemble size constrains ledger creation in ways that surprise teams during maintenance windows: you can have plenty of bookies but still be unable to create new ledgers.

Apache Pulsar entry log GC falling behind: reclaimed space that never comes back

Mon, 01 Jan 0001 00:00:00 +0000

A bookie’s ledger disk is filling. Retention policies have deleted ledgers, TTL has expired messages, and cursors have advanced past the data. By every logical measure, the space should be free. But disk usage keeps climbing, and the bookie is heading toward read-only.

BookKeeper does not store each ledger in a separate file. Entry logs are large append-only files that interleave entries from many ledgers, written sequentially as they arrive. When a ledger is deleted, its entries remain physically embedded in the entry log alongside data from other ledgers that are still active. The only way to reclaim that space is compaction: the garbage collector reads the live entries from an old entry log, writes them into a new file, then deletes the old one. If compaction stalls, throttles to a crawl, or never triggers, dead data accumulates indefinitely.

Apache Pulsar geo-replication backlog: replication lag and your real RPO window

Mon, 01 Jan 0001 00:00:00 +0000

Geo-replication in Pulsar is asynchronous by design. Messages are persisted locally and acknowledged to producers before they reach the remote cluster. Every message sitting in the replication backlog is a message that would be lost if you failed over right now. The replication backlog is your recovery point objective (RPO) window, measured in real time.

The common operational mistake is checking whether replication is “working” (connected, producing to the remote cluster) without checking how far behind it is. A replicator can be fully connected and actively shipping messages while being hours behind. The dashboard shows green. The failover plan assumes near-zero data loss. The reality is data loss measured in hours.

Apache Pulsar geo-replication disconnected: a remote cluster that stopped receiving

Mon, 01 Jan 0001 00:00:00 +0000

The symptom is unmistakable: pulsar_replication_disconnected_count is non-zero for all replicators to a specific remote cluster, pulsar_replication_connected_count has dropped to zero, and pulsar_replication_backlog is growing at the rate of local publish throughput. The remote cluster has stopped receiving messages. Local producers and consumers continue working normally because Pulsar geo-replication is asynchronous. Messages persist locally first, then replicate. The damage is silent and cumulative.

Every unreplicated message widens the recovery point objective (RPO) gap. If you fail over to the remote cluster right now, those messages are gone. The backlog is your exposure, measured in entries and in seconds via pulsar_replication_delay_in_seconds.

Apache Pulsar journal force write queue growing: the earliest write-saturation signal

Mon, 01 Jan 0001 00:00:00 +0000

bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE measures the depth of pending fsync batches inside a BookKeeper bookie’s journal write path. In a healthy system this gauge sits at or near zero, draining between write groups. When it sustains a non-zero depth, the journal disk cannot commit writes durably fast enough to keep up with incoming traffic.

This signal rises before bookie_journal_JOURNAL_SYNC latency spikes and before bookkeeper_server_ADD_ENTRY_IN_PROGRESS grows. If you are not watching the force write queue, your first indication of write-path saturation will be broker publish latency degradation or producer timeouts, which means you are already two or three steps into the cascade.

Apache Pulsar ledger rollover latency spikes: the periodic blip that is usually normal

Mon, 01 Jan 0001 00:00:00 +0000

You see a periodic sub-second spike in publish latency on your Pulsar brokers. It happens at regular intervals, lasts a fraction of a second, and then latency returns to baseline. No errors, no bookie failures, no GC pauses. The spike is visible in P99 publish latency but barely touches P50. It correlates across topics on the same broker but not across the entire cluster.

This is almost certainly a ledger rollover. When a managed ledger fills its current BookKeeper ledger (by entry count, size, or time), the broker seals that ledger and creates a new one. The creation step involves metadata store writes (ZooKeeper in most Pulsar 3.x deployments, Oxia in newer ones), which add latency to the write path for the duration of the operation. The result is a brief publish latency blip, typically under 100ms, that repeats at intervals determined by your managed ledger configuration.

Apache Pulsar LedgerFencedException: split ownership and fencing loops

Mon, 01 Jan 0001 00:00:00 +0000

LedgerFencedException appears when a broker tries to write to a BookKeeper ledger that another broker has already fenced. The error string is explicit: “Ledger has been fenced off. Some other client must have opened it to read.”

In most production environments, this is expected noise during planned failover. It becomes a problem when fencing occurs without a corresponding bundle transfer, or when two brokers fence each other’s ledgers in a loop.

Apache Pulsar managed ledger cache miss rate high: consumer reads falling through to bookies

Mon, 01 Jan 0001 00:00:00 +0000

The managed ledger cache is the broker’s off-heap buffer of recently written entries. Connected consumers read from it at sub-millisecond latency. On a cache miss, the broker issues a read to the BookKeeper bookie ensemble holding that ledger segment, which adds disk I/O, network bandwidth consumption between broker and bookie, and consumer read latency.

A sustained cache miss rate above 20% after warmup means the broker is serving a meaningful fraction of consumer reads from storage instead of memory. The extra bookie read traffic loads storage disks that also serve journal writes. If bookie read latency rises enough, publish latency follows. This is the leading edge of the Backlog Cascade: consumer reads and producer writes converging on the same physical disks.

Apache Pulsar message redelivery storm: poison messages and consumers making no progress

Mon, 01 Jan 0001 00:00:00 +0000

Subscription redelivery rate is climbing. Consumers are connected, the dispatch rate (pulsar_rate_out) looks healthy, but messages keep cycling back without being acknowledged. The backlog may even appear stable or zero, because pulsar_subscription_back_log measures messages not yet dispatched, not messages dispatched but unacked. The definitive signal is pulsar_subscription_msg_rate_redeliver trending upward relative to dispatch. When redelivery exceeds 10% of your dispatch rate, something is wrong. When it approaches 100%, you have zero forward progress: the broker burns memory and network re-sending messages that will never succeed.

Apache Pulsar messages expiring before consumers read them: TTL and silent loss

Mon, 01 Jan 0001 00:00:00 +0000

When pulsar_subscription_msg_rate_expired is non-zero on a topic where every message matters, messages are being silently deleted before consumers can read them. No errors appear in consumer logs. No producer failures occur. The broker’s TTL mechanism acknowledges messages on behalf of the subscription without ever delivering them to the consumer application.

The backlog can hide the problem. TTL expiry trims unacked messages from subscription cursors, so backlog can appear stable or declining while real consumer lag grows underneath. Operators monitoring backlog size alone see a healthy-looking number. The data loss is invisible unless you specifically monitor the expiry rate.

Apache Pulsar metadata store latency: the leading indicator before every cluster-wide failure

Mon, 01 Jan 0001 00:00:00 +0000

Metadata store latency is the earliest signal that a Pulsar cluster is heading toward a widespread outage. In most 3.x deployments the metadata store is ZooKeeper. Pulsar 3.3.0 introduced experimental Oxia support as an eventual replacement. Every broker, bookie, and load balancer operation that touches cluster topology, bundle ownership, schema lookups, ledger metadata, or cursor persistence routes through this store. When round-trip latency for those operations rises, the symptoms appear downstream in brokers and bookies, but the root cause is upstream.

Apache Pulsar monitoring checklist: the signals every production cluster needs

Mon, 01 Jan 0001 00:00:00 +0000

Pulsar fails differently from most systems you operate. A cluster can report every process “up” while the write path stalls on a bookie journal disk, while a broker GC-pauses its ZooKeeper session away, or while a subscription silently freezes because its unacked message count hit a limit. Monitoring that only checks “is the broker running” misses almost every real Pulsar incident.

This checklist organizes the signals that matter into four maturity levels: survival, operational, mature, and expert. Each level builds on the previous one. If you are standing up monitoring for a new cluster, start at Level 1 and work down. If you are auditing an existing deployment, jump to the level you think you are at and check for gaps.

Apache Pulsar monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Most Pulsar outages do not arrive without warning. The journal force write queue grows before publish latency spikes. ZooKeeper latency drifts upward for days before brokers lose session ownership. Direct memory climbs for weeks before the broker dies with an OutOfDirectMemoryError while heap dashboards look fine. The signals were there. The team just was not collecting them yet.

This is a four-level maturity model for Pulsar monitoring, from the bare minimum that tells you the cluster is alive to the deep signals operators add after their second or third incident. Use it to audit your current coverage and decide what to instrument next. The levels are cumulative: each one assumes everything below it is already in place.

Apache Pulsar NotEnoughBookiesException: new ledgers cannot be created and writes fail

Mon, 01 Jan 0001 00:00:00 +0000

NotEnoughBookiesException (BookKeeper error code -6, surfaced as BKNotEnoughBookiesException) fires when the ensemble placement policy cannot satisfy the ensemble size (E) and write quorum (Qw) requirements for a new ledger. The broker requests a new ledger on size or time rollover, during topic recovery, and during compaction. If the policy cannot form a valid ensemble, ledger creation fails, the managed ledger has nowhere to write, and messages cannot be persisted durably.

Apache Pulsar OutOfDirectMemoryError: the off-heap crash JVM heap dashboards never show

Mon, 01 Jan 0001 00:00:00 +0000

The broker is down. Your JVM dashboard shows heap at 55%, GC pauses normal, no heap OOM anywhere. Then someone opens the broker log and finds the real cause:

io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: ..., max: ...)

or the plainer JDK variant, java.lang.OutOfMemoryError: Direct buffer memory. In recent Pulsar versions the process may have exited deliberately: PulsarByteBufAllocator catches the allocation failure and, with the default -Dpulsar.allocator.exit_on_oom=true, kills the JVM rather than limp along half-broken. Either way, the broker is dead in the part your heap monitoring was not watching.

Apache Pulsar publish latency high: reading pulsar_broker_publish_latency P99

Mon, 01 Jan 0001 00:00:00 +0000

Your alert fired on sustained P99 elevation of pulsar_broker_publish_latency above 2x the rolling baseline. Producers are seeing slow acknowledgements.

The metric pulsar_broker_publish_latency is a Summary metric exposed on the broker Prometheus endpoint. It measures the time from when the broker receives a message from a producer through the BookKeeper write path (write quorum Qw, ack quorum Qa) and back to the client callback. It is broker-side only: it excludes client-to-broker network time and producer-side batching delay. The Summary type provides quantiles at 0.5, 0.95, 0.99, 0.999, 0.9999, and 1.0. Alert on P99. P50 can look healthy while P99 is spiking, and it is the tail that triggers producer timeouts.

Apache Pulsar subscription backlog growing: consumers falling behind producers

Mon, 01 Jan 0001 00:00:00 +0000

A growing subscription backlog means producers are publishing faster than consumers can acknowledge. The absolute backlog size matters less than its trajectory. A high but stable backlog is normal for lagged or replay consumers. A monotonically increasing backlog is a problem regardless of size: it consumes bookie disk until the bookie goes read-only, the backlog quota trips and throttles producers, or retention and TTL silently delete messages the consumer never saw.

Apache Pulsar throttled connections: the broker shedding load under pressure

Mon, 01 Jan 0001 00:00:00 +0000

Your dashboard shows pulsar_broker_throttled_connections climbing from zero, producers are complaining about elevated send latency, and some clients are timing out on new operations. The broker process is up, the health endpoint returns 200, and heap looks fine. The broker is not broken. It is defending itself.

Connection throttling is a protective mechanism. When the broker’s internal send queues on a connection back up beyond a configured ceiling, it stops reading new requests from that TCP connection (it disables auto-read on the Netty channel) until the backlog drains. The throttled connection count tells you how many connections are currently in that state. In a healthy cluster this gauge is zero. Any sustained non-zero value means the broker is at or near its capacity for the work being pushed through it.

Apache Pulsar TLS certificate expiry: the silent, total outage no metric warns you about

Mon, 01 Jan 0001 00:00:00 +0000

When every component in a Pulsar cluster uses TLS (brokers, bookies, ZooKeeper, proxies, clients), a single expired certificate causes immediate handshake failures. Existing connections may persist briefly, but every new connection attempt fails the TLS handshake. Producers cannot publish. Consumers cannot subscribe. Brokers cannot reach bookies. Brokers cannot reach ZooKeeper.

Pulsar does not expose a metric for certificate expiry. There is no pulsar_cert_days_until_expiry gauge. The first signals you see are indirect: authentication failures spike, connections drop, and SSL handshake exceptions fill the logs. By that point, the outage is already happening. The only reliable defense is external certificate expiry monitoring that alerts you days or weeks before the cert becomes invalid.

Apache Pulsar topic ownership oscillation: brokers fighting over the same bundle

Mon, 01 Jan 0001 00:00:00 +0000

Broker A takes ownership of a namespace bundle, opens managed ledgers, and begins serving clients. The load balancer sees A as overloaded and unloads the bundle. Broker B picks it up, fences the ledgers, creates new ones, and starts serving. The balancer now sees B as overloaded and moves it back. The cycle repeats indefinitely.

Each iteration forces ledger fencing, new ledger creation, ZK metadata writes, and client disconnection and reconnection storms. Producers see intermittent publish latency spikes. New clients fail lookups during the brief ownership gap between fencing and re-acquisition. The cluster looks healthy in aggregate because no single broker is down, but the affected topics are in a constant state of transition.

Apache Pulsar unacked messages at the limit: the silent dispatch freeze

Mon, 01 Jan 0001 00:00:00 +0000

Consumers are connected. Producers are publishing. Backlog looks flat or zero. But no messages are being processed.

Pulsar brokers enforce a per-subscription limit on unacknowledged messages (maxUnackedMessagesPerSubscription, default 200,000) and a per-consumer limit (maxUnackedMessagesPerConsumer, default 50,000). When unacked messages hit either ceiling, the broker stops dispatching new messages to that subscription or consumer. No error is returned to the client. No exception is thrown. Dispatch pauses silently.

Most monitoring focuses on backlog and connection count. Both can look healthy during a freeze. Messages were already dispatched to consumers, so backlog may read zero. Consumers remain connected at the TCP level. The definitive indicators are the unacked message count itself and the blockedSubscriptionOnUnackedMsgs flag in topic stats.

Apache Pulsar under-replicated ledgers: data at risk after a bookie failure

Mon, 01 Jan 0001 00:00:00 +0000

You lost a bookie. The broker layer kept serving traffic because your write quorum absorbed the failure. But now auditor_NUM_UNDER_REPLICATED_LEDGERS is climbing, and it is not coming back down. Every minute it stays elevated is a minute your data exists on fewer copies than your replication factor demands. If another bookie fails before AutoRecovery finishes rereplicating, those ledgers are gone permanently.

This metric is exposed only on the elected auditor node, not on every bookie. If you are scraping bookie metrics and do not see it, you are either scraping the wrong host or AutoRecovery is not running. In steady state, the count is zero. After a bookie restart or failure, a temporary spike is expected and should trend back to zero within minutes to hours depending on data volume. A count that grows across audit intervals means recovery is failing, or bookies are dying faster than recovery can complete.

Apache Pulsar write stall: bookie journal fsync latency and the blocked write path

Mon, 01 Jan 0001 00:00:00 +0000

The write stall is the most common performance failure in Pulsar. It starts at the bookie journal disk and cascades upward through the write path until producers are blocked or timing out.

Every persistent message write in Pulsar must survive a journal fsync before the bookie acknowledges it. The broker waits for ack quorum (Qa) acknowledgments before acknowledging the producer. When the journal disk cannot sync fast enough, pending fsync operations accumulate, add-entry operations queue up, brokers hold connections open waiting for acks, client buffers fill, and throughput collapses.

Apache Pulsar ZooKeeper session cascade: reconnect storms and the thundering herd

Mon, 01 Jan 0001 00:00:00 +0000

Multiple brokers lose their ZooKeeper sessions within seconds of each other. Bundle ownership churns across the cluster as surviving brokers acquire orphaned bundles. Every connected producer and consumer simultaneously discovers its topic has moved or its connection has dropped, triggering a mass reconnection event. Each reconnection generates topic lookups, ephemeral node registrations, and watch re-establishments that hit the already overloaded ZK ensemble. Latency rises further, more sessions expire, and the cascade tightens.

Apache Pulsar ZooKeeper session expired: brokers fenced and topics reassigned

Mon, 01 Jan 0001 00:00:00 +0000

A broker’s ZooKeeper session expires when it fails to send heartbeats within the session timeout window. The expiry triggers immediate fencing of the broker, loss of all namespace bundle ownership, and reassignment of those bundles to surviving brokers. Clients connected to the fenced broker see errors like “Topic is temporarily unavailable” or “Attempting to add producer to a fenced topic” until the topics are picked up elsewhere.

The most common root cause is a long JVM garbage collection pause on the broker that prevents ZK heartbeats from being sent in time. The second most common is elevated ZK server latency, where heartbeats are sent but not processed before the timeout. Both paths converge on the same outcome: the broker’s ephemeral znodes are deleted, ownership transfers, and the cluster enters a state of high metadata churn.

Apache Pulsar ZooKeeper watch explosion: thousands of watches killing metadata latency

Mon, 01 Jan 0001 00:00:00 +0000

Topic lookups slow down. Bundle ownership transfers stall. Broker sessions flicker between connected and disconnected. ZooKeeper reports thousands, sometimes tens of thousands, of registered watches. This is a watch explosion, and the root cause is rarely in ZooKeeper itself. It is in client connection churn.

Watches accumulate when consumers disconnect and reconnect in waves. Each reconnection registers metadata watches for topic ownership, subscription state, and policy changes. During a mass reconnection event (broker restart, network blip, load balancer cycle), thousands of ephemeral znodes are created and deleted in rapid succession. Each creation and deletion triggers watch registration and notification. ZK processes requests sequentially, so as notification load grows, request latency climbs. Once latency exceeds session timeout, brokers lose sessions, triggering bundle unloads and another round of client reconnections. The feedback loop tightens until the cluster thrashes.

How Apache Pulsar actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

Most Pulsar incidents are misdiagnosed in the first thirty minutes because the operator is looking at the wrong layer. A publish latency spike gets chased on the broker when the real bottleneck is a bookie journal disk. A “consumer problem” turns out to be a cursor that stopped advancing. A broker that looks healthy on its HTTP endpoint is fenced off from ZooKeeper and losing topic ownership in a loop.