Apache Pulsar ledger rollover latency spikes: the periodic blip that is usually normal

You see a periodic sub-second spike in publish latency on your Pulsar brokers. It happens at regular intervals, lasts a fraction of a second, and then latency returns to baseline. No errors, no bookie failures, no GC pauses. The spike is visible in P99 publish latency but barely touches P50. It correlates across topics on the same broker but not across the entire cluster.

This is almost certainly a ledger rollover. When a managed ledger fills its current BookKeeper ledger (by entry count, size, or time), the broker seals that ledger and creates a new one. The creation step involves metadata store writes (ZooKeeper in most Pulsar 3.x deployments, Oxia in newer ones), which add latency to the write path for the duration of the operation. The result is a brief publish latency blip, typically under 100ms, that repeats at intervals determined by your managed ledger configuration.

What this means

A managed ledger is Pulsar’s abstraction over BookKeeper ledgers. Each persistent topic has one managed ledger, which writes to a current open ledger. When that ledger hits a rollover threshold, the broker seals it (no more writes) and creates a new ledger to continue the append stream.

Three maximum thresholds can trigger rollover, but only after a minimum time gate has elapsed:

A ledger rolls over after the minimum time has passed AND any one of the three maximum thresholds is reached. A random jitter of up to 5% is added to the maximum rollover time (ManagedLedgerImpl.getMaximumRolloverTimeMs) to avoid multiple managed ledgers rolling over at exactly the same moment.

During rollover, the broker writes ledger metadata to the metadata store. This metadata operation adds latency to the write path for that topic. The spike is per-topic, not cluster-wide. If many topics share similar rollover timing (which the jitter mitigates but does not eliminate), the aggregate effect can produce a visible blip in broker-level publish latency.

After a broker restart or failover, all topics owned by that broker need new ledgers because the old ones were fenced during the ownership transfer. The first write to each topic opens a new ledger. With many topics, this creates a burst of simultaneous ledger creations against the metadata store.

Subscription cursors also have their own BookKeeper ledgers that roll over independently, controlled by managedLedgerCursorMaxEntriesPerLedger (default 50,000) and managedLedgerCursorRolloverTimeInSeconds (default 14,400, or 4 hours). Cursor ledger rollovers produce the same type of brief metadata-write latency blip, but on the ack/mark-delete path rather than the publish path.

flowchart TD
    A["Publish latency spike detected"] --> B{"Correlates with
LedgerSwitchLatency?"}
    B -->|Yes| C{"Periodic at
config intervals?"}
    B -->|No| D["Not rollover. Check
journal sync, GC, bookie health"]
    C -->|Yes| E["Normal periodic rollover
typically under 100ms"]
    C -->|No| F{"After broker restart
or failover?"}
    F -->|Yes| G["Post-failover batch
ledger creation. Self-resolving"]
    F -->|No| H{"Rollover rate
abnormally high?"}
    H -->|Yes| I["Check maxEntriesPerLedger
and maxSizePerLedgerMbytes"]
    H -->|No| D

Common causes

Quick checks

# Check ledger switch latency buckets (the definitive rollover signal)
curl -s http://<broker-host>:8080/metrics | grep pulsar_ml_LedgerSwitchLatencyBuckets

# Check publish latency quantiles
curl -s http://<broker-host>:8080/metrics | grep pulsar_broker_publish_latency

# Check total add-entry latency (includes ledger switch overhead)
curl -s http://<broker-host>:8080/metrics | grep pulsar_ml_AddEntryLatencyBuckets

# Check if journal sync latency is elevated (rules out disk issues)
curl -s http://<bookie-host>:8000/metrics | grep bookie_journal_JOURNAL_SYNC

# Check if add-entry queue is backing up (rules out write saturation)
curl -s http://<bookie-host>:8000/metrics | grep bookkeeper_server_ADD_ENTRY_IN_PROGRESS

# Check journal force write queue depth (earliest write-saturation signal)
curl -s http://<bookie-host>:8000/metrics | grep bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE

# Check metadata store health (requires 4lw whitelist in ZK 3.5+)
echo stat | nc <zk-host> 2181

How to diagnose it

1. Confirm the spike is ledger-switch-related. Check pulsar_ml_LedgerSwitchLatencyBuckets around the time of the spike. If you see entries in the histogram at the spike timestamp, the spike is rollover.

2. Check the publish latency distribution. Rollover spikes show up primarily in the P99 and higher quantiles, not in P50. If P50 is also elevated, the problem is not rollover alone.

3. Rule out disk issues. Compare the spike timing against bookie_journal_JOURNAL_SYNC P99 on the bookies in the write ensemble. If journal sync latency is flat but publish latency spiked, the spike is metadata-related (rollover), not disk-related.

4. Rule out write saturation. Check bookkeeper_server_ADD_ENTRY_IN_PROGRESS and bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE. Both should return to near-zero within seconds of the spike. If either stays elevated, the write path is saturated independently of rollover.

5. Determine the rollover frequency. Count ledger switch events per topic over a time window. If a single topic is rolling over more than once per minute, your managedLedgerMaxEntriesPerLedger or managedLedgerMaxSizePerLedgerMbytes may be set too low for your throughput.

6. Check for post-restart correlation. If the spike happened shortly after a broker restart or failover, it is the expected batch-ledger-creation behavior. The spike should resolve as each topic completes its first write. No action needed.

Metrics and signals to monitor

Fixes

If the spike is normal periodic rollover

Do nothing. Sub-100ms periodic spikes from ledger rollover are expected behavior. Alerting on them produces noise without value. Exclude rollover-correlated P99 spikes from your publish-latency alerting, or set the alert threshold with a sustained-duration condition so only persistent degradation fires.

If the spike is post-restart batch creation

Also do nothing. After a broker restart or failover, every topic must open a new ledger on its first write. The burst of metadata operations resolves within seconds to minutes depending on topic count. It is self-limiting because each topic creates its ledger exactly once.

If rollovers are abnormally frequent

Frequent rollovers stress the metadata store and generate metadata growth. Each closed ledger is a znode in ZooKeeper (or the equivalent in Oxia). Topics with long retention and frequent rollovers accumulate thousands of ledger segments, which slows metadata operations over time.

Check your configuration:

• If managedLedgerMaxEntriesPerLedger was set to a low value for testing and not reset, increase it back to the default of 50,000.
• If managedLedgerMaxSizePerLedgerMbytes is too small for your message throughput, increase it. The default of 2,048 MB is appropriate for most production workloads.
• If you are running standalone mode for any production-adjacent testing, note that standalone.conf ships the same managedLedgerMaxEntriesPerLedger=50,000 default as broker.conf, so extremely frequent rollovers in a cluster mean someone lowered the value for testing and did not reset it.

There is no built-in rate limiter for ledger rollover. Every rollover is an unthrottled burst of metadata writes. Keeping ledger sizes large enough that rollovers happen at most every few minutes (ideally every 10 minutes or longer, matching the minimum rollover time gate) is the right approach.

Direct-memory (ByteBuf) leaks during ledger rollover and under BookKeeper overload were real bugs fixed upstream in 2021 (apache/pulsar#10738; PRs #10755/#10773). Very low managedLedgerMaxEntriesPerLedger values (for example, 10) vastly increase rollover frequency; the default of 50,000 is safe. Do not lower it without a specific reason.

If the spike is not rollover

If pulsar_ml_LedgerSwitchLatencyBuckets does not correlate with your publish latency spike, the problem is elsewhere. Investigate these patterns:

• Bookie journal disk saturation: bookie_journal_JOURNAL_SYNC P99 is elevated, bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE is growing. The write path is backing up.
• Broker GC pauses: JVM GC pause times correlate with the spike. GC pauses can also cause ZooKeeper session expiration, which triggers bundle unloads and a cascade of reconnections.
• Metadata store latency storm: ZK latency elevated, bundle unloads spiking, client reconnections. Every metadata operation slows down, including rollover, but the root cause is the metadata store, not the rollover itself.
• Bookie failure cascade: Multiple bookies changing status, under-replicated ledger count spiking, recovery I/O competing with foreground writes.

Prevention

• Do not over-alert on publish latency P99 without duration. Rollover spikes will trigger any instantaneous P99 alert. Use a sustained-duration condition (P99 elevated for more than 5 minutes) or correlate with pulsar_ml_LedgerSwitchLatencyBuckets before alerting.
• Keep default ledger rollover settings unless you have a specific reason to change them. The defaults (50,000 entries, 2,048 MB, 4-hour max age, 10-minute minimum) are well-tuned for most workloads.
• Monitor metadata store latency. Rollover is only as expensive as the metadata store is slow. If ZK latency creeps above 10ms sustained, every rollover spike gets worse. Treat ZK degradation as a leading indicator for cluster-wide problems.
• Track ZK znode count over time. Each closed ledger creates a znode. If znode count grows monotonically, consider adjusting retention, enabling tiered storage offload, or increasing ledger rollover thresholds to reduce the rate of new ledger creation.
• Use a scrape interval that captures sub-second events. 15-second scrape intervals may miss sub-second latency spikes entirely. If you need to characterize rollover spikes precisely, use higher-resolution metrics collection for publish latency and ledger switch latency.

How Netdata helps

• Per-second metric collection captures rollover spikes that 15-second scrape intervals miss. You can see the exact shape and duration of each blip in pulsar_broker_publish_latency and correlate it against pulsar_ml_LedgerSwitchLatencyBuckets at the same resolution.
• ML-based anomaly detection baselines the normal periodic cadence of rollover spikes and flags deviations from the expected pattern, separating the normal blip from an emerging problem.
• Correlation across the stack lets you overlay publish latency, journal sync latency, add-entry queue depth, and metadata store latency in a single view. If the spike is rollover, journal sync stays flat while publish latency blips. If it is disk saturation, both move together.
• Configurable alerting with duration conditions reduces noise from periodic rollover spikes while still catching sustained degradation.

Related guides

• How Apache Pulsar actually works in production: a mental model for operators
• Apache Pulsar broker GC death spiral: heap pressure, stop-the-world pauses, and lost topic ownership
• Apache Pulsar journal force write queue growing: the earliest write-saturation signal
• Apache Pulsar bookie add-entry queue not draining: writes arriving faster than the disk can commit
• Apache Pulsar broker down: telling a dead broker from a fenced one
• Apache Pulsar bookie failure cascade: recovery I/O that topples surviving bookies
• Apache Pulsar entry log GC falling behind: reclaimed space that never comes back
• Apache Pulsar bookie disk filling: runway to read-only and how to reclaim space