$ guides / apache-pulsar / apache-pulsar-bookie-read-latency-high ▌

Operations Guides

Apache Pulsar bookie read latency high: catch-up reads competing with the write path

When bookkeeper_server_READ_ENTRY_REQUEST or bookie_BOOKIE_READ_ENTRY P99 latency rises well above baseline, the cause is usually not an isolated disk problem. It is consumers draining backlog (catch-up reads) generating a high volume of cold reads that bypass both the broker’s managed ledger cache and the bookie’s read cache, hitting ledger storage disks directly. If journal and ledger directories share a physical device, those read I/O operations compete with journal fsyncs and the write path degrades too.

On properly separated disks, you can saturate the read path without affecting write throughput or latency. Two conditions turn high read latency into a broader incident: a shared disk configuration that destroys I/O isolation, and a single bookie running slower than its peers due to a degraded disk or hot-spot. The ensemble still meets quorum and the cluster looks healthy in aggregate, but that bookie is a latent failure waiting to cascade.

What this means

BookKeeper exposes two layers of read latency metrics:

bookkeeper_server_READ_ENTRY_REQUEST (Summary): end-to-end read request latency, from request arrival to entry return. Includes queueing, thread scheduling, and the storage operation.
bookie_BOOKIE_READ_ENTRY (Summary): bookie-level read latency, measuring only the disk or cache operation to retrieve the entry.

On SSD-backed bookies, P99 should stay below 10ms. On HDD-backed bookies, P99 should stay below 50ms. Alert on sustained P99 above 2x the established baseline for the storage type.

DbLedgerStorage, the default storage backend, maintains a read cache (bookie_read_cache_hits / bookie_read_cache_misses) and a write cache (bookie_write_cache_hits / bookie_write_cache_misses). When consumers are tailing, entries are typically served from the broker’s managed ledger cache or the bookie’s write cache without touching disk. When consumers fall behind and read historical data, those reads miss both caches and hit ledger storage disks as random reads.

The failure cascade depends on disk topology:

flowchart TD
    A[Consumer falls behind] --> B[Backlog grows]
    B --> C[Broker ml cache miss]
    C --> D[Cold read hits bookie disk]
    D --> E{Journal and ledger
on same device?}
    E -->|Yes| F[Read I/O competes
with journal fsync]
    F --> G[Journal sync latency rises]
    G --> H[Publish latency rises
for all topics on bookie]
    E -->|No, separate disks| I[Read latency stays high
but writes unaffected]
    I --> J[Consumer catch-up slows
backlog drain takes longer]

With properly separated disks, high read latency degrades consumer catch-up but does not affect producers. When journal and ledger share a device, the read storm steals I/O from journal fsyncs and the write path stalls.

Common causes

Cause	What it looks like	First thing to check
Catch-up consumers draining backlog	Read latency rises on specific bookies; `pulsar_ml_cache_misses_rate` elevated; subscription backlog is non-zero and large	Check `pulsar_subscription_back_log` for subscriptions with large backlog
Shared journal and ledger disk	Read latency AND `bookie_journal_JOURNAL_SYNC` latency rise together; `iostat` shows one device serving both paths	Check `journalDirectories` and `ledgerDirectories` in `bookkeeper.conf`, then confirm the devices
Read cache thrashing	`bookie_read_cache_misses` elevated; multiple consumers reading at different positions in the same ledger	Compare read-ahead batch count to read entry count
BookKeeper compaction competing with reads	Periodic spikes correlated with GC activity; `bookie_gc_*` counters active	Check compaction schedule and GC metrics
Degraded disk on one bookie	One bookie’s read latency is 2x or more above peers; ensemble still meets quorum	Compare P99 read latency across all bookies in the ensemble

Quick checks

# Check bookie read latency metrics on the affected bookie
curl -s http://<bookie-host>:8000/metrics | grep -E "READ_ENTRY_REQUEST|BOOKIE_READ_ENTRY"

# Check read and write cache hit/miss rates
curl -s http://<bookie-host>:8000/metrics | grep -E "cache_hits|cache_misses"

# Check journal sync latency to see if reads are stealing I/O from writes
curl -s http://<bookie-host>:8000/metrics | grep bookie_journal_JOURNAL_SYNC

# Check broker managed ledger cache efficiency
curl -s http://<broker-host>:8080/metrics | grep pulsar_ml_cache

# Check subscription backlog for catch-up consumers
curl -s http://<broker-host>:8080/metrics | grep pulsar_subscription_back_log

# Check OS-level disk I/O on the bookie (identify journal vs ledger devices)
iostat -x 1

# Check the actual configured directories, then confirm whether they map to the same physical device
grep -E "journalDirectories|ledgerDirectories" <bookkeeper.conf path>
df -h <journal-directory> <ledger-directory>

# Compare read latency across all bookies in the ensemble
for host in bookie1 bookie2 bookie3; do
  echo -n "$host: "
  curl -s http://$host:8000/metrics | grep BOOKIE_READ_ENTRY | grep "quantile.*0.99"
done

How to diagnose it

Confirm the elevation is real and sustained. Pull bookkeeper_server_READ_ENTRY_REQUEST and bookie_BOOKIE_READ_ENTRY quantiles. Compare P99 against the device-type baseline (SSD < 10ms, HDD < 50ms). A single spike during ledger rollover or compaction is normal. Sustained elevation above 2x baseline is the problem.
Determine scope: single bookie or cluster-wide. If one bookie shows significantly higher read latency than its peers, it has a degraded disk, network issue, or hot-spot. The ensemble continues functioning because quorum is still met, but this bookie is a latent failure. Investigate proactively rather than waiting for it to cascade.
Check for catch-up consumer activity. Look at pulsar_subscription_back_log for subscriptions with large backlog. Check pulsar_ml_cache_misses_rate on the broker. If the broker cache miss rate is elevated and backlog is non-zero, consumers are reading historical data from bookies instead of from broker memory.
Check read cache effectiveness on the bookie. Compare bookie_read_cache_hits to bookie_read_cache_misses. A miss rate above 50% sustained indicates the bookie is serving cold reads from disk. For tailing workloads, hit rate should be above 90%. For catch-up workloads, lower hit rates are expected but still indicate disk pressure.
Check for read cache thrashing. When multiple catch-up consumers read at different positions in the same ledger, read-ahead entries can evict each other before being consumed. Each entry may then be read from disk multiple times. If bookie_readahead_batch_count significantly exceeds bookie_read_entry count, thrashing is likely.
Check for shared-disk I/O competition. This is the most consequential check. If journal and ledger directories share a physical device, catch-up reads directly compete with journal fsyncs. Check bookie_journal_JOURNAL_SYNC latency. If it rises in correlation with read latency, the write path is affected. Run iostat -x 1 and confirm whether journal and ledger I/O land on the same device by cross-referencing the paths configured in journalDirectories and ledgerDirectories.
Rule out BookKeeper compaction. Compaction (entry log rewriting for space reclamation) causes high disk read I/O that can be mistaken for a consumer read storm. Check bookie_gc_* counters. If read latency spikes correlate with compaction activity, the issue is GC competing with foreground reads, not consumer catch-up.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`bookkeeper_server_READ_ENTRY_REQUEST` P99	End-to-end read request latency. Directly affects consumer catch-up speed.	Sustained P99 > 2x baseline (SSD > 10ms, HDD > 50ms)
`bookie_BOOKIE_READ_ENTRY` P99	Bookie-level read latency. Isolates the storage operation from queueing delays.	Same threshold; divergence from `READ_ENTRY_REQUEST` indicates queueing
`bookie_read_cache_hits` / `bookie_read_cache_misses`	Read cache effectiveness. High miss rate means cold reads from disk.	Miss rate > 50% sustained
`bookie_journal_JOURNAL_SYNC` P99	Journal fsync latency. If this rises alongside read latency, shared disk is the problem.	P99 > 5ms (SSD) or > 20ms (HDD)
`pulsar_ml_cache_hits_rate` / `pulsar_ml_cache_misses_rate`	Broker managed ledger cache efficiency. High miss rate pushes reads to bookies.	Miss rate > 20% sustained
`pulsar_subscription_back_log`	Backlog size per subscription. Growing backlog drives catch-up reads.	Sustained growth or large stable backlog with active consumers
`bookie_readahead_batch_count`	Read-ahead batch volume. Shows how aggressively the bookie pre-fetches on cache miss.	Significantly higher than read entry count suggests thrashing

Fixes

Separate journal and ledger disks

If journal and ledger directories share a physical device, this is the root cause of write-path degradation during read storms. BookKeeper’s I/O isolation guarantee depends on separate physical devices for journal and ledger storage. Moving them apart eliminates the competition entirely.

This requires downtime for the affected bookie: stop the bookie, move ledger directories to a new device, update journalDirectories and ledgerDirectories in bookkeeper.conf, restart. Plan this as a rolling operation across the bookie fleet.

See Apache Pulsar bookie journal and ledger storage on one disk: the #1 architecture mistake for detailed guidance.

Throttle consumer catch-up

When the root cause is legitimate consumer catch-up (consumers draining a large backlog), the read storm is real traffic. The fix is to limit the rate at which consumers drain backlog, reducing disk read pressure.

Broker-side dispatch rate limiting controls how fast messages are dispatched to consumers. This trades slower catch-up for lower bookie I/O pressure, preventing the read storm from cascading into write-path degradation.

During an active incident where publish latency is affected, throttling catch-up consumers is the right call. Outside of incidents, evaluate whether the slower backlog drain is acceptable for the application.

Increase read cache size

DbLedgerStorage allocates the read cache and write cache from JVM direct memory, each defaulting to roughly 25% of available direct memory. If the working set of active reads exceeds the cache, increasing the read cache allocation can improve hit rates.

This is a tuning knob, not a fix for an architectural problem. If the read pattern is fundamentally random (many consumers reading widely separated positions), no cache size will help. Increasing cache is effective when the miss rate is high but the access pattern has some locality.

Tune read-ahead batch size

dbStorage_readAheadCacheBatchSize (default 1000 entries in Pulsar’s bookkeeper.conf) controls how many entries the bookie pre-fetches on a cache miss. A larger batch improves sequential read throughput but wastes I/O when consumers read sparsely. A smaller batch reduces wasted reads during thrashing but increases per-entry overhead.

If read cache thrashing is confirmed (read-ahead batch count significantly exceeds read entry count), reducing the batch size may help. Monitor bookie_readahead_batch_count before and after the change.

Schedule compaction outside peak windows

If BookKeeper compaction is the cause (read latency spikes correlate with GC activity), schedule compaction during low-traffic periods. Compaction thresholds (minorCompactionThreshold, majorCompactionThreshold) and intervals control when GC runs. Misconfigured thresholds can prevent compaction from triggering at all, or cause it to run too aggressively during peak traffic.

Investigate and address a single slow bookie

If one bookie shows P99 read latency 2x or more above its peers, it has a degraded disk, firmware issue, or hot-spot. The ensemble continues meeting quorum, so the cluster appears healthy, but this bookie is a latent failure. If another bookie fails, the degraded one may not survive the recovery I/O load.

Immediate actions: check SMART errors, check iostat for elevated await on the storage device, check for noisy neighbors on cloud storage (EBS throttling). If the disk is clearly degraded, decommission the bookie gracefully to force ledger recovery to healthy bookies. Note that decommissioning triggers ledger recovery I/O across the ensemble, so do it during a low-traffic window if possible.

Prevention

Separate journal and ledger disks on every bookie. This is the single most important architectural decision for I/O isolation. Without it, read storms during consumer catch-up will affect the write path.
Monitor read cache hit rates as a leading indicator. Declining hit rates signal that consumers are shifting from tailing reads to historical reads. Catch this before it becomes a full catch-up storm.
Track per-bookie read latency, not just cluster averages. A single slow bookie is invisible in aggregate metrics. Compare P99 read latency across all bookies in each ensemble.
Set dispatch rate limits proactively on high-fan-out topics. If certain topics generate catch-up storms (batch consumers, replay scenarios), configure rate limits before the backlog builds.
Monitor backlog growth rate, not just absolute size. A large but stable backlog is manageable. A growing backlog will eventually trigger catch-up reads when consumers restart or scale up.

How Netdata helps

Per-second read latency percentiles on bookkeeper_server_READ_ENTRY_REQUEST and bookie_BOOKIE_READ_ENTRY reveal P99 spikes that 15-second scrape intervals miss. Sub-second latency bursts from compaction or cache thrashing are visible without smoothing.
Cross-bookie comparison in a single dashboard view lets you overlay read latency for all bookies in an ensemble, making a slow outlier immediately obvious without manual curl commands against each host.
Correlation between read latency and journal sync latency on the same bookie reveals shared-disk I/O competition in seconds. If both rise together, the disk is shared. If only read latency rises, the write path is isolated.
Read cache hit/miss rates displayed alongside read latency show whether elevated latency comes from cache misses (catch-up reads) or disk degradation (hardware issue). This distinction determines whether the fix is consumer throttling or disk replacement.
ML-based anomaly detection flags read latency deviations from baseline without requiring static thresholds per device type. The baseline adapts to each bookie’s hardware and workload pattern.