$ guides / apache-pulsar / apache-pulsar-managed-ledger-cache-misses ▌

Operations Guides

Apache Pulsar managed ledger cache miss rate high: consumer reads falling through to bookies

The managed ledger cache is the broker’s off-heap buffer of recently written entries. Connected consumers read from it at sub-millisecond latency. On a cache miss, the broker issues a read to the BookKeeper bookie ensemble holding that ledger segment, which adds disk I/O, network bandwidth consumption between broker and bookie, and consumer read latency.

A sustained cache miss rate above 20% after warmup means the broker is serving a meaningful fraction of consumer reads from storage instead of memory. The extra bookie read traffic loads storage disks that also serve journal writes. If bookie read latency rises enough, publish latency follows. This is the leading edge of the Backlog Cascade: consumer reads and producer writes converging on the same physical disks.

The cache is per-broker and lives in direct (off-heap) memory. After a restart, it is cold. A high miss rate for the first 10 to 30 minutes while the working set warms up is expected. The signal only matters after the cache has had time to warm.

What this means

At 50% miss rate or higher, the cache is providing almost no benefit and the broker is effectively proxying reads to storage. The bookie read path was not designed for this volume in a tailing-consumer workload.

The diagnostic question is always: why is the cache not holding the entries consumers want? Three answers cover most cases: the cache is too small for the working set (thrashing), the access pattern changed (consumers reading old data the cache never held), or the configuration prevents effective caching (eviction too aggressive, catch-up reads excluded, or a known bug bypassing the cache for certain read patterns).

flowchart TD
    A[Cache miss rate > 20pct] --> B[Consumer reads fall through to bookies]
    B --> C[Bookie disk read I/O increases]
    C --> D[Reads compete with journal writes]
    D --> E[Publish latency rises]
    E --> F[Dispatch slows, consumers fall behind]
    F --> G[Backlog grows]
    G --> H[Cache evicts recent entries to hold older ones]
    H --> A

Common causes

Cause	What it looks like	First thing to check
Cache undersized for working set	High eviction rate concurrent with high miss rate; `pulsar_ml_cache_used_size` at or near configured limit	Compare cache size to topic count and throughput
Consumers reading historical data (catch-up)	Miss rate spikes when a slow consumer or new subscription reads from an old position; backlog grows on that subscription	Check `pulsar_subscription_back_log` for affected subscriptions
Cache cold after broker restart	Miss rate high for first 10-30 minutes, then declines; no eviction spike	Check broker uptime
Eviction configuration too aggressive	High evictions with moderate working set; CPU elevated on broker with many topics	Check eviction interval and threshold settings
Replay queue reads bypass cache (shared subscriptions)	Miss rate high specifically on shared subscriptions with batch messages and redelivery	Check redelivery rate and `preciseDispatcherFlowControl` setting

Quick checks

All commands are read-only. Run them on the affected broker and one of its bookies.

# Check cache hit/miss rates, evictions, and used size
curl -s http://<broker-host>:8080/metrics | grep pulsar_ml_cache

# Check broker uptime to rule out cold cache
curl -s http://<broker-host>:8080/metrics | grep process_start_time

# Check cache size and eviction configuration
grep -E "managedLedgerCacheSizeMB|managedLedgerCacheEviction" /path/to/broker.conf

# Check direct memory limit
grep -i MaxDirectMemorySize /path/to/pulsar_env.sh

# Check subscription backlogs for catch-up reads
curl -s http://<broker-host>:8080/metrics | grep pulsar_subscription_back_log

# Check redelivery rate (shared subscription replay bypass)
curl -s http://<broker-host>:8080/metrics | grep msg_rate_redeliver

# Check bookie read latency (are cache misses already impacting bookies?)
curl -s http://<bookie-host>:8000/metrics | grep -E "READ_ENTRY_REQUEST|BOOKIE_READ_ENTRY"

# Check bookie-level read cache hit rates
curl -s http://<bookie-host>:8000/metrics | grep -E "bookie_read_cache"

# Check broker direct memory via JMX
jcmd <broker-pid> VM.native_memory summary

How to diagnose it

1. Rule out cold cache. Check broker uptime. If the broker restarted within the last 30 minutes, the cache is cold and high miss rate is expected. Wait for warmup before diagnosing further. This is the single most common false positive.

2. Check whether the cache is full. Compare pulsar_ml_cache_used_size to the configured managedLedgerCacheSizeMB. If the cache is at capacity, the working set exceeds the cache. Evictions will be high because entries are removed to make room for new ones before consumers read them.

3. Check the eviction rate. Pull pulsar_ml_cache_evictions. High evictions concurrent with high miss rate is the signature of cache thrashing. Entries are evicted before consumers read them, then fetched again from bookies on the next request. This is a sizing problem, not a configuration problem.

4. Identify the access pattern. Determine whether consumers are tailing (reading recent entries) or catching up (reading old entries). Check pulsar_subscription_back_log for each subscription on the affected topics. Growing backlog on any subscription means that consumer is reading behind the write head, requesting entries the cache has already evicted or never held.

5. Check for catch-up read caching exclusion. Pulsar has configuration parameters that control whether entries are cached for backlogged subscriptions. If backlogged consumers are present and their reads are not being cached, every read goes to bookies regardless of cache size.

6. Check for replay queue bypass in shared subscriptions. On shared subscriptions with batch messages, redelivered messages may be read from storage even when they exist in cache. Check whether high miss rate correlates with high redelivery rate (pulsar_subscription_msg_rate_redeliver).

7. Check bookie impact. Look at bookkeeper_server_READ_ENTRY_REQUEST latency on bookies serving the affected topics. If bookie read latency is elevated alongside cache misses, the problem has progressed to the storage layer. Check journal sync latency on the same bookies to determine whether reads are already competing with writes.

8. Check direct memory headroom. The cache allocates from direct memory (MaxDirectMemorySize, default 4 GB on brokers ). If direct memory is near the limit, you cannot increase the cache size without also raising the direct memory ceiling or reducing other consumers of direct memory (Netty I/O buffers, BookKeeper client buffers). Check via JMX at java.nio:type=BufferPool,name=direct. Pulsar does not expose direct memory as a Prometheus metric.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`pulsar_ml_cache_misses_rate`	Direct measure of reads falling through to bookies	Greater than 20% of total cache requests sustained after warmup
`pulsar_ml_cache_hits_rate`	Counterpart to misses; hit ratio should exceed 80%	Declining trend over hours or days
`pulsar_ml_cache_evictions`	Indicates cache pressure and potential thrashing	Rising concurrently with miss rate
`pulsar_ml_cache_used_size`	Shows whether cache is at capacity	At or near configured `managedLedgerCacheSizeMB`
`pulsar_subscription_back_log`	Growing backlog drives catch-up reads that miss cache	Sustained growth on any subscription
`bookkeeper_server_READ_ENTRY_REQUEST`	Bookie read latency; cache misses add load here	P99 greater than 2x baseline
Broker direct memory (JMX)	Cache shares direct memory with Netty buffers	Above 75% of `MaxDirectMemorySize`
`pulsar_subscription_msg_rate_redeliver`	Redelivery in shared subscriptions may bypass cache	High rate correlated with miss rate

Fixes

Cache undersized for the working set

If pulsar_ml_cache_used_size is at capacity and evictions are high, the cache cannot hold the working set. Increase managedLedgerCacheSizeMB in broker.conf.

This is a direct-memory tradeoff. The cache allocates from MaxDirectMemorySize, which defaults to 4 GB on brokers . Netty I/O buffers and BookKeeper client buffers share this space. Raising the cache size without checking direct memory headroom risks OutOfDirectMemoryError.

Steps:

Check current direct memory usage via JMX (java.nio:type=BufferPool,name=direct).
If headroom exists, increase managedLedgerCacheSizeMB.
If direct memory is already constrained, raise -XX:MaxDirectMemorySize proportionally. Verify total process RSS fits within the host’s physical memory and your container memory limit if applicable.
Restart the broker for the change to take effect. Plan for 10 to 30 minutes of cold cache after restart.

Consumers reading historical data (catch-up reads)

If miss rate spikes coincide with a subscription backlog growing, consumers are reading entries the cache never held or has already evicted. The fix is on the consumer side, not the cache:

Identify which subscription has growing backlog.
Determine why the consumer fell behind. Check consumer application health, processing throughput, and downstream dependency latency.
Add consumer capacity or fix the consumer application.
If the backlog is unrecoverable and the consumer is dead, consider seeking the subscription cursor forward to stop the read storm. This permanently skips unprocessed messages for that subscription. Coordinate with the application team before doing this.

For topics where catch-up reads are expected as part of the workload (batch processing, replay, new subscriptions reading from earliest), consider tuning backlog caching parameters so that entries for backlogged consumers are retained in cache.

Eviction configuration too aggressive

The default eviction settings are known to be suboptimal for production brokers with many topics. The default eviction interval iterates over all managed ledgers frequently, and the default eviction time threshold may remove entries before consumers have a chance to read them.

Adjust these in broker.conf and restart. The tradeoff: a longer eviction interval reduces CPU overhead but means entries stay in cache longer before being considered for eviction. A longer threshold means entries survive longer, which helps tailing consumers but increases memory pressure if the working set is large.

Replay queue reads bypassing cache (shared subscriptions)

In shared subscriptions with batch messages, redelivered messages may be read from bookie storage even when they exist in the broker cache. This is a known issue where the replay path triggers storage reads instead of cache lookups.

Workaround: set preciseDispatcherFlowControl=true in broker.conf.

Longer term, PIP-430 introduces a new cache eviction strategy based on expected read counts that addresses this and other cache efficiency problems.

Prevention

Monitor cache hit ratio as a trend, not a threshold. A slowly declining hit ratio over hours or days is the earliest signal that the working set is growing or consumers are drifting behind. By the time miss rate crosses 20%, the cache has been degraded for a while.
Size the cache for peak working set, not average. Topic count, message rate, and consumer fan-out all affect the working set. Re-evaluate cache size when topic counts grow significantly or when new high-throughput topics are added.
Track direct memory as a first-class resource. The cache shares direct memory with Netty buffers. Monitor direct buffer pool usage via JMX alongside cache metrics. Pulsar does not expose direct memory as a Prometheus metric, so this requires JMX or process-level monitoring.
Set backlog alerts on critical subscriptions. Growing backlog is the precursor to cache miss storms. Catch it before consumers start reading old entries.
Review eviction configuration on upgrade. PIP-430 and subsequent changes may alter default eviction behavior. Validate cache performance after any broker upgrade.
Plan for cold cache after restarts. During rolling upgrades, the newly started broker will have high miss rates for 10 to 30 minutes. Stagger restarts and ensure surviving brokers can absorb the extra bookie read traffic during the warmup window.

How Netdata helps

Netdata’s per-second metrics collection lets you correlate managed ledger cache behavior with bookie read latency and consumer backlog without switching tools:

Cache efficiency at per-second resolution. Track pulsar_ml_cache_hits_rate, pulsar_ml_cache_misses_rate, pulsar_ml_cache_evictions, and pulsar_ml_cache_used_size side by side to distinguish thrashing from cold cache from access pattern change. Per-second granularity catches the moment miss rate starts climbing, not minutes later.
Cross-layer correlation. When cache miss rate spikes, immediately see whether bookie read latency (bookkeeper_server_READ_ENTRY_REQUEST) and journal sync latency are also affected. This tells you whether misses have progressed to the Backlog Cascade or are still contained at the cache layer.
Backlog tracking. Per-subscription backlog metrics show exactly which consumers are reading behind the write head and driving cache misses, narrowing the investigation from cluster-level to subscription-level in seconds.
Direct memory visibility. Monitor broker process RSS and JVM direct buffer pools to understand the tradeoff space when sizing the cache. This is the dimension Pulsar’s Prometheus endpoint does not expose.
Anomaly detection on hit ratio. ML anomaly flags on cache hit rate catch slow declines that static threshold alerts miss, giving you lead time before miss rate crosses the 20% line.