$ guides / apache-pulsar / apache-pulsar-publish-latency-high ▌

Operations Guides

Apache Pulsar publish latency high: reading pulsar_broker_publish_latency P99

Your alert fired on sustained P99 elevation of pulsar_broker_publish_latency above 2x the rolling baseline. Producers are seeing slow acknowledgements.

The metric pulsar_broker_publish_latency is a Summary metric exposed on the broker Prometheus endpoint. It measures the time from when the broker receives a message from a producer through the BookKeeper write path (write quorum Qw, ack quorum Qa) and back to the client callback. It is broker-side only: it excludes client-to-broker network time and producer-side batching delay. The Summary type provides quantiles at 0.5, 0.95, 0.99, 0.999, 0.9999, and 1.0. Alert on P99. P50 can look healthy while P99 is spiking, and it is the tail that triggers producer timeouts.

Healthy clusters typically see P99 below 10ms on SSD-backed bookies. The absolute threshold is workload-dependent; relative degradation (2x rolling baseline) is the more reliable alerting signal.

What this means

When P99 publish latency elevates, the broker is taking too long to persist messages through BookKeeper and acknowledge the producer. The bottleneck is in the write path: broker processing, BookKeeper journal fsync, quorum ack wait, GC pauses, or network contention between broker and bookies.

Publish latency is a downstream symptom. The root cause is almost always journal disk saturation, GC pressure, or network degradation, and it has been building for seconds or minutes before publish latency spikes. One slow bookie in the write quorum is enough to inflate latency for every topic whose ensemble includes that bookie.

Common causes

Cause	What it looks like	First thing to check
Journal sync saturation	One or more bookies show elevated JOURNAL_SYNC latency; force write queue growing; ADD_ENTRY_IN_PROGRESS not draining	`bookie_journal_JOURNAL_SYNC` P99 on each bookie
Broker GC death spiral	Publish latency spikes correlate with GC pauses; ZK sessions dropping; bundle unloads increasing	Broker GC logs or `jstat -gc`
Batching distortion	P99 elevated but throughput (bytes/sec) stable; message rate low relative to entry rate	Correlate with `pulsar_rate_in` and producer batch settings
Slow bookie in ensemble	Only topics using specific bookies affected; other topics normal; per-bookie ADD_ENTRY latency shows outlier	Per-bookie `bookkeeper_server_ADD_ENTRY_REQUEST` latency
Broker-bookie network contention	Journal sync latency normal on bookies but publish latency still high; TCP retransmits elevated	Network interface utilization and latency between broker and bookie hosts

Quick checks

These are safe, read-only checks. Run them in order.

# Check publish latency P99 on the broker
curl -s http://<broker-host>:8080/metrics | grep pulsar_broker_publish_latency

# Check publish rate to rule out batching distortion
curl -s http://<broker-host>:8080/metrics | grep -E "pulsar_(rate|throughput)_in"

# Check journal sync latency on each bookie
# <!-- TODO: verify default bookie HTTP metrics port - 8080 is the BookKeeper default, not 8000 -->
curl -s http://<bookie-host>:8000/metrics | grep bookie_journal_JOURNAL_SYNC

# Check journal force write queue depth (earliest saturation signal)
curl -s http://<bookie-host>:8000/metrics | grep JOURNAL_FORCE_WRITE_QUEUE_SIZE

# Check bookie write queue pressure
curl -s http://<bookie-host>:8000/metrics | grep bookkeeper_server_ADD_ENTRY_IN_PROGRESS

# Check raw journal disk I/O stats
iostat -x 1

# Check broker GC behavior
jstat -gc <broker-pid> 1000

# Check broker heap state
jcmd <broker-pid> GC.heap_info

# Check active connections for leak or storm patterns
curl -s http://<broker-host>:8080/metrics | grep pulsar_active_connections

How to diagnose it

The diagnostic flow narrows from “publish latency is high” to a specific root cause by checking the write path in order of likelihood.

flowchart TD
    A["P99 > 2x baseline"] --> B{"rate_in stable?"}
    B -- "No" --> C["Check producer errors
or throttling"]
    B -- "Yes" --> D{"P50 also elevated?"}
    D -- "Yes, large batches" --> E["Correlate with
producer batch config"]
    D -- "No, tail only" --> F{"Journal sync
P99 elevated?"}
    F -- "Yes" --> G["Check force write queue
and disk iostat"]
    F -- "No" --> H{"Broker GC
pauses > 1s?"}
    H -- "Yes" --> I["Investigate heap or
direct memory"]
    H -- "No" --> J["Check broker-bookie
network latency"]

Confirm the degradation is real, not a batching artifact. Pull pulsar_rate_in alongside the latency spike. If message rate dropped but throughput in bytes per second stayed stable, producers may have increased batch size. Higher batch sizes inflate per-batch publish latency because the broker waits for the batch to fill. If both rate and throughput dropped, the spike is real.
Isolate broker-side from bookie-side. Check bookie_journal_JOURNAL_SYNC on each bookie. If journal sync P99 is elevated on one or more bookies, the problem is in the storage layer. If all bookie journal sync latencies are normal, the bottleneck is on the broker (GC, thread contention, direct memory) or the network between broker and bookies.
If bookie-side: trace the write queue. Check bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE first. This is the earliest saturation signal in the stack. It rises before JOURNAL_SYNC latency spikes and before ADD_ENTRY_IN_PROGRESS grows. If the queue is sustained above zero for more than a few seconds, the journal disk cannot drain fsync operations fast enough. Run iostat -x 1 on the journal disk to confirm high %util and elevated await.
If broker-side: check GC and memory. Use jstat -gc <broker-pid> 1000 to check GC frequency and duration. Any full GC pause above 1 second will stall the broker event loop, causing publish latency spikes and potentially ZK session timeouts. Check direct memory separately: Pulsar does not expose it as a Prometheus metric. Use JMX (java.nio:type=BufferPool,name=direct) or compare process RSS against heap usage. A large gap between RSS and heap indicates direct memory consumption from Netty buffers.
If neither: check the network. If journal sync is normal and GC is clean, investigate broker-to-bookie network latency. Check TCP retransmit rates and NIC utilization on both broker and bookie hosts. The broker writes to Qw bookies in parallel and waits for Qa acks. Network latency directly adds to the quorum ack wait.
Check for the slow-bookie-in-ensemble pattern. Publish latency is dominated by the slowest bookie in the write quorum. You do not need all bookies to be slow. One degraded disk or one bookie with elevated journal latency is enough. Compare bookkeeper_server_ADD_ENTRY_REQUEST latency across all bookies in the ensemble. A single outlier bookie with 5x the latency of others will bottleneck every topic whose ensemble includes it.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`pulsar_broker_publish_latency` (P99)	Primary producer-experience SLI	Sustained above 2x rolling 1-hour baseline
`pulsar_rate_in`	Distinguishes batching artifacts from real degradation	Rate drops while throughput stays stable
`bookie_journal_JOURNAL_SYNC` (P99)	Physical limit of write throughput	Above 2x baseline; SSD P99 above 5ms indicates disk stress
`bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE`	Earliest write-path saturation signal	Sustained above 0 for more than 10 seconds
`bookkeeper_server_ADD_ENTRY_IN_PROGRESS`	Bookie write queue pressure	Queue not draining within 30 seconds after traffic bursts
Broker GC pause times	GC stalls block the event loop and spike latency	Any full GC above 1 second
`pulsar_active_connections`	Connection storms or leaks stress direct memory	Unexplained growth over days or weeks

Fixes

Journal disk saturation

If iostat shows the journal disk approaching bandwidth limits and the force write queue is not draining:

Short term: Reduce publish rate to give the journal disk room to drain. If a single bookie is the bottleneck, consider decommissioning it to trigger ledger recovery to other bookies. This is disruptive and generates heavy replication I/O: run bin/bookkeeper shell decommissionbookie only if the bookie is clearly degraded.
Medium term: Add bookies to the cluster to distribute write load across more journal disks. New ledgers will be placed on the new bookies automatically.
Long term: Dedicate an NVMe device exclusively for the bookie journal. Journal and entry log storage on the same physical disk is the most common architecture mistake in Pulsar deployments. See Apache Pulsar bookie journal and ledger storage on one disk.

Broker GC death spiral

If GC pauses correlate with publish latency spikes:

Short term: Reduce managedLedgerCacheSizeMB to free heap. This may increase bookie read load as cache evictions push reads to BookKeeper, so monitor pulsar_ml_cache_evictions after the change.
Medium term: Increase JVM heap (-Xmx) if the working set has outgrown the current allocation. Check for memory leaks in custom Pulsar Functions or IO connectors.
Consider the GC algorithm: If still running G1GC, switching to ZGC dramatically reduces pause times. ZGC pauses are sub-millisecond versus G1GC’s potential multi-second stops under heap pressure. See Apache Pulsar broker GC death spiral.

Network contention

If journal sync is clean and GC is healthy but publish latency is still elevated:

Check NIC utilization on broker and bookie hosts. Sustained utilization above 70% leaves no headroom for bursts.
Check TCP retransmit rates. Elevated retransmits indicate packet loss or congestion.
Verify that broker-to-bookie traffic is not competing with other workloads on shared network infrastructure.

Batching distortion

If the latency spike is a measurement artifact from producer batching:

This is not a real degradation. The broker is processing batches efficiently.
If the elevated latency violates producer-side SLAs, reduce the batch size or batch delay in the producer configuration to reduce batch fill time.

Slow bookie in ensemble

If one bookie shows consistently higher ADD_ENTRY latency than others:

Investigate the bookie’s journal disk health. Run SMART checks on the disk. Check for noisy neighbors on shared cloud storage (EBS latency spikes are common).
If the bookie is permanently degraded, decommission it gracefully and let auto-recovery redistribute its data. Monitor auditor_NUM_UNDER_REPLICATED_LEDGERS during recovery to ensure it trends to zero.

Prevention

Alert on P99, never P50. P50 can look healthy while P99 is spiking. Set the alert threshold relative to a rolling baseline (2x the 1-hour rolling average), not an absolute number.
Monitor the force write queue as a leading indicator. It rises before journal sync latency spikes and before publish latency degrades. See Apache Pulsar journal force write queue growing.
Keep journal and entry log on separate physical disks. This is the single highest-impact architecture decision for write latency.
Use a 15-second or shorter scrape interval for latency metrics. Sub-minute spikes are invisible at longer intervals.
Monitor direct memory alongside heap. Direct memory exhaustion crashes the broker with no heap-level warning. See Apache Pulsar OutOfDirectMemoryError.
Track per-bookie write latency distribution. A single slow bookie in the ensemble is the most common partial-degradation pattern. Standard deviation of ADD_ENTRY latency across bookies is the signal.

How Netdata helps

Netdata collects pulsar_broker_publish_latency at per-second resolution, so sub-minute P99 spikes are visible rather than averaged away.
Publish latency correlated with bookie_journal_JOURNAL_SYNC in a single view makes it immediately clear whether the bottleneck is storage or broker-side.
The force write queue depth appears alongside publish latency, exposing the leading indicator before latency degrades.
JVM GC pause metrics next to publish latency make GC-induced spikes obvious within seconds.
pulsar_rate_in plotted against publish latency distinguishes batching artifacts from real degradation at a glance.