$ guides / apache-pulsar / apache-pulsar-journal-write-stall ▌

Operations Guides

Apache Pulsar write stall: bookie journal fsync latency and the blocked write path

The write stall is the most common performance failure in Pulsar. It starts at the bookie journal disk and cascades upward through the write path until producers are blocked or timing out.

Every persistent message write in Pulsar must survive a journal fsync before the bookie acknowledges it. The broker waits for ack quorum (Qa) acknowledgments before acknowledging the producer. When the journal disk cannot sync fast enough, pending fsync operations accumulate, add-entry operations queue up, brokers hold connections open waiting for acks, client buffers fill, and throughput collapses.

The key metric is bookie_journal_JOURNAL_SYNC P99 latency. It is the physical limit of Pulsar’s write throughput. When it spikes, the entire write path blocks.

What this means

The bookie journal is a write-ahead log on a dedicated disk. Each write batch is flushed to the OS page cache by the Journal Thread, then the ForceWrite Thread calls fsync to durably commit it. With journalSyncData=true (the default), the bookie sends acknowledgment only after fsync completes. The journal is synchronous on the critical path of every write.

When fsync latency rises, the cascade follows a predictable path:

flowchart TD
    A[Journal disk fsync stalls] --> B[ForceWrite queue grows]
    B --> C[Journal thread blocks]
    C --> D[Add-entry operations queue]
    D --> E[Broker waits for Qa acks]
    E --> F[Publish latency spikes]
    F --> G[Producer buffers fill]
    G --> H[Throughput drops / timeouts]

The cascade is directional and the signals arrive in order. The first to rise is bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE, followed by bookie_journal_JOURNAL_SYNC latency, then bookkeeper_server_ADD_ENTRY_IN_PROGRESS, then pulsar_broker_publish_latency, then a drop in pulsar_rate_in. If you are alerting only on broker-side publish latency, you are already deep into the cascade.

Device-dependent thresholds from the playbook:

SSD: P99 journal sync latency should be under 5ms
HDD: P99 should be under 20ms
Alert on sustained 2x degradation from established baseline

Absolute thresholds matter less than relative degradation because they depend on the storage hardware. A cluster that runs at 2ms P99 normally is already in trouble at 6ms, even though that sits under the 20ms HDD ceiling.

Common causes

Cause	What it looks like	First thing to check
Journal disk shared with other workloads	Intermittent latency spikes, no clear hardware fault	Whether any non-Pulsar process writes to the journal mount
Cloud storage latency variability (EBS stalls)	Periodic fsync spikes with no disk error	`iostat -x 1` w_await on the journal device
Hardware disk degradation	Gradually increasing latency on one bookie	SMART errors, `iostat` w_await trend
Journal file size misconfiguration	Latency spikes at regular intervals	Journal rotation frequency in bookie logs
Single slow bookie dragging quorum	Publish latency spikes on some topics, not all	Compare `bookie_journal_JOURNAL_SYNC` across all bookies
Traffic spike exceeding disk bandwidth	Cluster-wide latency increase	Whether publish rate increased above baseline

Quick checks

Safe, read-only commands for triage.

# Journal sync latency P99 on a single bookie
curl -s http://<bookie-host>:8000/metrics | grep bookie_journal_JOURNAL_SYNC

# Force write queue depth (earliest warning signal)
curl -s http://<bookie-host>:8000/metrics | grep JOURNAL_FORCE_WRITE_QUEUE_SIZE

# Add-entry in-progress count (write queue depth)
curl -s http://<bookie-host>:8000/metrics | grep ADD_ENTRY_IN_PROGRESS

# Bookie server status (1 = writable, 0 = read-only)
curl -s http://<bookie-host>:8000/metrics | grep bookie_SERVER_STATUS

# Journal disk I/O at the OS level
iostat -x 1

# Broker publish latency
curl -s http://<broker-host>:8080/metrics | grep pulsar_broker_publish_latency

# Publish rate (is throughput dropping?)
curl -s http://<broker-host>:8080/metrics | grep pulsar_rate_in

# Whether any non-Pulsar process is writing to the journal mount.
# WARNING: lsof +D traverses the filesystem and can be slow on a busy journal disk.
# Consider `fuser -m /dev/<journal-device>` as a faster alternative.
lsof +D /path/to/journal | grep -v bookkeeper

How to diagnose it

Confirm the cascade is journal-bound. Check bookie_journal_JOURNAL_SYNC P99 across all bookies. If one bookie is an outlier, the problem is likely hardware on that node. If all bookies show elevated latency, the problem is traffic or shared infrastructure.
Verify at the OS level. Run iostat -x 1 on the affected bookie and focus on w_await for the journal device. A healthy NVMe or SSD journal should show low single-digit millisecond write await. Elevated w_await confirms the disk is the bottleneck, not the bookie JVM.
Rule out the bookie JVM. If journal sync latency is spiking but iostat shows no disk issue, check for GC pauses. The bookie’s Prometheus endpoint exposes JVM GC metrics (jvm_gc_collection_seconds) alongside the BookKeeper metrics. Long GC pauses in the bookie JVM block the write thread and can produce symptoms that look like journal latency. If GC pauses correlate with sync latency spikes, the JVM is the cause.
Rule out network. If broker-to-bookie network latency is also high, the problem is network, not disk. The distinguishing pattern: disk-only issues show elevated bookie_journal_JOURNAL_SYNC with normal network latency. Network issues show elevated latency on both journal sync and broker-to-bookie round trips.
Check scope. A single slow bookie points to hardware. Cluster-wide slowdown points to a traffic spike exceeding aggregate disk bandwidth, or shared storage infrastructure (power, storage array, rack).
Check the journal mount. The journal disk must be dedicated. If any other process writes to this disk (ledger storage, logs, OS swap, another bookie), latency will spike unpredictably. This is the most common architecture mistake in Pulsar deployments.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`bookie_journal_JOURNAL_SYNC` (P99)	Physical limit of write throughput; every write waits for fsync	Sustained 2x baseline elevation
`bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE`	Earliest warning signal; rises before sync latency spikes	Sustained above 0 for more than 10 seconds
`bookkeeper_server_ADD_ENTRY_IN_PROGRESS`	Write queue depth; shows writes backing up in the bookie	Sustained growth, queue not draining within 30 seconds
`pulsar_broker_publish_latency` (P99)	End-to-end write latency as seen by the broker	Sustained 2x rolling 1-hour average
`pulsar_rate_in`	Publish throughput; drops when the write path stalls	Drop below 50% of baseline
`bookie_SERVER_STATUS`	Whether the bookie is writable	Transition to 0 (read-only), typically disk full
`w_await` on journal device (iostat)	OS-level confirmation of disk latency	Elevated above device baseline

The force write queue size is your leading indicator. It rises before journal sync latency spikes and before add-entry in-progress grows. If you can only alert on one signal from the write path, alert on this one.

Related journal metrics worth tracking alongside bookie_journal_JOURNAL_SYNC: bookie_journal_JOURNAL_ADD_ENTRY (time to record entry in the journal), bookie_journal_JOURNAL_QUEUE_LATENCY (time spent waiting in the journal queue before processing), and bookie_journal_JOURNAL_FLUSH_LATENCY (flush from memory to filesystem, distinct from fsync). The journalIndex label distinguishes multiple journal directories if configured.

Fixes

Journal disk shared with other workloads

This is the most common cause and the most common architecture mistake. The journal disk must be dedicated. No other process should write to it.

Immediate: Identify and stop any non-Pulsar I/O on the journal mount. Check for ledger storage, OS logs, swap, or other bookies sharing the device.

Permanent: Move the journal to a dedicated physical disk. Many operators put journals on NVMe and entry logs (ledger storage) on SSDs. This is the standard production layout.

Cloud storage latency variability

Cloud block storage can have highly variable fsync latency. Periodic stalls are a known issue with some volume types.

Immediate: If transient, traffic-shape the producers to reduce write pressure while the storage recovers.

Permanent: Use provisioned IOPS volume types where available. Test your volume type under sustained fsync load before production deployment. Consider whether the cluster’s write throughput target is sustainable given the volume’s fsync characteristics.

Hardware disk degradation

One bookie shows gradually increasing journal sync latency while others are normal. SMART errors or sector remapping may be present.

Immediate: Identify the slow bookie from journal latency metrics. If the bookie is clearly degraded, decommission it gracefully to force ledger recovery to other bookies:

# WARNING: Destructive. Forces ledger re-replication to other bookies,
# which increases load on the remaining cluster. Only use if the bookie
# is clearly degraded and you have capacity headroom.
bin/bookkeeper shell decommissionbookie -bookieid <bookie-address:port>

Permanent: Replace the degraded disk. Validate the replacement under fsync load before returning the bookie to production.

Traffic exceeding disk bandwidth

All bookies show elevated journal sync latency. Publish rate increased above the cluster’s sustainable write bandwidth.

Immediate: Reduce producer load (rate limiting, traffic shaping) to bring write throughput back under the disk bandwidth ceiling.

Permanent: Add bookies to increase aggregate write bandwidth. Recalculate the cluster’s write capacity based on the journal disk’s sustained fsync throughput, not its peak rating.

Journal file size misconfiguration

If journal files are too small, frequent rotation causes I/O spikes at regular intervals.

Permanent: Review journalMaxGroupWaitMSec (controls max wait before flushing a batch) and journalBufferedWritesThreshold (max batch size before triggering a flush). Group commit via journalAdaptiveGroupWrites amortizes sync cost across multiple writes but widens the data loss window if a crash occurs before fsync completes.

Prevention

Dedicate journal disks. The journal disk must not serve any other I/O. Put journals on NVMe and entry logs on SSD for the standard production layout.
Alert on the force write queue, not just latency. bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE is the earliest signal. By the time bookie_journal_JOURNAL_SYNC P99 spikes, the cascade is already underway.
Set baseline-relative alerts. Absolute thresholds (5ms SSD, 20ms HDD) are starting points. Your real alert should be sustained 2x degradation from the rolling baseline. This catches degradation on hardware that normally runs well under the absolute thresholds.
Use short scrape intervals for critical metrics. Journal queues fill and empty rapidly. Sample-based metrics at 15-second intervals can miss short-lived spikes entirely. Use higher resolution for journal sync latency, force write queue size, and publish latency.
Track per-bookie write distribution. One hot bookie can drag down the entire cluster’s write performance. The standard deviation of write throughput across bookies reveals imbalance before it cascades.
Test failover behavior before production. Know what your metrics look like during a controlled bookie failure. When a real failure happens, you need a baseline for normal degradation versus a cascade in progress.

Correlating the cascade with Netdata

Per-second metric resolution captures journal fsync latency spikes that 15-second scrapes miss entirely. The force write queue fills and empties in sub-second windows; per-second collection is the minimum viable resolution for this signal.
Cross-tier correlation shows the cascade in a single view: bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE rising, then bookie_journal_JOURNAL_SYNC P99 spiking, then bookkeeper_server_ADD_ENTRY_IN_PROGRESS growing, then pulsar_broker_publish_latency degrading. The order and timing of these signals reveals whether the stall originated at the disk or upstream.
Per-bookie breakdown makes it obvious when one bookie is an outlier. Instead of aggregate journal latency, you see each bookie individually and can immediately distinguish a hardware-bound node from a cluster-wide traffic issue.
Anomaly detection on journal sync latency catches slow degradation that static thresholds miss. A bookie whose P99 drifts from 1ms to 3ms over a week has not crossed any absolute threshold but is heading toward failure.
OS-level disk metrics alongside application metrics let you correlate disk await, utilization, and queue depth with bookie journal metrics in the same dashboard, confirming whether the stall is disk-bound or JVM-bound without switching tools.