Apache Pulsar journal force write queue growing: the earliest write-saturation signal

bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE measures the depth of pending fsync batches inside a BookKeeper bookie’s journal write path. In a healthy system this gauge sits at or near zero, draining between write groups. When it sustains a non-zero depth, the journal disk cannot commit writes durably fast enough to keep up with incoming traffic.

This signal rises before bookie_journal_JOURNAL_SYNC latency spikes and before bookkeeper_server_ADD_ENTRY_IN_PROGRESS grows. If you are not watching the force write queue, your first indication of write-path saturation will be broker publish latency degradation or producer timeouts, which means you are already two or three steps into the cascade.

Brief spikes are normal, especially with journalAdaptiveGroupWrites enabled, which dynamically adjusts group commit batching under varying load. Sustained depth lasting more than 10 seconds is actionable: the journal disk is the bottleneck.

What this means

Each bookie journal maintains a force write queue between two internal components. The Journal Thread batches incoming entries and issues write() syscalls. When a batch meets a flush trigger, the Journal Thread enqueues a force write request. The Force Write Thread dequeues these requests and calls fsync() to durably commit them to disk. bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE measures how many force write requests are pending in that queue.

When the queue grows, the Force Write Thread cannot call fsync() fast enough to drain the batches the Journal Thread is producing. Each pending entry represents writes that have been handed to the operating system but are not yet durably committed. The bookie cannot acknowledge these writes back to the broker until fsync completes. As the queue deepens, every upstream component waits: add-entry operations stall, brokers hold producer connections open waiting for acks, and client buffers fill.

The metric carries a journalIndex label to distinguish between multiple journal directories if configured. If you run multiple journal volumes per bookie, check whether the queue depth is concentrated on one volume or spread across all of them. A single slow journal volume will inflate the aggregate, but the problem is isolated to one device.

This signal is the leading indicator in the write-path saturation cascade:

flowchart TD
    A["Journal disk cannot
fsync fast enough"] --> B["JOURNAL_FORCE_WRITE_QUEUE_SIZE
begins growing"]
    B --> C["JOURNAL_SYNC latency
P99 spikes"]
    C --> D["ADD_ENTRY_IN_PROGRESS
queue grows"]
    D --> E["Broker publish latency
P99 increases"]
    E --> F["Throughput drops
rate_in decreases"]
    F --> G["Producers time out
write errors appear"]

If you alert only on journal sync latency or broker publish latency, you detect the problem two or three stages later.

Common causes

Quick checks

Run these read-only commands to confirm the problem and localize it.

# Check current force write queue depth
curl -s http://<bookie-host>:8000/metrics | grep JOURNAL_FORCE_WRITE_QUEUE_SIZE

# Check journal fsync latency percentiles
curl -s http://<bookie-host>:8000/metrics | grep bookie_journal_JOURNAL_SYNC

# Check write queue depth (add-entry in-progress)
curl -s http://<bookie-host>:8000/metrics | grep bookkeeper_server_ADD_ENTRY_IN_PROGRESS

# Check for blocked add requests (non-zero means writes are being refused)
curl -s http://<bookie-host>:8000/metrics | grep bookkeeper_server_ADD_ENTRY_BLOCKED

# Check bookie server status (1 = writable, 0 = read-only)
curl -s http://<bookie-host>:8000/metrics | grep bookie_SERVER_STATUS

# Check journal disk utilization and await time
iostat -x 1

# Check for non-Pulsar processes writing to the journal mount
lsof +D /path/to/journal 2>/dev/null | head -20

# Check if impact has propagated to producers
curl -s http://<broker-host>:8080/metrics | grep pulsar_broker_publish_latency

# Check whether throughput is degrading
curl -s http://<broker-host>:8080/metrics | grep pulsar_rate_in

The first command tells you whether the queue is elevated. The second and third tell you whether the problem has progressed downstream. The iostat and lsof commands tell you why. If ADD_ENTRY_BLOCKED is non-zero, the bookie is already refusing new writes and the situation is urgent.

How to diagnose it

1. Confirm the queue depth is sustained, not transient. Sample JOURNAL_FORCE_WRITE_QUEUE_SIZE at 1-second intervals for 30 seconds. If it drains to zero between bursts, the system is handling traffic spikes normally. Sustained non-zero depth for more than 10 seconds means the journal disk is the bottleneck.

2. Check OS-level disk stats on the journal device. Run iostat -x 1 and focus on the journal disk. Look at %util and w_await. On SSD-backed journals, w_await above 5ms indicates degradation; on HDD, above 20ms. If %util is pinned at 100% and w_await is climbing, the device is saturated. Note: on NVMe and other multi-queue devices, %util can report 100% before the device is truly saturated because it measures time with at least one I/O in flight, not total command capacity.

3. Identify competing I/O on the journal mount. The journal disk must be dedicated. Run lsof or fuser on the journal mount point. If any process other than the bookie is writing to this device, that is the root cause. The most common mistake is placing the journal and entry log on the same disk, or co-locating the journal with OS logs, monitoring agents, or other workloads.

4. Determine scope: single bookie or cluster-wide. Check the force write queue on all bookies. If only one bookie shows sustained depth, the problem is localized to that host (hardware, misconfiguration, or noisy neighbor). If multiple bookies are affected, the cluster is experiencing a traffic spike exceeding aggregate journal disk capacity, or multiple bookies share degraded infrastructure (same storage backend, same rack).

5. Correlate with journal sync latency. If JOURNAL_FORCE_WRITE_QUEUE_SIZE is elevated but bookie_journal_JOURNAL_SYNC P99 latency is still within baseline, you are catching the problem early. If sync latency has already spiked, the cascade is underway and producer impact is likely imminent.

6. Check for GC pauses on the bookie JVM. If the journal disk is healthy (low w_await, low %util) but the queue still grows, the Force Write Thread may be blocked by JVM GC pauses. Check bookie GC logs for full GC events or pause times exceeding 1 second.

Metrics and signals to monitor

Fixes

Free the journal disk

If lsof or iostat reveals competing I/O on the journal device, move the competing workload or relocate the journal to a dedicated disk. Journal and entry log must be on separate physical disks. This is the most common architecture mistake in Pulsar deployments and the most frequent root cause of force write queue growth.

If you cannot immediately relocate the journal, reducing the competing I/O (disabling log shipping, moving monitoring agents, stopping compaction on the shared device) provides temporary relief.

Replace degraded hardware

If smartctl reports errors, dmesg shows disk failures, or w_await is consistently elevated on a device that should be fast, the disk is degraded. Replace it. A degrading SSD or a RAID controller with a failed cache battery (forcing write-through mode) will cause progressively worsening fsync latency that no configuration change can fix.

Add bookies to distribute load

If the queue grows because aggregate write throughput exceeds what the existing bookie fleet can handle, add bookies. New ledgers will be placed on the new bookies, distributing the write load. Existing ledgers do not automatically rebalance, but new ledger creation will use the expanded ensemble.

Adding bookies does not provide immediate relief for topics whose current open ledgers are on saturated bookies. Those topics benefit only when their managed ledgers roll over to new ledgers.

Reduce producer write rate

If the journal disk is healthy but the workload genuinely exceeds its capacity, traffic-shape the producers. Reduce the publish rate, increase batching, or redistribute topics across namespaces with different bookie ensembles. This buys time while you provision additional storage capacity.

Tune journal group commit settings

journalAdaptiveGroupWrites dynamically adjusts batching behavior: more aggressive flushing under low load for lower latency, more aggressive batching under high load for higher throughput. When this is enabled, brief force write queue spikes during bursts are expected. The flush triggers that control batch formation include maximum wait time, maximum accumulated bytes, and maximum accumulated entries.

Larger batches amortize fsync cost but widen the potential data loss window on crash. If your workload is write-heavy with small messages, increasing the batch size thresholds can reduce fsync frequency and help the Force Write Thread keep up.

Fence or decommission a chronically slow bookie

If a single bookie consistently shows elevated force write queue depth and cannot be remediated, consider decommissioning it. This forces AutoRecovery to replicate its ledgers to other bookies. Use:

# DANGER: Disruptive. Triggers recovery I/O on surviving bookies and can
# temporarily elevate their force write queues. Only run if the slow bookie
# is clearly degraded and dragging down write quorum acknowledgment.
bin/bookkeeper shell decommissionbookie -bookieid <bookie-address:port>

Recovery I/O competes with foreground traffic on surviving bookies. Only do this if the slow bookie is clearly degraded and dragging down write quorum acknowledgment for topics whose ensembles include it.

Prevention

• Dedicate journal disks. The journal must be on its own physical device with no competing I/O. This is the highest-impact preventive measure.
• Alert on sustained force write queue depth. A threshold of sustained non-zero depth for more than 10 seconds catches the problem before sync latency spikes. See the Apache Pulsar monitoring checklist for the full signal set.
• Track journal disk bandwidth against rated capacity. Monitor sustained write throughput as a percentage of device bandwidth. Maintain headroom below 50% of rated bandwidth to handle bursts.
• Monitor per-bookie write distribution. One bookie receiving disproportionate writes indicates placement imbalance that will eventually saturate that bookie’s journal.
• Verify journal and entry log separation after every deployment change. Misconfiguration introduced during scaling events or host replacements is a common root cause.
• Run regular disk health checks. smartctl scans and dmesg monitoring catch hardware degradation before it manifests as queue growth.

How Netdata helps

• Per-second collection of bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE catches the rising queue before JOURNAL_SYNC latency reacts and before producers feel the impact.
• The journalIndex label is preserved, so you can identify which journal volume is the bottleneck when multiple journals are configured per bookie.
• Correlating force write queue depth with journal sync latency, JVM GC pauses, and OS-level disk metrics on the same timeline confirms whether the disk is the root cause or whether the Force Write Thread is blocked by something else (GC, CPU contention), without a separate SSH session.
• Cross-bookie comparison views make it immediately obvious whether the problem is a single degraded bookie or a cluster-wide capacity shortfall.

Related guides

• Apache Pulsar active connections climbing: connection leaks and file descriptor exhaustion
• Apache Pulsar broker down: telling a dead broker from a fenced one
• Apache Pulsar broker GC death spiral: heap pressure, stop-the-world pauses, and lost topic ownership
• Apache Pulsar broker lookup failures: new clients cannot find their topic
• How Apache Pulsar actually works in production: a mental model for operators
• Apache Pulsar monitoring checklist: the signals every production cluster needs
• Apache Pulsar monitoring maturity model: from survival to expert
• Apache Pulsar OutOfDirectMemoryError: the off-heap crash JVM heap dashboards never show
• Apache Pulsar throttled connections: the broker shedding load under pressure