Apache Pulsar bookie journal and ledger storage on one disk: the #1 architecture mistake

Every BookKeeper bookie has two storage responsibilities with fundamentally different I/O profiles. The journal is a write-ahead log: sequential writes, fsync’d per entry or batch, on the critical path of every producer acknowledgment. Entry logs and ledger indexes (managed by DbLedgerStorage, the default storage backend) store message payloads and serve random reads whenever consumers catch up on historical data.

These workloads are incompatible on a single physical disk. When they share a device, consumer catch-up reads and BookKeeper garbage collection interrupt the journal’s sequential write pattern. Fsync latency spikes. Every producer writing through that bookie stalls. Because Pulsar’s write quorum requires acknowledgments from multiple bookies, one saturated bookie can bottleneck the entire topic.

The bookkeeper.conf template and the operational playbook are explicit: journal disks MUST be dedicated. It remains the most common and most damaging architecture mistake in production Pulsar deployments.

How disk contention causes the Backlog Cascade

The journal needs low-latency sequential writes with minimal interruption. On SSD-backed bookies, a single fsync should complete in under 5 milliseconds at P99. Entry log reads are random access, bursty, and driven by consumer catch-up behavior. When both land on the same physical disk, random reads compete with the journal’s fsync path. The disk’s I/O queue fills with mixed workloads, and the journal’s sequential write advantage disappears.

On spinning disks, the mechanism is obvious: the read head physically moves away from the journal write position. On SSDs, it is subtler but equally damaging. The flash controller’s internal queue fills with mixed read and write commands, the write buffer drains less efficiently, and fsync latency becomes erratic. The P99/P50 ratio of journal sync latency degrades long before the average does.

Group commit (journalAdaptiveGroupWrites) and the journalSyncData setting affect how the journal batches writes before issuing fsync. These can amortize sync cost across multiple entries, but no tuning overcomes physical disk contention between two incompatible workloads sharing a device.

The write quorum amplifies the problem. When a broker writes to an ensemble of bookies, it waits for acknowledgments from Qa bookies (the ack quorum) before acknowledging to the producer. If one bookie in the ensemble has a stalled journal, the broker waits for it. Every producer writing through that topic experiences the latency of the slowest bookie in its ensemble. One saturated bookie is enough.

flowchart TD
    A[Consumer falls behind] -->|grows| B[Subscription backlog]
    B -->|cache misses| C[Broker reads from bookies]
    C -->|random I/O| D[Entry log reads on shared disk]
    D -->|contends with| E[Journal fsync latency spikes]
    E -->|queues build| F[Force write queue grows]
    F -->|backs up| G[Add entry in-progress grows]
    G -->|waits for acks| H[Broker publish latency rises]
    H -->|slows all producers| I[More consumers fall behind]
    I --> A

The cascade crosses topic boundaries. Normal consumer backlog affects only the subscription with the slow consumer. When the journal disk is shared:

Consumer catch-up reads on the entry log compete with journal fsyncs
Journal sync latency spikes on the affected bookie
Broker publish latency rises for all topics whose ledgers include that bookie
All producers using those topics slow down, not just those with slow consumers
More consumers fall behind because dispatching slows across the cluster
More reads hit bookies, compounding the I/O pressure

Where it shows up in production

The diagnostic test: if publish latency is rising alongside a consumer backlog, and journal sync latency is elevated on the bookie serving that consumer’s reads, you are in the cascade. If only dispatch is affected and publish latency is normal, it is a consumer problem, not the cascade.

Cloud environments make this harder to catch. Cloud persistent disks (AWS EBS gp3, GCP persistent disk, Azure managed disk) introduce their own latency variability. When journal and entry log share a cloud volume, periodic disk latency spikes hit both paths simultaneously, and the journal has no dedicated headroom to absorb them. NVMe instance storage, with its predictable low-latency profile, is the preferred choice for journal disks in cloud deployments.

Common misuses

Same physical disk, different mount points. Mounting the journal at /mnt/journal and the entry log at /mnt/ledgers on the same underlying disk provides no isolation. The OS block layer serves both from the same device queue. The allowMultipleDirsUnderSameDiskPartition setting (default false) catches directories on the same filesystem partition, but it cannot detect separate filesystem mounts backed by the same physical device.

Separate directories, same partition. Configuring journalDirectories (plural form available since BookKeeper 4.5.0) and ledgerDirectories to different paths on the same filesystem is functionally identical. The partition-level check in allowMultipleDirsUnderSameDiskPartition catches this case on startup.

Cloud shared storage. A single cloud persistent disk for both journal and entry log is the same mistake with added latency variability. The journal should be on the fastest available storage, dedicated exclusively to journal writes.

Disabling fsync as a workaround. Setting journalSyncData=false skips journal fsyncs and boosts throughput (no official multiplier is documented; community reports vary widely by hardware and workload). Some operators use this to work around shared-disk contention. This trades durability for throughput: on power failure or process crash, acknowledged messages that were not fsync’d to the journal can be lost. This is a data-loss risk disguised as a performance improvement.

Using HDDs for the journal. Spinning disks should not be used for journal storage in production. HDD P99 journal sync latency targets are under 20ms in ideal conditions, but random read contention from entry log access pushes it far beyond that with high variance.

Relying on the default flushInterval. The bookkeeper.conf default flushInterval of 60000ms controls how frequently entry log data is flushed to disk. When journal and ledger directories share a device, frequent flushing degrades performance. No official guidance promotes tuning flushInterval as a mitigation, so treat it as a workaround: increasing the interval may reduce I/O contention but extends the recovery window after a crash, since more unflushed data must be replayed from the journal on restart.

Signals to watch in production

Signal	Why it matters	Warning sign
`bookie_journal_JOURNAL_SYNC` P99 (with `journalIndex` label)	Journal fsync latency is the physical limit of write throughput. The `journalIndex` label isolates per-journal-directory health.	SSD P99 above 5ms, or sustained 2x degradation from baseline.
`bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE`	Depth of the pending fsync queue. The earliest warning signal: rises before sync latency spikes and before add-entry queues build.	Sustained depth above 0 for more than 10 seconds.
`bookkeeper_server_ADD_ENTRY_IN_PROGRESS`	Number of add-entry operations in progress on the bookie.	Queue not draining within 30 seconds after a traffic burst.
`pulsar_broker_publish_latency` P99	End-to-end write latency as seen by the broker. Rising alongside journal latency confirms the cascade is affecting producers.	Sustained 2x elevation over rolling baseline.
OS-level disk metrics (`iostat -x 1`)	Per-device utilization, await time, and queue depth. On a shared disk, mixed read and write contention appears on a single device.	`%util` above 70% sustained, `w_await` elevated on the journal device.
`bookie_read_cache_hits` / `bookie_read_cache_misses`	Bookie read cache efficiency. High miss rate during consumer catch-up means more random reads hitting entry log files.	Read cache miss rate above 50% sustained during backlog catch-up.
Subscription backlog (`pulsar_subscription_back_log`)	The root cause input. Growing backlog drives catch-up reads that trigger the cascade.	Continuous growth for more than 15 minutes with active consumer connections.

How Netdata helps

Netdata’s per-second collection is the main advantage for catching the Backlog Cascade early. The force write queue depth (bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE) is the first signal to rise when the journal disk cannot keep up. Per-second collection catches transient spikes that 15-second Prometheus scrape intervals miss entirely.

When bookie_journal_JOURNAL_SYNC P99 spikes, Netdata’s single-timeline view lets you correlate journal sync latency with per-device disk I/O on the same chart. This distinguishes “disk is failing” from “entry log reads are contending with journal writes” without switching between dashboards. For bookies with multiple journal directories, the journalIndex label is preserved, isolating which journal disk is degraded.

Cross-layer correlation completes the picture: subscription backlog on the broker, journal sync latency on the bookie, and broker publish latency appear in a single timeline. The cascade pattern is immediately visible rather than requiring a multi-dashboard investigation.