Apache Pulsar ensemble, write quorum, and ack quorum: what E, Qw, and Qa actually guarantee

Every persistent message in Apache Pulsar passes through BookKeeper’s quorum system, governed by three parameters: ensemble size (E), write quorum (Qw), and ack quorum (Qa). Configured per namespace and inherited by topics, they determine how many bookies receive each entry, how many must confirm a durable write before the producer is acknowledged, and how the cluster behaves when bookies fail, slow down, or are taken offline.

The nuance is in what each parameter does not guarantee. Write quorum is not a durability floor for acknowledged entries. Ack quorum is not the number of copies that will exist after the write completes. Ensemble size constrains ledger creation in ways that surprise teams during maintenance windows: you can have plenty of bookies but still be unable to create new ledgers.

These three parameters sit behind most write-path incidents: publish latency spikes, NotEnoughBookiesException during rolling restarts, broker OOM under degraded bookie conditions, and gaps between expected and actual replication depth.

What it is and why it matters

BookKeeper persists each managed ledger’s data across a set of bookies called an ensemble. Three values define the persistence policy.

Ensemble size (E) is the total number of bookies assigned to a single ledger, fixed at ledger creation time. Every entry in the ledger is distributed across a subset of these E bookies.

Write quorum (Qw) is the number of bookies that receive each individual entry. This is the intended replication factor for each entry. When Qw is less than E, entries are striped across different subsets of the ensemble.

Ack quorum (Qa) is the number of bookies that must acknowledge a durable write (journal fsync complete) before the broker acknowledges the producer. This is the minimum replication factor guaranteed to the producer. The system tolerates up to Qa - 1 bookie failures without data loss for acknowledged entries.

The invariant is strict: E >= Qw >= Qa. Ledger creation fails if this is violated.

Set the policy per namespace:

# Set persistence policy for a namespace
pulsar-admin namespaces set-persistence <tenant/namespace> -e <E> -w <Qw> -a <Qa>

# Check the current policy
pulsar-admin namespaces get-persistence <tenant/namespace>

Changes apply to new ledgers only. Existing ledgers keep their current ensemble until they roll over.

When get-persistence returns all zeros, the namespace inherits broker-level defaults — managedLedgerDefaultEnsembleSize=2, managedLedgerDefaultWriteQuorum=2, managedLedgerDefaultAckQuorum=2 in broker.conf.

How it works

The striping algorithm

When Qw is less than E, entries are striped across the ensemble. For a given entry ID, the write quorum is the subsequence of the ensemble starting at bookie index (entry_id mod E), with length Qw. Wrapping is modular.

For example, with E=5 and Qw=3:

There are exactly E distinct write quorums in any ensemble. The pattern repeats every E entries. When Qw equals E, every entry goes to all bookies and no striping occurs.

The write path

When a producer sends a message, the broker appends it to the managed ledger, which writes the entry to Qw bookies in parallel. Each bookie writes the entry to its journal and fsyncs. Once Qa of those Qw bookies confirm the fsync, the broker acknowledges the producer.

flowchart TD
    P[Producer sends message] --> B[Broker: managed ledger append]
    B --> S[Select Qw bookies from ensemble of E]
    S --> W1[Bookie A: journal fsync]
    S --> W2[Bookie B: journal fsync]
    S --> W3[Bookie C: journal fsync]
    W1 -->|ack| GATE{Qa of Qw acks received?}
    W2 -->|ack| GATE
    W3 -->|ack| GATE
    GATE -->|Yes| ACK[Broker acks producer]

Publish latency is bounded by the Qa-th fastest fsync among the Qw bookies in the write set. One slow bookie that happens to be the Qa-th responder is enough to drag latency for every entry that includes it.

Where it shows up in production

NotEnoughBookiesException requires E bookies, not Qw

BookKeeper requires at least E available bookies to create a new ledger, because the ensemble is selected at ledger creation time, before any entries are written.

If you set E=5 and Qw=3, you might expect that losing 2 bookies is safe since you only need 3 for writes. But the broker cannot create new ledgers until at least 5 bookies are available. Existing ledgers continue to accept writes (their ensemble is already assigned), but ledger rollover fails. During rolling bookie restarts, this causes NotEnoughBookiesException in broker logs and publish errors for affected topics.

Setting E higher than Qw reduces write availability without improving durability for any individual entry. It spreads entries across more bookies (load distribution) at the cost of stricter availability requirements for ledger creation.

The slowest bookie in the write set drags publish latency

The broker waits for Qa acks out of Qw writes. Publish latency for each entry depends on which specific bookies are in its write set and how fast they fsync. If one bookie’s journal disk degrades, every entry that includes that bookie sees elevated latency.

The impact is partial: only topics whose ledgers include the degraded bookie are affected. Other topics continue normally. This partial impact makes the problem harder to detect in aggregate metrics. Per-bookie journal sync latency is the signal that isolates the culprit — the Prometheus metric is bookie_journal_JOURNAL_SYNC (there is no ..._LATENCY variant).

Entries can remain at Qa copies, never reaching Qw

When Qa is less than Qw, an entry is acknowledged after Qa bookies confirm. The remaining (Qw - Qa) bookies may still be processing the write. Under normal conditions, they complete and the entry reaches Qw copies. But several scenarios can leave entries permanently at Qa replication depth:

• Ensemble changes during ledger recovery replace bookies, and pending writes to the removed bookies are abandoned.
• Ledger closure races with in-flight writes.
• A bookie fails after Qa acks but before all Qw writes complete.

This is a legal state of the BookKeeper protocol. Entries that reach Qa but not Qw remain at Qa copies indefinitely. The “Guaranteed Write Quorum” protocol, which would enforce Qw copies for all acknowledged entries, has been formally verified in TLA+ (Jack Vanlightly’s BookKeeper TLA+ specifications) but has not been adopted by any released BookKeeper as of 2026.

Qw is an intended replication factor, not a guaranteed one. Qa is the only replication depth you can rely on for acknowledged entries.

Broker memory pressure when Qa < Qw and a bookie is slow

When Qa is less than Qw and a bookie responds slowly to add-entry requests, the broker holds pending operations in memory waiting for all Qw writes to complete or time out. Entries that have already reached Qa but have not received all Qw acks keep their pending add operations queued. Under sustained slow-bookie conditions, these pending operations accumulate in the broker’s direct memory, eventually causing OOM.

This is documented in Pulsar issue #14861 (“Broker OOM when the bookie is slowly to respond and AQ is smaller than WQ”, reproduced with E=2, W=2, A=1). The root cause is that the gap between Qa and Qw creates a window where operations are acknowledged to the producer but cannot be fully retired on the broker side. The wider the gap (Qw minus Qa), the more pending state can accumulate.

Rack-awareness constrains placement

When rack-aware placement policy is enabled, the BookKeeper client selects bookies from different failure domains. The relevant bookkeeper.conf settings are enforceMinNumRacksPerWriteQuorum (default false) and minNumRacksPerWriteQuorum (default 2). With E=3, Qw=3 and minNumRacksPerWriteQuorum=3, the ensemble must include bookies from at least 3 distinct racks. If the required number of racks is not available, ledger creation fails.

Before decommissioning a bookie, verify that the E >= Qw >= Qa invariant still holds with one fewer bookie in the cluster. If decommissioning drops available bookies below E, all new ledger creation for that namespace fails.

Tradeoffs and when to use it

Common configurations

Why E > Qw usually hurts more than it helps

Striping (E > Qw) distributes entries across more bookies, which can smooth write load. But it carries two costs that usually outweigh the benefit.

Read performance degrades. BookKeeper optimizes for sequential reads from a single bookie. When entries are striped across different bookie subsets, a consumer reading sequentially must hit multiple bookies instead of reading from one. This increases read fan-out and latency, particularly for catch-up reads that bypass the broker cache.

Availability decreases. Ledger creation requires E available bookies. Setting E higher than necessary means more bookies must be up to create new ledgers, which makes rolling restarts and maintenance windows riskier.

BookKeeper’s sticky read optimization (bookkeeperEnableStickyReads=true) only takes effect when E equals Qw; with striping enabled, sticky reads provide no benefit. The default is true in current Pulsar broker.conf, which states the caveat explicitly.

Multiple BookKeeper committers recommend setting E = Qw unless you have a specific, measured reason to stripe.

Why Qa = 1 is dangerous

Setting ack quorum to 1 means the producer is acknowledged after a single bookie confirms. If that bookie fails before the remaining Qw - 1 writes complete, the entry exists on only one node. Recovery cannot always determine whether that bookie holds the entry, which can block ledger recovery entirely.

Qa = 1 should be reserved for non-critical, ephemeral data where data loss is acceptable and recovery speed does not matter.

The metadata persistence parameters are dead code

The ManagedLedgerConfig fields metadataEnsembleSize, metadataWriteQuorumSize, and metadataAckQuorumSize have had zero effect since Pulsar 2.2 (2018): PR #2535 refactored ledger creation so cursor ledgers use the data-ledger settings, and the fields’ getters are never called (see Pulsar issue #26212). The dead fields were removed on the master/5.0 development line by PR #26293 (August 2026), but still exist, unused, in the 3.x and 4.x lines. If you are setting them expecting them to control cursor ledger replication, they do nothing.

Signals to watch in production

Metric names in this table were checked against BookKeeper’s Prometheus output and the Pulsar 3.0.x/4.0.x references (bookie_SERVER_STATUS, bookie_journal_JOURNAL_SYNC, bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE, bookkeeper_server_ADD_ENTRY_IN_PROGRESS, auditor_NUM_UNDER_REPLICATED_LEDGERS, pulsar_broker_publish_latency).

How Netdata helps

• Per-bookie journal sync latency at per-second resolution isolates which bookie in an ensemble is the Qa-th slow responder. Correlate journal sync spikes with broker publish latency to identify a degraded journal disk.
• Add-entry in-progress and force write queue depth surface write-path saturation before journal sync latency spikes. These are the earliest indicators that a bookie cannot keep up.
• Bookie server status changes fire immediately when a bookie goes read-only. Compare available bookie count against your configured E to assess ledger creation risk before producers see errors.
• Under-replicated ledger count tracked over time shows whether auto-recovery is keeping up after a bookie failure. A growing count means durability risk is accumulating.
• Anomaly detection on publish latency flags the partial, topic-specific latency spikes caused by a single degraded bookie, even when aggregate cluster metrics look normal.

Related guides

• How Apache Pulsar actually works in production: a mental model for operators
• Apache Pulsar journal force write queue growing: the earliest write-saturation signal
• Apache Pulsar bookie add-entry queue not draining: writes arriving faster than the disk can commit
• Apache Pulsar bookie read-only: disk full and bookie_SERVER_STATUS at zero
• Apache Pulsar bookie disk filling: runway to read-only and how to reclaim space
• Apache Pulsar bookie journal and ledger storage on one disk: the #1 architecture mistake
• Apache Pulsar broker down: telling a dead broker from a fenced one