NATS JetStream AckWait tuning: matching the ack timeout to processing time

AckWait is the timer the JetStream server starts when it delivers a message to a consumer. If the client does not ack before the timer expires, the server redelivers the message. That is the entire mechanism, and most “JetStream consumer is stuck” incidents that are not crashes come down to this timer being mismatched to the actual processing time of the work behind it.

Too short, and messages redeliver while they are still being processed: num_redelivered climbs, num_ack_pending stays pinned near the ceiling, and the consumer makes net-zero progress. Too long, and a genuinely dead worker holds a slot in MaxAckPending for minutes while nothing flows. This article covers how to size AckWait against p99 processing time, how it interacts with MaxAckPending and MaxDeliver, and when to stop stretching the timeout and use in-progress acks instead.

What AckWait actually does

JetStream consumers using explicit acks track every delivered-but-unacked message. The server keeps a redelivery timer per outstanding message, sized by the consumer’s AckWait. The lifecycle is:

Server delivers the message to the consumer and starts the AckWait timer.
Client processes and acks. Timer cancelled, message removed from ack-pending state.
If the timer expires first, the server marks the message for redelivery and delivers it again, incrementing its delivery count.
Repeat until the message is acked, terminated (AckTerm), or the delivery count reaches MaxDeliver.

sequenceDiagram
    participant S as JetStream server
    participant C as Consumer
    S->>C: deliver message (start AckWait timer)
    alt ack before timeout
        C->>S: ack
        S->>S: cancel timer, clear ack-pending
    else timeout
        S->>C: redeliver (delivery count + 1)
        note over S: if count reaches MaxDeliver, stop
    else long job
        C->>S: ack in progress (+WPI)
        S->>S: reset AckWait timer
    end

Properties that matter for tuning:

AckWait is per message, not per consumer. Each outstanding message has its own timer. A burst of deliveries creates a burst of timers that all expire together if processing is uniformly slow.
AckWait is editable after consumer creation. You can tune it on a live durable without recreating the consumer.
BackOff overrides AckWait. If the consumer has a BackOff list set (available since NATS Server 2.7.1), the first BackOff value becomes the effective AckWait, and later values space out subsequent redeliveries. BackOff applies only to ack timeouts, not to explicit NAKs. A NAK triggers immediate redelivery unless you use NakWithDelay.
Redelivery is bounded by MaxDeliver, which defaults to -1 (unlimited). A poison message with an unlimited delivery count will cycle forever, re-consuming a slot on every AckWait expiry.

Two server-side behaviors are worth knowing before you build assumptions on top of this. First, late acks are accepted: if the client acks after the timer has expired and the message has been redelivered, the server still accepts the ack rather than rejecting it. This is documented as intended behavior, and it means the same logical work can execute twice unless your handler is idempotent. Second, for pull consumers the server does not actively push redeliveries on timeout; redelivery and the corresponding ack-pending accounting happen when the next pull request arrives. A stalled pull consumer that stops fetching will show elevated num_ack_pending with no visible redelivery traffic.

How AckWait interacts with MaxAckPending and MaxDeliver

These three knobs form one system. Tuning AckWait in isolation is the most common mistake.

MaxAckPending (default 1000) is the maximum number of delivered-but-unacked messages the server will tolerate before it stops delivering new ones. When num_ack_pending reaches it, delivery stalls completely. See the dedicated guide on consumers stalled at MaxAckPending for that failure mode in depth.

The interaction with AckWait: every message that expires without an ack holds its slot for the full AckWait duration before the server even attempts redelivery. So the effective capacity of the consumer pipeline is bounded by:

sustainable throughput >= MaxAckPending / AckWait

If your handler takes 45 seconds and AckWait is 30 seconds, then with MaxAckPending at the default 1000, up to 1000 messages are in flight, all expiring before completion, all redelivering and expiring again. The consumer is busy the whole time and acks almost nothing. This is the composite failure mode where MaxAckPending combined with a short AckWait leaves messages cycling between delivered and redelivered without ever being processed.

MaxDeliver decides what happens to messages that never get acked. With the default -1, a message that always fails (or always takes longer than AckWait) redelivers forever, consuming a slot each cycle. A finite MaxDeliver plus a dead-letter destination turns pathological messages into observable, bounded failures instead of silent pipeline drag.

Sizing AckWait against real processing time

The sizing rule from Synadia’s operational guidance: set AckWait to at least 2 to 3 times your p99 processing time, to absorb variance from GC pauses, downstream latency, and noisy neighbors. The reasoning:

If AckWait is below p99, then by definition at least 1 percent of messages redeliver while still being processed. On a busy consumer that 1 percent competes for slots and handler capacity, which raises processing latency, which pushes more messages past the timeout. The feedback loop is self-reinforcing.
The 2-3x multiplier covers the gap between your measured p99 in a quiet environment and the real p99 during the incident you are actually tuning for.

You need a real processing-time distribution, measured in the handler from message receipt to ack, not inferred. NATS exposes no built-in metric for message processing latency, so application-side timing is the only reliable source.

# Check current redelivery pressure per consumer
nats consumer info STREAM CONSUMER --json | jq '{ack_pending: .num_ack_pending, redelivered: .num_redelivered, delivered: .delivered.stream_seq}'

# Same data via the monitoring endpoint (costlier on servers with many consumers)
curl -s 'http://localhost:8222/jsz?consumers=true' | \
  jq '.account_details[].stream_detail[].consumer_detail[]? | {name, num_ack_pending, num_redelivered}'

Interpretation heuristics:

num_redelivered climbing steadily while the application logs no errors: AckWait is shorter than real processing time. Raise it or speed up the handler.
num_ack_pending pinned at MaxAckPending with num_redelivered climbing: the pipeline is saturated with expiring deliveries. Raising AckWait helps only if handlers actually finish; otherwise lower MaxAckPending so fewer messages are in flight at once.
num_redelivered climbing on a subset of messages only: a bimodal workload. The tail (large payloads, expensive records) exceeds AckWait. Split the stream, or switch to in-progress acks below.
Redeliveries with num_ack_pending low and handlers idle: likely a NAK loop or handler error, not an AckWait problem. Check application logs first.

A redelivery ratio above roughly 10 percent of deliveries is a widely used line for “systemic issue”; a healthy consumer sits near zero.

Long jobs: use in-progress acks instead of a huge AckWait

When processing time is legitimately minutes long (report generation, video transcode, large batch calls), stretching AckWait to cover it creates the opposite problem: a worker that dies mid-message holds that message and its slot for the entire inflated timeout before redelivery. Recovery time from a real failure becomes as bad as the timeout.

The correct mechanism is the in-progress ack. The client sends an ack-with-progress (+WPI, exposed as InProgress() or equivalent in client libraries) before AckWait expires, and the server resets the timer by another full AckWait period. The handler sends one periodically while working:

// Pattern: short AckWait, periodic progress signals for long jobs
ticker := time.NewTicker(ackWait / 2)
defer ticker.Stop()
for {
    select {
    case <-ticker.C:
        msg.InProgress() // resets the server-side AckWait timer
    case <-done:
        msg.Ack()
        return
    }
}

This gives you both properties at once: redelivery after a real crash happens quickly (one AckWait, kept short), and long jobs survive as long as the worker is alive to report progress. One caveat: in some client libraries (the Java client, for example) the progress ack is fire-and-forget with no confirmation, so a lost progress message is invisible. Sending progress at half the AckWait interval, as above, tolerates one lost update.

Signals to watch in production

Signal	Why it matters	Warning sign
`num_redelivered` rate per consumer	Direct measure of AckWait expiry volume	Any sustained positive rate on a consumer that should be keeping up
Redeliveries / deliveries ratio	Normalizes redelivery volume to load	Above ~10 percent indicates a systemic mismatch
`num_ack_pending` vs `MaxAckPending`	Shows pipeline saturation from unacked deliveries	Pinned at the ceiling, especially combined with rising redeliveries
`num_pending` growth	Consumer falling behind producers, the downstream effect of stalled progress	Sustained growth while handlers appear busy
Application-side handler p99	The input to AckWait sizing	p99 above roughly one third of AckWait

How Netdata helps

Netdata’s NATS collector polls the server’s HTTP monitoring endpoints, so JetStream aggregate state (streams, consumers, API totals) is captured alongside host-level CPU, memory, and disk latency on one timeline.
Correlating consumer-side symptoms with host metrics answers the most common tuning question: is processing slow because AckWait is short, or because the worker host is saturated? Handler latency rising in step with host CPU or iowait points at capacity, not configuration.
api.inflight and api.errors from /jsz distinguish consumer-side ack timeout storms from server-side JetStream distress (Raft or disk issues), which produce superficially similar stalls.
Per-consumer fields like num_ack_pending and num_redelivered are not yet collected per consumer, so pair Netdata’s server-level view with periodic nats consumer info snapshots or application-side handler timing metrics for the per-consumer picture.