$ guides / nats / nats-consumer-redelivery-loop ▌

Operations Guides

NATS JetStream redelivery loop: num_redelivered climbing and messages reprocessed

A JetStream consumer’s num_redelivered counter is climbing and your application is processing the same messages over and over. Nothing is lost, but nothing completes either: deliveries happen, acks do not, and the server keeps trying again.

This is a redelivery loop. The server delivers a message, the consumer fails to acknowledge it within AckWait, the server redelivers it, and each pass increments num_redelivered. Every redelivery is wasted work: duplicate side effects if your handler is not idempotent, duplicate load on downstream systems, and a num_ack_pending count that never drains.

The loop has three common roots: the consumer is crash-looping between delivery and ack, one poison message kills the handler every time it is delivered, or processing simply takes longer than AckWait. A fourth possibility is a server-side bug in specific versions, covered below. The job is to figure out which one you have, then break the cycle without losing the stuck messages.

What this means

JetStream’s at-least-once guarantee works like this: when the server delivers a message to a consumer, it starts an AckWait timer (default 30 seconds). If no acknowledgement arrives before the timer expires, the server redelivers the message. An ack can be a positive ack, a +TERM (terminate, stop redelivering), a NAK (request redelivery, optionally with a delay), or a +WPI in-progress signal that extends the window by another AckWait.

A redelivery loop is what happens when the ack never lands, repeatedly:

flowchart LR
    D[Server delivers message] --> W{Ack within AckWait?}
    W -- ack received --> OK[Forward progress]
    W -- timeout or crash --> R[Redeliver, num_redelivered +1]
    R --> D
    W -- NAK with delay --> RD[Delayed redelivery] --> D
    W -- TERM --> T[Terminated, no more attempts]
    R -.MaxDeliver reached.-> MD[Advisory on MAX_DELIVERIES subject]

Two interactions make the loop worse:

MaxAckPending. Redelivered messages still count against the consumer’s outstanding-ack budget. If the loop pushes num_ack_pending to MaxAckPending, the server stops delivering entirely and the loop becomes a full stall. See NATS consumer stalled at MaxAckPending.
Unlimited MaxDeliver. If MaxDeliver was never set, the consumer gets unlimited redelivery (server default -1). Unless someone explicitly set a bound, a poison message loops forever.

Common causes

Cause	What it looks like	First thing to check
Consumer crash loop	Redeliveries cluster around process restarts; `num_ack_pending` resets when the consumer dies	Consumer application logs and restart count; correlate with redelivery timing
Poison message	Redeliveries concentrate on one or a few stream sequences; same message redelivered to MaxDeliver or forever	Which stream sequences are being redelivered; consumer error/panic logs at delivery time
Processing slower than AckWait	Messages eventually ack on redelivery, or acks arrive just after the timer; handler latency p99 near or above AckWait	Handler duration metrics vs configured `AckWait`
NAK without delay on a persistent error	Immediate, rapid redelivery; high redelivery rate in short bursts	Consumer code: is `Nak()` called without a delay on errors that cannot clear instantly?
Server-side bug (version-specific)	Loop persists after the consumer is verified healthy; ack floor frozen	Server version against known issues (see below)

Quick checks

All of these are read-only.

# Per-consumer redelivery, ack-pending, and pending counts across the server
curl -s 'http://localhost:8222/jsz?consumers=true' | \
  jq '.account_details[].streams[].consumer[] | {name, num_pending, num_ack_pending, num_redelivered}'

# Detail for the suspect consumer: ack floor, delivered cursor, redelivered count
nats consumer info STREAM CONSUMER --json | \
  jq '{num_ack_pending, num_redelivered, num_waiting, ack_floor, delivered}'

Things to look for in the output:

num_redelivered growing between two polls a minute apart. Confirms an active loop, not historical residue. The counter is cumulative; only the rate matters.
ack_floor.consumer_seq frozen. The ack floor not advancing means no message is being completed. That points at a crash loop or poison message, not slow processing (slow processing still acks eventually, so the floor creeps).
num_ack_pending at or near MaxAckPending. The loop has saturated the consumer’s budget and delivery is about to stop or already has.
num_pending growing while redeliveries climb. The consumer is reprocessing old messages AND falling behind on new ones. The backlog will still be there after you fix the loop.

Also check the consumer’s configured AckWait and MaxDeliver, and the handler’s observed processing time. If p99 handler latency is anywhere near AckWait, you have your answer.

# Confirm the redelivered count is actively increasing, not historical
nats consumer info STREAM CONSUMER --json | jq .num_redelivered
sleep 60
nats consumer info STREAM CONSUMER --json | jq .num_redelivered

How to diagnose it

Localise the loop to one consumer. Use the /jsz?consumers=true query above to find which consumer has the rising num_redelivered. In a multi-consumer stream, other consumers are usually fine, which already tells you the stream and server are healthy and the problem is consumer-specific.
Check whether the consumer process is restarting. If the consumer application crashes after receiving a message but before acking, every restart replays the unacked window. Container restart counts, supervisor logs, or deployment events that line up with redelivery bursts confirm this. Fix the crash; the loop resolves itself.
Check whether it is one message or all messages. If the ack floor never moves and redeliveries sit on a narrow range of stream sequences, you have a poison message: one payload that deterministically kills or hangs the handler. If redeliveries are spread across many sequences and the floor advances slowly, it is systemic: processing too slow, or NAKs on a downstream error.
Compare handler latency to AckWait. If p99 processing time approaches or exceeds AckWait, redelivery is guaranteed under any latency jitter. The redelivered copy then competes with the original work, making everything slower, which triggers more redeliveries. This is self-reinforcing.
Check the server version. Two known bugs produce redelivery-loop symptoms on healthy consumers:
- Interest Policy streams on v2.10.25: messages that reached MaxDeliver without an ack could be silently deleted by any ephemeral consumer that fetched them. Fixed in v2.11.0.
- LastPerSubject + Explicit Ack on v2.11.0 through v2.11.4: consumers could stall with acks not registering, messages stuck in redelivery, and the ack floor frozen. Fixed in v2.11.5. If you match either version window and the consumer application is verified healthy, upgrade before debugging the application further.
Rule out acks arriving out of window. An ack sent after AckWait expired can be interpreted as acking a different redelivery. If your handler does exactly-once bookkeeping with local deduplication, verify it is not acking stale work after long pauses (GC, blocking I/O).

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`num_redelivered` rate per consumer	The loop itself; cumulative counter, so only the rate is meaningful	Any sustained positive rate on a consumer that should ack cleanly
`ack_floor.consumer_seq` movement	Distinguishes “nothing completes” (crash loop, poison) from “slow but progressing”	Frozen while deliveries continue
`num_ack_pending` vs `MaxAckPending`	Approaching the cap means the loop is about to freeze delivery entirely	Ratio above 0.8; equality means stalled
`num_pending` growth	Backlog accumulating while the consumer burns capacity on duplicates	Sustained growth alongside rising redeliveries
Consumer process restarts	Crash loop is the top cause of redeliver-before-ack	Restarts correlated with redelivery bursts
Handler processing latency	Latency near `AckWait` guarantees timeout redelivery under jitter	p99 above roughly half of `AckWait`

The per-consumer fields require /jsz?consumers=true, which is costlier on servers with many consumers. Poll the specific consumer with nats consumer info during an incident rather than scraping all consumers aggressively.

Fixes

Consumer crash loop

Fix the crash. This is application-side: unhandled panic, OOM in the consumer, a bad deploy. Until the process survives long enough to ack, no JetStream tuning will help. The redeliveries stop as soon as the first delivery window is processed to completion.

Poison message

Bound the loop, then quarantine:

Set MaxDeliver to a finite value. If it is unset, the consumer gets unlimited redelivery (-1) and the poison message loops forever. Pick a value that reflects real transient-retry needs, typically single digits to low tens.
Watch the advisory subjects. When a message exhausts MaxDeliver, the server publishes an advisory on $JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.<STREAM>.<CONSUMER> containing the stream_seq. A +TERM ack publishes on $JS.EVENT.ADVISORY.CONSUMER.MSG_TERMINATED.<STREAM>.<CONSUMER>. Subscribe to these to catch poison messages as they are ejected.
Build a dead-letter path. JetStream has no built-in DLQ, by design. The standard pattern is: subscribe to the advisory subjects, extract the stream_seq, and copy the message into a dedicated DLQ stream for inspection and replay.
Use AckTerm (+TERM) for permanently unprocessable messages. TERM tells the server to stop redelivering without pretending the work succeeded. That is the correct response to a message that will never parse, as opposed to NAK, which asks for another attempt.
Manual cleanup. Messages that exhaust MaxDeliver remain in the stream (on WorkQueue and Limits retention) and must be deleted or acked explicitly via the JetStream API.

Processing slower than AckWait

You have three levers, in order of preference:

Make the handler faster. The redelivery is a symptom; the latency is the disease.
Send in-progress signals. AckProgress (+WPI) extends the window by another AckWait. Use it for legitimately long jobs instead of inflating the timeout.
Raise AckWait. Set it comfortably above p99 handler latency, not marginally above the mean. Too short creates loops; too long delays legitimate redelivery after real failures. BackOff (a sequence of durations) can replace the fixed AckWait with escalating intervals if you want fast first retries and patient later ones.

NAK storms on transient errors

NAK without a delay triggers immediate redelivery. If the downstream failure lasts seconds (database failover, dependency restart), an immediate NAK just spins the loop at full speed. Use Nak() with a delay (or NakWithDelay() where the client exposes it) so redelivery happens after the dependency has had time to recover.

One known wrinkle: the server does not immediately redeliver unacked messages when a client disconnects; it waits for the AckWait timer. Also, for pull consumers with no active fetch requests, AckWait expiry does not advance messages toward MaxDeliver; the advisory only fires once a client actually fetches and the message is redelivered. Do not expect a dead pull consumer to burn through MaxDeliver on its own.

Prevention

Always set a finite MaxDeliver on every consumer. Unlimited redelivery turns any poison message into a permanent loop.
Pair MaxDeliver with a DLQ path from day one: advisory subscription plus a DLQ stream. A bounded loop with nowhere to put the message is just silent stalling.
Make handlers idempotent. At-least-once means duplicates happen even outside incidents. Idempotency turns a redelivery loop from “corrupting downstream state” into “wasting compute”.
Size AckWait against measured p99 handler latency, with headroom for jitter, and use +WPI for long jobs.
NAK with delay, always. Immediate NAK is only appropriate for errors you know clear instantly.
Alert on the num_redelivered rate per critical consumer, not on lag alone. A redelivery loop can burn for hours while num_pending looks acceptable if the publish rate is low.
Pin and track server versions against the known redelivery bugs above, and upgrade out of the affected windows.

How Netdata helps

Netdata’s NATS collector polls the server’s HTTP monitoring endpoints, which gives you the server-level context around a redelivery loop:

JetStream state from /jsz: aggregate stream and consumer counts and API error rates, so you can confirm the server itself is healthy while one consumer loops.
API error rate (api.errors vs api.total): helps separate consumer-side loops from JetStream subsystem instability.
Storage and message accumulation: a looping consumer that also falls behind shows up as stream growth, which correlates with the redelivery signal.
Server health and uptime: rules out crash loops and restarts on the server side before you dig into the consumer application.

The per-consumer num_redelivered, num_ack_pending, and ack-floor fields come from /jsz?consumers=true or nats consumer info; during an incident, poll those directly alongside the Netdata dashboards to correlate the consumer loop with server-level signals.