$ guides / nats / nats-consumer-max-ack-pending ▌

Operations Guides

NATS consumer stalled at MaxAckPending: delivery stops until messages are acked

Your JetStream consumer was processing messages fine, and then deliveries just stopped. No error on the client. No error in the server log. The stream keeps growing, CPU and memory are normal, and the consumer application is still connected. Everything is green except the one thing that matters: no messages are moving.

This is the most common cause of “JetStream consumer stopped receiving messages,” and it is silent by design. When a consumer’s count of delivered-but-unacknowledged messages (num_ack_pending) reaches its configured MaxAckPending limit, the server stops delivering new messages to that consumer. No error is raised, no advisory is emitted. Delivery pauses until pending messages are acknowledged, negatively acknowledged, or expire past AckWait.

The default MaxAckPending is 1000 for consumers with an explicit ack policy, so this limit is in play even if you never set it. This guide covers how to confirm the stall, the root causes, and the alert that catches it before the stream backs up.

What this means

Every JetStream consumer with an ack policy maintains a pool of in-flight messages: messages the server has delivered but the client has not yet acknowledged. MaxAckPending is the cap on that pool. It bounds how much unacked state the server tracks and how much redelivery work can pile up behind a slow client.

When the pool is full, the server holds back further deliveries. Messages that would have been delivered stay in the stream and show up as num_pending on the consumer:

flowchart TD
  P[Producer publishes] --> S[Stream stores messages]
  S -->|deliver while num_ack_pending less than MaxAckPending| C[Consumer]
  C -->|ack, nak, or AckWait expiry| A[Pool slot freed]
  A --> S
  C -->|processing stalls| F[num_ack_pending equals MaxAckPending]
  F -->|server stops delivering, no error raised| X[Consumer idle, stream num_pending grows]

Two properties make this failure mode nasty:

It is silent. The server does not tell the consumer it has been paused. Client libraries see a healthy connection and simply receive nothing. There is no server log line that says “consumer X stalled.”
It is shared in queue groups. If the consumer has a deliver group (queue group), MaxAckPending is shared across all members. Adding more subscriber instances does not raise the ceiling; the group as a whole can hold at most MaxAckPending unacked messages.

For push consumers, MaxAckPending is effectively the only form of flow control. Pull consumers request messages explicitly, so they do not stall the same way, but they still track pending acks against the same configured limit. See the consumer concepts in the NATS documentation for the full field semantics.

Common causes

Cause	What it looks like	First thing to check
Consumer not acking (bug, crash mid-processing, hung handler)	`num_ack_pending` pinned at `MaxAckPending`, `num_redelivered` flat, application alive but silent	Is the application processing? Thread dump, handler logs, recent deploys
Processing time exceeds `AckWait`	`num_redelivered` climbing steadily, same messages processed more than once, duplicates downstream	Compare p99 handler latency to `AckWait`
`MaxAckPending` too low for real processing latency	Consumer makes progress but throughput is capped; `num_ack_pending` oscillates at the limit	Compute required in-flight depth: message rate x processing time
Queue group sharing the pool	Stall appeared after adding load, all group members idle together	Check `deliver_group` on the consumer; the limit is group-wide

The first two are application problems the limit is correctly exposing. The third is a sizing problem. The fourth is the one that surprises teams who scale out expecting more in-flight capacity.

Quick checks

All of these are read-only. Substitute your stream and consumer names.

# Check ack-pending state for the consumer (the direct confirmation)
nats consumer info STREAM CONSUMER --json | jq '{
  ack_pending: .num_ack_pending,
  pending: .num_pending,
  redelivered: .num_redelivered,
  max_ack_pending: .config.max_ack_pending,
  ack_wait: .config.ack_wait,
  deliver_group: .config.deliver_group
}'

What you are looking for: num_ack_pending exactly equal to max_ack_pending. That is the stalled state. If max_ack_pending is absent from .config in the output, the consumer is on the server default of 1000.

# Same data via the HTTP monitoring endpoint (no CLI needed)
curl -s 'http://localhost:8222/jsz?consumers=true' | \
  jq '.account_details[].streams[].consumer[] | {name, num_pending, num_ack_pending, num_redelivered}'

# Is the stream growing because of this consumer?
nats stream info STREAM --json | jq '{messages: .state.messages, bytes: .state.bytes, first_seq: .state.first_seq, last_seq: .state.last_seq}'

# Are messages timing out and being redelivered? Watch over 60 seconds.
nats consumer info STREAM CONSUMER --json | jq .num_redelivered
sleep 60
nats consumer info STREAM CONSUMER --json | jq .num_redelivered

# Is the consumer's client connection itself healthy?
curl -s 'http://localhost:8222/connz?sort=pending&limit=10' | \
  jq '.connections[] | {cid, name, ip, pending_bytes, subscriptions}'

# Aggregate JetStream pressure building behind the stall
curl -s http://localhost:8222/jsz | jq '{bytes, messages, storage, api_errors: .api.errors}'

/jsz?consumers=true returns substantially more data than the aggregate /jsz and is costlier on servers with many consumers. Use it for targeted diagnosis, not as a fast poll loop.

How to diagnose it

Confirm the stall. num_ack_pending == max_ack_pending on the consumer, with num_pending growing. Both signals are required: if num_ack_pending is at the limit but num_pending is zero, the stream has simply caught up and nothing is waiting.
Decide: ack timeout churn or a dead consumer. Watch num_redelivered over a minute. If it climbs, messages are being delivered, timing out past AckWait, and redelivered. The consumer is alive but too slow per message, or it acks after AckWait has already expired. If num_redelivered is flat while num_ack_pending sits at the limit, the client is holding messages it will never ack: a hung handler, a crash between delivery and ack where the connection stayed open, or a code path that swallows the message without acking.
Check the application, not the server. This is the step teams skip. slow_consumers on the server will be zero for this failure: the server is not struggling to write to the client, it has deliberately stopped. Look at the consumer process: is the handler blocked on a database, an HTTP call, a lock? Recent deploys? GC death spiral or a dead downstream?
Verify the queue group assumption. If the consumer has a deliver_group, all members share the MaxAckPending pool. Ten instances do not give you 10,000 in-flight slots; the group still has 1000 (or whatever is configured). If the stall started right after you scaled out, this is why.
Check for an unacked pull delivery before trusting the numbers. A delivered pull-consumer message stays in num_ack_pending until the client acks or terminates it, the message is removed by retention, or delivery exceeds MaxDeliver. A pull consumer that exits without acking and never issues another request therefore leaves the counter above zero; maintainers closed issue #6093 as intended behavior, not a counter bug. Separately, servers older than 2.9.22 had a bug where push consumers could exceed MaxAckPending through redelivery logic, causing a permanent stall (issue #4112). If you are on anything older, upgrade.
Check for caps above the consumer. If you raise MaxAckPending and the effective behavior does not change, look upward: account JetStream limits expose max_ack_pending, and stream consumer limits expose max_ack_pending; a value above either configured cap is rejected, and an unset value is reduced to the lower applicable cap.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`num_ack_pending / MaxAckPending` ratio	The direct stall predictor	Greater than 0.8 sustained; equal to 1.0 is stalled
`num_pending` growth rate	Backlog accumulating behind the stalled consumer	Positive growth sustained while the consumer should be draining
`num_redelivered` rate	Ack timeouts churning; separates slow processing from a dead consumer	Any sustained positive rate
Stream `messages` / `bytes` growth	Storage pressure building while retention cannot proceed	Steady climb with static `first_seq`
`/jsz` `storage` vs limits	A stalled consumer on an interest- or workqueue-retention stream blocks cleanup	Storage approaching 90% of reserved
`/jsz` `api.errors`	Secondary effect if storage fills and publishes get rejected	Sustained positive rate

The ratio alert is the one that matters: warn above 0.8, page at 1.0. A consumer that habitually rides at 90% or more of its limit is one slow processing cycle away from stalling, and that high-water pattern deserves a ticket even without a full stall.

Fixes

Consumer is hung or crashed without acking

Fix the application. The limit is doing its job; the consumer is the fault. Restart the stuck process if it is genuinely wedged, then find out why: blocked handler, deadlock, hung downstream. If the consumer code has a path where it returns early without acking or naking (an error branch, a filter that drops messages), that path leaks pool slots until the consumer stalls. Audit the handler so every received message ends in exactly one of ack, nak, or term.

If the consumer is genuinely dead and the stream is under storage pressure, deleting the consumer makes its pending messages eligible for retention cleanup on interest- or workqueue-retention streams. Understand the data-loss implication before doing that on a stream whose messages you still need.

Processing time exceeds AckWait

Either make processing faster or raise AckWait so it comfortably exceeds p99 handler latency, including GC pauses and downstream call timeouts. Redelivery is not free: each redelivered message consumes a pool slot again, so chronic ack-timeout churn keeps num_ack_pending pinned near the limit and effectively halves throughput. MaxAckPending and AckWait are editable on an existing consumer; you do not need to recreate it to tune these.

MaxAckPending too low for the workload

Size it from the throughput you need: required in-flight depth is roughly message rate multiplied by per-message processing time, per consumer (or per queue group, since the pool is shared). If you process 200 messages per second with 200 ms handlers, you need about 40 slots to keep the pipeline full; 1000 is generous. But at 5000 messages per second with 500 ms handlers, the default 1000 is your throughput ceiling and no application tuning gets you past it. Raise MaxAckPending to cover the depth with headroom. To disable the ceiling, set it to -1; 0 means “apply the applicable default or cap”, not unlimited.

There is no per-subject MaxAckPending; each consumer has one shared pool across all its filter subjects. If you need independent flow control per subject, that means separate consumers.

Queue group out of capacity

Because the pool is shared across the deliver group, scaling out members spreads work but does not raise in-flight depth. If the group collectively needs more in-flight messages, raise MaxAckPending on the consumer itself. If different instances need independent limits, they need independent consumers, which changes delivery semantics from queue-group load balancing to fan-out; make that choice deliberately.

Prevention

Alert on the ratio, not the absolute number. num_ack_pending / MaxAckPending > 0.8 warns while you still have runway; == MaxAckPending means you are already stalled. Absolute thresholds break the moment someone changes the configured limit.
Track num_redelivered as a rate. It distinguishes slow processing from dead consumers and catches AckWait misconfiguration before anyone notices the throughput ceiling.
Correlate consumer lag with stream growth. A consumer stalled at the limit on a limits-retention stream is a backlog problem; on an interest- or workqueue-retention stream it is also a storage-exhaustion fuse. Know which retention policy each stream uses before you decide how urgent a stall is.
Set AckWait from measured latency. Derive it from p99 handler latency with margin, and re-derive it when handler behavior changes. Do not leave it at a default you never measured against.
Test the failure mode. Kill a consumer mid-processing in staging, watch num_ack_pending climb to the limit, and confirm your alert fires before the stream backs up. Teams that only test the happy path discover this interaction during incidents.

How Netdata helps

Netdata’s NATS collector polls the server’s HTTP monitoring endpoints, so the aggregate effects of a stalled consumer show up directly: JetStream bytes, messages, and storage climbing while producers keep publishing, and api.errors if the backlog eventually fills storage and publishes start being rejected.
Correlating stream growth against server throughput (in_msgs vs out_msgs) distinguishes a consumer stall (publishes continue, deliveries drop) from a producer or routing failure (publishes drop too).
Server-level signals like slow_consumers and per-connection pending_bytes stay flat in a MaxAckPending stall, which is itself diagnostic: when the stream grows but the server shows no backpressure distress, the delivery pause is consumer ack state, not a slow connection.
Know the granularity limit: Netdata currently collects aggregate JetStream figures, not per-consumer num_ack_pending. The ratio alerting in this guide needs per-consumer polling (/jsz?consumers=true or the CLI) alongside Netdata’s server-level view; the two together cover both the fuse and the explosion.