RabbitMQ connection in flow state: credit-based backpressure explained

You opened the management UI or ran rabbitmqctl list_connections state and saw one or more connections sitting in state: flow. No alarm fired. Nothing is blocked cluster-wide. Is it a problem? Usually no, and treating it like one is a common source of alert noise.

Connection state flow is RabbitMQ’s internal, per-connection, credit-based backpressure doing what it was designed to do. A specific publishing connection is outrunning something downstream: a queue process, the message store, disk I/O, or an inter-node link. The broker throttles just that connection, transiently, toggling between blocked and unblocked many times per second. It is surgical. It is not a resource alarm.

This article explains the mechanism, how to tell healthy transient flow from a systemic write-path bottleneck, and which signals to correlate before you touch anything. For the broader picture of how the broker’s moving parts fit together, see How RabbitMQ actually works in production.

What the flow state actually is

RabbitMQ has two independent throttling systems, and operators routinely confuse them:

  1. Credit-based flow control (per-connection). Internal pipeline backpressure. A fast publisher is temporarily throttled when a downstream component cannot keep up. Only that connection is affected. The connection shows state: flow in the API and management UI. No alarm is involved.
  2. Resource alarms (cluster-wide). The emergency stop. When memory or disk thresholds are breached on any node, all publishers on all nodes are blocked. Connections show state: blocked (tried to publish, frozen) or state: blocking (alarm active, has not tried to publish yet).

The distinction matters operationally. blocked and blocking mean the memory or disk alarm is active and publishing has halted cluster-wide; that is a page. flow means one publisher is being paced by a slow downstream; most of the time, that is the system working correctly. The official documentation is blunt about it: a connection in flow control should see no difference from normal running, and the flow state exists to inform the operator that the publishing rate is being restricted.

Four connection states you will see:

StateMeaningUrgency
runningNormal operationNone
flowCredit-based backpressure on this connection onlyInformational unless sustained and widespread
blockingResource alarm active, connection has not attempted to publishCovered by memory/disk alarm signals
blockedResource alarm active, connection tried to publish and is frozenCovered by memory/disk alarm signals

How the credit chain works

Inside the broker, every entity on the write path is an Erlang process: one per connection, one per channel, one per queue, plus shared message store processes. These processes talk to each other through credit-based flow control.

When a process sends work to the next process in the chain, it spends credit. When the receiver processes the work, it grants credit back. When credit runs out, the sender blocks until more credit is granted. The chain looks like this:

flowchart LR
    P[Publisher] -->|AMQP publish| C[Connection process]
    C -->|spends credit| CH[Channel process]
    CH -->|spends credit| Q[Queue process]
    Q -->|spends credit| MS[Message store / disk]
    MS -->|grants credit| Q
    Q -->|grants credit| CH
    CH -->|grants credit| C
    C -->|socket backpressure| P

The backpressure cascades upward. If the queue process is slow (for example, it is paging messages to disk under memory pressure, or disk I/O is saturated), it stops granting credit to the channel. The channel’s credit runs out, so it stops granting credit to the connection. The connection process stops reading from the TCP socket, the publisher’s TCP buffer fills, and the publisher sees reduced throughput. The connection’s state shows as flow.

Key properties:

  • Per-process and per-connection. A single slow queue can put every connection publishing to it into flow, while connections publishing to other queues stay in running.
  • Highly transient. Credit is granted and consumed continuously. A connection can flip in and out of flow many times per second. Any single sample showing flow tells you almost nothing.
  • Invisible to the application. Most AMQP client libraries do not surface flow control to the publisher. The application simply observes lower throughput with no error.
  • Healthy by design. Flow control prevents unbounded memory growth when publishers outrun consumers or the persistence layer. The problem is never the backpressure itself; the problem is whatever is causing it.

The default credit values are set as an initial credit of 200 messages, with additional credit granted as the receiver processes work, tunable via the credit_flow_default_credit configuration key.

One reporting nuance: the state reported for a process stays flow for a short hysteresis window (on the order of a second) after the last block event, even if the process has already been unblocked. The management UI can therefore show flow briefly after throttling has resolved.

Quorum queues are different at the tail of the chain

For quorum queues, the credit chain effectively ends at the channel. Instead of credit to a queue process, the channel tracks pending Raft commands. When the number of pending commands exceeds quorum_commands_soft_limit, the channel stops granting credit upstream, and the publishing connection throttles. The symptom looks the same (state: flow), but the bottleneck is Raft replication: follower disk I/O, inter-node latency, or a lagging member.

Where backpressure can originate

  • Queue process bottleneck. A classic queue is a single Erlang process and cannot scale beyond one core’s worth of work. A hot queue slows down, credit stalls, and publishers to that queue throttle.
  • Disk I/O saturation. Persistent message writes, queue index files, and quorum WAL segments compete for the disk. Slow persistence means slow credit return.
  • Memory pressure and paging. When the broker crosses the paging threshold, queues start writing messages to disk. The added I/O slows queue processes and flow state appears before any memory alarm fires. See RabbitMQ paging messages to disk.
  • CPU saturation. A persistently elevated Erlang run queue delays every process in the chain, including credit processing.
  • Cluster link congestion. Publishing to a queue owned by a remote node routes through the Erlang distribution link between nodes. If that link saturates, flow triggers on the local publishing connection even though the local node has spare resources.

When flow is healthy and when it is a bottleneck

Flow state is expected during bursts. A batch publisher dumping a backlog, a burst of persistent messages, or a wave of consumers joining with large prefetch (the queue must read and dispatch many messages at once, briefly stalling the publish path) will all produce transient flow. It resolves on its own.

It becomes a real signal when it is sustained and widespread. A practical threshold: alert (ticket, not page) when more than 10% of connections are in flow state sustained for more than 5 minutes. That pattern indicates a systemic write-path bottleneck, typically disk I/O saturation or a hot queue, and it is often the earliest warning before memory pressure escalates into a full alarm. Flow is the leading edge; the memory resource limit alarm is the cliff.

A rough decision guide:

ObservationInterpretationAction
A few connections flicker through flow during burstsNormal backpressureNone
Same one or two connections persistently in flowA specific slow queue or slow consumer pathInvestigate that queue’s depth, paging, disk I/O
>10% of connections in flow for >5 minSystemic write-path bottleneckCheck disk I/O, run queue, paging activity
Any connections in blocked or blockingMemory or disk alarm activeTreat as the alarm incident, not a flow issue

Checking connection states

Both of these are read-only and safe to run:

# Count connections by state via the management API
curl -s -u guest:guest http://localhost:15672/api/connections \
  | jq 'group_by(.state) | map({state: .[0].state, count: length})'

# Or via the CLI
rabbitmqctl list_connections state | sort | uniq -c

Note that the default guest user only authenticates over localhost; use real management credentials for remote checks.

Caveats:

  • There is no aggregate metric for flow or blocked connection counts in the Management API or the Prometheus plugin. The count must be derived by iterating /api/connections and counting states, which is expensive on deployments with thousands of connections. Sample at a reasonable interval rather than polling aggressively, and remember that the management API itself imposes overhead.
  • A single sample is nearly meaningless because of how transient flow is. You need repeated samples over minutes to distinguish flicker from sustained throttling.
  • Publisher confirm latency, measured from the client side, rises before flow becomes visible at the broker. If you have client instrumentation, it is an earlier warning than connection state.

Signals to correlate before acting

SignalWhy it mattersWarning sign
Connections in flow (derived count)The symptom itself>10% of connections, sustained >5 min
Disk I/O on the data partitionPersistence bottleneck is the most common root causeHigh write latency, saturation during flow events
messages_paged_out per queuePaging slows queue processes and stalls creditPage-outs climbing while flow is active
Erlang run_queueCPU saturation delays the whole credit chainSustained above core count
mem_used / mem_limitFlow often precedes the memory alarmRatio > 0.7 while flow is widespread
Publish rateSustained heavy flow shows up as throttled ingestPublish rate sagging below baseline without a client cause
Quorum queue follower healthRaft replication lag throttles publishers to that queueLagging or offline members, growing WAL

The correlation pattern that matters: widespread sustained flow + rising messages_paged_out + climbing mem_used / mem_limit means you are on the path to a memory alarm. Act on queue depth and consumers now, before the cliff.

If your applications publish and consume on the same connection

Because flow control throttles at the connection level, a publishing channel being throttled can interfere with other operations sharing that connection. Applications that regularly experience flow control should use separate connections for publishing and consuming. Connections that only consume are not affected by publisher flow control. This is a cheap client-side change that isolates consumers from publish-path backpressure entirely.

What not to do: do not alert on “any connection in flow” the way you alert on resource alarms. That is the most common mistake teams make with this signal, and it produces pager noise for a mechanism that is working as designed. And do not “fix” flow by restarting the broker. Flow is a symptom of a downstream bottleneck; a restart destroys the diagnostic state and often makes things worse during queue recovery.

How Netdata helps

Netdata surfaces the signals you need to put flow state in context, without hand-deriving counts from the API:

  • Connection state tracking from the management API, so you can trend how many connections sit in flow versus running over time instead of relying on one-off samples.
  • Per-queue messages_paged_out and memory ratio charts, letting you see whether flow events line up with paging activity and approach to the memory watermark.
  • Publish, deliver, and ack rates alongside queue depth, so you can tell whether throttled ingest is caused by a downstream drain problem or a genuine publisher burst.
  • Disk I/O and system metrics from the node itself, correlated on the same dashboard, which is how you confirm or rule out persistence saturation as the root cause of sustained flow.
  • Erlang run queue visibility, catching the CPU-saturation variant where the credit chain stalls because schedulers are overloaded.

The correlation is the point: flow count on its own is noise; flow count aligned with paging, disk latency, and a climbing memory ratio is an early warning that buys you minutes before a cluster-wide alarm.