RabbitMQ connection storm: reconnect loops, FD pressure, and CPU spent on handshakes

A connection storm happens when many clients open connections to RabbitMQ at the same time, or when a smaller set of clients reconnect in a tight loop. The broker tolerates thousands of idle connections far better than it tolerates creating them. Connection setup is expensive: TCP handling, TLS handshake, AMQP negotiation, authentication, a new Erlang process per connection, plus one process per channel the client opens.

The symptom pattern is distinct. churn_rates.connection_created spikes far above baseline. File descriptor and socket usage climb fast. The Erlang run queue rises because scheduler time goes to handshakes instead of message routing and heartbeats. In the worst case, healthy connections start timing out because the VM cannot service them, those clients reconnect, and the storm feeds itself.

The steps below assume RabbitMQ 3.13+ with the management plugin available. For the broader signal model, see How RabbitMQ actually works in production.

What this means

Every AMQP connection costs the broker:

• One file descriptor and one socket. FD exhaustion is a cliff edge: new connections are refused, and queue file operations can fail.
• One Erlang process per connection, one per channel. Process table exhaustion crashes the node outright.
• Scheduler CPU. TLS termination, AMQP negotiation, and authentication all run on the Erlang schedulers. A burst of simultaneous handshakes can saturate them.
• Memory per connection. Kilobytes per idle connection process, more with buffers and channels.

The dangerous property of a storm is positive feedback. As scheduler load rises, heartbeat processing is delayed. Heartbeat timeouts kill existing connections. Those clients reconnect, adding more handshakes. A sustained run queue above the core count delays everything including heartbeats, and in clusters it can trigger false network partition detection.

flowchart TD
  A[Trigger: crash loop, deploy, LB failover, broker restart] --> B[Mass reconnect attempts]
  B --> C[Handshake CPU: TLS, AMQP negotiation, auth]
  B --> D[FD, socket, and process count climb]
  C --> E[Run queue rises]
  E --> F[Heartbeats delayed]
  F --> G[Existing connections time out]
  G --> B
  D --> H[New connections refused at FD or socket limit]
  E --> I[False partition detection in clusters]

Common causes

Quick checks

All read-only and safe during an incident.

# 1. Connection churn rate (cumulative counters; sample twice, ~30s apart)
curl -s -u guest:guest http://localhost:15672/api/overview | \
  jq '.churn_rates | {connection_created, connection_closed, channel_created, channel_closed}'

# 2. Object totals: connections and channels now vs. baseline
curl -s -u guest:guest http://localhost:15672/api/overview | jq '.object_totals'

# 3. FD, socket, and process headroom
curl -s -u guest:guest http://localhost:15672/api/nodes | \
  jq '.[] | {name, fd_used, fd_total, sockets_used, sockets_total, proc_used, proc_total}'

# 4. Erlang run queue (CPU saturation proxy)
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, run_queue}'

# 5. Who is connecting: count connections per peer host
rabbitmqctl list_connections peer_host | sort | uniq -c | sort -rn | head -20

# 6. Connection states (blocked/flow are a different problem; here you want churn)
rabbitmqctl list_connections state | sort | uniq -c

# 7. Authentication failures feeding the loop
grep -c "login refused" /var/log/rabbitmq/rabbit@$(hostname).log

# 8. Node uptime: did the broker just restart and trigger a fleet-wide reconnect?
curl -s -u guest:guest http://localhost:15672/api/nodes | \
  jq '.[] | {name, uptime_seconds: (.uptime / 1000)}'

Two caveats. The churn and message-rate fields are cumulative counters; a single reading is meaningless, so take two samples and compute the delta per second. And /api/connections iteration is expensive on nodes with many connections; prefer rabbitmqctl list_connections peer_host over repeated full API dumps during the incident.

How to diagnose it

1. Confirm churn, not just count. A stable connection count with high connection_created and connection_closed rates means a reconnect loop: connections open and close at equal rates and the totals look fine. A rising count with a high creation rate means a genuine flood or leak. This distinction decides the fix.
2. Check headroom. From check 3: fd_used / fd_total and sockets_used / sockets_total above 0.8 mean you are near the cliff where new connections get refused. proc_used / proc_total above 0.8 means process-table risk. If all three ratios are comfortable, the storm is expensive but survivable and will likely resolve on its own.
3. Check scheduler saturation. run_queue is aggregated across all schedulers, so compare it to the total core count. Sustained values above the core count mean every operation, including heartbeats, is delayed. This is the point where the storm starts damaging healthy connections.
4. Find the source. Group connections by peer_host (check 5). In a reconnect loop, one host or one subnet dominates. If the top peer hosts are load balancer IPs, suspect LB health checks or an LB failover that moved all client traffic at once. If clients set connection names, rabbitmqctl list_connections peer_host user_provided_name identifies the offending application directly.
5. Check for auth failures. A client with wrong credentials fails fast and retries fast, producing very high churn with very short connection lifetimes. The log pattern login refused at high rate from one source is the signature. Fixing the credential ends the storm.
6. Rule out the normal cases. Rolling deployments cause bounded churn that decays within the deployment window. A broker restart causes a one-time thundering herd as every client reconnects; churn spikes and then falls. If churn is not decaying and uptime is not recent, you have a loop, not an event.
7. In clusters, watch for the secondary effect. A saturated run queue delays inter-node heartbeats and can produce false partition detection. If a partition appears during the storm, treat the storm as the cause, not a network fault, until proven otherwise.

Metrics and signals to monitor

Fixes

Stop the offending client

If one peer_host or deployment dominates, the fastest relief is to stop it at the source: scale the crash-looping deployment to zero, or block it at the network layer (firewall, security group, LB rule) while you fix the application. This is disruptive to that client only, and it is what actually ends the loop. Restarting the broker does not help: the clients will reconnect to the restarted node and re-trigger the herd.

Fix authentication failures

If the loop is credential-driven, correct the credential in the client and redeploy. The storm stops as soon as connections start succeeding.

Let a bounded storm burn out

If the trigger was a broker restart or LB failover, FD and socket headroom is comfortable, and churn is decaying, do nothing aggressive. Adding capacity or rate-limiting new connections mid-herd can make recovery slower and harder to reason about. Watch fd_used, sockets_used, and run_queue until they settle.

Raise limits only as headroom, not as the fix

If storms are recurring and legitimate (large fleets, real failover events), raise the OS FD limit and the Erlang process and port limits together; raising one without the others has no effect. This buys capacity for the thundering herd but does nothing for a true reconnect loop, which will consume any limit eventually.

Prevention

• Client-side backoff with jitter. Every client library reconnect path must use exponential backoff, and retries must be jittered so a fleet does not reconnect in lockstep after a broker restart. A retry loop with fixed sub-second delays is a storm generator. This is the single highest-value fix.
• Provision FDs for the herd. Size the FD limit for reconnection events, not steady state: expected peak concurrent connections with margin for the post-restart overlap window where old and new connections coexist briefly.
• Watch LB health checks. Health checks that open test connections add churn continuously. Tune check intervals, prefer lightweight checks, and confirm whether your LB health check completes an AMQP handshake or just opens TCP.
• Plan rolling restarts. Restart nodes one at a time and let the herd from each node settle before proceeding. Suppress resource alerts for a short warm-up window after each restart so the expected churn does not page anyone.
• Alert on the churn rate itself. A baseline-relative alert on connection_created rate (sustained above 3x baseline, outside deployment windows) catches loops in minutes instead of at FD exhaustion.
• Separate connection-count alerts from churn alerts. A stable count with violent churn underneath is the case count-based alerts always miss.

How Netdata helps

• Netdata collects the RabbitMQ overview counters, including churn_rates.connection_created and connection_closed, and computes per-second rates automatically, so you see the storm as a rate spike rather than having to diff cumulative counters by hand.
• Per-node fd_used, sockets_used, and proc_used against their limits are charted together, which makes the “which limit hits first” question answerable at a glance during the incident.
• The Erlang run_queue chart next to connection churn shows the causal link in this failure mode: handshake load driving scheduler saturation.
• Correlating churn with node uptime on the same dashboard separates a post-restart thundering herd (uptime low, churn decaying) from a true reconnect loop (uptime high, churn sustained), which is the central diagnostic fork in this article.
• Anomaly detection on the churn rate flags the early phase of a loop, before FD or socket ratios reach alert thresholds.

Related guides

• RabbitMQ connection blocked / blocking: publishers frozen by a resource alarm
• RabbitMQ connection in flow state: credit-based backpressure explained
• RabbitMQ flow control vs resource alarms: the two throttling mechanisms operators confuse
• RabbitMQ memory resource limit alarm: publishers blocked across the whole cluster
• RabbitMQ monitoring checklist: the signals every production broker needs
• How RabbitMQ actually works in production: a mental model for operators