RabbitMQ file descriptor exhaustion: fd_used near fd_total and refused connections

Clients that were connected keep working, but every new connection attempt fails. Publishers in a reconnect loop after a deployment start timing out. The broker log shows too many open files, and the management API shows fd_used sitting at or near fd_total. This is RabbitMQ file descriptor exhaustion, and it is a cliff-edge failure: no graceful degradation, just a hard refusal.

This is also one of the most preventable RabbitMQ incidents. The default OS limit (often 1024) is absurdly low for any real deployment, and fd_used / fd_total is a ratio you can watch climbing for hours or days before anything breaks. Mid-incident, the fix is cutting connection pressure now and raising limits properly afterward. Proactively, the monitoring section is the part that matters.

This failure gets misdiagnosed as a disk problem because too many open files surfaces around file operations, or as a network problem because the visible symptom is refused connections. It is neither. The node has run out of kernel-level handles it needs for sockets and files alike.

What this means

Every RabbitMQ node runs inside the Erlang VM as a single OS process, and that process needs file descriptors for almost everything it does:

• One FD per TCP connection: client connections, inter-node cluster links, and federation/shovel links.
• Queue index and message store files: classic queues with persistence open segment files; busy queues hold several.
• Raft WAL segments: quorum queues keep write-ahead log files open per queue.
• Log files, management listeners, and internal runtime handles.

The FD limit is per-node, not cluster-wide. When the node hits it, three things fail at once: new connections are refused, the node cannot open queue index or store files, and paging messages to disk can fail. That last one is what turns an FD problem into a memory incident, because a node that cannot page to disk under memory pressure has no safety valve left.

There is also a trap inside the trap. RabbitMQ tracks sockets separately from total FDs, and the socket limit is roughly 90% of the FD limit. You can exhaust sockets_total and start refusing connections while fd_used / fd_total still looks acceptable. Always check both ratios.

flowchart TD
  A[Connections and queue files consume FDs] --> B[fd_used climbs toward fd_total]
  B --> C{sockets_used near sockets_total?}
  C -->|Yes, sockets hit first| D[New connections refused]
  C -->|No| E{fd_used at fd_total?}
  E -->|Yes| D
  D --> F[Cannot open queue index/store files]
  D --> G[Paging to disk may fail]
  G --> H[Memory pressure escalates with no relief valve]
  F --> I[Node effectively bricked]

Common causes

Quick checks

All read-only. Run these before touching anything.

# 1. RabbitMQ's own view: fd and socket usage plus limits
rabbitmq-diagnostics status | grep -A5 "File Descriptors"

# 2. Same data via the management API, with the ratio computed
curl -s -u guest:guest http://localhost:15672/api/nodes | \
  jq '.[] | {name, fd_used, fd_total, fd_ratio: (.fd_used / .fd_total),
             sockets_used, sockets_total, sock_ratio: (.sockets_used / .sockets_total)}'

# 3. Ground truth from the OS: actual open FDs for the beam process
BEAM_PID=$(pgrep -f beam.smp | head -1)
ls /proc/$BEAM_PID/fd | wc -l
grep "Max open files" /proc/$BEAM_PID/limits

# 4. Who is holding connections: group by peer host to find the noisy client
rabbitmqctl list_connections peer_host state | sort | uniq -c | sort -rn | head -20

# 5. Connection churn: are connections being created faster than closed?
curl -s -u guest:guest http://localhost:15672/api/overview | \
  jq '.churn_rates | {connection_created, connection_closed}'

# 6. Queue count: is queue proliferation the real FD consumer?
curl -s -u guest:guest http://localhost:15672/api/overview | jq '.object_totals'

# 7. Confirm the refused-connection symptom in the broker log
grep -i "too many open files\|file descriptor limit" /var/log/rabbitmq/rabbit@$(hostname).log | tail -20

Note on step 3: the OS count from /proc/<pid>/fd and the broker-reported fd_used can differ slightly because of how the Erlang VM accounts for handles. Trust the trend and the ratio, not an exact match between the two numbers.

How to diagnose it

1. Confirm the symptom is FD exhaustion, not something else. Refused connections can also come from a resource alarm (memory or disk), which blocks publishers but does not refuse the TCP accept. Check mem_alarm and disk_free_alarm in /api/nodes. If an alarm is active, you are dealing with publisher blocking from a resource alarm, and the FD numbers are secondary.

2. Read both ratios. Get fd_used / fd_total and sockets_used / sockets_total from /api/nodes. If the socket ratio is near 1.0 while the FD ratio is lower, the socket limit hit first. Either way, new connections are dead. If fd_total is around 1024, the limit itself is the root cause and everything else is just what tipped you over it.

3. Determine whether the growth is connections or files. Compare object_totals.connections against the FD count from /proc/<beam_pid>/fd. If connections roughly explain the FD usage, you have a connection problem (leak or storm). If connections are modest but FDs are high, look at queue count and queue type: persistent classic queues open index segment files, and quorum queues hold Raft WAL files open.

4. Find the source of connection growth. rabbitmqctl list_connections peer_host grouped by host usually identifies one or two offending clients. A connection storm (churn rate spiking, short connection lifetimes) points at a crash-looping application or a reconnect loop without backoff. A slow, steady climb over days points at a leak: connections opened and never closed.

5. Check the log for the smoking gun. too many open files in the broker log confirms exhaustion events. The broker also logs when the file descriptor limit alarm sets, with a notice that new connections will not be accepted until the alarm clears.

6. Assess blast radius beyond connections. If the node is under memory pressure and cannot page to disk because file opens are failing, treat this as more urgent than a plain connection limit. Watch mem_used / mem_limit and queue-level paged-out message counts while you work.

Metrics and signals to monitor

Headroom guidance for the FD ratio: comfortable below 0.6, concerning between 0.6 and 0.8, critical above 0.8. The failure itself is binary, so the ratio is your runway: (fd_total - fd_used) / fd_growth_rate tells you roughly how long you have.

Fixes

Immediate relief during an incident

Cut connection pressure. If one or two clients are responsible for most connections (step 4 in diagnosis), stop or firewall those clients while you fix them. Closing zombie or leaked connections via the management API frees FDs immediately without restarting the broker. Do not restart the node as a first move: a restart triggers a reconnection storm from every client at once, which is exactly the load profile that exhausted FDs in the first place, and queue recovery will open a burst of index and store files at the same time.

Drain a queue leak if that is the cause. If thousands of abandoned auto-declared queues are holding segment files open, deleting the unused queues frees their FDs. Confirm the queues are genuinely unused (zero consumers, zero publish rate) before deleting.

Raise the limit properly

Fix the systemd unit, not just limits.conf. This is the mistake most teams make. If RabbitMQ runs under systemd, /etc/security/limits.conf does not apply to the service. You need a unit override:

# Create a systemd override for the FD limit
systemctl edit rabbitmq-server

[Service]
LimitNOFILE=65535

Then systemctl daemon-reload and restart the service during a maintenance window. The restart is required: the limit is set at process start.

Raise the OS ulimit too. Raising only the OS limit or only the Erlang-side limit has no effect; both must move together. Verify after restart that fd_total in /api/nodes reflects the new value, and confirm with /proc/<beam_pid>/limits.

Size the limit for the workload, not the default. RabbitMQ’s production checklist recommends allowing at least 65,536 file descriptors for the broker user. A practical sizing rule from the official networking documentation: take the expected peak concurrent connections per node, multiply by 1.5, and make sure the limit is at least that high. Add headroom for queue index, store, and WAL files if you run many persistent or quorum queues. If you raise the limit well above the Erlang VM’s default port ceiling, check ERL_MAX_PORTS as well, since the VM has its own internal cap that must be raised alongside.

Prevention

• Alert on the ratio, not the failure. Alert at fd_used / fd_total > 0.8 and sockets_used / sockets_total > 0.8, sustained. The failure mode gives you no warning at the moment it happens; the ratio gives you hours or days.
• Provision the limit at deploy time. Bake LimitNOFILE into the unit (or container runtime equivalent) so no node ever runs with the 1024 default. See the monitoring checklist for where FD signals sit relative to the rest of the broker’s baseline signals.
• Watch connection and queue churn. A stable connection count can hide heavy churn, and slow leaks only become visible over days. Trend churn_rates and object_totals so you catch the buildup, not the outage.
• Fix client behavior. Connection-per-operation patterns, reconnect loops without backoff, and channels opened without closing are client bugs that the broker pays for. Enforce connection pooling and exponential backoff in shared client libraries.
• Clean up queue proliferation. Auto-delete and exclusive queues should disappear with their connections. If queue count grows monotonically, something is declaring queues without a lifecycle. Review the mental model for how queues and connections consume broker resources when auditing.

How Netdata helps

• Netdata’s RabbitMQ collector surfaces fd_used, fd_total, sockets_used, and sockets_total per node, so both ratios are charted continuously rather than discovered during an outage.
• Correlating the FD ratio against connection count and connection churn on the same dashboard separates a leak (slow correlated climb) from a storm (vertical ramp with churn spike) at a glance.
• Because FD exhaustion disables disk paging, seeing mem_used / mem_limit alongside the FD ratio tells you whether the incident is about to compound into a memory alarm.
• Per-second collection catches the fast ramps that 5-second management API polling windows smooth over, which matters during post-deploy reconnection storms.
• Alerts on the ratio thresholds (0.8 warning) fire while you still have runway, since the failure itself is a cliff-edge with no gradual symptom.

Related guides

• RabbitMQ connection blocked / blocking: publishers frozen by a resource alarm
• RabbitMQ connection in flow state: credit-based backpressure explained
• RabbitMQ consumer timeout: delivery acknowledgement timed out and the channel is closed
• RabbitMQ consumer_utilisation low: consumers attached but not keeping up
• RabbitMQ consumers connected but not acknowledging: the consumer black hole
• RabbitMQ disk free limit alarm: free disk space insufficient and publishing halted
• RabbitMQ disk_free_limit at the default 50MB: the production footgun
• RabbitMQ flow control vs resource alarms: the two throttling mechanisms operators confuse
• RabbitMQ head message age: the queue latency that depth alone cannot show
• How RabbitMQ actually works in production: a mental model for operators
• RabbitMQ memory resource limit alarm: publishers blocked across the whole cluster
• RabbitMQ monitoring checklist: the signals every production broker needs