RabbitMQ Mnesia growth: too many queues and bindings, slow declarations and rejoins

Your cluster still routes messages fine, but everything else has gone soft. queue.declare calls that used to return in milliseconds now take a second or more. The management UI takes ten seconds to render the queue list. A node restarted for a routine upgrade has been “starting” for five minutes and is serving no traffic. No resource alarm has fired, and yet the system is clearly sick.

This is what Mnesia metadata growth looks like in production. All RabbitMQ metadata (queues, exchanges, bindings, vhosts, users, permissions, policies) lives in Mnesia, Erlang’s built-in distributed database, replicated across cluster nodes and held in RAM as well as on disk. Mnesia does not store message data, but every declaration, binding, policy change, and node rejoin goes through it, synchronously. When the tables get large, every one of those operations gets slower.

The trigger is almost never a single big queue. It is accumulation: applications that auto-declare a queue per user, per session, or per request and never clean up; reply-queue patterns without auto-delete; topic exchange topologies that grow a binding per entity. By the time declarations are visibly slow, the table counts are usually well into five figures.

What this means

Mnesia operations are synchronous for transactional consistency. When a client declares a queue, that declaration is a Mnesia transaction that must replicate across the cluster before it completes. Small tables make these transactions cheap. Large tables make them expensive in three distinct ways:

1. Declaration and binding latency climbs. Creating, deleting, or binding a queue requires a metadata transaction. At community-observed thresholds, rabbit_queue table sizes past roughly 50,000 entries start to hurt, and past roughly 100,000 entries, declarations and the management API become noticeably slow. These are operational heuristics, not hard RabbitMQ limits, and your mileage depends on hardware and binding density.
2. The management API degrades. Listing queues, exchanges, and bindings means scanning those tables. The management plugin’s stats aggregation is itself a resource consumer on nodes with very large object counts, and some teams disable detailed per-queue stats on huge deployments for exactly this reason.
3. Node rejoins become outages. When a node restarts and rejoins a cluster, it syncs its Mnesia tables from a peer. With 100,000 or more queues, that sync can take minutes, during which the rejoining node serves no traffic. Every rolling restart or recovery event carries a metadata-sync tax proportional to table size.

flowchart TD
  A[Apps auto-declare queues and bindings without cleanup] --> B[rabbit_queue and rabbit_route tables grow]
  B --> C[Declaration and binding transactions slow down]
  B --> D[Management API scans become expensive]
  B --> E[Node rejoin: Mnesia table sync from peer]
  E --> F[Minutes of no traffic served during rolling restarts]
  C --> G[Client timeouts and declaration retries]
  G --> B
  D --> H[Monitoring collection itself stresses the node]

Note the feedback loop: slow declarations cause client timeouts, clients retry declarations, and retries add more transaction load. High transient and auto-delete queue churn also stresses Mnesia, since each create and delete is a replicated transaction. Churn hurts as much as raw count.

Common causes

Quick checks

All read-only and safe to run during an incident.

# Total object counts: queues, exchanges, connections, channels, consumers
curl -s -u guest:guest http://localhost:15672/api/overview | jq '.object_totals'

# Queue churn: is creation outpacing deletion?
curl -s -u guest:guest http://localhost:15672/api/overview | \
  jq '.churn_rates | {queue_declared, queue_created, queue_deleted}'

# Mnesia table sizes (the direct measurement)
rabbitmqctl eval '[{T, mnesia:table_info(T, size)} || T <- mnesia:system_info(tables)].'

# Memory share held by Mnesia metadata
rabbitmq-diagnostics memory_breakdown

# Count empty, consumer-less queues
rabbitmqctl list_queues name messages consumers | \
  awk '$2 == 0 && $3 == 0' | wc -l

# Is a node mid-rejoin? Check uptime and cluster view
rabbitmq-diagnostics cluster_status
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, uptime}'

A note on the table-size eval: reading table sizes is cheap, but on a severely overloaded node any rabbitmqctl eval competes with broker work. Run it once, record the numbers, and do not put it in a tight loop.

How to diagnose it

1. Confirm metadata is the problem, not messages. Check mem_used and the memory breakdown. If the mnesia category is a large share of node memory while queue message counts are modest, you have a metadata problem, not a backlog problem.
2. Get the table sizes. From the eval above, look at rabbit_queue, rabbit_durable_queue, rabbit_route, and the binding tables. Rough guidance: rabbit_queue past 50,000 is where you start planning; past 100,000 you should expect slow declarations and slow management API responses.
3. Determine whether the driver is count or churn. Compare queue_created against queue_deleted over 30 minutes. A positive gap sustained over hours means a leak. High rates of both mean churn, which stresses Mnesia transactions even at stable table sizes.
4. Identify the source application. Group queues by name prefix or vhost. Auto-generated queue names almost always share a pattern that points at one application or client library. Check which connections are issuing declarations.
5. Check rejoin exposure. If you are in a rolling restart or recently recovered a node, correlate the slow window with node uptime. A rejoining node syncing a 100k-queue metadata set explains minutes of unavailability all by itself.
6. Verify the management API is a victim, not the cause. Probe the AMQP listener (TCP connect to 5672) separately. If AMQP publish/consume is healthy but the management API is slow, the data plane is fine and the problem is metadata scale plus stats collection overhead.

Metrics and signals to monitor

Fixes

Stop the leak at the application

Find the declaring application and change the pattern. Queues should be declared once at deploy time or bounded to a known set, not per request, per user, or per session. Where ephemeral queues are genuinely needed (RPC reply queues), use auto-delete or exclusive queues so cleanup happens on connection close, and make sure client libraries actually close connections rather than leaking them. Note the tradeoff: auto-delete queues add churn, so this helps with table size but not with transaction rate.

Delete the accumulated queues

Once the source is fixed, remove the backlog of dead queues in batches during low traffic, not with a single mass delete. Bulk deletes are themselves Mnesia transactions, and deleting thousands of queues in one shot can stall declarations cluster-wide while the transactions replicate. Script a loop that deletes a few hundred queues, pauses, and repeats, while watching declaration latency.

Reduce binding count

If the problem is bindings rather than queues, simplify the topology. A single topic exchange with a binding per entity is a classic growth pattern; consolidating routing keys or splitting traffic across a small number of direct exchanges can collapse the binding table dramatically. Test routing behavior before deleting bindings in production: a missing binding silently drops non-mandatory messages.

Plan restarts around the sync cost

Until table sizes come down, treat every node restart as carrying a multi-minute rejoin penalty. Restart one node at a time, wait for it to fully rejoin and serve traffic before touching the next, and drain clients from a node before restarting it so the sync window does not look like an outage.

Evaluate Khepri on supported versions

Khepri is RabbitMQ’s Raft-based replacement for Mnesia as the metadata store. It became the default for new clusters in RabbitMQ 4.1, and RabbitMQ 4.2 removed Mnesia support entirely. Existing clusters upgraded from 3.x keep using Mnesia until the khepri_db feature flag is explicitly enabled, and enabling it is one-way: the only rollback path is blue-green deployment. Khepri changes the replication and recovery characteristics of the metadata store, but it does not make unbounded queue and binding proliferation free; metadata scale still costs RAM and operation latency. Treat migration as its own planned project, not as a hotfix for an active incident.

Prevention

• Alert on queue and binding count trends, not just absolute values. A queue count that grows 5% week over week is a leak you can fix while it is cheap.
• Set per-vhost limits where appropriate. RabbitMQ supports per-vhost max-queues and max-connections limits, which turn silent proliferation into an immediate, visible declaration error.
• Gate declaration patterns in code review. Any code path that declares a queue or binding inside a request handler, loop, or per-entity provisioning flow should be questioned.
• Exercise restarts. Periodically restart a node in a staging cluster with production-scale metadata and measure rejoin time. That number is your real rolling-restart budget.
• Clean up automatically. TTLs, queue expiry (x-expires), and auto-delete on genuinely ephemeral queues keep table sizes bounded without operator toil.

How Netdata helps

• Object counts over time: Netdata’s RabbitMQ collector tracks connections, channels, consumers, queues, and exchanges from the overview endpoint, so queue-count growth shows up as a visible trend long before declarations get slow.
• Churn correlation: queue declared/created/deleted rates on the same dashboard as object totals let you distinguish a leak (created > deleted) from churn (both high) at a glance.
• Memory breakdown context: correlating node memory usage with queue counts and message depths separates metadata-driven memory growth from backlog-driven growth, which have completely different fixes.
• Restart and rejoin visibility: node uptime resets next to cluster-wide declaration latency and management API responsiveness show exactly how expensive your last rolling restart was.
• Per-queue cardinality control: on deployments with tens of thousands of queues, you can scope per-queue metric collection so monitoring itself does not add to the management plugin’s load.

Related guides

• RabbitMQ channel leak and churn: the channel-per-message anti-pattern
• RabbitMQ classic mirrored queues removed: migrating to quorum queues
• RabbitMQ connection blocked / blocking: publishers frozen by a resource alarm
• RabbitMQ connection in flow state: credit-based backpressure explained
• RabbitMQ connection storm: reconnect loops, FD pressure, and CPU spent on handshakes
• RabbitMQ consumer timeout: delivery acknowledgement timed out and the channel is closed
• RabbitMQ consumer_utilisation low: consumers attached but not keeping up
• RabbitMQ consumers connected but not acknowledging: the consumer black hole
• RabbitMQ disk free limit alarm: free disk space insufficient and publishing halted
• RabbitMQ disk_free_limit at the default 50MB: the production footgun
• RabbitMQ Erlang distribution port saturation: the single link that carries all cluster traffic
• RabbitMQ Erlang process exhaustion: proc_used near the process limit