ProxySQL backend connection pool exhausted: queries queuing for a free connection

Queries are queuing. Client-facing latency is climbing. Applications are reporting timeouts or “too many connections” errors. You look at ProxySQL’s backend pool and see ConnFree at zero across one or more backends. The natural assumption is that the pool is exhausted.

It might not be. ConnFree == 0 alone is not saturation. ProxySQL can still create new backend connections up to the max_connections limit configured in mysql_servers. The pool is under pressure, but ProxySQL has headroom to open more connections if the backend can accept them.

True saturation means ProxySQL cannot create new backend connections (either max_connections in mysql_servers is reached, or the backend MySQL is rejecting connections, shown by rising ConnERR) AND has no free ones to reuse. At that point queries queue, latency rises, and without intervention the queue deepens until clients time out.

What this means

Each backend entry in mysql_servers has a max_connections cap that limits how many connections ProxySQL will open to that server. When you see ConnFree == 0 sustained for more than 60 seconds with ConnUsed > 0, the pool is under pressure. But ProxySQL will still attempt to open new connections up to max_connections. The key question is whether those attempts are succeeding or failing:

• Pool pressure without saturation: ProxySQL is creating connections on demand and keeping up, but the free pool is empty. This happens under burst load, batch processing, or when the idle connection floor (mysql-free_connections_pct, default 10) is low relative to demand. Queries may see slightly elevated latency from connection establishment overhead, but they are not failing.
• True saturation: ProxySQL has hit its max_connections cap for a backend, or the backend MySQL itself is rejecting connections. ConnPool_get_conn_failure is rising. Server_Connections_delayed is positive. Queries are genuinely queuing for a free connection.

The single most important diagnostic signal is whether ConnPool_get_conn_failure is rising. That counter measures how often ProxySQL tried to get a backend connection from the pool and could not. It is more actionable than ConnFree == 0 alone.

flowchart TD
    A["ConnFree == 0
sustained > 60s"] --> B{"ConnPool_get_conn_failure
rising?"}
    B -->|No| C["Not saturation
ProxySQL still opening conns
up to max_connections"]
    B -->|Yes| D{"ConnERR rising?"}
    D -->|No| E["ProxySQL max_connections
cap reached for this backend"]
    D -->|Yes| F["Backend rejecting connections
check MySQL max_connections"]
    C --> G["Check multiplexing ratio
and Active_Transactions"]
    E --> H["Raise max_connections
or reduce pinning"]
    F --> I["Check backend MySQL
max_connections
and SHOW PROCESSLIST"]

Common causes

Quick checks

All commands connect to the admin interface on port 6032. Replace credentials as appropriate. These are read-only queries against stats tables and are safe to run during an active incident.

# Pool state per backend: ConnUsed, ConnFree, ConnOK, ConnERR
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup, srv_host, srv_port, status, ConnUsed, ConnFree, ConnOK, ConnERR, Latency_us FROM stats_mysql_connection_pool ORDER BY hostgroup, srv_host;"

# Global pool operation counters: get_conn_failure is the key saturation signal
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name IN ('ConnPool_get_conn_success','ConnPool_get_conn_failure','ConnPool_get_conn_immediate','Server_Connections_delayed','Server_Connections_aborted','Server_Connections_connected');"

# Active transactions and multiplexing health
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name IN ('Active_Transactions','Client_Connections_connected','Client_Connections_non_idle','Client_Connections_hostgroup_locked');"

# Top 20 queries by total execution time
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup, schemaname, username, digest_text, count_star, sum_time, ROUND(sum_time/count_star) AS avg_time_us, max_time FROM stats_mysql_query_digest ORDER BY sum_time DESC LIMIT 20;"

# Per-backend max_connections from runtime config
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup_id, hostname, port, status, max_connections, weight FROM runtime_mysql_servers ORDER BY hostgroup_id;"

# Longest-running client sessions
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT user, cli_host, hostgroup, db, command, time_ms FROM stats_mysql_processlist ORDER BY time_ms DESC LIMIT 20;"

How to diagnose it

1. Confirm whether this is true saturation. Check ConnPool_get_conn_failure and Server_Connections_delayed. If both are rising, queries are queuing for backend connections. If ConnPool_get_conn_failure is flat and only ConnFree == 0, the pool is under pressure but not saturated.

2. Check ConnERR on the saturated backend. Rising ConnERR means ProxySQL tried to connect and was rejected. This points to either max_connections in mysql_servers being reached, or the backend MySQL itself refusing connections. If ConnERR is flat but ConnUsed equals max_connections, ProxySQL’s own per-backend cap is the limit.

3. Compare ConnUsed against max_connections per backend. Cross-reference stats_mysql_connection_pool with runtime_mysql_servers. If ConnUsed is at or near max_connections, the per-backend cap is the bottleneck.

4. Identify what is consuming the connections. Check stats_mysql_query_digest ordered by sum_time DESC. Slow queries hold backend connections longer, reducing pool turnover. A single query type dominating total execution time is your optimization target.

5. Check multiplexing health. Compute Client_Connections_hostgroup_locked / Client_Connections_connected. If this ratio exceeds 0.5, more than half of client sessions have pinned backend connections. The pool cannot multiplex effectively, and ConnUsed will scale linearly with client count instead of sublinearly.

6. Check for unbalanced routing. If one backend in a hostgroup has ConnFree == 0 while peers have idle connections, check weight in runtime_mysql_servers and rule hit patterns in stats_mysql_query_rules.

7. Check the backend MySQL directly. Connect to the backend (bypassing ProxySQL) and run SHOW PROCESSLIST and SHOW STATUS LIKE 'Threads_connected'. If the backend is near its own max_connections, the problem is on the MySQL side, not the proxy side.

Metrics and signals to monitor

Fixes

Slow queries consuming the pool

If stats_mysql_query_digest shows one or few query types dominating sum_time, those queries are holding backend connections long enough to starve the pool. Fixing the query (adding an index, rewriting the SQL, partitioning data) is the correct long-term fix.

Short-term mitigation: increase max_connections in mysql_servers for the affected backend, if the backend MySQL can handle the additional connections. This buys headroom but does not address the underlying query performance.

-- Increase per-backend max_connections (active after LOAD + SAVE)
UPDATE mysql_servers SET max_connections=200 WHERE hostname='10.0.0.5' AND port=3306;
LOAD MYSQL SERVERS TO RUNTIME;
SAVE MYSQL SERVERS TO DISK;

This change takes effect immediately after LOAD MYSQL SERVERS TO RUNTIME. Without SAVE MYSQL SERVERS TO DISK, the change is lost on restart.

Multiplexing collapse

If Client_Connections_hostgroup_locked / Client_Connections_connected is high, sessions are pinning backend connections. Common causes: ORMs setting session variables (SET NAMES, SET sql_mode, SET time_zone) on every connection, long-running transactions, GET_LOCK(), temporary tables, or user-defined variables. When multiplexing is disabled for a session, it remains disabled until the client disconnects.

Diagnose by checking stats_mysql_processlist for sessions with multiplexing disabled. The extended_info column may show the specific reason. Consider moving session variable initialization to MySQL server-side defaults or ProxySQL’s mysql-init_connect so they do not require per-session SET commands.

Short-term: increase max_connections on affected backends to absorb the near 1:1 mapping until the application can be fixed. Long-term: identify which SET commands or session features are disabling multiplexing and eliminate them from the application’s connection initialization path.

max_connections too low in mysql_servers

If ConnUsed is at max_connections but the backend MySQL has available capacity, raise ProxySQL’s per-backend max_connections. Verify the backend can handle the increase by checking SHOW STATUS LIKE 'Threads_connected' against SHOW VARIABLES LIKE 'max_connections' on the MySQL side.

Critical sizing rule: the sum of all ProxySQL instances’ max_connections for a given backend should not exceed approximately 80% of the backend MySQL’s actual max_connections. The remaining 20% covers direct admin connections, replication threads, monitoring tools, and any non-proxy application connections that share the backend.

Unbalanced routing

If one backend is saturated while peers in the same hostgroup are idle, check routing weights in runtime_mysql_servers. A backend with a higher weight receives proportionally more traffic. Also review mysql_query_rules hit patterns: a rule may be routing specific query patterns to one backend disproportionately.

Backend MySQL at its own connection limit

If the backend MySQL is at its own max_connections, ProxySQL cannot help by raising its own limit. The fix is on the MySQL side: raise max_connections in MySQL, reduce idle connections from other consumers, add more read replicas, or identify and terminate runaway sessions consuming connections.

Prevention

• Monitor ConnPool_get_conn_failure rate as your primary saturation signal. It is more actionable than ConnFree == 0 alone because it tells you ProxySQL actually tried and failed to get a connection.
• Track the multiplexing ratio (Client_Connections_hostgroup_locked / Client_Connections_connected) over time. A gradual decline indicates application changes are silently breaking multiplexing long before it becomes an incident.
• Set max_connections headroom properly. The sum of all ProxySQL instances’ per-backend max_connections should stay below 80% of the backend MySQL’s actual max_connections.
• Alert on ConnFree == 0 sustained for more than 60 seconds combined with ConnPool_get_conn_failure above zero. ConnFree == 0 without get_conn_failure is pool pressure, not saturation.
• Watch Server_Connections_delayed. Any sustained positive value means queries are already waiting for backend connections. This is a leading indicator that appears before ConnPool_get_conn_failure starts climbing.
• Reset stats_mysql_query_digest periodically. The table grows unboundedly and resets on restart. Use SELECT * FROM stats_mysql_query_digest_reset for periodic collection; reading that table resets the stats at the end of the SELECT.
• Verify config changes are saved to disk. A runtime-only change to mysql_servers will be lost on restart. Always run SAVE MYSQL SERVERS TO DISK after LOAD MYSQL SERVERS TO RUNTIME.

How Netdata helps

Netdata collects ProxySQL admin metrics per second and retains them across restarts, which matters because ProxySQL’s stats tables reset on restart.

• Per-second ConnPool_get_conn_failure and Server_Connections_delayed: rate of change is more informative than point-in-time snapshots from manual admin queries.
• Per-backend correlation of ConnUsed, ConnFree, and ConnERR: immediately shows whether one backend or all are saturated, and whether errors accompany the zero-free-connection state.
• Active_Transactions and Client_Connections_hostgroup_locked alongside pool metrics: reveals whether multiplexing collapse is driving the exhaustion without manual joins across admin tables.
• Anomaly detection on ConnPool_get_conn_failure and the multiplexing ratio: surfaces gradual degradation before it crosses a static threshold, useful for multiplexing collapse that develops slowly over weeks.
• Historical retention across restarts: stats_mysql_connection_pool and stats_mysql_query_digest reset on restart. Without external persistence, you lose the baseline needed to distinguish normal peak load from degradation.

Related guides

• ProxySQL backend flapping between ONLINE and SHUNNED: monitor-induced oscillation
• ProxySQL OFFLINE_SOFT vs OFFLINE_HARD vs SHUNNED: what each backend status means
• ProxySQL backend SHUNNED: why a healthy backend gets pulled out of rotation
• How ProxySQL actually works in production: a mental model for operators
• ProxySQL monitor check failures: connect, ping, read-only, and replication-lag probes failing
• ProxySQL monitoring checklist: the signals every production proxy needs
• ProxySQL monitoring maturity model: from survival to expert
• ProxySQL zero ONLINE backends in a hostgroup: total outage for that traffic class