ProxySQL error 1040 Too many connections: the backend MySQL rejecting the pool

Your application queries are failing with ERROR 1040 (HY000): Too many connections. ProxySQL is up, the admin interface responds, and some backends show ONLINE. But queries against certain hostgroups error out, and the problem is spreading. The error is coming from the backend MySQL server itself, not from ProxySQL’s own pool ceiling.

Four independent limits can produce error 1040 or something visually identical to the client: ProxySQL’s global mysql-max_connections (default 2048), per-user max_connections in mysql_users, per-backend max_connections in mysql_servers, and the backend MySQL’s server-level max_connections. When the backend’s limit is hit, ProxySQL tries to open a connection, the backend refuses, ConnERR climbs, and the backend may be shunned after enough failures. This article covers that fourth case.

The most common root cause is a sizing mistake: per-backend max_connections is set close to MySQL’s max_connections, leaving no room for other ProxySQL instances, admin sessions, replication threads, monitoring tools, or direct application connections that also consume backend slots. The backend runs out of connections while ProxySQL still believes it has headroom.

What this means

The critical diagnostic distinction is when the error surfaces. If the client sees “Too many connections” during connection establishment (the MySQL handshake phase), the limit was hit on the ProxySQL side. If the error appears during query execution, the backend MySQL rejected the connection attempt. This maps to the MySQL C API: mysql_real_connect() failures point to ProxySQL’s own limits, while mysql_query() failures point to the backend.

When the backend returns 1040, several things happen in sequence:

1. ProxySQL records the failed connection attempt as an increment to ConnERR in stats_mysql_connection_pool for that backend.
2. If failures accumulate past mysql-shun_on_failures (default 5) consecutive errors, ProxySQL shuns the backend, marking it SHUNNED and routing traffic to remaining ONLINE backends.
3. The backend stays shunned for mysql-shun_recovery_time_sec (default 10 seconds) before ProxySQL retries it.
4. If remaining backends are also near their limits, the cascade continues until no ONLINE backends remain in the hostgroup.

Common causes

Quick checks

All commands below are read-only. Connect to the admin interface on port 6032.

# Check backend connection pool for ConnERR and saturation
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup, srv_host, srv_port, status, ConnUsed, ConnFree, ConnOK, ConnERR FROM stats_mysql_connection_pool;"

# Check which ProxySQL limit was hit (if any)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name IN ('Access_Denied_Max_Connections','Access_Denied_Max_User_Connections','Access_Denied_Wrong_Password');"

# Check per-errno error breakdown (most granular signal)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT * FROM stats_mysql_errors ORDER BY last_seen DESC LIMIT 20;"

# Check configured per-backend max_connections
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup_id, hostname, port, status, max_connections FROM runtime_mysql_servers;"

# Check delayed and aborted server connections
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name IN ('Server_Connections_delayed','Server_Connections_aborted','ConnPool_get_conn_failure','ConnPool_get_conn_success');"

# Check backend MySQL directly (run on the MySQL host)
mysql -e "SHOW VARIABLES LIKE 'max_connections'; SHOW STATUS LIKE 'Threads_connected';"

# See what is consuming connections on the backend
mysql -e "SELECT user, COUNT(*) as cnt FROM information_schema.processlist GROUP BY user ORDER BY cnt DESC;"

How to diagnose it

The first step is determining whether the error originates from the backend or from ProxySQL itself. The decision tree below narrows the four possible sources to the one you are actually hitting.

flowchart TD
    A["Client sees error 1040"] --> B{"Connect time or query time?"}
    B -->|"Connect time"| C["ProxySQL-side limit hit"]
    B -->|"Query time"| D["Backend MySQL rejecting connections"]
    C --> E["Check Access_Denied_Max_Connections"]
    C --> F["Check per-user utilization in stats_mysql_users"]
    D --> G["Check ConnERR in stats_mysql_connection_pool"]
    D --> H["Check MySQL Threads_connected vs max_connections"]
    G --> I["Backend may go SHUNNED"]

1. Confirm the error source. Query stats_mysql_errors for errno 1040 and note which hostgroup and backend it appears against. If the error is tied to a specific backend host, the backend is the source. If Access_Denied_Max_Connections in stats_mysql_global is rising, ProxySQL rejected the client at its own global limit. These can coexist.

2. Check ConnERR rate. Read ConnERR from stats_mysql_connection_pool twice, a few seconds apart. A rising delta on a specific backend confirms that ProxySQL is actively failing to connect to that backend. If ConnERR is rising but ConnOK is not, the backend is refusing connections.

3. Check the backend MySQL directly. Compare Threads_connected to max_connections on the backend. If Threads_connected is at or near max_connections, the backend is saturated. The question is who is consuming those threads.

4. Identify all connection consumers. On the backend, run SELECT user, COUNT(*) FROM information_schema.processlist GROUP BY user ORDER BY cnt DESC. Look for: ProxySQL’s monitor user, ProxySQL’s app user (possibly multiple entries if multiple ProxySQL instances), replication threads (usually system user), admin sessions, and any direct application connections that bypass the proxy.

5. Sum the ProxySQL demand. For each backend host, sum the configured max_connections across all ProxySQL instances that route to it. This sum should stay below 80% of the backend MySQL’s actual max_connections. If you have two ProxySQL instances each configured with max_connections=400 pointing at a MySQL backend with max_connections=500, the sum (800) vastly exceeds the backend capacity (500).

6. Check for multiplexing collapse. If Client_Connections_hostgroup_locked is a high fraction of Client_Connections_connected, multiplexing has degraded and each client is pinning a backend connection. This inflates backend connection demand far beyond what the pool sizing assumed.

7. Check the ProxySQL version. If you are running v2.0.6 through v2.0.15, a known bug caused ProxySQL to create far more backend connections than mysql_servers.max_connections allowed. Upgrade to v2.0.16 or later.

Metrics and signals to monitor

Fixes

Right-size per-backend max_connections

The primary fix is ensuring the sum of all ProxySQL instances’ per-backend max_connections stays below the backend MySQL’s max_connections minus a reserve for non-ProxySQL consumers.

The reserve must account for: admin sessions connecting directly to MySQL, replication threads (one IO thread and one SQL thread per replica, or more for multi-source replication), monitoring tools that connect directly to MySQL (Prometheus mysqld_exporter, Percona PMM, etc.), and any application connections that bypass ProxySQL.

A practical approach: if MySQL has max_connections=500 and non-ProxySQL consumers use approximately 50 connections, the ProxySQL budget is 450. With two ProxySQL instances, set per-backend max_connections to roughly 200 each, leaving headroom for spikes.

-- Reduce per-backend max_connections (adjust to your capacity)
UPDATE mysql_servers SET max_connections=200 WHERE hostname='10.0.0.10' AND port=3306;
LOAD MYSQL SERVERS TO RUNTIME;
SAVE MYSQL SERVERS TO DISK;

This is safe to apply at runtime. Existing connections are not killed. New connections are subject to the lower limit.

Increase backend MySQL max_connections

If the backend has CPU and memory headroom, raising MySQL’s max_connections may be appropriate. But this is often treating the symptom. Each MySQL thread consumes memory (thread stack plus per-connection buffers). Raising max_connections without understanding why demand is high risks trading connection exhaustion for memory exhaustion.

-- On the backend MySQL (requires SUPER privilege)
SET GLOBAL max_connections=600;

This takes effect immediately but does not persist across MySQL restarts. Add it to my.cnf for persistence.

Address multiplexing collapse

If the root cause is multiplexing degradation (backend connections scaling 1:1 with clients), reducing per-backend max_connections alone will not fix the problem. Queries will still queue and time out. The underlying issue is that session state is pinning backend connections.

Check stats_mysql_processlist for sessions with disabled multiplexing and the reason. Common culprits include ORMs that issue SET commands on every connection, applications using temporary tables or user-defined variables, and long-running transactions.

Handle connection storms

After a ProxySQL restart, the backend connection pool is empty. The first burst of queries creates backend connections synchronously. If enough clients reconnect at once, the backend can be overwhelmed.

If your ProxySQL version supports it, mysql-connection_warming (default false) pre-creates connections up to the free pool threshold. Also ensure mysql-free_connections_pct (default 10) maintains a warm pool of idle connections.

Prevention

• Sum before you set. Before configuring max_connections in mysql_servers, sum the intended value across all ProxySQL instances and verify it is below 80% of the backend MySQL’s max_connections.
• Account for all consumers. Periodically audit the backend SHOW PROCESSLIST grouped by user to verify that non-ProxySQL connection counts match expectations.
• Monitor ConnERR as a rate, not a counter. The cumulative value is not useful without delta. Alert on sustained rate increases.
• Watch multiplexing ratio. Track Client_Connections_hostgroup_locked / Client_Connections_connected. A rising ratio means backend demand is inflating and the sizing model no longer holds.
• Keep ProxySQL current. The v2.0.6 through v2.0.15 connection leak bug is resolved in v2.0.16 and later. Do not run these versions in production.
• Gate cold start in alerts. Use ProxySQL_Uptime > 600 as a condition on connection-pressure alerts to exclude the restart window when pools are empty and connection churn is expected.

How Netdata helps

Netdata’s ProxySQL collector surfaces the signals that distinguish backend-originated 1040 from ProxySQL-side limits, at per-second resolution:

• ConnERR per backend from stats_mysql_connection_pool shows exactly which backend is refusing connections and when the rate started climbing.
• Access_Denied_Max_Connections and Access_Denied_Max_User_Connections are tracked separately, so you can tell whether ProxySQL itself rejected the client or the backend did.
• Server_Connections_delayed and ConnPool_get_conn_failure provide direct evidence of pool starvation before the backend reaches its limit.
• Backend status transitions (ONLINE to SHUNNED) correlate with ConnERR spikes, showing the cause-and-effect chain from backend rejection to traffic rerouting.
• Client_Connections_hostgroup_locked alongside backend ConnUsed reveals multiplexing collapse as a contributing factor, even when the immediate symptom is backend 1040.

Related guides

• ProxySQL backend flapping between ONLINE and SHUNNED: monitor-induced oscillation
• ProxySQL OFFLINE_SOFT vs OFFLINE_HARD vs SHUNNED: what each backend status means
• ProxySQL backend SHUNNED: why a healthy backend gets pulled out of rotation
• How ProxySQL actually works in production: a mental model for operators
• ProxySQL monitor check failures: connect, ping, read-only, and replication-lag probes failing
• ProxySQL monitoring checklist: the signals every production proxy needs
• ProxySQL monitoring maturity model: from survival to expert
• ProxySQL zero ONLINE backends in a hostgroup: total outage for that traffic class