ProxySQL error 9001 Max connect timeout reached while reaching hostgroup

Your application is receiving MySQL error 9001 (HY000): “Max connect timeout reached while reaching hostgroup N after Mms.” ProxySQL tried to obtain a working backend connection for the client query and failed within the configured timeout window (mysql-connect_timeout_server_max, default 10000ms).

Every backend in the target hostgroup is either down, SHUNNED, OFFLINE, or saturated to the point where no connection could be borrowed or created in time. Queries are failing.

The “after Mms” portion tells you how long ProxySQL spent trying. If M is close to your mysql-connect_timeout_server_max value, ProxySQL exhausted the full retry window. If M is 0, the timeout fired immediately, which can indicate a configuration or version-specific issue rather than a genuine backend problem.

What this means

ProxySQL maintains a backend connection pool per hostgroup. When a client query arrives, ProxySQL borrows a connection from the pool, routes the query, and returns the connection on completion. If no free connection exists and the backend’s max_connections limit (configured in mysql_servers) has been reached, ProxySQL cannot create a new one. It retries according to mysql-connect_retries_on_failure (default 10) with mysql-connect_retries_delay (default 1ms) between attempts. If the total elapsed time hits mysql-connect_timeout_server_max, error 9001 is returned to the client.

Error 9001 is generated by ProxySQL, not the backend MySQL server. The backend may be healthy but unreachable, saturated, or misconfigured in ProxySQL’s tables.

ConnPool_get_conn_failure in stats_mysql_global is not a direct count of error 9001s. It tracks every time the pool could not immediately provide a connection, including transient shun events that resolve before the client sees an error. Use stats_mysql_errors and max_connect_timeouts to measure actual client-facing failures.

flowchart TD
    A["Client receives error 9001"] --> B{"Any ONLINE backends
in the target hostgroup?"}
    B -- "No" --> C["Backend outage
or all SHUNNED"]
    B -- "Yes" --> D{"ConnERR rising
on backends?"}
    D -- "Yes" --> E["Backend unreachable or
rejecting connections"]
    D -- "No" --> F{"ConnUsed at
max_connections?"}
    F -- "Yes" --> G["Pool saturation or
multiplexing collapse"]
    F -- "No" --> H["Check query rules,
config drift, or
ProxySQL version bugs"]
    C --> I["Check monitor results,
backend reachability"]
    E --> I
    G --> J["Check multiplexing ratio,
raise max_connections"]
    H --> K["Check apply flags,
runtime vs memory config"]

Common causes

Quick checks

Run these against the ProxySQL admin interface (default port 6032). All are read-only.

# Check backend status per hostgroup
mysql -u admin -p<admin_pass> -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup, srv_host, srv_port, status, ConnUsed, ConnFree, ConnOK, ConnERR FROM stats_mysql_connection_pool ORDER BY hostgroup;"

# Check error 9001 occurrences by hostgroup and user
mysql -u admin -p<admin_pass> -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup, username, err_code, count_star, last_seen FROM stats_mysql_errors ORDER BY last_seen DESC LIMIT 20;"

# Check global timeout and pool failure counters
mysql -u admin -p<admin_pass> -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name IN ('max_connect_timeouts','ConnPool_get_conn_failure','ConnPool_get_conn_success','Server_Connections_delayed');"

# Check runtime server config including max_connections
mysql -u admin -p<admin_pass> -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup_id, hostname, port, status, weight, max_connections, max_replication_lag FROM runtime_mysql_servers ORDER BY hostgroup_id;"

# Check connect timeout settings
mysql -u admin -p<admin_pass> -h 127.0.0.1 -P 6032 \
  -e "SELECT variable_name, variable_value FROM global_variables WHERE variable_name LIKE '%connect_timeout%';"

# Check monitor check results
mysql -u admin -p<admin_pass> -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name LIKE 'MySQL_Monitor_%check%';"

# Check multiplexing health
mysql -u admin -p<admin_pass> -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name IN ('Client_Connections_connected','Client_Connections_hostgroup_locked','Active_Transactions');"

# Check recent monitor ping log for failures
mysql -u admin -p<admin_pass> -h 127.0.0.1 -P 6032 \
  -e "SELECT * FROM monitor.mysql_server_ping_log ORDER BY time_start_us DESC LIMIT 20;"

How to diagnose it

1. Identify the hostgroup from the error message. The error says “hostgroup N.” Focus your investigation on that hostgroup. Error 9001 on one hostgroup does not necessarily mean other hostgroups are affected.

2. Check backend status in stats_mysql_connection_pool. If all backends in the hostgroup are SHUNNED, OFFLINE_SOFT, or OFFLINE_HARD, you have a backend availability problem. If all are ONLINE, the issue is likely pool saturation, multiplexing collapse, or a ProxySQL-level problem.

3. Check stats_mysql_errors for the error code breakdown. This table gives you the count and last_seen timestamp for error 9001 per hostgroup and user. Confirm the error is being generated for the hostgroup you expect, and check whether it correlates with a specific user.

4. Examine max_connect_timeouts in stats_mysql_global. This is a cumulative counter of connection attempts that timed out. Take two readings a few seconds apart to compute the rate. A rising rate confirms active connection failures.

5. Check ConnERR per backend. Rising ConnERR means ProxySQL is actively trying and failing to connect to backends. If ConnERR is rising on all backends in the hostgroup, suspect a network partition, credential issue, or backend MySQL outage. If ConnERR is flat but error 9001 persists, suspect pool saturation.

6. Compare ConnUsed against max_connections and check Server_Connections_delayed. Join stats_mysql_connection_pool with runtime_mysql_servers to check whether backends are at their configured connection limit. If ConnUsed equals max_connections, ConnFree is 0, and Server_Connections_delayed is above 0, the pool is saturated and queries are waiting.

7. Verify query rule configuration if backends are healthy. If all backends are ONLINE, ConnERR is flat, and the pool is not saturated, check whether query rules are misrouting traffic or whether a ProxySQL version bug is involved. Compare mysql_servers with runtime_mysql_servers to detect configuration drift.

8. Check the “after Mms” value. If M is consistently near mysql-connect_timeout_server_max, ProxySQL is exhausting the full retry window, pointing to genuine backend unavailability or saturation. If M is 0, investigate client driver interactions or version-specific bugs.

Metrics and signals to monitor

Fixes

All backends down or unreachable

Connect directly to the backend MySQL instances, bypassing ProxySQL, to verify they are running and accepting connections. Check network paths between the ProxySQL host and backends. If credentials have changed, update both mysql_users (for frontend auth) and mysql-monitor_username / mysql-monitor_password (for monitor checks), then load to runtime:

UPDATE mysql_users SET password='<new_password>' WHERE username='<user>';
LOAD MYSQL USERS TO RUNTIME;
SAVE MYSQL USERS TO DISK;

SET mysql-monitor_password='<new_monitor_password>';
LOAD MYSQL VARIABLES TO RUNTIME;
SAVE MYSQL VARIABLES TO DISK;

If monitor credentials are stale, ProxySQL cannot verify backend health and will SHUN backends it considers unreachable, even if they are actually up.

All backends SHUNNED

If backends are SHUNNED due to replication lag, check whether lag exceeds the max_replication_lag threshold configured in mysql_servers. The SHUN self-corrects when replication catches up. If the lag is persistent, investigate the replication topology.

If backends are SHUNNED due to connection errors (controlled by mysql-shun_on_failures, default 5 consecutive failures, and mysql-shun_recovery_time_sec, default 10s), check whether the failures are real or caused by aggressive monitor intervals on a network with jitter. Increasing shun_on_failures or mysql-monitor_ping_interval can reduce false shunning.

Backend pool saturation

If ConnUsed is at max_connections and ConnFree is 0, you need more backend connection capacity. Options:

• Increase max_connections in mysql_servers if the backend MySQL can handle more connections. ProxySQL’s per-backend max_connections is separate from MySQL’s global max_connections. The sum of all ProxySQL instances’ per-backend limits plus other consumers should not exceed the backend MySQL’s capacity.

UPDATE mysql_servers SET max_connections=<new_value> WHERE hostgroup_id=<hg> AND hostname='<host>';
LOAD MYSQL SERVERS TO RUNTIME;
SAVE MYSQL SERVERS TO DISK;

• Address multiplexing collapse if the root cause is pinned sessions. Common culprits: ORM connection initialization setting session variables, prepared statements, GET_LOCK(), temporary tables, or user-defined variables.
• Scale horizontally by adding more backend replicas to the hostgroup or more ProxySQL instances.

Single-backend hostgroup with a SHUNNED backend

If a hostgroup has only one backend and it is SHUNNED, ProxySQL continues retrying until mysql-connect_timeout_server_max is reached, then returns error 9001. The retry count in this code path is not controlled by mysql-connect_retries_on_failure in the same way as multi-backend hostgroups. Adding a second backend provides failover capacity and prevents the full timeout delay.

ProxySQL version bugs

If all backends are ONLINE, ConnERR is flat, the pool is not saturated, and error 9001 persists, check your ProxySQL version:

• Pre-2.0.7: A bug affected mysql-connect_timeout_server_max values above 10 minutes. Upgrade if running an old version.
• Pre-3.0.9: A query rule with apply=1 could bypass fast routing, sending traffic to the wrong hostgroup or failing with error 9001. Fixed in ProxySQL 3.0.9 and 3.1.9.
• Pre-3.0.9: Connection pool starvation under contention could cause error 9001 even when backends had capacity. The backend-pool session scheduler introduced in 3.0.9 addresses this by prioritizing the session closest to hitting connect_timeout_server_max.

Error 9001 with “after 0ms”

If the error reports “after 0ms,” the timeout fired immediately. This can indicate a client driver interaction issue, particularly with mysql-default_query_timeout set very high. Check the ProxySQL GitHub issues for your specific client driver if this pattern appears.

Prevention

• Monitor max_connect_timeouts and stats_mysql_errors per hostgroup. These are the most direct measures of error 9001 conditions. Alert on any sustained rate above zero.
• Track multiplexing ratio. A declining ratio means backend connection demand is growing faster than client count. Address it before saturation hits.
• Keep at least two backends per hostgroup. Single-backend hostgroups cannot fail over and force the full timeout window on failure.
• Verify config layer consistency. Compare mysql_servers with runtime_mysql_servers and ensure changes are both loaded to runtime and saved to disk.
• Keep ProxySQL updated. The backend-pool session scheduler and query rule fast routing fixes in 3.0.9 address common error 9001 causes under contention.
• Validate max_connections holistically. The sum of ProxySQL per-backend limits across all instances should leave headroom for direct admin connections, replication threads, and monitoring tools.

How Netdata helps

• Per-second collection on max_connect_timeouts, ConnPool_get_conn_failure, ConnERR, and Server_Connections_delayed provides the resolution needed to catch transient spikes.
• Backend status tracking from stats_mysql_connection_pool surfaces SHUNNED and OFFLINE transitions alongside error counters in a single timeline, correlating shun events with error 9001 spikes.
• Anomaly detection on connection pool metrics flags unusual ConnPool_get_conn_failure patterns before they escalate to client-facing errors.
• Multiplexing ratio computed from Client_Connections_hostgroup_locked and Client_Connections_connected, making multiplexing collapse visible without manual queries.
• Monitor check result tracking (MySQL_Monitor_connect_check_OK/ERR, MySQL_Monitor_ping_check_OK/ERR) correlates with backend status changes, helping distinguish real backend failures from monitor-induced flapping.
• Cross-layer correlation connects ProxySQL CPU saturation, file descriptor usage, or memory pressure with error 9001 events in the same dashboard.

Related guides

• ProxySQL backend flapping between ONLINE and SHUNNED: monitor-induced oscillation
• ProxySQL OFFLINE_SOFT vs OFFLINE_HARD vs SHUNNED: what each backend status means
• ProxySQL backend SHUNNED: why a healthy backend gets pulled out of rotation
• How ProxySQL actually works in production: a mental model for operators
• ProxySQL monitor check failures: connect, ping, read-only, and replication-lag probes failing
• ProxySQL monitoring checklist: the signals every production proxy needs
• ProxySQL monitoring maturity model: from survival to expert
• ProxySQL zero ONLINE backends in a hostgroup: total outage for that traffic class