$ guides / proxysql / proxysql-connerr-backend-connection-errors ▌

Operations Guides

ProxySQL ConnERR climbing: backend connection errors and how to localise them

ConnERR is a per-backend cumulative counter in stats_mysql_connection_pool that increments every time ProxySQL fails to open a connection to a backend MySQL server. Because it is cumulative, a single nonzero value is historical: it means at least one attempt failed since the last stats reset or ProxySQL restart. Take two readings 10 to 30 seconds apart and compute the delta. A positive sustained delta means ProxySQL is actively failing to establish backend connections right now.

The first diagnostic question is scope: is ConnERR climbing on one backend, some backends, or every backend in a hostgroup? One backend with rising errors while peers stay healthy points to a backend-specific failure. All backends failing simultaneously points to network, DNS, or proxy-side problems.

What this means

Each row in stats_mysql_connection_pool represents one backend in one hostgroup. ConnERR tracks failures on that specific backend. The counter coexists with ConnOK, which counts successful connection attempts. A backend can have both high ConnOK and high ConnERR: the backend is intermittently reachable. This pattern produces sporadic query failures, occasional latency spikes, and may trigger shunning if errors accumulate fast enough.

ProxySQL’s shunning threshold is controlled by mysql-shun_on_failures (default 5). If more than this many errors occur within one second, the backend is shunned for mysql-shun_recovery_time_sec seconds (default 10). A backend that fails repeatedly may cycle between ONLINE and SHUNNED, and each recovery cycle adds ConnERR as ProxySQL retries connections.

flowchart TD
    A["ConnERR rising
delta stats_mysql_connection_pool"] --> B{"All backends
in hostgroup affected?"}
    B -- Yes --> C["Network, DNS, or
proxy-side issue"]
    B -- One backend --> D{"Monitor checks
also failing?"}
    B -- Some backends --> E["Compare per-backend
error codes in
stats_mysql_errors"]
    D -- Yes --> F["Backend down or
overloaded"]
    D -- No --> G["Data-plane only:
firewall, port,
or credentials"]
    E --> H["Match errno to
root cause"]

Common causes

Cause	What it looks like	First thing to check
Backend MySQL down or unreachable	`ConnERR` climbing on one backend, `ConnOK` flat, monitor connect checks failing	Connect directly to that backend from the proxy host
Backend `max_connections` exhausted (error 1040)	`ConnERR` climbing alongside active `ConnOK`, error 1040 in `stats_mysql_errors`	Backend `SHOW STATUS LIKE 'Threads_connected'` vs its `max_connections`
Credential mismatch	`ConnERR` on all or specific backends, monitor check ERR counters rising, `Access_Denied_Wrong_Password` increasing	Verify `mysql-monitor_username` / `mysql-monitor_password` match backend grants
Network or DNS failure	`ConnERR` rising on all backends simultaneously, monitor connect and ping failing across the board, `Server_Connections_aborted` rising	DNS resolution and network path from proxy host to backends
Stale connections closed by backend	Intermittent `ConnERR`, errors clustered after idle periods, `ConnOK` still happening between failures	Backend `wait_timeout` vs ProxySQL `mysql-connection_max_age_ms`

Quick checks

Run these read-only queries against the admin interface (default port 6032). The examples use the default admin credentials (admin/admin); replace with your actual credentials.

# Delta ConnERR across two readings to confirm active failure rate
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup, srv_host, srv_port, status, ConnUsed, ConnFree, ConnOK, ConnERR FROM stats_mysql_connection_pool ORDER BY hostgroup, srv_host;"
# Wait 10-30 seconds, run again, compare ConnERR values.

# Check backend status and configured limits
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup_id, hostname, port, status, max_connections, max_replication_lag FROM runtime_mysql_servers ORDER BY hostgroup_id;"

# Check monitor check results (connect and ping)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name IN ('MySQL_Monitor_connect_check_OK','MySQL_Monitor_connect_check_ERR','MySQL_Monitor_ping_check_OK','MySQL_Monitor_ping_check_ERR','MySQL_Monitor_Workers');"

# Check per-error-code breakdown (ProxySQL 2.0.6+)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT * FROM stats_mysql_errors ORDER BY last_seen DESC LIMIT 20;"

# Check server-side connection aborts and delays
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name IN ('Server_Connections_aborted','Server_Connections_delayed','Access_Denied_Wrong_Password','Access_Denied_Max_Connections');"

# Verify the backend is reachable directly from the proxy host
mysql -u <monitor_user> -p -h <backend_host> -P <backend_port> -e "SELECT 1;"

# Check DNS resolution if backends are configured by hostname
getent hosts <backend_hostname>

# Check file descriptor usage on the proxy host (FD exhaustion causes connection failures)
ls /proc/$(pidof proxysql)/fd | wc -l
cat /proc/$(pidof proxysql)/limits | grep "open files"

How to diagnose it

Confirm the rate is actively climbing. Take two readings of stats_mysql_connection_pool 10 to 30 seconds apart. A static nonzero value is historical noise. Only a positive delta indicates an active problem.
Localise by scope. Is ConnERR climbing on one backend, several backends, or every backend in the hostgroup? One backend with errors while peers are healthy isolates the culprit to that backend or its network path. All backends failing simultaneously points to a shared dependency: DNS, the network, file descriptor limits, or proxy-side configuration.
Cross-reference monitor check results. The monitor module runs its own connect and ping checks on separate threads using mysql-monitor_username and mysql-monitor_password. If monitor connect checks are also failing (MySQL_Monitor_connect_check_ERR rising), the backend itself is unreachable. If monitor checks pass but ConnERR is still climbing, the problem is on the data plane: application-facing connections use different credentials, a different port, or hit a firewall rule that does not apply to the monitor.
Check stats_mysql_errors for specific error codes. This table provides per-hostgroup, per-user, per-schema error breakdown with MySQL error numbers. Error 1040 means the backend rejected the connection due to its own max_connections. Error 1045 means access denied (credential mismatch). ProxySQL-generated errors in the 9000+ range indicate internal proxy failures. Available in ProxySQL 2.0.6 and later.
Check whether ConnOK is also rising on the same backend. If both ConnOK and ConnERR are climbing, the backend is intermittently reachable. Common causes: DNS flakiness, network microdrops, backend thread pool exhaustion under load, or wait_timeout closing idle pooled connections that ProxySQL has not yet detected as dead.
Verify the backend directly. From the ProxySQL host, connect to the backend MySQL using the same hostname or IP and port the proxy uses. If the direct connection fails, the problem is below the proxy layer.
Check for shun cycling. A backend that repeatedly transitions between ONLINE and SHUNNED accumulates ConnERR during each shun recovery. Check runtime_mysql_servers status and the monitor ping log for alternating success and failure patterns.
Check for CLOSE_WAIT socket accumulation. Many backend connections stuck in CLOSE_WAIT indicate a file descriptor leak that will eventually cause new connection attempts to fail.

ss -tnp | grep ':3306' | grep CLOSE-WAIT | wc -l

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`ConnERR` delta per backend	Direct measure of active connection failures	Positive sustained delta on any backend
`ConnOK` alongside `ConnERR`	Distinguishes total outage from intermittent reachability	Both rising means intermittent, harder to diagnose
Backend status in `runtime_mysql_servers`	Shows whether errors have triggered shunning	ONLINE to SHUNNED transitions correlate with ConnERR spikes
Monitor connect/ping check ERR	Tells you whether the monitor sees the same failures	Monitor ERR rising alongside ConnERR confirms backend unreachability
`stats_mysql_errors` errno	Maps the failure to a specific MySQL or ProxySQL error code	New error codes appearing that were absent from baseline
`Server_Connections_aborted`	Backend-side connection aborts, broader than per-backend ConnERR	Sustained rate above zero
`Server_Connections_delayed`	Queries waiting for a free backend connection	Sustained above zero means pool pressure
`ConnPool_get_conn_failure`	Pool could not provide a connection (often because backend was shunned)	Spikes correlate with shun events from ConnERR
File descriptor count on proxy host	FD exhaustion causes all new connections to fail	Approaching `ulimit -n`

Fixes

Backend MySQL is down or unreachable

If direct connection to the backend fails, the fix is on the MySQL side. Check the MySQL error log, verify the process is running, and confirm the listener is bound to the expected address and port. If ProxySQL connects via Unix socket, verify the socket path exists and has correct permissions.

If the backend was removed from orchestration (Kubernetes, Consul) but remains in mysql_servers, remove the stale entry. This redirects traffic to remaining backends in the hostgroup:

DELETE FROM mysql_servers WHERE hostname='<stale_host>' AND port=<port>;
LOAD MYSQL SERVERS TO RUNTIME;
SAVE MYSQL SERVERS TO DISK;

Backend max_connections is exhausted

The backend MySQL’s max_connections is shared among ProxySQL, other ProxySQL instances, direct admin connections, replication threads, and monitoring tools. If ProxySQL’s per-backend max_connections plus all other consumers exceeds the backend’s limit, new connections get rejected with error 1040.

Check the backend directly:

SHOW VARIABLES LIKE 'max_connections';
SHOW STATUS LIKE 'Threads_connected';
SHOW PROCESSLIST;

If the backend can handle more connections, increase ProxySQL’s per-backend limit:

UPDATE mysql_servers SET max_connections=<new_value> WHERE hostname='<host>' AND port=<port>;
LOAD MYSQL SERVERS TO RUNTIME;
SAVE MYSQL SERVERS TO DISK;

max_connections in mysql_servers is ProxySQL’s self-imposed limit, not the backend MySQL’s limit. The sum of all ProxySQL instances’ max_connections for a given backend should stay below 80% of the backend MySQL’s actual max_connections to leave headroom for other consumers.

Credential mismatch

Credential problems affect two separate paths. The monitor uses mysql-monitor_username and mysql-monitor_password. Data-plane connections use credentials from mysql_users. Both must match the backend MySQL grants.

Fix monitor credentials:

SET mysql-monitor_username='<user>';
SET mysql-monitor_password='<password>';
LOAD MYSQL VARIABLES TO RUNTIME;
SAVE MYSQL VARIABLES TO DISK;

Fix data-plane credentials:

UPDATE mysql_users SET password='<password>' WHERE username='<user>';
LOAD MYSQL USERS TO RUNTIME;
SAVE MYSQL USERS TO DISK;

Network or DNS failure

If ConnERR is climbing on all backends simultaneously, check the shared dependencies. Run DNS resolution from the proxy host. Check the network path with mtr or traceroute. Verify firewall and security group rules have not changed.

Connection lifecycle issues (wait_timeout)

Backend MySQL closes idle connections after wait_timeout (default 28800 seconds). If ProxySQL holds pooled connections that the backend silently closes, the next query on that dead connection triggers a reconnect attempt counted in ConnERR. Under bursty load with many simultaneously expired connections, the error rate can spike and trigger shunning.

Set mysql-connection_max_age_ms to a value lower than the backend’s wait_timeout so ProxySQL proactively recycles connections before the backend closes them:

SET mysql-connection_max_age_ms=<value_below_wait_timeout_ms>;
LOAD MYSQL VARIABLES TO RUNTIME;
SAVE MYSQL VARIABLES TO DISK;

mysql-connect_retries_on_failure (default 10) controls retries on connection failure. This setting is not effective for hard failures like connection refused. For a hostgroup with a single backend that goes down, ProxySQL returns error 9001 (“Max connect timeout reached while reaching hostgroup”) after mysql-connect_timeout_server_max (default 10000ms) elapses.

Prevention

Alert on the ConnERR rate, not the absolute value. The cumulative counter is meaningless without a time window. Alert on positive sustained deltas.
Track ConnOK alongside ConnERR per backend. The combination reveals intermittent reachability, which is the pattern most likely to cause prolonged undiagnosed incidents.
Keep ProxySQL’s per-backend max_connections within the backend’s capacity. Account for all connection consumers, not just this ProxySQL instance.
Verify monitor credentials independently. The monitor runs on separate threads with separate credentials. If monitor credentials expire, ProxySQL shuns backends it cannot verify, even if they are healthy.
Align connection lifecycle settings. Set mysql-connection_max_age_ms below the backend’s wait_timeout to avoid stale-connection errors.
Watch file descriptor usage on the proxy host. Each client and backend connection consumes one file descriptor. FD exhaustion causes all connection attempts to fail simultaneously, mimicking a network outage.
Remove stale backend entries promptly. If orchestration removes a backend, clean up mysql_servers. Stale entries produce ConnERR and shunning against a backend that no longer exists.

How Netdata helps

Per-second ConnERR rate per backend, with the cumulative counter already delta’d, so you see the failure rate without manual two-poll arithmetic.
ConnOK displayed alongside ConnERR for the same backend row, making intermittent reachability immediately visible on a single chart.
Backend status transitions (ONLINE to SHUNNED) correlated with ConnERR spikes on a shared timeline, showing whether shunning follows the errors or coincides with a different root cause.
Monitor connect and ping check failure rates displayed next to ConnERR, so you can distinguish backend-side unreachability from data-plane-only failures.
Server_Connections_aborted and ConnPool_get_conn_failure rates that tell you whether ConnERR is translating into client-visible query failures or remaining an internal retry that self-heals.