$ guides / proxysql / proxysql-client-connections-max-reached ▌

Operations Guides

ProxySQL client connections at mysql-max_connections: frontend saturation and rejected clients

When Client_Connections_connected reaches mysql-max_connections (default 2048), ProxySQL stops accepting new client sessions. Already-connected clients continue working, but new sessions fail with “Too many connections” until existing ones close.

This is a cliff-edge failure. There is no queueing at the frontend and no backpressure signal. Because mysql-max_connections is a global limit shared across all users and hostgroups, a single application with a connection leak can consume every available slot and lock out every other user of the proxy.

This failure occurs on the frontend (client-to-proxy) layer. It is distinct from backend pool exhaustion, where ProxySQL accepts client connections but cannot obtain connections to MySQL backends. Both produce client-facing errors, but the counters, root causes, and fixes differ.

What this means

mysql-max_connections is ProxySQL’s hard cap on concurrent client sessions. It is dynamic at runtime (changeable without restart). Every client connection, regardless of user or hostgroup, consumes one slot against this limit. Internal connections (admin interface, monitor module) are tracked separately.

When the limit is reached, ProxySQL rejects new connections before authentication completes. The client receives a “Too many connections” error. On the ProxySQL side, Access_Denied_Max_Connections increments in stats_mysql_global, and Client_Connections_aborted rises.

Three distinctions matter for diagnosis:

Global limit vs per-user limit. mysql_users has its own max_connections column (default 10000). Hitting the per-user limit increments Access_Denied_Max_User_Connections, not Access_Denied_Max_Connections. The client-facing error message does not distinguish between the two limits. If you only monitor Client_Connections_aborted, you cannot tell which ceiling was hit without checking the specific Access_Denied_* counters.

Frontend limit vs backend pool limit. Backend pool exhaustion means queries enter ProxySQL but fail because no backend connection is available. Frontend saturation means the client cannot even establish a session. If clients are connecting but queries fail with timeout errors, the problem is downstream. See ProxySQL backend connection pool exhausted.

The limit is shared. One user, one hostgroup, or one application can consume all 2048 slots. There is no per-hostgroup reservation. This is why per-user limits in mysql_users are essential for multi-tenant ProxySQL deployments.

flowchart TD
    A["Clients receive 'Too many connections'"] --> B{"Client_Connections_connected\nnear mysql-max_connections?"}
    B -- "Yes" --> C["Global frontend saturation"]
    B -- "No" --> D{"Access_Denied_Max_User_Connections\nrising?"}
    D -- "Yes" --> E["Per-user limit hit"]
    D -- "No" --> F["Check backend pool:\nConnPool_get_conn_failure,\nServer_Connections_delayed"]
    C --> G["Check stats_mysql_users\nfor dominant user"]
    G --> H{"One user holds\ndisproportionate slots?"}
    H -- "Yes" --> I["Connection leak or\nmissing per-user limit"]
    H -- "No" --> J["Capacity gap:\nraise mysql-max_connections"]

Common causes

Cause	What it looks like	First thing to check
Application connection leak	`Client_Connections_connected` grows monotonically and never drops. `Client_Connections_created` keeps climbing while connected stays flat at max.	`stats_mysql_users` for which user holds the most frontend connections
Connection storm (thundering herd)	Sharp spike in `Client_Connections_created` correlated with a deployment, restart, or failover event. Connected hits max within seconds, then recovers as connections drain.	Timeline correlation with deployment or infrastructure events
Insufficient global limit	`connected` sits near max during normal peak load with no anomaly in application behavior. All users affected proportionally.	Historical trend of peak `connected` vs `mysql-max_connections`
Per-user limit hit (same symptom, different counter)	`connected` well below global max. `Access_Denied_Max_User_Connections` rising while `Access_Denied_Max_Connections` stays flat. One user at its `mysql_users.max_connections` ceiling.	`stats_mysql_users` per-user utilization

Quick checks

All commands connect to the ProxySQL admin interface on port 6032. These are read-only SELECTs and are safe to run during an active incident.

# Check current connected count and rejection counters
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global \
      WHERE Variable_Name IN ('Client_Connections_connected','Client_Connections_created', \
      'Client_Connections_aborted','Client_Connections_non_idle');"

# Check the global limit
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT variable_name, variable_value FROM global_variables \
      WHERE variable_name = 'mysql-max_connections';"

# Distinguish global-limit rejections from per-user-limit rejections
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global \
      WHERE Variable_Name LIKE 'Access_Denied%';"

# Per-user frontend connection distribution
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT username, frontend_connections, frontend_max_connections \
      FROM stats_mysql_users ORDER BY frontend_connections DESC LIMIT 20;"

# Active client sessions, sorted by idle time (leaked connections have high time_ms)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT user, cli_host, hostgroup, db, command, time_ms \
      FROM stats_mysql_processlist ORDER BY time_ms DESC LIMIT 30;"

# Check for ProxySQL-generated error packets (not specific to frontend rejections)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global \
      WHERE Variable_Name = 'generated_error_packets';"

How to diagnose it

Confirm the global limit is the source. Compare Client_Connections_connected against the mysql-max_connections value from global_variables. If connected is at or near max, you have global frontend saturation. If connected is well below max but clients still see “Too many connections,” check Access_Denied_Max_User_Connections for a per-user limit hit.
Classify the growth pattern. Take two readings of Client_Connections_connected 60 seconds apart. If the value is monotonically increasing, suspect a connection leak. If it spiked and is now stable at max, suspect a connection storm that has not drained. If it has been sitting near max for hours during normal load, suspect a capacity gap.
Identify which user is consuming slots. Query stats_mysql_users ordered by frontend_connections descending. One user holding a disproportionate share of the global limit is the signature of a single application leak or a missing per-user cap. Note that the frontend_connections counter in stats_mysql_users may be unreliable when the global limit is actually reached; cross-reference with stats_mysql_processlist for a ground-truth count per user.
Check for idle sessions. Query stats_mysql_processlist for sessions with command = Sleep and high time_ms. These are connected clients doing nothing. A large number of long-sleeping sessions from one user indicates the application is opening connections and never closing them.
Understand active vs idle ratio. Check Client_Connections_non_idle. Without the --idle-threads startup flag, this value always equals Client_Connections_connected and provides no additional signal. With --idle-threads enabled, non_idle tracks only connections actively handled by worker threads, letting you see how many of the saturated slots are actually doing work versus sitting idle.
Correlate with deployment events. If Client_Connections_created shows a sharp spike, match the timestamp against application deployments, ProxySQL restarts, or load balancer failovers. A connection storm typically resolves itself as the pool stabilizes, but it can cause a brief total outage if the spike pushes connected past the limit.
Check OS file descriptors. Each client and backend connection consumes one file descriptor. If you plan to raise mysql-max_connections, verify the OS ulimit can accommodate the increase. Check with:

# Current FD usage for the ProxySQL process
ls /proc/$(pidof proxysql)/fd | wc -l

# FD limit
cat /proc/$(pidof proxysql)/limits | grep "Max open files"

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`Client_Connections_connected` / `mysql-max_connections`	Direct utilization of the global frontend limit	Ratio above 80% sustained (TICKET), above 95% (PAGE)
`Access_Denied_Max_Connections`	Confirms rejections are due to the global limit, not a per-user cap	Any sustained increase from zero
`Access_Denied_Max_User_Connections`	Distinguishes per-user limit hit from global limit	Rising while `Access_Denied_Max_Connections` is flat
`Client_Connections_aborted`	Connections rejected before or during authentication	Rate above zero sustained for more than 2 minutes
`Client_Connections_created` rate	Connection churn velocity, indicates storms or retry loops	Spike during a non-deployment window, or continuous high rate
Per-user `frontend_connections` in `stats_mysql_users`	Shows which user is consuming global slots	One user holding a disproportionate share

Fixes

Application connection leak

Root cause: the application opens connections to ProxySQL and never closes them, or its connection pool grows without bound. Each leaked connection permanently occupies a global slot until the application process exits.

Immediate mitigation: identify long-idle sessions in stats_mysql_processlist (look for command = Sleep with high time_ms from a single user). You can terminate individual client sessions through the ProxySQL admin interface to free slots. Restarting the leaking application process will also release its connections, though this causes a brief error spike for that application’s users.

Permanent fix: fix the application-side connection pool configuration. Set maximum pool size, maximum connection lifetime, and idle connection eviction timeout. Ensure the pool closes connections that exceed the lifetime rather than holding them open indefinitely.

Tradeoff: requires an application code or configuration change and a redeployment. The fix is not instant, but without it the problem will recur.

Connection storm

Root cause: a mass reconnect triggered by a ProxySQL restart, load balancer failover, or simultaneous application pod restart. All clients attempt to reconnect at once, and the burst exceeds the global limit before the pool stabilizes.

Immediate mitigation: connection storms are typically self-resolving as clients back off and retry. The main risk is backend max_connections exhaustion during the same window. Monitor backend pool metrics during the storm.

Permanent fix: stagger application restarts (rolling deployments). Configure mysql-free_connections_pct (default 10) to maintain a warm pool of free backend connections so the backend side does not also saturate. Consider whether raising mysql-max_connections gives enough headroom to absorb the storm.

Tradeoff: staggering restarts slows deployment velocity. The headroom approach consumes more memory and file descriptors at idle.

Insufficient global limit

Root cause: legitimate peak workload exceeds the configured mysql-max_connections. This is a capacity problem, not a bug.

Fix: raise the limit at runtime:

# Raise the global frontend connection limit (runtime + persistent)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SET mysql-max_connections = 4096; LOAD MYSQL VARIABLES TO RUNTIME; SAVE MYSQL VARIABLES TO DISK;"

Tradeoff: each additional client connection consumes memory (connection buffers, session state) and one file descriptor. Before raising the limit, verify the ProxySQL host has sufficient memory and that the OS file descriptor limit can accommodate the increase. A rough heuristic from the playbook: set the OS FD limit to at least 2x (max client connections + max backend connections + 1000 overhead). Also ensure the backend MySQL instances can handle the increased connection pressure, since more frontend connections can translate to more backend connections if multiplexing is degraded.

Missing per-user limits

Root cause: mysql_users entries have no max_connections set (or it is set to the default 10000), so any single user can consume the entire global pool. This turns a single misbehaving application into a full outage for all tenants.

Fix: set max_connections per user in mysql_users based on each application’s expected peak usage:

# Set a per-user connection limit
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "UPDATE mysql_users SET max_connections=200 WHERE username='app_reader'; \
      LOAD MYSQL USERS TO RUNTIME; SAVE MYSQL USERS TO DISK;"

Tradeoff: if set too low, legitimate traffic spikes for that user will be rejected. Tune based on observed peak usage with headroom. The sum of all per-user limits should be less than mysql-max_connections to leave headroom for other users and administrative connections.

Prevention

Monitor the ratio. Alert on Client_Connections_connected / mysql-max_connections at 80% (TICKET) and 95% (PAGE). The playbook recommends peak headroom below 70% of the limit to absorb traffic spikes.
Set per-user limits. Every user in mysql_users should have a max_connections value proportional to its expected workload. This prevents any single application from monopolizing the global pool.
Track the peak trend. Plot peak Client_Connections_connected over weeks. A slowly rising peak without a corresponding traffic increase suggests a slow connection leak or multiplexing degradation causing connection accumulation.
Verify OS limits. Ensure the file descriptor limit can handle 2x the peak expected total connections (client plus backend plus overhead).
Monitor Client_Connections_aborted rate. Sustained above zero means clients are being rejected right now. This is the most direct signal that the frontend is rejecting connections.
Distinguish counters in alerting. Alert on Access_Denied_Max_Connections and Access_Denied_Max_User_Connections separately. They indicate different problems and require different fixes.

How Netdata helps

Netdata collects these ProxySQL metrics at per-second granularity, providing immediate visibility during fast-moving connection storms.

Per-second Client_Connections_connected collection with automatic ratio calculation against mysql-max_connections, so you see saturation approaching before clients start receiving errors.
Access_Denied_Max_Connections and Access_Denied_Max_User_Connections tracked separately, letting you distinguish global-limit rejections from per-user-limit rejections without manual admin queries during an incident.
Client_Connections_aborted rate as a leading indicator of active rejection, with per-second granularity to catch brief spikes that minute-level polling misses.
Anomaly detection on connection count trends, flagging slow monotonic growth (the signature of a connection leak) well before it reaches the hard limit.
Correlation of Client_Connections_created spikes with deployment events, making it faster to confirm a connection storm versus a leak as the root cause.
Cross-correlation with backend pool metrics (ConnPool_get_conn_failure, ConnERR, Server_Connections_delayed) to immediately distinguish frontend saturation from backend pool exhaustion, since both produce client-facing errors but require different fixes.