ProxySQL connection storm after restart: an empty pool meeting a mass reconnect

ProxySQL just restarted. Within seconds, client connections pour in. The backend pool is empty, so every query demands a new connect() to MySQL. Client_Connections_created and backend ConnOK spike in lockstep. The backend’s CPU jumps as it authenticates connections and spins up threads instead of executing queries. If TLS is enabled, each handshake multiplies the cost.

The storm is usually self-resolving in 30 to 120 seconds as the pool fills and steady-state multiplexing resumes. During that window, latency spikes, queries may queue, and if backend max_connections is exhausted, new connections fail outright with MySQL error 1040.

This is not a bug. It is the expected consequence of ProxySQL creating backend connections on demand after losing its pool state. The operator’s job is to distinguish it from a real outage, ride it out if it is harmless, and prevent it from escalating on the next restart.

What this means

When ProxySQL restarts, its backend connection pool is empty. Stats tables reset: stats_mysql_connection_pool, stats_mysql_query_digest, stats_mysql_commands_counters all start at zero. The query cache is cold. The monitor module needs at least one full check interval to re-establish backend health.

Every client that reconnects sends a query. ProxySQL has no free backend connection to borrow, so it opens a new one. Under normal conditions, the pool fills gradually and multiplexing takes over: N clients share M backends where M is much smaller than N. But when all clients reconnect at once, that gradual fill becomes a burst of simultaneous connect() calls hitting the backend MySQL server.

The defining signature is correlated spikes across ProxySQL and the backend:

• Client_Connections_created jumps as clients reconnect to the proxy.
• Backend ConnOK jumps simultaneously as the proxy opens connections to backends.
• Backend MySQL Threads_created climbs as MySQL creates threads for each new connection.

If backend max_connections is hit, ConnERR starts climbing and ProxySQL may shun the backend after enough consecutive failures.

flowchart TD
    A["ProxySQL restart / LB failover / mass redeploy"] --> B["Backend pool empty"]
    B --> C["All clients reconnect simultaneously"]
    C --> D["Each query opens new backend connection"]
    D --> E["Client_Connections_created + ConnOK spike"]
    D --> F["Backend MySQL CPU spikes on auth + thread creation"]
    E --> G{"Backend max_connections hit?"}
    G -->|No| H["Pool fills in 30-120s, storm self-resolves"]
    G -->|Yes| I["ConnERR + Access_Denied_Max_Connections climb"]
    I --> J["Backend may be SHUNNED, queries fail"]

Common causes

Quick checks

All queries below are read-only against the admin interface (default port 6032). Replace admin/admin with your configured credentials.

# Check if this is a cold start (uptime in seconds)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name = 'ProxySQL_Uptime';"

# Confirm the correlated spike: client creates and server connection counts
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name IN ('Client_Connections_created','Client_Connections_connected','Server_Connections_created','Server_Connections_connected');"

# Check pool state per backend
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup, srv_host, srv_port, status, ConnUsed, ConnFree, ConnOK, ConnERR FROM stats_mysql_connection_pool;"

# Check for connection acquisition failures (pool starvation during the storm)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name IN ('ConnPool_get_conn_failure','ConnPool_get_conn_success','Server_Connections_delayed');"

# Check for max_connections exhaustion (the escalation condition)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name IN ('Access_Denied_Max_Connections','max_connect_timeouts');"

# Verify all backends are ONLINE (storm should not cause shunning unless errors accumulate)
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup_id, hostname, port, status FROM runtime_mysql_servers;"

How to diagnose it

1. Confirm the trigger. Check ProxySQL_Uptime. If it is under 120 seconds, you are in the cold-start window. Correlate with deployment logs, LB failover events, or ProxySQL restart timestamps. A storm without a corresponding infra event is a different problem.

2. Verify the signature. Take two readings of Client_Connections_created and backend ConnOK a few seconds apart. If both deltas are large and climbing in lockstep, the pool is filling on demand. This is the storm, not a backend outage.

3. Check backend health. All backends should remain ONLINE during a pure connection storm. If backends are going SHUNNED, check ConnERR rates: the backend may be rejecting connections due to its own max_connections limit or an authentication backlog. See ProxySQL backend SHUNNED: why a healthy backend gets pulled out of rotation for shunning causes beyond the storm.

4. Assess severity. The storm is self-resolving if backend capacity holds. Check whether any escalation signals are present:

   • ConnERR climbing: backend is rejecting connections.
   • Access_Denied_Max_Connections incrementing: backend max_connections is exhausted.
   • max_connect_timeouts incrementing: backends cannot accept connections fast enough.
   • Server_Connections_delayed greater than zero: queries are queuing for a free backend connection.

5. Monitor recovery. Watch ConnFree rise and ConnUsed stabilize. Once the pool is warm, new client queries borrow existing connections instead of creating new ones. The spike should flatten within 30 to 120 seconds. If it does not, the problem is not a simple cold-start storm.

Metrics and signals to monitor

Fixes

During the storm

If the storm is ongoing and backend capacity is holding (no ConnERR, no Access_Denied_Max_Connections, all backends ONLINE), the correct action is to wait. The pool will fill within 30 to 120 seconds. Restarting ProxySQL again or changing configuration mid-storm adds churn without solving the empty-pool problem.

If backend max_connections is being hit:

• Check whether the backend MySQL has headroom. On the backend, run SHOW VARIABLES LIKE 'max_connections' and SHOW STATUS LIKE 'Threads_connected' to see the gap.
• Consider temporarily raising the backend’s max_connections if the server has CPU and memory headroom. This is safe if the spike is transient. Each MySQL thread consumes memory (typically 256 KB to several MB depending on sort buffers and thread stack), so verify the server can absorb the additional threads.
• If multiple ProxySQL instances share the same backends, the aggregate connection count matters. Each instance’s per-backend max_connections in mysql_servers should be sized so the sum across all instances stays within the backend’s limit minus a reserve for admin, replication, and monitoring connections.

After the storm

Once the pool is warm and steady state resumes, review the trigger:

• Planned restart: add pre-warming and pool configuration (see Prevention below).
• LB failover: review health check configuration. A TCP check on port 6033 passes even if ProxySQL cannot reach any backend. Use a MySQL protocol check that actually routes a query through the proxy.
• App redeploy: stagger pod restarts so clients reconnect gradually instead of simultaneously.

Prevention

Stagger restarts and deployments. The root cause is simultaneity: all clients reconnecting at once into an empty pool. Rolling deploys, jittered connection retry backoff on the application side, and staged ProxySQL restarts all reduce the burst amplitude. This is the most reliable prevention and requires no ProxySQL configuration changes.

Set mysql-free_connections_pct to maintain a warm pool. Default is 10. This tells ProxySQL to keep idle connections open per backend as a percentage of mysql_servers.max_connections. The formula is mysql-free_connections_pct * mysql_servers.max_connections / 100. With max_connections=200 and free_connections_pct=10, ProxySQL keeps approximately 20 idle connections warm in steady state.

SET mysql-free_connections_pct = 10;
LOAD MYSQL VARIABLES TO RUNTIME;
SAVE MYSQL VARIABLES TO DISK;

This keeps connections warm during normal operation but does not pre-create them at startup. It maintains steady-state pool warmth; it does not eliminate the initial cold-start burst.

Enable mysql-connection_warming (with caveats). Default is false. When enabled, ProxySQL attempts to open connections for all servers in all hostgroups until the expected warm connection count is reached, using the same formula as free_connections_pct.

SET mysql-connection_warming = true;
LOAD MYSQL VARIABLES TO RUNTIME;
SAVE MYSQL VARIABLES TO DISK;

mysql-connection_warming has known reliability issues. Operators have reported that enabling it produced far fewer connections than the formula predicts. Test it in your environment before relying on it for production restarts.

Set mysql-throttle_connections_per_sec_to_hostgroup. Default is 1000000, effectively unlimited. Lowering this rate-limits new backend connection creation per hostgroup per second, smoothing the storm into a controlled fill. With a value of 100, each hostgroup is limited to 100 new backend connections per second.

SET mysql-throttle_connections_per_sec_to_hostgroup = 100;
LOAD MYSQL VARIABLES TO RUNTIME;
SAVE MYSQL VARIABLES TO DISK;

This is the most direct storm-mitigation knob. It trades a slightly longer warm-up period for a lower peak load on backend MySQL. Tune the value based on how many connections per second your backends can authenticate without CPU saturation.

Gate cold-start-sensitive alerts. Use ProxySQL_Uptime > 600 as a condition on connection-related alerts to avoid paging on the expected storm. Alerting should focus on whether the storm escalates (backend max_connections exhaustion, sustained ConnERR, ConnPool_get_conn_failure) rather than on the storm itself.

How Netdata helps

• Per-second resolution on Client_Connections_created and backend ConnOK: the correlated spike signature is visible at 1-second granularity, making it easy to distinguish a transient cold-start storm from a sustained backend outage.
• ProxySQL_Uptime correlation: Netdata overlays uptime with connection metrics, so the cold-start window is immediately apparent without manual cross-referencing against restart logs.
• ConnPool_get_conn_failure and Server_Connections_delayed: pool-starvation signals appear at per-second resolution, showing exactly when queries start queuing and when the pool recovers.
• Backend ConnERR and escalation signals: the conditions that turn a transient storm into a page-worthy incident are tracked continuously alongside the storm indicators.
• Backend MySQL CPU correlation: when Netdata monitors both the ProxySQL host and the MySQL backends, the CPU spike on the backend during connection creation is visible alongside the ProxySQL-side signals, confirming the cause-and-effect chain without switching tools.

Related guides

• ProxySQL backend connection pool exhausted: queries queuing for a free connection
• ProxySQL backend flapping between ONLINE and SHUNNED: monitor-induced oscillation
• ProxySQL OFFLINE_SOFT vs OFFLINE_HARD vs SHUNNED: what each backend status means
• ProxySQL backend SHUNNED: why a healthy backend gets pulled out of rotation
• ProxySQL ConnERR climbing: backend connection errors and how to localise them
• ProxySQL ConnPool_get_conn_failure rising: the most direct pool-starvation signal
• ProxySQL hostgroup_locked connections: reading the multiplexing-health ratio
• How ProxySQL actually works in production: a mental model for operators
• ProxySQL error 9001 Max connect timeout reached while reaching hostgroup
• ProxySQL monitor check failures: connect, ping, read-only, and replication-lag probes failing
• ProxySQL MySQL_Monitor_Workers is zero: health checks stopped and status is stale
• ProxySQL monitoring checklist: the signals every production proxy needs