$ guides / proxysql / proxysql-server-connections-delayed ▌

Operations Guides

ProxySQL Server_Connections_delayed above zero: queries waiting on the backend pool

When Server_Connections_delayed in ProxySQL’s stats_mysql_global table climbs above zero, queries are waiting for a backend connection that was not immediately available. This is not an error counter. It is a pressure signal: ProxySQL’s backend connection pool could not instantly satisfy a connection request, so the requesting session had to wait. A brief blip during a traffic burst is normal. A sustained increase means the pool is undersized, multiplexing has degraded, or backends are disappearing from rotation faster than the pool can adapt.

The counter is cumulative since ProxySQL start. To assess current pressure, you need the rate of change, not the absolute value. A value of 10,000 accumulated over weeks is noise. A delta of 500 in five minutes is the signal you need to investigate.

What this means

ProxySQL’s backend connection pool is the multiplexing core. When a client sends a query, ProxySQL borrows a backend connection, routes the query, returns the result, then returns the connection to the pool. When no free connection is available and the pool cannot create a new one, the query must wait. That wait increments Server_Connections_delayed.

A critical nuance: ConnFree == 0 alone does not mean saturation. ProxySQL may still create new connections up to the backend’s max_connections limit (configured in mysql_servers). True saturation means ProxySQL cannot create new connections and has no free ones. The symptom is ConnERR increasing (ProxySQL tried to open a new connection and was rejected) or ConnPool_get_conn_failure climbing.

The companion metric to watch alongside this is Server_Connections_aborted. When backends close connections unexpectedly, often because the backend MySQL’s wait_timeout fires on an idle pooled connection that ProxySQL still thinks is valid, the next query on that connection hits an error. ProxySQL handles this by reconnecting, but the aborted connection reduces effective pool size and can contribute to delayed connections in the next burst.

flowchart TD
    A[Query arrives] --> B{Free backend connection?}
    B -- Yes --> C[Borrow, execute, return to pool]
    B -- No --> D{Can create new connection?}
    D -- Yes --> E[Create, execute, return to pool]
    D -- No, at max_connections --> F[Query waits]
    F --> G[Server_Connections_delayed++]
    G --> H{Connection frees before timeout?}
    H -- Yes --> C
    H -- No --> I[max_connect_timeouts++]
    I --> J[Error 9001 to client]

Common causes

Cause	What it looks like	First thing to check
Pool at max_connections	ConnFree is zero, ConnUsed at limit, ConnERR stable	Compare ConnUsed to max_connections in runtime_mysql_servers
Multiplexing collapse	hostgroup_locked high relative to connected, Active_Transactions elevated	Check Client_Connections_hostgroup_locked ratio
Backend SHUNNED	One or more backends SHUNNED in stats_mysql_connection_pool, ConnERR rising on that backend	Check monitor check results and backend status
Backend dropping pooled connections	Server_Connections_aborted rising, no corresponding backend issue	Compare backend wait_timeout to ProxySQL mysql-wait_timeout
Prepared statements pinning connections	Stmt_Server_Active_Total high, multiplexing ratio degraded	Check application prepared statement usage

Quick checks

All commands connect to the ProxySQL admin interface on port 6032 and run read-only queries. Replace -u admin -padmin with your configured admin credentials.

# Check the delayed and aborted counters
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global \
      WHERE Variable_Name IN ('Server_Connections_delayed','Server_Connections_aborted', \
      'Server_Connections_connected','Server_Connections_created');"

# Check per-backend pool state
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup, srv_host, srv_port, status, ConnUsed, ConnFree, ConnOK, ConnERR \
      FROM stats_mysql_connection_pool ORDER BY hostgroup, srv_host;"

# Cross-reference pool usage with configured limits
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT cp.hostgroup, cp.srv_host, cp.srv_port, cp.ConnUsed, cp.ConnFree, rs.max_connections \
      FROM stats_mysql_connection_pool cp \
      JOIN runtime_mysql_servers rs \
      ON cp.hostgroup = rs.hostgroup_id AND cp.srv_host = rs.hostname AND cp.srv_port = rs.port;"

# Check pool operation outcomes
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global \
      WHERE Variable_Name IN ('ConnPool_get_conn_success','ConnPool_get_conn_failure', \
      'ConnPool_get_conn_immediate');"

# Check multiplexing health
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global \
      WHERE Variable_Name IN ('Client_Connections_connected','Client_Connections_hostgroup_locked', \
      'Active_Transactions');"

# Check backend status across hostgroups
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup_id, hostname, port, status, max_connections FROM runtime_mysql_servers;"

# Check for connect timeout errors
mysql -u admin -padmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global \
      WHERE Variable_Name = 'max_connect_timeouts';"

How to diagnose it

Establish whether the counter is actively increasing. Take two readings of Server_Connections_delayed a few seconds apart. If the value is not changing, the pressure was transient and may have resolved. If it is climbing, proceed.
Check per-backend pool saturation. Join stats_mysql_connection_pool with runtime_mysql_servers to compare ConnUsed against max_connections. If ConnUsed equals or approaches max_connections and ConnFree is zero, the pool for that backend is saturated.
Check ConnPool_get_conn_failure. This metric is the most direct indicator of pool starvation. It counts how often ProxySQL tried to get a backend connection from the pool and failed at that instant. A rising rate confirms the pool cannot satisfy demand. Note that ConnPool_get_conn_failure does not directly translate to client-facing errors. The MySQL thread retries according to mysql-connect_retries_on_failure (default 10). A high failure count can accumulate in seconds without client impact if retries eventually succeed.
Check backend status. Look for SHUNNED backends. When a backend is SHUNNED, its connections are not available for new queries, effectively reducing pool capacity for that hostgroup. Even if ConnFree shows free connections at other times, during a shunning window the pool cannot provide connections for that hostgroup.
Check multiplexing health. If Client_Connections_hostgroup_locked is approaching Client_Connections_connected, multiplexing has collapsed. Each client session pins a dedicated backend connection, making the effective pool size equal to the client count rather than the configured max_connections. Check Active_Transactions to see if long-running transactions are the cause.
Check Server_Connections_aborted. If this is rising alongside Server_Connections_delayed, backends are closing connections out from under ProxySQL. The most common cause is the backend MySQL’s wait_timeout firing on idle pooled connections. ProxySQL keeps the connection in its pool thinking it is valid, but the next query that tries to use it gets an error.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Server_Connections_delayed (rate)	Direct evidence of pool pressure	Sustained rate above zero
ConnPool_get_conn_failure (rate)	ProxySQL tried and failed to get a connection	Rising rate correlates with delayed
ConnFree per backend	Idle connections available for reuse	Zero with ConnUsed at max_connections
ConnUsed / max_connections	Pool utilization per backend	Above 80% sustained
Server_Connections_aborted (rate)	Backends closing connections unexpectedly	Rising rate, especially with delayed
Client_Connections_hostgroup_locked ratio	Multiplexing degradation	Above 50% of connected
Active_Transactions	Connections pinned by transactions	High relative to client count
Backend status	SHUNNED reduces effective pool	Any backend not ONLINE
max_connect_timeouts	Queries waiting beyond timeout limit	Any sustained increase

Fixes

Pool at max_connections

If ConnUsed is at max_connections and backends are otherwise healthy, the cap is too low for the workload. Increase max_connections in mysql_servers for the affected backend.

UPDATE mysql_servers SET max_connections = <new_value>
  WHERE hostgroup_id = <hg> AND hostname = '<host>';
LOAD MYSQL SERVERS TO RUNTIME;
SAVE MYSQL SERVERS TO DISK;

The sum of max_connections across all ProxySQL instances for a given backend must not exceed what the backend MySQL can handle. ProxySQL shares the backend’s connection capacity with direct admin connections, replication threads, monitoring tools, and any non-proxy applications. Leave headroom.

Also check whether the backend MySQL itself is rejecting connections. If ConnERR is rising and Server_Connections_aborted is climbing, the backend may have hit its own max_connections limit.

Multiplexing collapse

If hostgroup_locked is high relative to connected, the application is disabling multiplexing. Common causes: ORMs setting session variables (SET NAMES, SET sql_mode, SET time_zone) on every connection, long-running transactions, GET_LOCK(), temporary tables, user-defined variables, or prepared statements in some configurations.

Short-term: increase max_connections on backends to absorb the 1:1 mapping. Long-term: identify which SET commands are breaking multiplexing. Move session variable initialization to server-side defaults or ProxySQL’s mysql-init_connect so they do not need per-session SET commands. Check stats_mysql_processlist for sessions with disabled multiplexing and the reason.

Backend SHUNNED reducing pool capacity

When a backend is SHUNNED, its connections are unavailable for new queries. If enough backends in a hostgroup are SHUNNED, the remaining backends absorb all traffic and may saturate their own pools. Check the monitor check results to understand why the backend was shunned. Common causes: connection errors exceeding mysql-shun_on_failures (default 5), or replication lag exceeding the configured max_replication_lag.

If shunning is transient and self-corrects within mysql-shun_recovery_time_sec (default 10 seconds), the pool pressure is temporary. If shunning is persistent or flapping, investigate the underlying backend health or adjust monitor thresholds.

Backend MySQL wait_timeout closing pooled connections

If the backend MySQL’s wait_timeout is lower than ProxySQL’s mysql-wait_timeout (default 28800000ms, approximately 8 hours), the backend may close idle connections that ProxySQL still tracks as valid. The next query on that connection fails, increments Server_Connections_aborted, and forces ProxySQL to create a new connection. During bursts, this churn contributes to pool pressure.

Prevention

Track ConnUsed / max_connections per backend as a trend. The leading indicators are ConnFree trending down and ConnPool_get_conn_failure appearing before Server_Connections_delayed rises. Aim for at least 20% pool capacity free during peak.
Monitor the multiplexing ratio. Client_Connections_connected / Server_Connections_connected should be well above 1:1. A declining ratio means the effective pool is shrinking even without traffic growth.
Alert on ConnPool_get_conn_failure rate, not just ConnFree. This is the most direct indicator that ProxySQL tried and failed to get a connection. It surfaces pressure before delayed connections become visible.
Align timeout settings. Ensure the backend MySQL’s wait_timeout is not silently closing pooled connections. Set mysql-connection_max_age_ms if needed.
Size max_connections with all consumers in mind. The sum of ProxySQL’s per-backend max_connections across all instances must leave room for direct connections, replication threads, and monitoring tools on the backend MySQL.
Watch for connection churn after deployments. Application restarts create connection storms. Stagger restarts or configure connection warming if available in your version.

How Netdata helps

Netdata’s ProxySQL collector surfaces these signals at per-second resolution, which matters because pool pressure can be transient and invisible to minute-level polling:

Server_Connections_delayed as a rate, not just a cumulative counter, so you can see when pressure is actively building versus historical accumulation.
ConnPool_get_conn_failure correlated with per-backend ConnUsed and ConnFree, letting you see which specific backend is saturating before the delayed counter confirms it.
Server_Connections_aborted trended alongside delayed connections, surfacing the backend wait_timeout interaction without a separate investigation.
Multiplexing ratio (hostgroup_locked / connected) trended over time, so you can catch the gradual multiplexing collapse that turns into pool starvation weeks later.
ML anomaly detection on pool operation rates, catching the early deviation from baseline that precedes a visible saturation event.