$ guides / proxysql / proxysql-connpool-get-conn-failure ▌

Operations Guides

ProxySQL ConnPool_get_conn_failure rising: the most direct pool-starvation signal

When ConnPool_get_conn_failure in stats_mysql_global starts climbing, ProxySQL’s internal connection pool failed to hand a backend connection to a worker thread on request. ProxySQL asked the pool for a connection to a hostgroup and got nothing back. That makes it the most actionable signal for backend pool starvation, more direct than watching ConnUsed and ConnFree in isolation.

This counter does not mean clients are seeing errors. It increments on every internal pool-lookup miss, even if ProxySQL subsequently creates a new backend connection or retries successfully. A spike of hundreds of thousands of failures in a few seconds can correspond to zero client-visible errors. What it tells you: the pool was pressured enough to miss, and sustained rising rates mean that pressure is not resolving on its own.

Pair this metric with two siblings in the same stats table: ConnPool_get_conn_success (the pool handed back a connection) and ConnPool_get_conn_immediate (the pool had a connection ready in the thread’s local cache with no wait). The ratio between immediate gets, successful gets, and failed gets tells you whether the pool is warm and healthy, working but contended, or genuinely starved.

What this means

Each ProxySQL worker thread maintains a local cache of backend connections. When a query arrives, the thread first checks its local cache (immediate get). If that misses, it asks the shared backend connection pool for a connection to the target hostgroup. If the pool has a free connection or can create one within the configured limits, the request succeeds. If it cannot, ConnPool_get_conn_failure increments.

The counter is cumulative since ProxySQL start. You must compute a rate (delta over time) to get actionable signal. A high absolute number is meaningless on a long-running instance; a rising rate is what matters.

The pool can fail to provide a connection for several distinct reasons, and the diagnostic path forks accordingly:

flowchart TD
    A["ConnPool_get_conn_failure
rate rising"] --> B{"Any backends
SHUNNED or OFFLINE?"}
    B -- "Yes" --> C["Capacity removed
from hostgroup"]
    B -- "No" --> D{"ConnUsed near
max_connections?"}
    D -- "Yes" --> E["Pool at capacity
ceiling"]
    D -- "No" --> F{"hostgroup_locked ratio
above 50%?"}
    F -- "Yes" --> G["Multiplexing collapse
pinning backend connections"]
    F -- "No" --> H{"Recent restart or
LB health check noise?"}
    H -- "Yes" --> I["Cold pool or
spurious failures"]
    H -- "No" --> J["Check Server_Connections_delayed
and backend ConnERR"]

Common causes

Cause	What it looks like	First thing to check
Backend SHUNNED or OFFLINE	Failures spike, then drop when backend recovers; `ConnFree` was 0 during the window even though connections exist after recovery	`status` column in `stats_mysql_connection_pool`
Pool at capacity ceiling	`ConnUsed` equals or approaches `max_connections` per backend; failures sustained under load	Compare `ConnUsed` to `max_connections` in `runtime_mysql_servers`
Multiplexing collapse	`Client_Connections_hostgroup_locked` high relative to `connected`; `ConnUsed` tracks client count linearly	`hostgroup_locked / connected` ratio
Cold pool after restart	Failures spike briefly post-restart, then settle as pool warms; `ProxySQL_Uptime` low	`ProxySQL_Uptime` in `stats_mysql_global`
LB health check noise	Failures increase steadily with no query impact; `Questions` rate stable	Check external health check configuration hitting port 6033
Backend failover transition	Failures spike during writer switchover; no ONLINE backend in target hostgroup briefly	`runtime_mysql_servers` status during failover window

Quick checks

Run these against the admin interface (default port 6032). All are read-only.

-- Check the three pool-get counters and compute context
SELECT Variable_Name, Variable_Value
FROM stats_mysql_global
WHERE Variable_Name IN (
  'ConnPool_get_conn_failure',
  'ConnPool_get_conn_success',
  'ConnPool_get_conn_immediate',
  'ProxySQL_Uptime'
);

-- Per-backend pool state: what is available, what is used, what errored
SELECT hostgroup, srv_host, srv_port, status,
       ConnUsed, ConnFree, ConnOK, ConnERR
FROM stats_mysql_connection_pool
ORDER BY hostgroup, srv_host;

-- Compare ConnUsed to ProxySQL's per-backend max_connections ceiling
SELECT cp.hostgroup, cp.srv_host, cp.srv_port, cp.status,
       cp.ConnUsed, cp.ConnFree, rs.max_connections
FROM stats_mysql_connection_pool cp
JOIN runtime_mysql_servers rs
  ON cp.hostgroup = rs.hostgroup_id
 AND cp.srv_host = rs.hostname
 AND cp.srv_port = rs.port;

-- Multiplexing health: are too many sessions pinning backend connections?
SELECT Variable_Name, Variable_Value
FROM stats_mysql_global
WHERE Variable_Name IN (
  'Client_Connections_connected',
  'Client_Connections_hostgroup_locked',
  'Active_Transactions',
  'Server_Connections_delayed'
);

-- Backend status across all hostgroups
SELECT hostgroup_id, hostname, port, status, max_connections
FROM runtime_mysql_servers
ORDER BY hostgroup_id, status;

-- Server-side delayed connections: direct evidence queries waited for a backend
SELECT Variable_Name, Variable_Value
FROM stats_mysql_global
WHERE Variable_Name = 'Server_Connections_delayed';

How to diagnose it

Confirm the rate is real. Take two readings of ConnPool_get_conn_failure spaced 60 seconds apart and compute the delta. If the delta is zero or negligible, the counter is not actively rising and you may be looking at stale historical accumulation.
Check the success-to-failure ratio. Compare ConnPool_get_conn_failure to ConnPool_get_conn_success over the same interval. If failures are a tiny fraction of successes (less than 1%), the pool is handling the vast majority of requests and the misses may be noise. If failures approach or exceed 10% of successes, the pool is genuinely pressured.
Check ConnPool_get_conn_immediate. This counter tracks connections served from the thread’s local cache without touching the shared pool. A high immediate count relative to total gets means the pool architecture is working well. A dropping immediate count with rising failures means threads are increasingly losing their local cache and hitting the shared pool, which is contended.
Identify which hostgroup is affected. ConnPool_get_conn_failure is global, not per-hostgroup. Cross-reference with per-backend ConnUsed, ConnFree, and status in stats_mysql_connection_pool to find the hostgroup where capacity is missing.
Check for SHUNNED backends. A backend can be SHUNNED for brief windows controlled by shun_recovery_time . During that window, ConnFree drops to 0 for that backend and failures spike. By the time you query, the backend may already be ONLINE again with free connections visible. Check the monitor logs (monitor.mysql_server_ping_log, monitor.mysql_server_connect_log) for alternating success/failure patterns around the time of the failure spike.
Check for cold pool. If ProxySQL_Uptime is low (under 120 seconds), the pool is still warming. Backend connections are created on-demand after restart. Failures during this window are expected and self-resolving.
Check for multiplexing collapse. If Client_Connections_hostgroup_locked is more than 50% of Client_Connections_connected, sessions are pinning backend connections. The pool is starved not because backends are slow but because too many clients have disabled multiplexing via transactions, SET variables, temporary tables, or prepared statements.
Rule out LB health check noise. External load balancer health checks (AWS NLB, HAProxy, etc.) hitting port 6033 can cause ConnPool_get_conn_failure to increment because those connections open and close without a valid backend session context. If Questions rate is stable and clients report no errors, check whether health check traffic is inflating the counter.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`ConnPool_get_conn_failure` rate	Direct evidence the pool could not satisfy a request	Sustained non-zero rate
`ConnPool_get_conn_success` rate	Context for the failure rate	Failures exceeding 10% of successes
`ConnPool_get_conn_immediate` rate	Local cache hit rate; measures pool warmth	Dropping while failures rise
`Server_Connections_delayed`	Queries that had to wait for a free backend connection	Any sustained non-zero value
`ConnUsed` vs `max_connections` per backend	Whether the pool is at its configured ceiling	Ratio above 80% sustained
Backend `status`	Whether capacity was removed by SHUNNED or OFFLINE	SHUNNED flapping or sustained
`Client_Connections_hostgroup_locked` ratio	Multiplexing health	Above 50% of connected clients
`ConnERR` per backend	Backend rejecting ProxySQL’s connection attempts	Sustained non-zero rate

Fixes

Backend at capacity ceiling

If ConnUsed is at or near max_connections for backends in the affected hostgroup, the pool has no room to grow within the current limit.

Check whether the backend MySQL can handle more connections. If it can, raise the per-backend limit:

Warning: LOAD MYSQL SERVERS TO RUNTIME is immediately effective on live traffic. SAVE MYSQL SERVERS TO DISK persists the change across restarts. Verify the backend MySQL can absorb the increased connection count before applying.

UPDATE mysql_servers SET max_connections = <new_value>
  WHERE hostgroup_id = <hg> AND hostname = '<host>' AND port = <port>;
LOAD MYSQL SERVERS TO RUNTIME;
SAVE MYSQL SERVERS TO DISK;

The sum of max_connections across all ProxySQL instances pointing at a single backend should stay below 80% of the backend MySQL’s max_connections setting. Other consumers (direct admin connections, replication threads, other proxies, monitoring tools) need headroom too.

Backends SHUNNED

If failures correlate with SHUNNED transitions, the fix depends on why backends are being shunned:

Replication lag: Check max_replication_lag in runtime_mysql_servers. If replicas are being shunned for brief lag spikes that self-correct, consider whether the threshold is too aggressive for the workload.
Connection errors: Check ConnERR on the affected backend. If the backend is rejecting connections due to its own max_connections, fix that on the MySQL side.
Monitor flapping: If the backend alternates between ONLINE and SHUNNED rapidly, check monitor intervals (mysql-monitor_ping_interval, mysql-monitor_connect_interval) and shun_on_failures (default 5). Too-sensitive thresholds on a network with jitter cause oscillation.

See ProxySQL backend flapping between ONLINE and SHUNNED: monitor-induced oscillation for detailed flapping diagnosis.

Multiplexing collapse

If hostgroup_locked is high, the pool is starved because backend connections are pinned to client sessions instead of being shared. Identify which sessions have disabled multiplexing:

-- Check for sessions with disabled multiplexing
SELECT * FROM stats_mysql_processlist WHERE extended_info != '';

Common causes: ORMs setting session variables (SET NAMES, SET sql_mode, SET time_zone) on every connection, long-running transactions, temporary tables, GET_LOCK(), user-defined variables, or prepared statements.

Fix options:

Move session variable initialization to MySQL server-side defaults or ProxySQL’s mysql-init_connect so it does not disable multiplexing per session.
Review mysql-multiplexing and related variables (mysql-forward_autocommit, mysql-autocommit_false_not_reusable).
If a specific application is responsible, coordinate with the application team.

See How ProxySQL actually works in production: a mental model for operators for the multiplexing model.

Cold pool after restart

If failures spike immediately after restart and settle within 30 to 120 seconds, this is expected warmup behavior. The pool starts empty and fills on-demand as queries arrive.

If the spike causes client timeouts (aggressive application timeouts), consider connection warming. The mysql-connection_warming variable, if available in your version, pre-creates backend connections at startup. Alternatively, configure mysql-free_connections_pct to maintain a warm pool of free connections.

LB health check inflation

If ConnPool_get_conn_failure is rising but Questions rate is stable and clients report no errors, external health checks may be inflating the counter. Load balancer probes that connect to port 6033 and immediately disconnect can cause pool-lookup misses because they open without a valid backend session context.

Fix: configure the load balancer health check to use a proper MySQL protocol check that executes a query through the proxy, or accept the inflation and alert on the ratio of failures to successes rather than the absolute failure rate.

Prevention

Alert on the failure rate, not the absolute counter. ConnPool_get_conn_failure is cumulative since start. Set up delta-based rate alerts, not threshold-on-raw-value alerts.
Alert on the failure-to-success ratio. A 1% failure rate during peak may be normal for your workload. A sudden jump from 0.1% to 5% is the signal, regardless of absolute numbers.
Track ConnPool_get_conn_immediate as a leading indicator. When the local cache hit rate drops, shared-pool contention follows. A declining immediate ratio is an early warning before failures appear.
Monitor multiplexing health routinely. Multiplexing collapse is the most common cause of gradual pool starvation. Track hostgroup_locked / connected as a standing dashboard metric, not just during incidents.
Correlate with Server_Connections_delayed. When this counter is non-zero, queries are already waiting. It is the symptom that precedes client-visible latency.
Keep per-backend max_connections below 80% utilization at peak. Headroom absorbs traffic spikes without triggering pool misses.

How Netdata helps

Collects ConnPool_get_conn_failure, ConnPool_get_conn_success, and ConnPool_get_conn_immediate per second, so rate changes are visible immediately rather than on a slow polling interval.
Surfaces Server_Connections_delayed alongside the pool-get counters, so you can correlate pool misses with query queuing in a single view.
Tracks per-backend ConnUsed, ConnFree, ConnOK, and ConnERR so you can pinpoint which backend in which hostgroup is driving the failures.
Monitors backend status transitions (ONLINE to SHUNNED and back) with per-second resolution, making brief shun windows visible that polling-based monitoring would miss.
Correlates pool-starvation signals with Client_Connections_hostgroup_locked and Active_Transactions so multiplexing collapse is identifiable without manual cross-referencing.
ML-based anomaly detection on the failure rate baseline helps distinguish a genuine shift from normal daily variation.