$ guides / proxysql ▌

PROXYSQL · OPERATIONS PLAYBOOK

ProxySQL's quiet failures: multiplexing that collapses to 1:1, backends that shun themselves, and a config that reverts on restart

A MySQL-protocol-aware proxy that multiplexes thousands of client connections onto a handful of backend connections, routes every query through an ordered rule chain, and health-checks backends into and out of rotation. We trace how that design behaves under load, where the pooling silently disappears, and what to do when a backend, a rule, or a forgotten LOAD TO RUNTIME takes the service down.

> Start with the monitoring checklist → # Jump to the full guide list

ProxySQL's defaults get you a working read/write split in an afternoon, then hand you a set of silent failure modes that only surface under real load or on the next restart.

The defaults work. Until an ORM starts sending SET NAMES utf8mb4 on every connection, multiplexing quietly collapses to 1:1, and the connection pooling you deployed ProxySQL for simply stops happening. Until a replica drifts past max_replication_lag and the monitor SHUNS it, then restores it, then shuns it again on a jittery network. Until a query rule missing apply=1 routes a write to a read-only replica and returns 1290 - The MySQL server is running with the --read-only option — or worse, the replica is not read-only and the write silently diverges. Until you fix a rule in RUNTIME during an incident, forget SAVE TO DISK, and lose the fix on the next restart.

These guides are written for engineers who already run ProxySQL, not for people choosing a proxy. The goal is the mental model of how the proxy actually behaves under load, the failure patterns that keep recurring, the monitoring story that catches them before they page anyone, and the runbooks you wish someone had handed you before your last incident. ProxySQL speaks the MySQL wire protocol only — none of this applies to PostgreSQL.

How ProxySQL actually runs in production

ProxySQL is not a TCP relay. It parses the MySQL wire protocol, evaluates every query against an ordered rule chain, borrows and returns backend connections to multiplex many clients onto few, and continuously health-checks backends into and out of hostgroups — all driven by a three-layer config that is live only after LOAD TO RUNTIME and durable only after SAVE TO DISK. Most production failures live between these layers, not inside any one of them.

clients / frontend pool

Client connections land on port 6033 and authenticate against ProxySQL's own <code>mysql_users</code> table, independently of backend credentials. Each session carries state — user, schema, transaction status, session variables, autocommit — and <code>mysql-max_connections</code> (default 2048) is a single global cap across all users and hostgroups.

CLIENT

query processor + rules

Every query is matched against <code>mysql_query_rules</code> in <code>rule_id</code> order; the first match with <code>apply=1</code> wins and decides hostgroup, caching, rewriting, and timeout. A missing <code>apply=1</code> or a broad regex is a silent mis-routing bug, and complex <code>match_pattern</code> regex is a CPU tax on every single query.

ROUTE

query cache

An in-memory, TTL-only result cache keyed by query digest + user + schema. There is no invalidation on data change — it is a time-bounded stale-read cache, not a coherent one. Only queries matched by a rule with <code>cache_ttl > 0</code> are cached at all.

CACHE

multiplexing engine

The core abstraction: borrow a backend connection, run the query, return it to the pool. Session state — open transactions, <code>SET</code> variables, temp tables, <code>LOCK TABLES</code>, <code>GET_LOCK()</code>, user-defined variables — pins a backend connection to one client (<code>Client_Connections_hostgroup_locked</code>) and defeats pooling.

MUX

backend connection pool

Persistent connections to each backend, organised by hostgroup, reused across clients when session state permits. Each backend has its own <code>max_connections</code> in <code>mysql_servers</code> — ProxySQL's self-imposed limit, entirely separate from the backend MySQL's own <code>max_connections</code>.

POOL

monitor module

Background threads run connect, ping, read-only, and replication-lag checks using separate <code>mysql-monitor_*</code> credentials. Results drive automatic ONLINE/SHUNNED/OFFLINE transitions and writer/reader role assignment. If monitor workers stall or the monitor password is wrong, health decisions go stale or shun healthy backends.

MONITOR

hostgroups + backends

Backends live in hostgroups with a status — <code>ONLINE</code>, <code>SHUNNED</code>, <code>OFFLINE_SOFT</code>, or <code>OFFLINE_HARD</code>. Galera and Group Replication use special hostgroup tables that auto-route writes to the PRIMARY and reads to the SECONDARY based on cluster node state.

BACKEND

admin + three-layer config

The admin interface on port 6032 exposes runtime state as SQL tables. Changes flow MEMORY to RUNTIME (via <code>LOAD ... TO RUNTIME</code>) to DISK (via <code>SAVE ... TO DISK</code>). A change is not active until loaded and not durable until saved — the single most common source of operational surprise.

CONFIG

Why this matters: 'ProxySQL is slow' or 'connections are exhausted' can come from multiplexing collapsed to 1:1, a backend pool at its per-backend max_connections, the backend MySQL's own limit rejecting ProxySQL, worker threads saturated on regex rules, a replica shunned for lag redirecting traffic, or a config that reverted on restart. The symptom rhymes but each layer has a different signal — and a different fix.

The failures you'll actually see

Most ProxySQL incidents fall into a small set of recurring patterns. Recognise the shape, and triage gets dramatically faster.

ACTIVE

Multiplexing collapse

An application or ORM starts issuing SET commands on every connection, ProxySQL pins a backend connection per client, and the multiplexing ratio drifts toward 1:1. Backend ConnUsed now tracks client count linearly, the pool fills, and the 10:1 pooling your capacity plan assumed is fiction — all with no error until the backend runs out of connections.

Client_Connections_hostgroup_locked climbing toward connected
Multiplexing ratio (client / backend connections) approaching 1:1
ConnUsed rising in lockstep with client connection count
Latency rising uniformly across all query types after a deploy

Investigate →

CRITICAL

Backend connection starvation

Every backend connection in a hostgroup is in use and ProxySQL cannot open more — either it hit its own per-backend max_connections or the backend MySQL rejected it with 1040 - Too many connections. New queries queue, then fail with Max connect timeout reached while reaching hostgroup. A cliff-edge once the pool is full.

ConnFree == 0 with ConnUsed > 0 sustained on all backends
ConnPool_get_conn_failure and Server_Connections_delayed rising
ConnERR climbing as the backend refuses new connections
Clients seeing error 9001 Max connect timeout reached

Investigate →

ACTIVE

Backends shunning themselves

The monitor module marks a backend SHUNNED after consecutive connection errors or replication lag over threshold. It is a protection mechanism and usually self-corrects — but aggressive check intervals or a wrong monitor password make healthy backends flap ONLINE and SHUNNED, oscillating traffic and disrupting queries mid-flight.

status = SHUNNED in stats_mysql_connection_pool
More than 3 ONLINE/SHUNNED transitions in 10 minutes
MySQL_Monitor check ERR rising while the backend is healthy directly
Periodic latency spikes and Questions-rate dips

Investigate →

IMMINENT

Silent read/write-split failure

A query rule missing apply=1, or a wrong rule_id order, routes writes to the reader hostgroup. If the replica is read-only you get 1290 - ... running with the --read-only option; if it is not, the write succeeds on a replica and never reaches the primary — silent data divergence with no error at all.

Error 1290 in stats_mysql_errors from a reader hostgroup
INSERT / UPDATE / DELETE digests appearing in the reader hostgroup
Write-routing rule hits dropping, catch-all rule hits rising
Application reporting intermittent read-only errors

Investigate →

IMMINENT

The three-layer config revert

A change made in the admin interface lands in MEMORY, gets loaded to RUNTIME during an incident, and is never saved to DISK — so the next restart (upgrade, OOM, reboot) silently reverts to stale config. Or a change is saved but never loaded, and everyone wonders why it isn't working. No metric flags it until the restart.

runtime_mysql_servers matches memory but differs from disk
Behaviour not matching the config someone believes is active
Config reverting after a ProxySQL restart or upgrade
Per-module LOAD/SAVE done for some modules but not others

Investigate →

CRITICAL

Authentication cascade

A credential rotation reaches the application or the backend but not ProxySQL's mysql_users (or its separate mysql-monitor_password). Frontend logins fail with 1045 - Access denied and the Questions rate collapses, or the monitor cannot authenticate and shuns healthy backends. When all auth fails, the service is down for new connections.

Access_Denied_Wrong_Password rising with Questions rate dropping
Error 1045 Access denied for user in client errors
Client_Connections_aborted spiking
Monitor check ERR rising after a password change (monitor creds stale)

Investigate →

Choosing a tool

Best ProxySQL Monitoring Tools: 6 Ranked for 2026

A ranked review of the tools teams actually shortlist here, what each one is genuinely good at, and how the pricing behaves as you scale.

Read the buyer's guide →

ProxySQL monitoring maturity levels

ProxySQL observability works in four practical levels. Each is a complete operation, not a stepping stone. Pick the level that matches how much your proxy tier matters. Most production deployments should land at the second level.

Level 1: Survival

Know that something is wrong

Survival monitoring is the floor. With these signals you can answer one question: is the proxy accepting clients and can it reach at least one backend? You will not learn what broke, but you will learn that something broke before users do. Survival is enough for dev proxies and non-critical tiers.

ProxySQL process running Host-level: is the proxysql process alive at all?
Data-plane port 6033 accepting An external TCP/MySQL check — the proxy can be running while the listener hangs.
At least 1 ONLINE backend per hostgroup Zero ONLINE in a hostgroup is a total outage for that traffic class.
Backend ConnERR per backend ProxySQL tried to reach a backend and failed.
Client_Connections_aborted Clients being rejected or crashing on connect.
ProxySQL process RSS Memory footprint and OOM risk.

Level 2: Operational

Diagnose most incidents on your own

Operational monitoring is what most production deployments should target. Survival tells you something is wrong; operational tells you what. With this coverage your team can usually diagnose an incident on its own: pool pressure, backend flapping, auth failures, slow queries, routing.

Client_Connections_connected vs max Frontend saturation; a cliff at mysql-max_connections.
ConnUsed / ConnFree per backend Backend pool pressure per server.
Questions rate Throughput baseline; a sudden drop means traffic stopped or is failing.
Slow_queries rate The ratio to Questions matters more than the absolute count.
Monitor check OK/ERR per type Connect, ping, read-only, and replication-lag probe health.
Per-user connection utilisation One app hitting its cap can starve the rest of the instance.
Access_Denied_* counters Wrong password, max connections, and max user connections.
ProxySQL process CPU Worker-thread saturation slows every query uniformly.

Level 3: Mature

Catch problems before they become incidents

Mature monitoring catches problems before they wake anyone up. Multiplexing eroding, the cache churning, a backend drifting slower than its peers, connection acquisition starting to fail. None of these page you on day one. They become page-out incidents on day thirty.

Multiplexing ratio and hostgroup_locked Is the pooling you deployed ProxySQL for actually happening?
Query cache hit rate, memory, purge Effective offload versus pure churn.
Per-command latency histograms The real query-latency distribution, not the monitor ping.
Per-backend Latency_us divergence One backend far slower than peers in the same hostgroup.
Active_Transactions Open transactions pin backend connections and break multiplexing.
ConnPool_get_conn_failure rate The most direct pool-starvation signal there is.
backend_lagging / offline_during_query Queries that failed in flight, not theoretical risk.
Detailed memory breakdown stats_memory_metrics: jemalloc_resident and fragmentation.

Level 4: Expert

Reactive instrumentation after real incidents

Expert signals enter your stack the day after a specific incident proved you needed them. Per-rule hit distribution, query digests, cluster checksums, monitor freshness, config-layer drift. Most teams never need every signal here. Add the ones your incident history says you do.

Per-query-rule hits stats_mysql_query_rules: which rules fire, and whether the write rule fires at all.
Query digest top-N stats_mysql_query_digest: workload profile and new/anomalous queries.
ProxySQL cluster checksums Config divergence between peers (split-brain routing).
Monitor freshness Gap between intended and actual check interval (thread starvation).
Config layer drift Memory vs runtime vs disk — the change that reverts on the next restart.
Per-thread CPU (ps -L) Hot-thread detection under load.
File-descriptor usage vs ulimit A host-level cliff ProxySQL does not expose in stats.
stats_mysql_processlist snapshots Per-session state: how many are idle, executing, or locked.

Operating mistakes worth avoiding

The traps ProxySQL teams keep falling into. Each has a clear, well-known fix. Most teams only learn it after an incident.

Monitoring the frontend but ignoring the backend

Teams check 'can I connect to ProxySQL?' and forget 'can ProxySQL reach the backends?'. Backend <code>ConnERR</code> and monitor check results are the signals that actually predict an outage, and they are the ones most often missing from dashboards.

Treating SHUNNED as always critical

<code>SHUNNED</code> is ProxySQL's protection mechanism, and it usually self-corrects within <code>mysql-shun_recovery_time_sec</code>. Paging on every shun event is pure alert fatigue. Only page when zero backends remain ONLINE in a hostgroup; a single shunned backend is a ticket.

Never measuring multiplexing efficiency

The whole point of ProxySQL is multiplexing, yet almost no one tracks the ratio. An ORM setting session variables silently drops it to 1:1, so the proxy adds latency while providing no pooling — and the capacity model that assumed 10:1 is fiction. Watch <code>Client_Connections_hostgroup_locked / connected</code>.

Trusting Latency_us as query latency

<code>Latency_us</code> in <code>stats_mysql_connection_pool</code> is the monitor's ping latency, not query execution time. Dashboards built on it do not represent user-facing performance. Real query latency lives in the <code>stats_mysql_commands_counters</code> histograms and <code>stats_mysql_query_digest</code>.

Ignoring the three-layer config model

Changes live in MEMORY, activate only on <code>LOAD ... TO RUNTIME</code>, and persist only on <code>SAVE ... TO DISK</code> — per module. Fix a rule during an incident, forget the save, and a restart days later reverts it. Nobody monitors config-layer divergence until it bites.

Confusing ProxySQL's max_connections with MySQL's

The per-backend <code>max_connections</code> in <code>mysql_servers</code> is ProxySQL's self-imposed limit; the backend MySQL has its own. Set them equal and you forget that other ProxySQL instances, admin sessions, replication, and monitoring also consume backend slots — so MySQL hits <code>1040</code> while ProxySQL still thinks it has headroom.

Using absolute thresholds instead of ratios

'ConnUsed > 100' or 'latency > 100ms' fails across environments and workloads — dev instances, batch windows, and peak traffic all differ. Alert on ratios (utilisation percentages, pool fraction) and baseline deviation, not fixed numbers, or you drown in false positives and miss the real shift.

Not monitoring per-user connection counts

<code>mysql_users</code> supports per-user <code>max_connections</code>, but teams set only the global limit. One misbehaving application then consumes every connection and starves every other app sharing the instance. Set per-user caps and watch utilisation against them in <code>stats_mysql_users</code>.

ProxySQL runbooks in this section

Each guide is a focused runbook for one symptom or topic. Pick one when you have an incident, or use the categories to learn the area.

▸

Start here

▸

Backend health and the monitor module

▸

Backend connection pool and starvation

▸

Connection multiplexing and session pinning

▸

Client connections, limits, and connection storms

▸

Query rules, routing, and read/write split

▸

Query cache effectiveness and memory

▸

Authentication and error tracking

▸

Configuration layers, drift, and cluster sync

▸

Threads, CPU, memory, and file descriptors

▸

Replication lag, hostgroups, and Galera/Group Replication

WHERE TO GO NEXT

Setting up ProxySQL monitoring, or putting out a fire?

If you're starting from scratch, the monitoring checklist is the path of least regret. If you're mid-incident, jump straight to the symptom that matches what you're seeing.

> Start with the checklist > Back to Operations Guides