ProxySQL Operations Guides on Netdata

How ProxySQL actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

When ProxySQL fails in production, the layer where it fails determines what you see: connection exhaustion at the frontend pool, mis-routing in the query processor, pool starvation from multiplexing collapse, or false health decisions in the monitor module. Understanding these layers is prerequisite to debugging any of them.

This article covers the request path from client connection to backend response, the components that make routing and pooling decisions, and the failure modes characteristic of each. The ProxySQL runbooks in this section build on the terminology and component relationships described here.

ProxySQL Active_Transactions pinning backend connections: multiplexing lost to open transactions

Mon, 01 Jan 0001 00:00:00 +0000

Active_Transactions is climbing in stats_mysql_global. ConnUsed is growing across backends. But the Questions rate hasn’t changed. You check for slow queries, backend errors, new traffic patterns. Nothing explains it.

An open transaction pins a backend connection for its entire duration. ProxySQL disables multiplexing on that session until the client issues COMMIT or ROLLBACK. If transactions are long-running, or if sessions hold transactions open and never commit, backend connections accumulate in ConnUsed with no corresponding query throughput increase. ConnFree trends toward zero. Eventually queries queue, ConnPool_get_conn_failure starts climbing, and new queries hit the max connect timeout.

ProxySQL all replicas lagging: reads falling back onto the writer

Mon, 01 Jan 0001 00:00:00 +0000

You configured a read/write split with replicas in the reader hostgroup, each with max_replication_lag set to keep stale reads away from clients. Now all of them are SHUNNED and the reader hostgroup has zero ONLINE backends. Read queries are either failing or falling back onto the writer, which absorbs the full read and write workload.

The cascade is the danger. As each replica lags and shuns, its traffic redirects to the remaining ONLINE replicas, increasing their load. The last standing replica absorbs all read traffic, its lag climbs, and it shuns too. The reader hostgroup goes empty and every read hits the writer.

ProxySQL backend connection pool exhausted: queries queuing for a free connection

Mon, 01 Jan 0001 00:00:00 +0000

Queries are queuing. Client-facing latency is climbing. Applications are reporting timeouts or “too many connections” errors. You look at ProxySQL’s backend pool and see ConnFree at zero across one or more backends. The natural assumption is that the pool is exhausted.

It might not be. ConnFree == 0 alone is not saturation. ProxySQL can still create new backend connections up to the max_connections limit configured in mysql_servers. The pool is under pressure, but ProxySQL has headroom to open more connections if the backend can accept them.

ProxySQL backend flapping between ONLINE and SHUNNED: monitor-induced oscillation

Mon, 01 Jan 0001 00:00:00 +0000

A ProxySQL backend alternating between ONLINE and SHUNNED more than 3 times in 10 minutes is a distinct failure pattern from a persistent SHUNNED state. The backend MySQL is healthy when checked directly, but ProxySQL’s monitor module produces false-negative health decisions on a jittery network path or under transient backend pressure. Each oscillation kills active connections, causing ConnERR spikes, latency spikes on surviving backends, and dips in Questions as traffic is disrupted and redistributed.

ProxySQL backend SHUNNED: why a healthy backend gets pulled out of rotation

Mon, 01 Jan 0001 00:00:00 +0000

You see status=SHUNNED on a backend in stats_mysql_connection_pool or runtime_mysql_servers. The backend itself looks healthy when you connect directly. Queries may be failing or routing to fewer backends than expected, but the MySQL server is up and accepting connections.

SHUNNED is ProxySQL’s protection mechanism, not a bug. When a backend generates more than mysql-shun_on_failures connection errors within one second, or when its replication lag exceeds the configured max_replication_lag, ProxySQL temporarily removes it from rotation. The backend typically returns to ONLINE on its own within mysql-shun_recovery_time_sec (default 10 seconds), provided there is activity in the connection pool for that hostgroup.

ProxySQL backend_lagging_during_query and backend_offline_during_query: in-flight query failures

Mon, 01 Jan 0001 00:00:00 +0000

Two counters in stats_mysql_global track queries that were in flight when the backend changed state underneath them. backend_lagging_during_query counts queries that may have returned stale data. backend_offline_during_query counts queries that failed. Both are cumulative since the last ProxySQL restart; a nonzero value after steady-state means something disrupted a running query.

backend_offline_during_query sustained increase is a TICKET by the playbook severity model. Each increment means a query was routed to a backend that then disappeared mid-execution: crash, network partition, SHUNNED transition, or operator-triggered OFFLINE_HARD. The diagnostic work is correlating counter increments with the specific backend transitions that caused them.

ProxySQL client connections at mysql-max_connections: frontend saturation and rejected clients

Mon, 01 Jan 0001 00:00:00 +0000

When Client_Connections_connected reaches mysql-max_connections (default 2048), ProxySQL stops accepting new client sessions. Already-connected clients continue working, but new sessions fail with “Too many connections” until existing ones close.

This is a cliff-edge failure. There is no queueing at the frontend and no backpressure signal. Because mysql-max_connections is a global limit shared across all users and hostgroups, a single application with a connection leak can consume every available slot and lock out every other user of the proxy.

ProxySQL Client_Connections_aborted rising: clients rejected or crashing on connect

Mon, 01 Jan 0001 00:00:00 +0000

Client_Connections_aborted is a cumulative counter in stats_mysql_global. What matters is the rate of change, not the absolute value. A sustained non-zero rate means clients are failing to establish or maintain connections through ProxySQL.

The diagnostic question is whether aborts come from ProxySQL actively rejecting connections (authentication failures, connection limits), clients disconnecting improperly (timeouts, crashes, protocol mismatches), or the Client Error Limit feature banning source addresses. The Access_Denied_* counters in the same stats table identify which. If aborts rise while the Questions rate drops, clients are experiencing a real outage. If Questions is stable, the impact may be limited to a subset of clients.

ProxySQL Cluster checksum mismatch: split-brain routing across proxy peers

Mon, 01 Jan 0001 00:00:00 +0000

ProxySQL Cluster peers exchange configuration via a pull-based sync protocol. When sync stalls or conflicts, peers diverge silently. Clients on proxy A get one set of backends; clients on proxy B get a different set. The same query routes to different hostgroups. Credentials differ. Query rules match differently.

The detection signal is stats_proxysql_servers_checksums. Each peer computes a checksum per config module. When peers agree, checksums match. When they disagree, diff_check climbs for the affected module. Persistent mismatch means the cluster is not converging.

ProxySQL config changes not applied: the LOAD TO RUNTIME / SAVE TO DISK trap

Mon, 01 Jan 0001 00:00:00 +0000

You INSERT or UPDATE a row in mysql_servers, mysql_users, or mysql_query_rules through the admin interface. The SQL succeeds with no error. But traffic behavior does not change. Or it works fine for a day, then ProxySQL restarts and everything reverts.

The root cause is the three-layer configuration model. Every admin SQL change lands in MEMORY, the staging area. That change is not active until you explicitly run LOAD ... TO RUNTIME. And it does not survive a restart until you explicitly run SAVE ... TO DISK. No transition is automatic. No error fires if you skip a step.

ProxySQL config lost after restart: runtime never saved to disk

Mon, 01 Jan 0001 00:00:00 +0000

ProxySQL separates live configuration from persistent configuration. If you apply a change to RUNTIME but never run SAVE ... TO DISK, that change survives only until the next restart. When ProxySQL restarts, it loads from proxysql.db on disk, and everything unsaved is gone.

The three-layer model means making a change live and making it durable are two separate, explicit steps. There is no single “save everything” command. If you forget any module’s SAVE ... TO DISK, that module reverts on the next restart.

ProxySQL connection storm after restart: an empty pool meeting a mass reconnect

Mon, 01 Jan 0001 00:00:00 +0000

ProxySQL just restarted. Within seconds, client connections pour in. The backend pool is empty, so every query demands a new connect() to MySQL. Client_Connections_created and backend ConnOK spike in lockstep. The backend’s CPU jumps as it authenticates connections and spins up threads instead of executing queries. If TLS is enabled, each handshake multiplies the cost.

The storm is usually self-resolving in 30 to 120 seconds as the pool fills and steady-state multiplexing resumes. During that window, latency spikes, queries may queue, and if backend max_connections is exhausted, new connections fail outright with MySQL error 1040.

ProxySQL ConnERR climbing: backend connection errors and how to localise them

Mon, 01 Jan 0001 00:00:00 +0000

ConnERR is a per-backend cumulative counter in stats_mysql_connection_pool that increments every time ProxySQL fails to open a connection to a backend MySQL server. Because it is cumulative, a single nonzero value is historical: it means at least one attempt failed since the last stats reset or ProxySQL restart. Take two readings 10 to 30 seconds apart and compute the delta. A positive sustained delta means ProxySQL is actively failing to establish backend connections right now.

ProxySQL ConnPool_get_conn_failure rising: the most direct pool-starvation signal

Mon, 01 Jan 0001 00:00:00 +0000

When ConnPool_get_conn_failure in stats_mysql_global starts climbing, ProxySQL’s internal connection pool failed to hand a backend connection to a worker thread on request. ProxySQL asked the pool for a connection to a hostgroup and got nothing back. That makes it the most actionable signal for backend pool starvation, more direct than watching ConnUsed and ConnFree in isolation.

This counter does not mean clients are seeing errors. It increments on every internal pool-lookup miss, even if ProxySQL subsequently creates a new backend connection or retries successfully. A spike of hundreds of thousands of failures in a few seconds can correspond to zero client-visible errors. What it tells you: the pool was pressured enough to miss, and sustained rising rates mean that pressure is not resolving on its own.

ProxySQL error 1040 Too many connections: the backend MySQL rejecting the pool

Mon, 01 Jan 0001 00:00:00 +0000

Your application queries are failing with ERROR 1040 (HY000): Too many connections. ProxySQL is up, the admin interface responds, and some backends show ONLINE. But queries against certain hostgroups error out, and the problem is spreading. The error is coming from the backend MySQL server itself, not from ProxySQL’s own pool ceiling.

Four independent limits can produce error 1040 or something visually identical to the client: ProxySQL’s global mysql-max_connections (default 2048), per-user max_connections in mysql_users, per-backend max_connections in mysql_servers, and the backend MySQL’s server-level max_connections. When the backend’s limit is hit, ProxySQL tries to open a connection, the backend refuses, ConnERR climbs, and the backend may be shunned after enough failures. This article covers that fourth case.

ProxySQL error 1045 Access denied for user: credential rotation not propagated

Mon, 01 Jan 0001 00:00:00 +0000

Error 1045 is MySQL’s “Access denied for user.” In a ProxySQL environment it can surface at three independent authentication boundaries, and fixing the wrong one wastes the incident window. The typical trigger is a password rotation applied to one layer (backend MySQL, application config, or ProxySQL itself) without propagating it through ProxySQL’s three-layer configuration model.

The error string carries an immediate clue. (using password: YES) means a password was sent but did not match. (using password: NO) means no password was sent at all, which can indicate a client misconfiguration or, rarely, a version-specific bug. Most credential rotation incidents produce (using password: YES).

ProxySQL error 1290 The MySQL server is running with the --read-only option

Mon, 01 Jan 0001 00:00:00 +0000

MySQL error 1290 in a ProxySQL environment is almost always a read/write split misconfiguration: a write query that should have gone to the primary was routed to a read-only replica, and the replica refused it. In ProxySQL, this surfaces in stats_mysql_errors with errno=1290, attributed to a reader hostgroup backend. The proxy is not malfunctioning. It is routing queries exactly as its mysql_query_rules dictate. The rules are wrong.

The fix has two phases: stop the bleeding by deactivating the offending rule, then fix the rule chain so writes route correctly.

ProxySQL error 1317 Query execution was interrupted: timeout-killed queries

Mon, 01 Jan 0001 00:00:00 +0000

Error 1317 (“Query execution was interrupted”) means ProxySQL killed a backend query that exceeded a configured timeout. When a query runs past its limit, ProxySQL opens a separate connection to the backend, issues KILL QUERY <thread_id>, and returns error 1317 to the client.

The question is whether 1317 is a healthy safety valve catching runaway queries, or a symptom of backend degradation killing queries that should normally succeed. Raising a timeout is cheap, but masking backend slowness with a higher ceiling only delays the problem.

ProxySQL error 9001 Max connect timeout reached while reaching hostgroup

Mon, 01 Jan 0001 00:00:00 +0000

Your application is receiving MySQL error 9001 (HY000): “Max connect timeout reached while reaching hostgroup N after Mms.” ProxySQL tried to obtain a working backend connection for the client query and failed within the configured timeout window (mysql-connect_timeout_server_max, default 10000ms).

Every backend in the target hostgroup is either down, SHUNNED, OFFLINE, or saturated to the point where no connection could be borrowed or created in time. Queries are failing.

The “after Mms” portion tells you how long ProxySQL spent trying. If M is close to your mysql-connect_timeout_server_max value, ProxySQL exhausted the full retry window. If M is 0, the timeout fired immediately, which can indicate a configuration or version-specific issue rather than a genuine backend problem.

ProxySQL Galera and Group Replication hostgroups: automatic writer/reader routing

Mon, 01 Jan 0001 00:00:00 +0000

ProxySQL’s mysql_galera_hostgroups and mysql_group_replication_hostgroups tables automate writer/reader hostgroup assignment for synchronous replication clusters. Instead of manually assigning nodes to writer and reader hostgroups in mysql_servers, you define the hostgroup IDs and let the monitor module reassign nodes based on live cluster topology. When a Galera cluster’s active writer shifts during a split-brain resolution, or a Group Replication member wins a primary election, ProxySQL re-routes traffic without operator intervention.

Both cluster types expose their own state. Galera nodes report wsrep_local_state, wsrep_desync, and wsrep_reject_queries. Group Replication nodes report membership state and role through performance_schema.replication_group_members (MySQL 8.0+) or a user-installed sys.gr_member_routing_candidate_status view on older versions. ProxySQL polls these on its monitor interval and moves nodes between writer, reader, backup, and offline hostgroups. This differs from async replication read/write splitting, where ProxySQL relies on read_only checks against statically configured backends. With these tables, the cluster’s own state source is authoritative.

ProxySQL hostgroup_locked connections: reading the multiplexing-health ratio

Mon, 01 Jan 0001 00:00:00 +0000

ProxySQL exists to multiplex: N client connections served by M backend connections, where M is ideally much less than N. When that ratio degrades, the proxy adds latency without providing pooling benefit. The single metric that captures this degradation is Client_Connections_hostgroup_locked divided by Client_Connections_connected.

ORMs and connection libraries can silently disable multiplexing by emitting SET commands or opening transactions on every connection. The proxy runs at near 1:1 for months, and the capacity plan that assumed 10:1 multiplexing is fiction.

ProxySQL memory growth and OOM: what drives RSS and how to tell a leak from load

Mon, 01 Jan 0001 00:00:00 +0000

When ProxySQL RSS climbs toward system limits, the first question is whether growth is load-driven (more connections, cached result sets, or query digests) or a genuine leak. The fixes are different: load-driven growth requires tuning buffer sizes, cache limits, or digest configuration. A leak requires identifying the subsystem that is not releasing memory and possibly upgrading to a patched version.

This guide covers the memory consumers inside ProxySQL, the admin-table queries that break down where bytes are going, and the correlation tests that separate a leak from load.

ProxySQL monitor check failures: connect, ping, read-only, and replication-lag probes failing

Mon, 01 Jan 0001 00:00:00 +0000

When MySQL_Monitor_connect_check_ERR, MySQL_Monitor_ping_check_ERR, MySQL_Monitor_read_only_check_ERR, or MySQL_Monitor_replication_lag_check_ERR start climbing in stats_mysql_global, the ProxySQL monitor module has lost visibility into one or more backends. These counters are a leading indicator. The actual user-facing impact shows up later as backend status transitions, typically SHUNNED, when ProxySQL can no longer verify a backend is healthy and pulls it from rotation.

The monitor module runs four independent check types, each on its own schedule, each writing results to its own log table in the monitor schema. Monitor connections use credentials from mysql-monitor_username and mysql-monitor_password, which are completely separate from the data-plane credentials in mysql_users. This separation is the single most common source of confusion: application traffic can continue flowing normally while the monitor silently fails in the background.

ProxySQL monitor password invalid: healthy backends shunned because health checks fail

Mon, 01 Jan 0001 00:00:00 +0000

Backends are going SHUNNED, but direct connections to the MySQL servers work fine. The servers are up, responsive, and serving queries normally. Yet ProxySQL has pulled them from rotation.

The backends are not the problem. The Monitor module is.

When mysql-monitor_password is wrong, expired, or out of sync, every monitor connect and ping check fails with an authentication error. ProxySQL interprets these failures as backend health failures. After mysql-monitor_ping_max_failures (default: 3) consecutive ping failures, it SHUNS the backend and closes its connections. The backend is healthy, but ProxySQL cannot verify that, so it stops sending traffic.

ProxySQL monitoring checklist: the signals every production proxy needs

Mon, 01 Jan 0001 00:00:00 +0000

ProxySQL sits between your applications and MySQL-compatible backends, fully parsing the MySQL wire protocol and making per-query routing, caching, and connection-pooling decisions.

This checklist defines the monitoring signals every ProxySQL deployment needs, organized into four maturity levels: survival, operational, mature, and expert. Each level builds on the previous one. If you track query digest anomalies at level 4 but cannot tell whether your backends are ONLINE at level 1, you are debugging blind.

ProxySQL monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

ProxySQL deployments tend to land at one of two monitoring extremes: a process check and a port check, or a full dashboard nobody acts on. The gap between “is it running” and “is it healthy” is where most incidents live. This article maps four monitoring maturity levels, each adding signals that catch failure modes the previous level cannot.

Use this as a self-assessment. Find the highest level where you have every signal covered, then look at what the next level adds. All signals come from ProxySQL’s own stats_* tables and host-level metrics, queryable through the admin interface on port 6032.

ProxySQL multiplexing collapse: when connection pooling silently drops to 1:1

Mon, 01 Jan 0001 00:00:00 +0000

You deployed ProxySQL for connection pooling. The capacity plan assumed a 10:1 multiplexing ratio: 1000 client connections served by 100 backend connections. Instead, the backend pool keeps filling up, ConnUsed tracks the client count almost linearly, and the MySQL backend is hitting max_connections even though traffic has not changed. ProxySQL is running, backends are ONLINE, queries are succeeding, but the proxy is providing zero pooling benefit. It has silently degraded into a 1:1 connection relay with overhead.

ProxySQL MySQL_Monitor_Workers is zero: health checks stopped and status is stale

Mon, 01 Jan 0001 00:00:00 +0000

When MySQL_Monitor_Workers in stats_mysql_global reads zero and stays there, ProxySQL’s monitor module has stopped probing backends. No connect checks, no ping checks, no read-only checks, no replication lag checks. The status values in runtime_mysql_servers and stats_mysql_connection_pool are frozen at whatever they were when the last check ran. A backend that crashed five minutes ago still shows ONLINE.

This state is silent. The data plane keeps routing queries. Nothing crashes, nothing logs an error visible to most dashboards. The only outward signal is that ProxySQL stops reacting to real backend health changes: a writer failover goes undetected, a lagging replica stays in rotation, a dead backend keeps receiving queries until clients time out.

ProxySQL OFFLINE_SOFT vs OFFLINE_HARD vs SHUNNED: what each backend status means

Mon, 01 Jan 0001 00:00:00 +0000

ProxySQL tracks four backend statuses: ONLINE, SHUNNED, OFFLINE_SOFT, and OFFLINE_HARD. OFFLINE_SOFT and OFFLINE_HARD are operator-controlled drain states. SHUNNED is automatic and monitor-driven. The distinction determines who sets the state, how it recovers, and which table shows the truth.

A backend that is OFFLINE_SOFT is deliberately draining and will not recover until you change it back. A backend that is SHUNNED is temporarily avoided by the monitor and self-corrects. Conflating the two leads to unnecessary intervention (restarting backends that would recover on their own) or dangerous neglect (assuming a draining backend will fix itself during a failover).

ProxySQL per-user max_connections: one application starving the shared proxy

Mon, 01 Jan 0001 00:00:00 +0000

ProxySQL enforces two layers of frontend connection limits: a global ceiling (mysql-max_connections, default 2048) and an optional per-user cap (mysql_users.max_connections, default 10000). The per-user cap is the one most teams leave at default, which is effectively no limit. A single application with a connection leak or misconfigured pool can consume connections toward the global ceiling. Once the global pool is exhausted, every other application sharing that ProxySQL instance is denied new connections.

ProxySQL query cache hit rate dropped: backend load about to surge

Mon, 01 Jan 0001 00:00:00 +0000

A query cache hit rate drop is a leading indicator. By the time backend CPU spikes or the connection pool saturates, the cache has already stopped absorbing read load.

ProxySQL’s query cache is a TTL-based in-memory result set cache with no invalidation on data change. When it works, it shields backends from repetitive read queries. When it stops, every previously-cached query becomes a real backend query, and backend load increases proportionally to the miss delta.

ProxySQL query cache memory growth: high-cardinality caching toward OOM

Mon, 01 Jan 0001 00:00:00 +0000

Query_Cache_Memory_bytes sits at the mysql-query_cache_size_MB ceiling, the purge rate is rising, and hit rate is dropping despite a full cache. This is the high-cardinality caching pattern: the cache is churning through unique keys that never repeat, and the soft limit cannot keep RSS bounded because it only triggers eviction of expired entries.

The worst case is OOM kill. The kernel terminates ProxySQL, clients disconnect, and on restart the cache is empty, stats tables reset, and the cycle starts again. If you are mid-incident, run PROXYSQL FLUSH QUERY CACHE for immediate relief. Then fix the root cause before memory grows back.

ProxySQL query cache stampede: a hot entry expires and the herd hits the backend

Mon, 01 Jan 0001 00:00:00 +0000

Backend connection pool usage spikes suddenly. Backend latency climbs. Query cache hit rate drops to near zero for a brief window, then recovers minutes later. If this pattern repeats at regular intervals, a cache stampede is the likely cause.

ProxySQL’s query cache is stale-read, keyed by query digest, user, and schema. It uses TTL-based expiration only. There is no request coalescing, no lock-based regeneration, and no mechanism to ensure only one request repopulates the cache after a miss. When a hot entry expires, every concurrent request for that query independently hits the backend MySQL.

ProxySQL query digest memory growth: stats_mysql_query_digest growing unbounded

Mon, 01 Jan 0001 00:00:00 +0000

ProxySQL’s stats_mysql_query_digest table grows without bound. There is no built-in memory cap. On high-cardinality workloads where queries embed unique identifiers (timestamps, UUIDs, session tokens, savepoint names), the digest hash table can balloon to gigabytes, tracked as query_digest_memory in stats_memory_metrics.

The symptoms: ProxySQL runs normally for days or weeks, then develops periodic latency spikes that correlate with monitoring scrapes. RSS creeps upward. In extreme cases, the process is OOM-killed. The digest table is rarely inspected for size, only for query content, so the root cause stays hidden.

ProxySQL query rule CPU overload: expensive regex saturating the worker threads

Mon, 01 Jan 0001 00:00:00 +0000

ProxySQL host CPU is pegged. Client query latency is climbing. But the MySQL backends are idle: ConnFree is greater than zero across the pool, backend ping latency is normal, and there are no slow queries on the database side. The proxy itself is the bottleneck.

When the proxy’s worker threads are saturated, every query slows down uniformly regardless of which backend it targets or how complex the SQL is. The symptom looks like a backend problem from the application’s perspective, but the databases are fine.

ProxySQL query rule not matching: zero hits, silent regex failures, and broken routing

Mon, 01 Jan 0001 00:00:00 +0000

A critical query rule (write routing, read/write splitting, query caching) shows zero hits in stats_mysql_query_rules. Queries that should match it are falling through to the default hostgroup or a catch-all rule instead. The rule exists in runtime_mysql_query_rules, it looks correct, and there are no errors in the ProxySQL log.

The proxy parsed your rule, compiled the regex, and loaded it to runtime. It just never matches any query. Meanwhile, writes may be landing on read-only replicas (MySQL error 1290) or reads may be piling onto the writer hostgroup unnecessarily. If replicas are not configured read_only, writes can silently succeed on a non-primary backend and never reach the authoritative source.

ProxySQL query rule order and apply=1: how rule chains silently mis-route

Mon, 01 Jan 0001 00:00:00 +0000

ProxySQL query rules look deceptively simple: write a regex, point it at a hostgroup, done. But mysql_query_rules is an ordered chain, and the chain’s behavior depends on three interacting variables that most operators never think about together: rule_id order, the apply flag, and flagIN/flagOUT chaining. Get any of these wrong and queries route to the wrong backend silently, with no error.

The mis-route may not manifest until weeks after the configuration change. A rule chain can work correctly by accident, because a later rule happens to set the same hostgroup the operator intended. Then someone adds a new rule, the last match changes, and writes start landing on read-only replicas.

ProxySQL read/write split misrouting: writes silently reaching replicas

Mon, 01 Jan 0001 00:00:00 +0000

ProxySQL read/write split routes queries based on mysql_query_rules evaluated in ascending rule_id order. When that chain is misconfigured, write queries can match a read-routing rule and land on a replica hostgroup. If the replica enforces super_read_only, the write fails with MySQL error 1290 and the client sees an error. If it does not, the write succeeds on the replica, never reaches the primary, and the client receives a success response. Data diverges silently with no error at any layer.

ProxySQL replica shunned for replication lag: max_replication_lag and stale-read protection

Mon, 01 Jan 0001 00:00:00 +0000

When a MySQL replica falls behind its primary, ProxySQL’s monitor module detects the lag and pulls that replica out of read rotation by marking it SHUNNED. This is deliberate stale-read protection: applications should not read from a replica that has not applied recent writes, because they would see stale results or violate read-after-write consistency.

The mechanism is governed by max_replication_lag, a per-backend column in mysql_servers measured in seconds. When the monitor’s replication lag check finds Seconds_Behind_Master exceeding that threshold, the backend is shunned until lag drops back below it. The tradeoff is direct: shunning protects consistency but reduces read capacity. In a topology with two or three replicas, losing one shifts 33 to 50 percent of read traffic onto the remaining backends, which can push them over the same threshold.

ProxySQL runtime vs memory drift: the unapplied change that surfaces on restart

Mon, 01 Jan 0001 00:00:00 +0000

ProxySQL’s three-layer configuration model makes changes safe and atomic, but every edit requires explicit promotion through two transitions. A missed step creates silent drift between what is staged (MEMORY), what is active (RUNTIME), and what persists on disk (DISK). The drift produces no error, no log entry, and no metric. The first visible symptom is often a restart that reverts to a stale configuration.

Two failure patterns dominate. First: an operator edits a MEMORY table but forgets LOAD ... TO RUNTIME. The change sits staged but inactive. The admin interface accepted it without complaint, so the operator assumes it took effect. Later, a restart reloads from DISK and the unsaved MEMORY change is discarded. Second: the operator loads to RUNTIME (the change takes effect) but forgets SAVE ... TO DISK. Everything works until a restart, and the change vanishes. Because RUNTIME matched MEMORY during that window, there was no signal that DISK was behind. Neither form of drift has a native ProxySQL metric. Detection requires comparing MEMORY tables against RUNTIME tables directly.

ProxySQL Server_Connections_delayed above zero: queries waiting on the backend pool

Mon, 01 Jan 0001 00:00:00 +0000

When Server_Connections_delayed in ProxySQL’s stats_mysql_global table climbs above zero, queries are waiting for a backend connection that was not immediately available. This is not an error counter. It is a pressure signal: ProxySQL’s backend connection pool could not instantly satisfy a connection request, so the requesting session had to wait. A brief blip during a traffic burst is normal. A sustained increase means the pool is undersized, multiplexing has degraded, or backends are disappearing from rotation faster than the pool can adapt.

ProxySQL SET statements disabling multiplexing: session state that pins connections

Mon, 01 Jan 0001 00:00:00 +0000

ProxySQL multiplexes N client sessions across M backend connections, where M is smaller than N. When an application issues a SET statement that changes session state, ProxySQL can no longer safely reuse that backend connection for other clients. The connection pins to that client session until the client disconnects.

This is correct behavior. A backend connection carrying a modified session variable (such as sql_mode or time_zone) would produce different query results if handed to another client expecting the default. ProxySQL detects this state and pins the connection to protect correctness.

ProxySQL Too many open files: file descriptor exhaustion takes down connections and health checks

Mon, 01 Jan 0001 00:00:00 +0000

ProxySQL hits a host-level cliff that its own stats tables cannot see. Every client, backend, monitor, and admin session consumes a file descriptor. When the process reaches its RLIMIT_NOFILE ceiling, accept() and connect() fail simultaneously: new client connections are rejected, backend connections cannot be established, and the monitor module cannot open sockets to check backend health. The proxy appears to hang while the process keeps running.

The first visible symptom is often not a connection error. ProxySQL’s internal SQLite database also needs file descriptors. When the pool is exhausted, SQLite fails first with unable to open database file, which operators frequently misdiagnose as disk corruption or a permissions problem. Connection errors surface after that: 2004: Can't create TCP/IP socket (24) in the ProxySQL log, where errno 24 is EMFILE.

ProxySQL worker thread CPU saturation: mysql-threads as a hard parallelism ceiling

Mon, 01 Jan 0001 00:00:00 +0000

When every query through ProxySQL slows down simultaneously, regardless of backend or query digest, the proxy itself is the bottleneck. The most common cause is worker thread CPU saturation: all mysql-threads worker threads are pegged at or near 100% CPU, and their epoll event loops can no longer service connections without adding queuing delay.

mysql-threads (default 4) cannot be changed at runtime. It is set at startup and caps query-processing parallelism absolutely. Adding backends or client connections does not raise this ceiling.

ProxySQL zero ONLINE backends in a hostgroup: total outage for that traffic class

Mon, 01 Jan 0001 00:00:00 +0000

When ProxySQL reports zero ONLINE backends in a hostgroup, every query routed to that hostgroup fails immediately. If the affected hostgroup handles writes, all INSERT, UPDATE, and DELETE operations fail. If it is a reader hostgroup, read queries either error out or, depending on query rules, flood a writer hostgroup not sized for that load.

Clients see the error “Hostgroup X has no servers available!” or generic connection timeouts. Application error rates spike. Questions drops toward zero while Client_Connections_aborted rises.