ProxySQL monitoring maturity model: from survival to expert

ProxySQL deployments tend to land at one of two monitoring extremes: a process check and a port check, or a full dashboard nobody acts on. The gap between “is it running” and “is it healthy” is where most incidents live. This article maps four monitoring maturity levels, each adding signals that catch failure modes the previous level cannot.

Use this as a self-assessment. Find the highest level where you have every signal covered, then look at what the next level adds. All signals come from ProxySQL’s own stats_* tables and host-level metrics, queryable through the admin interface on port 6032.

Scope note: this model covers ProxySQL as a MySQL-wire-protocol proxy. ProxySQL does not support PostgreSQL.

flowchart TD
    L1["Level 1: Survival
Is ProxySQL up?"]
    L2["Level 2: Operational
Is traffic flowing correctly?"]
    L3["Level 3: Mature
Is the proxy itself healthy?"]
    L4["Level 4: Expert
Is the proxy optimal?"]

    L1 --> L2
    L2 --> L3
    L3 --> L4

Level 1: survival

The question: is ProxySQL alive and able to serve traffic? Without these signals, you rely on users to report outages.

Key points:

• Port 6033 is an external check, not a stats metric. ProxySQL does not expose listener health in its stats tables. You need a TCP connect check or a synthetic MySQL protocol check from outside the process. If the port stops accepting connections, page unconditionally. The proxy is down for all clients.
• ConnERR is cumulative. Compute a rate (delta over time) for it to be meaningful. A raw counter of 5000 tells you nothing without knowing how fast it is growing.
• Zero ONLINE backends is the only true PAGE condition at this level, but needs conditions to avoid false positives: sustained for more than 60 seconds, ProxySQL_Uptime > 600 to exclude cold start, and Client_Connections_non_idle > 0 or Questions rate above zero to exclude idle instances.
• SHUNNED is not a page. SHUNNED is ProxySQL’s protection mechanism. A backend in SHUNNED state often self-corrects when replication lag subsides or connection errors stop. Page only when zero backends remain ONLINE in a hostgroup.

Level 2: operational

The question: is traffic flowing through the proxy correctly, and are resources under control? This is the minimum level for production.

Key points:

• ConnFree == 0 does not mean saturation. ProxySQL can still create new connections up to the backend’s max_connections configured in mysql_servers. True saturation means ProxySQL cannot create new connections AND has no free ones. The symptom is ConnERR increasing because the backend rejected the connection attempt.
• Server_Connections_delayed above zero is direct evidence of pool pressure. Queries had to wait for a free backend connection. More actionable than ConnFree alone.
• Monitor check results are a leading indicator. Sustained failure rate increases across check types (connect, ping, read_only, replication_lag) precede backend status changes. Watch MySQL_Monitor_Workers == 0 specifically: it means monitoring is disabled and all status decisions are stale.
• Access_Denied has three distinct counters. Access_Denied_Wrong_Password indicates credential mismatch or brute force. Access_Denied_Max_Connections means global connection limits are hit. Access_Denied_Max_User_Connections means per-user limits are hit. Different causes, different responses.
• Use ratios, not absolute thresholds. Setting “ConnUsed > 100” fails across environments. A dev instance with max_connections=50 and production with max_connections=1000 need different thresholds. Track ConnUsed / max_connections as a percentage.
• Slow_queries can spike during batch operations and schema migrations. It is a TICKET signal, not a PAGE. The actual service impact is better measured by whether Questions rate drops.

Level 3: mature

The question: is the proxy itself healthy, not just passing traffic? This level catches failure modes that look fine at Level 2 until they cascade.

Key points:

• ConnPool_get_conn_failure is the signal most teams miss. ConnUsed and ConnFree tell you about pool state. ConnPool_get_conn_failure tells you ProxySQL actually tried to get a connection and failed. More actionable than watching ConnFree approach zero.
• Latency_us is ping latency, not query latency. The Latency_us column in stats_mysql_connection_pool comes from the monitor module’s ping check. It represents network round-trip time, not query execution time. For real query latency distribution, use the histogram buckets in stats_mysql_commands_counters (cnt_100us through cnt_INFs).
• backend_lagging_during_query and backend_offline_during_query count actual failures. These are not theoretical risks. They count queries that were already in flight when the backend became unavailable or was detected as lagging. The query may have returned stale data or failed entirely.
• Query_Cache_Purged rate distinguishes healthy saturation from harmful churn. A cache at capacity with a stable hit ratio is normal LRU behavior. If purge rate is high and hit ratio is declining simultaneously, the cache is too small for the working set and is churning with no benefit.
• Querying stats_memory_metrics has a performance cost. It can cause latency spikes and CPU load. ProxySQL’s Prometheus exporter uses a separate, longer collection interval for memory metrics for this reason. Do not scrape this table at high frequency.
• query_digest_memory grows with unique query patterns and is not bounded. The stats_mysql_query_digest table accumulates entries in memory. The only ways to reclaim this memory are restarting ProxySQL or querying stats_mysql_query_digest_reset. Warning: stats_mysql_query_digest_reset reads and then clears all digest statistics. Use it only when you can afford to lose historical digest data.
• Active_Transactions pin backend connections. A high count relative to total client connections means multiplexing is degraded. Each active transaction holds a dedicated backend connection that cannot be reused.

Level 4: expert

The question: is the proxy operating optimally, and is its configuration correct? These are the signals you add after your third ProxySQL incident.

Key points:

• Multiplexing ratio = Client_Connections_connected / Server_Connections_connected. A healthy proxy should have this ratio well above 1, meaning many clients share few backend connections. When it approaches 1:1, ProxySQL provides no pooling benefit while adding latency overhead. The pinning ratio (Client_Connections_hostgroup_locked / Client_Connections_connected) tells you what fraction of sessions have broken multiplexing. A pinning ratio above 50% sustained is serious degradation.
• stats_mysql_query_rules.hits resets on rule load. Every time you run LOAD MYSQL QUERY RULES TO RUNTIME, the hits counter resets to zero. Historical trend data is lost silently. If a critical write-routing rule shows zero hits after a reload, routing may be broken.
• Cluster checksum divergence is latent risk, not immediate outage. Nodes continue operating with their own config. But if checksums disagree for mysql_servers or mysql_query_rules, traffic may be routed differently depending on which ProxySQL instance a client connects to. This only applies to ProxySQL Cluster deployments using proxysql_servers.
• Config drift is invisible to metrics. There is no stat that says “your runtime config differs from disk.” You detect it by comparing tables: mysql_servers (MEMORY layer) vs runtime_mysql_servers (RUNTIME layer). If they differ, someone made a change that was not loaded to runtime, or loaded a change that was not saved to disk. A restart will revert to disk state.
• Per-thread CPU reveals hot-spotting. ProxySQL distributes connections across worker threads (mysql-threads, default 4). If one thread is pinned at 100% CPU while others are idle, you have a load distribution problem. Thread count is a hard ceiling on parallelism and cannot be changed without restart.
• stats_mysql_errors gives the most granular error information. It breaks down errors by MySQL error number, hostgroup, user, and schema. Error 1040 (too many connections) from a backend is a different problem from error 1045 (access denied) or error 9001+ (ProxySQL internal errors). ProxySQL-generated errors in the 9000+ range always warrant investigation.

How Netdata helps

• Per-second collection from ProxySQL’s admin interface. Netdata queries the stats_* tables at high frequency, giving rate-based views of cumulative counters like ConnERR, Questions, and ConnPool_get_conn_failure without manual delta calculation.
• Correlation across levels. A multiplexing ratio drop (Level 4) that correlates with a ConnUsed spike (Level 2) and an Active_Transactions increase (Level 3) points to transaction-driven degradation. Viewing these signals on a single timeline shortens diagnosis from “which metric changed” to “why did it change together.”
• Backend health and monitor checks on the same dashboard. Netdata surfaces per-backend status, ConnERR rates, and monitor check OK/ERR ratios together, making it obvious whether a SHUNNED backend is caused by network issues, replication lag, or credential problems.
• Host-level signals alongside ProxySQL metrics. Process RSS, per-thread CPU, and file descriptor counts from the host layer correlate directly with ProxySQL internal metrics. An RSS spike that coincides with a query_digest_memory growth trend tells you where memory is going without a separate investigation.
• Anomaly detection on query rates and error counters. Netdata’s ML-based anomaly detection flags deviations from established baselines without requiring static thresholds, which matters for ProxySQL metrics where absolute thresholds are environment-dependent.