ProxySQL monitor check failures: connect, ping, read-only, and replication-lag probes failing

When MySQL_Monitor_connect_check_ERR, MySQL_Monitor_ping_check_ERR, MySQL_Monitor_read_only_check_ERR, or MySQL_Monitor_replication_lag_check_ERR start climbing in stats_mysql_global, the ProxySQL monitor module has lost visibility into one or more backends. These counters are a leading indicator. The actual user-facing impact shows up later as backend status transitions, typically SHUNNED, when ProxySQL can no longer verify a backend is healthy and pulls it from rotation.

The monitor module runs four independent check types, each on its own schedule, each writing results to its own log table in the monitor schema. Monitor connections use credentials from mysql-monitor_username and mysql-monitor_password, which are completely separate from the data-plane credentials in mysql_users. This separation is the single most common source of confusion: application traffic can continue flowing normally while the monitor silently fails in the background.

What this means

ProxySQL’s monitor module continuously probes every backend to answer four questions:

• Connect checks: Can ProxySQL establish a TCP connection to the backend?
• Ping checks: Is the backend responsive to mysql_ping() on an existing monitor connection?
• Read-only checks: Is @@read_only set? This drives automatic read/write splitting between hostgroups.
• Replication lag checks: How far behind is this replica? When lag exceeds max_replication_lag for a backend, ProxySQL shuns it.

Each check type has its own interval, timeout, and error threshold. After mysql-monitor_ping_max_failures (default 3) consecutive ping failures, the monitor signals the Hostgroups Manager to shun the backend. After mysql-monitor_read_only_max_timeout_count (default 3) consecutive read-only check timeouts, the backend is assumed to have read_only=1 and may be moved to a reader hostgroup. These are not soft warnings. They are direct inputs to routing decisions.

flowchart TD
    A["*_check_ERR rising in stats_mysql_global"] --> B["Check monitor.*_log tables"]
    B --> C{Error pattern?}
    C -->|"Access denied"| D["Monitor credentials invalid"]
    C -->|"Connection refused"| E["Backend MySQL down
or port blocked"]
    C -->|"Timeout"| F["Network latency or
SSL connect_timeout rounding"]
    C -->|No entries / stale| G["Monitor workers
starved or stopped"]
    D --> H["Fix grants, reload variables"]
    E --> I["Restore backend"]
    F --> J["Tune timeouts"]
    G --> K["Check MySQL_Monitor_Workers"]

Common causes

Quick checks

Run these against the admin interface on port 6032. All queries are read-only. Avoid passing the password on the command line in production; use MYSQL_PWD or a credentials file.

# Error and success counters, plus worker thread status
mysql -u radmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name LIKE 'MySQL_Monitor_%' ORDER BY Variable_Name;"

mysql -u radmin -h 127.0.0.1 -P 6032 \
  -e "SELECT Variable_Name, Variable_Value FROM stats_mysql_global WHERE Variable_Name IN ('MySQL_Monitor_Workers','MySQL_Monitor_Workers_Started','MySQL_Monitor_Workers_Aux');"

# Monitor credentials and intervals
mysql -u radmin -h 127.0.0.1 -P 6032 \
  -e "SELECT variable_name, variable_value FROM global_variables WHERE variable_name LIKE 'mysql-monitor_%' ORDER BY variable_name;"

# Check log tables for actual error strings (one per check type)
mysql -u radmin -h 127.0.0.1 -P 6032 \
  -e "SELECT * FROM monitor.mysql_server_connect_log ORDER BY time_start_us DESC LIMIT 20;"

mysql -u radmin -h 127.0.0.1 -P 6032 \
  -e "SELECT * FROM monitor.mysql_server_ping_log ORDER BY time_start_us DESC LIMIT 20;"

mysql -u radmin -h 127.0.0.1 -P 6032 \
  -e "SELECT * FROM monitor.mysql_server_read_only_log ORDER BY time_start_us DESC LIMIT 20;"

mysql -u radmin -h 127.0.0.1 -P 6032 \
  -e "SELECT * FROM monitor.mysql_server_replication_lag_log ORDER BY time_start_us DESC LIMIT 20;"

# Backend status, connection pool errors, and mysql_users conflict check
mysql -u radmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup_id, hostname, port, status FROM runtime_mysql_servers ORDER BY hostgroup_id;"

mysql -u radmin -h 127.0.0.1 -P 6032 \
  -e "SELECT hostgroup, srv_host, srv_port, status, ConnOK, ConnERR FROM stats_mysql_connection_pool ORDER BY hostgroup;"

mysql -u radmin -h 127.0.0.1 -P 6032 \
  -e "SELECT username FROM mysql_users WHERE username = (SELECT variable_value FROM global_variables WHERE variable_name = 'mysql-monitor_username');"

How to diagnose it

1. Identify which check types are failing. Compare the _OK and _ERR counters for each check type in stats_mysql_global. If all four types are failing across all backends, the problem is systemic: credentials, monitor configuration, or monitor threads. If only one check type fails (for example, replication_lag_check_ERR rising while others stay flat), the problem is privilege-specific.

2. Read the monitor log tables. Each check type writes to its own log table in the monitor schema. The error string is the fastest path to root cause: “Access denied for user” means credentials; “Connection refused” means the backend is down or the port is blocked; empty or stale entries mean the monitor threads are not running checks.

3. Verify monitor credentials. Check global_variables for mysql-monitor_username and mysql-monitor_password. Both default to empty strings; monitor checks will not run until you set them. Connect directly to a backend MySQL and verify the user exists with the matching password.

4. Verify privileges on the backend. Connect and ping checks need only USAGE. Read-only checks run SELECT @@read_only, which also requires only USAGE. Replication lag checks run SHOW SLAVE STATUS or equivalent, which requires REPLICATION CLIENT. If replication_lag_check_ERR is the only counter rising, the monitor user is missing REPLICATION CLIENT.

5. Confirm the monitor user is not in mysql_users. ProxySQL explicitly forbids using the same username in both mysql-monitor_username and mysql_users. If the monitor username appears in mysql_users, ProxySQL cannot authenticate monitor checks correctly.

6. Check network connectivity to affected backends. If only specific backends are failing, run nc -zv <host> <port> or mtr <host> from the ProxySQL host. Firewall rule changes, security group updates, and DNS resolution failures are common culprits.

7. Check monitor worker thread health. Query MySQL_Monitor_Workers in stats_mysql_global. If it is 0, the monitor module is not running. If checks are arriving late (large gaps between time_start_us entries in the log tables relative to configured intervals), the monitor threads may be starved by too many backends or too-aggressive intervals.

8. Check for SSL-related timeout behavior. When backends have use_ssl enabled, mysql-monitor_connect_timeout is rounded up to the nearest whole second with a minimum of 1 second. A configured timeout of 200ms effectively becomes 1000ms. This can cause confusion when comparing configured timeouts to observed check latencies.

Metrics and signals to monitor

Fixes

Monitor credentials invalid or missing

Update the monitor password in ProxySQL and ensure the user exists on every backend.

-- In ProxySQL admin interface
SET mysql-monitor_password = '<correct_password>';
LOAD MYSQL VARIABLES TO RUNTIME;
SAVE MYSQL VARIABLES TO DISK;

On each backend MySQL, create or fix the monitor user:

-- On the backend MySQL server
-- Required for connect, ping, and read-only checks
CREATE USER IF NOT EXISTS 'monitor'@'<proxysql_host>' IDENTIFIED BY '<password>';
ALTER USER 'monitor'@'<proxysql_host>' IDENTIFIED BY '<password>';
GRANT USAGE ON *.* TO 'monitor'@'<proxysql_host>';
-- Required additionally for replication lag checks
GRANT REPLICATION CLIENT ON *.* TO 'monitor'@'<proxysql_host>';
FLUSH PRIVILEGES;

After fixing, verify by checking the monitor log tables. New entries should show success. The error counters are cumulative and will not reset, so compare the rate of increase before and after the fix.

Monitor user lacks privileges

If only replication_lag_check_ERR is rising, grant REPLICATION CLIENT on the backend. If read-only checks are failing with privilege errors, verify USAGE is granted. The grants must be applied on every backend the monitor probes. The monitor module does not retry with escalated privileges; it simply fails and increments the error counter.

Monitor user duplicated in mysql_users

Remove the monitor username from mysql_users. The monitor user must be exclusive to the monitor module.

-- WARNING: this removes a user from the ProxySQL runtime configuration.
-- Confirm the username is not serving application traffic before deleting.
DELETE FROM mysql_users WHERE username = '<monitor_username>';
LOAD MYSQL USERS TO RUNTIME;
SAVE MYSQL USERS TO DISK;

Network or backend unreachable

If the backend MySQL is down, restore it. If the network is blocked, fix the firewall rule, security group, or DNS resolution. Remove the backend from mysql_servers if it is permanently decommissioned, otherwise stale entries will continue generating check errors and consuming monitor thread cycles.

Read-only and replication-lag check edge cases

Read-only check timeouts can falsely take a healthy writer offline. If the monitor’s connection to the backend goes stale and the read-only check times out, ProxySQL may move the backend to a reader hostgroup even though it is a healthy writer. Increasing mysql-monitor_read_only_timeout and mysql-monitor_connect_timeout can mitigate this by giving the monitor more time to get a response.

Replication-lag check timeouts on already-shunned replicas can cause ProxySQL to re-enable the server with a bogus lag value, then immediately shun it again, creating a flapping cycle. If you see a backend oscillating between SHUNNED and ONLINE with replication-lag errors in the log, investigate whether the lag check itself is timing out rather than reporting actual lag.

Prevention

• Treat monitor credential rotation as a deployment step. When database passwords change, update both mysql_users and mysql-monitor_password in the same change. A partial rotation creates a delayed failure mode that is harder to diagnose.
• Alert on sustained check error rate. The *_check_ERR counters are leading indicators. A sustained non-zero rate for any check type should trigger a ticket before the downstream SHUNNED event.
• Track MySQL_Monitor_Workers. Verify it is non-zero after restarts and config reloads. Check the freshness of entries in the monitor log tables against configured intervals to detect thread starvation.
• Confirm config layer consistency after changes. Verify that monitor variable changes are loaded to RUNTIME and saved to DISK. A restart with stale disk config will silently revert the fix.
• Grant only USAGE and REPLICATION CLIENT. Over-privileging the monitor user is a security risk; under-privileging breaks specific check types silently.
• Never put the monitor username in mysql_users. This is a hard restriction. Using the same username in both places causes authentication failures with error messages that do not clearly point to the conflict.

How Netdata helps

• Netdata collects all MySQL_Monitor_*_check_OK and *_check_ERR counters from stats_mysql_global at per-second resolution. This shows the exact moment check failures begin, not just the aggregated state minutes later.
• Correlating monitor check error rates with backend status changes (ONLINE to SHUNNED transitions in runtime_mysql_servers) on a single timeline makes the cause-and-effect relationship visible without manually cross-referencing separate tools.
• MySQL_Monitor_Workers is tracked alongside the check counters, so you can distinguish “monitor threads are not running” from “monitor threads are running but checks are failing.”
• Backend connection pool metrics (ConnERR, ConnUsed, ConnFree) are collected in the same view, letting you determine whether monitor failures and data-plane connection errors share a common root cause.

Related guides

• ProxySQL backend SHUNNED: why a healthy backend gets pulled out of rotation
• How ProxySQL actually works in production: a mental model for operators
• ProxySQL monitoring checklist: the signals every production proxy needs
• ProxySQL monitoring maturity model: from survival to expert